Past vs Present: How We’re Evolving the Speaker Selection Process

The spirit of WordCamps is to hear from a diverse group of people on a diverse group of topics relevant to those who use and make WordPress.

WordCamps are organized by volunteers. The number of volunteer hours that go into planning, organizing, and producing any WordCamp are vast. For flagship camps, like WordCamp US, the number of organizers and volunteer hours goes through the roof.

In an effort to select the best content possible for WCUS 2019 and to be a good steward of volunteer hours, we (the Programming Team) are experimenting with a new speaker selection process.

Because transparency is important, we’d like to share the new process with you in advance of our presenter announcement later this month.

Speaker submissions

In the past, there was no limit on the number of sessions a potential speaker could submit. While this allowed for a single speaker to present multiple talk ideas for consideration, the result was 500+ talks submitted in 2018 — each one requiring review and consideration by multiple organizers.

For WCUS 2019, we decided to limit speaker submissions to two talks per person. The idea was to encourage speakers to submit only their strongest ideas. The result was a considerable reduction in talk submissions (230 talks submitted) but an overall increase in the quality of those submissions.

The review team

In the past, all organizers were part of the review process. While this ensured diversity of experience among the reviewers, there were a couple of less than ideal issues this produced:

A lot of organizers participating meant each individual contributing additional volunteer hours to covering reviewing over 500 talks.
Organizers were reviewing talk submissions on topics outside of their core areas of expertise.

For WCUS 2019, we took each talk category (i.e. Development, Design, Accessibility, etc.) and requested the organizers to review only talks within their sphere of experience. This accomplished two things:

Reduction in overall hours needed (i.e. each volunteer is reviewing talks in a maximum of 3-4 categories)
Increase in diversity of topics that “rise to the top.” With volunteers only reviewing topics in their area of expertise, it was easier not only to identify the best in each category of submissions, but also to get a strong representation across all categories.

Are the reviews really blind?

A “blind review” means that any personally-identifying information is stripped from the talk submission, giving reviewers a chance to honestly assess the quality of the submission without the bias of knowing who submitted it.

Naturally, talks are submitted with names. One of the lead organizers exported all of this data into a spreadsheet and removed the speaker names and locations. The programming team leads then reviewed the spreadsheet and removed any “rogue” references to speakers or locale mentioned in the talk description or notes.

While it’s inevitable that a reviewer who’s been around the WordPress community for awhile might recognize a topic or description and associate it with a speaker, every effort was made to make the talk selection process about the quality of the content, not the identity of the speaker.

Scoring, Round 1

In the past, voting on talk submissions was binary: Thumbs up or thumbs down.

This year, voting was conducted on a 3-point scale:

1 – Must attend this session
2 – Might attend this session
3 – Would not attend this session

This system helped float the strongest talks to the top and allowed for a “neutral” vote vs. an outright dismissal if the reviewer didn’t feel strongly one way or the other.

Reviews at this initial level were completely “blind.”

Scoring and Review, Round 2

Spreadsheets of submissions were distributed to each member of the review team for voting and, once complete, those copies were merged back into a central document.

At this point, “best in category” talks are identified based on results from Scoring, Round 1.

As part of a related, but separate, effort the Programming team calculated a percentage of Personas based upon how people self-identified when signing up for WCUS 2018 (i.e. Designer, Developer, Blogger, etc.)

Those persona percentages were then used to inform how many talks per category should be in the final schedule. For example, with nearly 60% of previous-year attendees identifying as a “Developer”, it makes sense to dedicate a majority of content to developer-oriented categories.

The second highest-identifying group was Designers, so that informed the next biggest chunk of topics that should be represented on the schedule. And so forth.

The criterion for a talk ultimately includes (in this order):

Desirability, as ranked against other in-category submissions
Relevance, as compared to how previous attendees self-identified their role with WordPress
Diversity, as a final element of “human judgement” used to ensure representation across a broad spectrum of experience

Joanna Barsh working the crowd while speaking at WCUS 2018

A note on invited speakers

Each year there are invited speakers — people who didn’t necessarily submit speaker applications, but are invited to take the stage at WordCamp U.S.

At first this may seem odd, but there are a few compelling reasons why we choose to do this:

Matt Mullenweg is able to invite influential folks of various industries to share their knowledge and experience with WCUS attendees. Allotting invitational spots to Matt, enables WCUS attendees to hear from speakers who otherwise would not know of the conference, or traditionally submit an application.
“Hot topics” in WordPress inevitably change as the year progresses (and official speaker submissions have passed). Leaving a few spots open, enables the programming team to bring in speakers in a more “last minute” fashion to address timely topics.
Speaker submissions do not always reflect the diversity we strive to represent at WordCamp U.S. Invited speakers allow the programming team an opportunity to fill in gaps in both content and speakers that become obvious once official submissions are closed.

This year, there will be approximately 10 invited speakers and the breakdown looks like this:

3-5 speakers invited personally by Matt Mullenweg
3-5 speakers invited by the programming team

A final note

If you’ve participated in planning a WordCamp, you know that final speaker selection is a process of invites being extended, some accepted and some declined.

From there, a schedule is roughly planned, always with the contingency that some speakers who accept a position may need to decline at the last minute (life takes unexpected turns, after all). Those spots are typically backfilled with speakers who submitted talks, but were not extended an invitation initially.

There’s a delicate dance of creating a schedule that meets the following guidelines:

Presents topics most relevant to attendees (as they self-identify their interests during ticket sales)
Ensure that attendees have a “clear” choice of which talk to attend in any given timeframe (i.e. don’t make attendees choose between two talks at the same time about React.js)
Ensure that speakers meet community guidelines and respect the GPL

Ultimately, our mission as the programming team is to bring the best possible content to the conference so that attendees leave with increased knowledge and exposure to WordPress (and related) topics as well as a sense of belonging within this thing we call the WordPress Community.

Your WCUS 2019 Programming Team:

Co-leads: Tessa Kriesel and Carrie Dils
Raquel Landefeld
David Bisset
Bret Phillips
Adam Silver
Sandy Edwards

You don’t want to miss out on WordCamp U.S. 2019! We will be having speakers, workshops, childcare, an incredible sponsor hall and more. If you haven’t already, get your tickets to WordCamp US today! If you need a hotel, we also have a hotel block at the Marriott that is across the street from the venue. Rooms are limited, reserve today.