Allen Kinsel - SQL DBA

Since you're speaking in public, knowing what feedback others have seems to make sense. If you were in the session, you already know, so are we somehow thinking that partial public is different than on-demand later?

As far as «knowing exactly how you compare» isn't true. There isn't necessarily the same people, or sample of people in all sessions. I think seeing a good rating means that people liked the information. Or they didn't. But you don't know if the people were overwhelmed (100 ppl in a 400 session or vice versa).

What I do think you ought to consider is requiring speakers to
a) have presented the session elsewhre, like user group or sql saturday
b) have had ratings

I would also look to license, or even embed speaker rate into the PASS site and track speakers and presentations over time.

Great post, Allen!

We need to overcome the inertia of the old 20th Century style of doing business in which the first impulse is «do not share, it's too much work» and replace it with the 21st Century style of «information wants to be free».

Keep driving for this!

-Kev
-Twitter @kekline

Steve - about the 100-level-people-in-a-400-level-session, I sincerely believe that's the speaker's fault. It's the speaker's responsibility to write an accurate abstract that makes sure the right people are in the seats.

I'm totally okay with my scores being released, but if there's pushback, maybe we turn around and ask why we're releasing them. If the goal is to get attendees to go to the best sessions, then next to each abstract in the guide, have gold, silver, and bronze award winner ratings next to the highest rated speakers. That way, attendees can say, «I wasn't sure which one to decide out of these two, and I don't know these guys personally, but this guy was an award-winner last year.» It helps people make better «buying» decisions without insulting the people on the bottom.

And I can understand why people wouldn't wanna be at the bottom. Nobody wants to be last place in the Olympics - even though you're in an elite group, you're still last, and that can feel bad.

I really like Steve's point, that the session has been presented elsewhere. I think PASS is the wrong place to be giving a presentation for the first time, and especially in the case where a presenter's ratings will be released to the public, this could be a recipe for disaster (at least for the confidence and self-esteem of the presenter). At least if the session has been given at a code camp or SQL Saturday, they've had a chance to see how the presentation is received, learn from feedback and questions, etc. Even setting the presenter's feelings aside, the audience at PASS deserves a presentation that has been rehearsed and learned from at least once before, IMHO.

I agree that I'd hate to be last on the list. I think I'd lean towards only publishing the scores and not the comments (I'd hate to have to screen them anyway). Attendance at the various sessions is a good metric to share too. In some ways I think of publishing it as a way to support the decisions made for the next year schedule, but I guess more than that I just don't see a lot of danger in keeping it confidential.

Keep blogging on this stuff Allen!

Its essential to record scores, however people have bad days and bad audiences and so making the scores public is not something that should be mandated. Speakers can allow for the scores to be made public but thats as far as I would go.

Its also essential to have aggregate numbers for say 1st time speakers, DBA sessions, Dev Sessions, and maybe by topic, SSIS, SSRS etc. that way a speaker can compare there scores with others.

Personally its the comments that matter, scores give you a guide but are useless. Comments are gold.

@Simon - I don't agree that scores are useless. When I was first attending the Summit and I didn't know any of the speakers, I would have loved to have had at least a rough ballpark guide that pointed me to some of the «don't-miss» sessions. I had a tough time deciding between sessions because there was so much material, and I walked away from some of the sessions saying to myself, «Man, that person had no idea how the hell to do a presentation, and I just wasted an hour of my life.» We need to maximize the value that new attendees get. I'm totally open to ideas on how to guide them. Comments don't really work because they take a huge amount of space - you can't put those on the guides.

I'm not sure that it's always the speakers fault for a 100-400 mismatch. I have seen experienced, and good speakers, think they were presenting at one level and people think it's another.

I agree that comments perhaps are not appropriate, especially as some people are jerks about things. As Buck Woody mentioned, he got banged for presenting the same thing at two conferences, which is a poor comment.

However, perhaps a gross average of ratings, or even a count of the ratings at each level, would be good. I might also include an average of what level (100, 200, 300, 400) people think the session was.

My other comment is that I'm not sure most people rate speakers on a spectrum. They look at sessions and will make the decision as Brent mentioned. Do I go to Buck or Steve? And Buck is rated higher, so go there. Speakers look at the who's first, who's last, but does the general audience do that? I'd think they don't bother. I wouldn't publish ratings as a list, but rather on the DVD, and on the abstract list next to each description

I meant the scores are useless for the speaker.

Oh yeh I've been in my fair share of awfull sessions. Many of them however have been from very well known people, bad day or a bad session.

Scores also get really screwed, for instance if you compare scores from a ug event and a conference the UG event its much higher often because the attendees are not as distant from the speaker, at a ug event people tend to vote higher because they feel the speaker would know its them that voted.
Making voting anonymous also skews the results greatly.

Its a big challenge and I don't know the answer

Interesting point Steve. We ask if people vote based on the person or the session. The bias is always towards the session, I guess PASS would be differenct as there are more big names.

I would do it in the book because it seems like so many people are wandering around the conference at the last minute looking for places to go. If the session you planned on going to is suddenly full (or canceled), then that's when you need the speaker guidance the most.

Publish two things:

1) List of top 5 speakers for an event after all ratings tabulated.
2) List of exceptional speakers over time. Not the «merely good.» Not even the excellent. I'm talking about the can't miss list. Make sure there's at least X number of ratings to make this list. And that the average rating is Y.

Otherwise, distribute to the speakers their scores and if they wish to publish their own, they can.

Allen,

I'm including the SQL Saturdays, if those get rated by speakerrate.com or something of that sort. And as far as exceptional speakers, I would have a list, regardless of event. So if I show up at SQL Saturday - New Orleans, don't know who to see, I can hit on my BB and see that Buck Woody is an exceptional speaker and he has a slot in the next session.

I don't think PASS should necessarily publish speaker ratings, although if PASS wants to encourage the use of or somehow integrate with SpeakerRate, that would be okay.

Allen, in one of your comments, I think you hit on a key idea...make a bigger deal out of the fact someone was given a Spotlight session. It wasn't until I became a member of the Program Committee that I understood what a Spotlight was and why. So, I suspect that many of the attendees don't really know. I think it needs to be communicated better and more frequently.

I also like Brent's idea about giving a rating/award to speakers that is published in the guide (ideally included on the short-list, not just the full blown book). That could be Gold/Silver/Bronze or something like «PASS All-Star» or «PASS Veteran». After all, if the Program Committee keeps accepting someone to speak year after year then that says something about either the speaker or the expected content.

Because it is part of PASS's objective, as communicated to the Program Committee each year, to include some relatively new speakers, I think the biggest challenge will be finding something that does not penalize new speakers. That's why I like the idea of highlighting veterans or gold medalists better than publishing ratings for everyone.

If PASS wants to provide additional details in the feedback that goes back to the speakers so they know better how they compared to others, that is fine, but I think separate from helping attendees with the next event's program guide.

Ratings are very subjective, and citizen reviewers all enter the room with different expectations. I think that after a while the ratings could suffer from sabotage and pique, because PASS seems to support «rock star» posturing among its experts, which promotes unhealthy competition. I don't support publication, but I would support highlighting popular speakers and noting their accomplishments, such as publication or MVP status.

Jay - whoa, that's scary to hear. When you say «PASS seems to support rock star posturing among its experts, which promotes unhealthy competition,» can you elaborate on that? I'm interested in hearing more about both how PASS promotes that, and how it's unhealthy.

Brent,
I got an email this morning from a PASS chapter saying that a «mystery rock star» would show up at an upcoming chapter event.
Don't get me wrong. I like SQL rock stars. I have personally talked to a SQL rock star, and he is a smart, helpful guy, even humble! He gave me a copy of his book, no charge. He answered questions for me via email. Sql and .Net rock stars help me every day via Google, Safari and my book collection. They are great. But I do think that there are drawbacks to star systems, and this discussion -- about whether to publish speaker ratings -- highlights one of them. I'll call it the «Who do you think you are?» problem. That is, who do you think you are to believe that you can speak in front of other programmers - you're no Itzik Ben-Gan! Organizations create, promote and use star systems, in part, to market themselves. Dude, would you go to Seattle to hear me speak? Of course not. You might go to hear Itzik, though. He's a star, in the best sense, and when PASS is trying to sell pricey tickets to its events, it markets the speakers. The brochures and web pages tell this story. Call them rock stars, marquee names or celebrities. And I believe that SQL rock stars work for their fees and fame. But the star system can be a problem when the less-than-luminous want to get involved. At a recent programming event I attended, a SQL speaker spent the first five minutes telling us that he would never have applied to speak had he known that Rock Star 1 and Rock Star 2 and Rock Star 3 were going to be speaking at the same event. Yet after he unburdened himself of hero worship and personal inadequacy and so forth, he gave one of the best presentations I've seen - coherent structure, strong examples, understandable metaphors. Let's get him on Sql Idol! If you take him at his word, he is saying he wouldn't have come forward had he known the stars would have been there. It would have been too humiliating, etc. What a shame that would have been. Perhaps he was speaking rhetorically, and if so I suppose his message was twofold: he really is in awe, and - in case the rest of us didn't know - we should be too! I can't say to what degree this speaker represents the hoi polloi, but I can say that the star system can inhibit those who might have a few good presentations in them. If PASS makes the ratings public, this will scare off even more of those who are not in the firmament. There is a degree of intimidation that comes from the star system. You may disagree. It's a pretty subjective area, and well discussed, so I'll leave it there.

Jay, You get extra points for using the words firmament and luminous!

I'm with you on this one overall. The compromise I might offer would be to publish the Top 10 from each track, as PASS has been doing, then leave the rest private. It's sort of extending the old management axiom of «Praise in public, discipline in private».

Jay,

There is something to what you say, but there is also some merit to showing off the speakers that the peers find more interesting. You might have gotten one of the exceptions there in terms of an unknown speaker giving a great presentation. I've seen lots of unknowns that didn't do a good job.

I think we can push some people as RockStars, but more we ought to be trying to raise the quality of all speakers, choosing them based on not only content, but the ratings from PASS *and* local events. Get local chapter leaders and others to submit ratings of user group presentations and SQL Saturday presentations.

I would encourage more people to speak, but you have to have a thick skin, and you can't get caught up in worrying about what other speakers do. Do your best job and teach people something.

Jay - wow, there's so many good thoughts in there.

I've heard other people rant against some of the «hero worship» that's been happening lately. To some extent, things like Twitter follower counts and webcast attendee metrics build up a cult of personality, and that's not healthy.

You've inspired me to do a blog post for tomorrow! Off I go...

Late to the party, but I say publish the scores (web and in the book are both good ideas), and that there's nothing wrong with the Rock Star-ness. Everyone wants to know that they're getting value for attending an event, so you want to know that someone worthwhile is speaking, not just show up and find me on the podium.

Plus, it would help all speakers to determine how they need to improve, and there could be things they can learn from the high scoring speakers. I don't think publishing the numbers will scare anyone off any more than currently is happening, just knowing that Paul/Kimberly/Itzik/Kevin/Brent/Allen/Buck/etc will be submitting sessions.