Posts tagged Evaluations
PASS Summit 2011 Preconference Session selection info
The other day I detailed as much info as I could about changes to this years session selection processes. The call for ALL Sessions at the Summit is officially open until May 5, 2011 that means we’re accepting abstracts for regular sessions as well as for the Preconference sessions.
For many of the more experienced speakers in the SQL Community the opportunity to give a full day highly technical session at the summit is attractive. Not only do they get to showcase their technical skills, they also get to showcase their own unique training style. Additionally, the pay isnt half bad either. Of course, most of these speakers will tell you that based on the amount of time needed to generate an all day session the pay is actually half bad, not half good
For the past 3 years we’ve had a set of requirements in place that qualify speakers to present a preconference session, those requirements are looked at every year by a set of volunteers to make sure they are still valid and usually only minor adjustments are made. This year however I made a larger adjustment. In order to clarify the requirement that giving a precon requires you to also give a spotlight session, I added #11 to the list of criteria you should meet to deliver a precon. Since that is a gimme criteria, I bumped the total criteria needed to meet from 4 to 5
After you’ve done your thing and submitted a terrific abstract for a precon session its up to us to try and be fair in the selection process for these sessions. Over the last few years we have tried many things to both open up the selection process and to open up submissions to a competitive process. There have been many ups and downs in this iterative process. Sometimes we do well, and sometimes… not so much. The point is that we’re always trying to produce a better process. I’m excited that hopefully this year we’ve ironed out a few more of the wrinkles we exposed last year.
Process Part one
After the Call for abstracts closes we are going to release a “preferencer” tool that we’re going to use to allow PASS Members the ability to communicate what their favorite sessions are. We’re going to use the data gathered from this tool to assist us in making the session selections. Additionally, we are tentatively going to use this same info to prepopulate your attendee schedule builder after the selections are made. I’d like to think we’ll be able to use the data from this tool in a 1-1 fashion where we take the top X precons per track and build the session schedule that way. In reality though, I expect we’re going to find several overlapping session subjects at the top of the list. I’m thinking things like 3 “indexing sessions” in the top 5 for the DBA track, or 2 Sessions from a single speaker for instance. We’ll need a a team to help make those decisions.
Introducing the team that will make the decisions
K. Brian Kelley Author/Blogger/Speaker Extraordinaire – A man with his credentials doesn’t need an intro.
Me - Yeah, Im going to have a seat at the table this year. I havent participated in an actual abstract selection team in a couple of years so this should be a fun diversion.
In building this small team, I’m expecting that there wont be a heck of a lot of “work” to do. I’m really expecting that we will be able to rely mainly on the members preferences but I wanted to have insurance in case there was more to the selection than simply relying on preferences.
The multiple similar session problem
In the past one dilemma that Ive been looking for creative ways to solve goes something like this.
“If I specialize in Architecture for SQL Flux capacitors and Veteran speaker XYZ always presents a precon on SQL flux capacitor architecture how will I ever get chosen for a precon”
This year, I’m hoping that with the membership showing which sessions they’d like to see if a particular topic has an apparent huge interest, we can potentially give 2 precons on a similar topic (1 on Monday, and 1 on Tuesday). The way the tool is being built, we should get good data about which precons a group of people would most like to see. I dont know if this will work or not but, its a small enough risk that I cant see why we wouldnt at least try to see what the data tells us.
After thinking about the data this tool should generate, I wonder where I can find a good data analysis person to volunteer and tell me what it means. Maybe I should ask someone in the community for help with that before mucking it up myself *hint*
The results are in!!!
After tabulating over ten thousand distinct session evaluations for the 2010 PASS Summit we are pleased to release the top 10 sessions overall and the top 5 sessions per track.
Getting these session results generated and out to the speakers in a timely manner is always challenging. After taking until the second week of January 2010 to return Speaker Evaluations for the 2009 Summit we put in sweeping changes to prevent that from happening again in 2010.
Fortunately we were very successful in getting the data, We (Community Volunteers) designed and built a database to house the eval info, and designed a system that could be used to enter the evaluations quickly during and shortly after the Summit. This was a resounding success. Unfortunately where we fell short was in executing on delivering the data to the speakers and the community. When we designed these systems, the process to send out the evaluations wasnt really discussed, or possibly just wasnt finished (the perils of distributing work include less insight into exact issues). Either way, I wound up in the 23rd hour reworking last years SSIS package to fit the new database schema.
We delivered Speaker evaluations to the speakers a full 3 weeks earlier than last year. This included additional info about overall speaker scores that we had never provided in the past. I realize a success to me (3 weeks sooner) is still a failure to others (4 weeks after the summit to get the data to the speakers) We’re going to be working on improving this for next years summit but for now, Ill take the wins where I can get them!
Getting the top 10 sessions posted has taken an extra 3 weeks. I take full responsibility on this one. I had the data on my laptop for the entire time, at first it was the holidays, then it was something shiny, after that I kept running into issues trying to make queries that werent just usable for this years summit, but would be able to generate similar results for any event we enter into this database. In the end though, I have a set of queries for this process that will be reused.
This database/process was one of the projects a large group of OUTSTANDING Community members chipped in and worked on under the umbrella of the program committee in 2010. I have big plans to round up another set of volunteers and put a web based front end on the db and push its use out to all SQL events that would like to use it. The information that we’re gathering will be invaluable to both the speakers and to the community in the future.
Oh, Hey, Dear reader if you’re still reading this far into the babble I guess your looking for the Best of PASS Summit 2010 Link right? Without further ado…..
PS: No, Adam Machanic it didnt take 257 weeks to get this out
I’ve been kicking around several ideas in the program committee and a couple of them have to do with what information PASS releases, specifically information about the speakers. The general question I’ve been trying to come up with ideas about, and the subject of this blog post is:
Should PASS Release Speaker Evaluation Scores to the public
As with all things, there are goods and bad’s to releasing this data. And there are even more possible ways to release the data.
Personally, I’d like to release aggregate scores for every session at every summit. I’m not sure that is a practical option though. I’m hoping some people out in the community might tell me that I’m thinking this through too much and no one cares, after all many of our speakers use speakerrate.com which is totally open.
Everyone who purchased the Summit DVD’s would know when looking at their session lists which to focus on first
Every speaker would know exactly how they compared to others, I think this would be especially helpful to speakers who are just starting their craft.
New conference attendee’s would know which speakers have better ratings and could plan their itineraries appropriately.
Every speaker may not appreciate their scores being published (especially lower rated speakers)
Privacy: See above
We could simply require that if you want to speak at the Summit, we will release the results. We could put some language in the terms that are agreed to by the speakers, actually I think we could do this without changing the current terms but, I do not think this is the right thing to do, it just doesn’t pass the smell test for me.
We could include an opt-in on the speaker terms, which would allow disclosure. While this would be the easiest option to implement with the least amount of backlash, I don’t think it really accomplishes much, mainly because without the complete picture of the scores you wouldn’t know how the speakers ranked or if you were a speaker it would make it tough to know for sure where your session stood.
We could release the complete score list to every speaker, this would solve the speakers knowing where they rate in the crowd. Unfortunately It wouldn’t help those in the community know which speakers were the best.
As with all things PASS I’m open to suggestions, and maybe I’m missing something, let me know