Like most in the SQL community who are fortunate enough to attend the annual summit I am looking forward to next Sunday.  But, unlike most I am probably looking forward to this years summit for slightly different reasons. I’m definitely ready for the plethora of sessions, the great networking and all of the regular things everyone looks forward to (SQLKARAOKE!) but, when you are as heavily involved as I am in leading the Summit Program Committee, just getting it “started” is a huge relief.

For the last four years Ive looked forward to the Sunday before the Summit like ive looked forward to no other Sunday of the year.  However this year is going to be different!  This year will be my last year of leading the Summit “Program Committee”.  Its truly bittersweet for me as Ive really loved doing this but Its time for me to focus on doing other things for our great community.

I plan on handing the reigns of the summit over to _____ after the end of this years Summit.  I wish I knew who was taking over so I could tell you, but as it is no one has volunteered and PASS doesn’t exactly have a structure in place to allow for transnational hand off(more on this later).  I wish I could say it will be all Rainbows and Unicorns for the next person leading the program committee but, the reality is it is a lot of work.  The payoff is quite large though in knowing that you lead a group of 30ish volunteers to do a large portion of the work in making the summit a success but, even with a dedicated support staff (PASSHQ) its still a challenge.  After doing every job from the bottom to the top of this small piece of PASS I  feel like ive literally “Been there, done that, got the T-shirt”

I’d guess Ive spent at least 10 hours a week on average working on the Summit for the last several years and this year I added on regular PASS Board member tasks and am regularly exceeding those 10 hours a week, unfortunately, its just not sustainable for me any longer.  Im happy to spend the time for the community but, Its time for a new challenge and fortunately being on the PASS Board affords me all of the community challenges I could ever need (including figuring out to handle transition within PASS mentioned earlier). 

If you spot me at the Summit, feel free to stop over and introduce yourself, I promise not to try and convince you to volunteer to take my PASS job!

My favorite PASS memory isnt actually a single memory at all, its actually the conglomeration of all the SHARED experiences I have with all the hundreds(?) of people that I have gotten to know rather well over the years.  One of the most important things about our community of SQL professionals that I can never seem to put into words are those “Shared experiences”, they are what makes us unique.  Some first timers will listen to the conversations and think “what in the world is so funny” or “really, did that actually happen” or more likely, “what in the heck are they talking about” and I can see that there would be some intimidation factor with that but, the reality is you just have to jump in and start making those shared experiences of your own.  While some of my favorite shared experiences are from 2004-2006 that doesnt mean that I dont have some great ones from 2010 with people I had just met.  My point really is those experiences (which un/fortunately become memories) whether made at the Summit, a SQL Rally, SQL Saturday, or a local UG meeting are the things that actually tie us together as a group of professionals.  Sure, we all talk SQL and geek about it but, without the shared experience factor we’d all just be names behind posts on a Newsgroup/Forum which wouldnt be nearly as much fun!  

just choose one

If I had to list a single memory though it would be from 2004, where it all started for me.  You see I had just met Pat Wright, and Tom Larock and somehow they convinced me to attend the early morning (7AM If I recall) PASS volunteer session.  Im not an early morning person AT ALL so I definelty felt some peer pressure to go but, I can honestly say sitting in that session hosted by Wayne Snyder has probably changed my professional career as much or more than anything else.  Wayne is a very dynamic speaker and he had us all believing we could change the world, or at least the SQL portion of it.  I really miss those sessions, and wish we could find a way to ressurect them because this introvert would have never become involved with PASS had it not been for that 1 single session (and the encouragement of Tom and Pat) Thanks to the 3 of you for that!

I apologize for the tardiness of this post, its been a busy week with PASS finalizing the Summit Sessions.

As always, there were some awesome posts this month!  If youve ever wondered why you need to prepare to recover your databases, or your life for that matter I suggest reading through the huge amount of content below.

The good stuff

Rob Farley (B | T) Writes us a two part post with half being technical about migrations, downtime and high availability and the other half being personal with regards to dealing with and controlling life’s disasters.  Hats off to Rob for pouring it all out there.  (sometimes it just feels better to write it all down and put it in perspective)

Noel McKinney (B | T) recounts a bad situation where he played the part of message queue during a human disaster where a developers spouse unplugged the telephone in the middle of the night (surprising this didnt cost someone a job)

John Pertell (B | T) tells us about times where he learned lessons the hard way about backups and restores.  His stories hit home for me and im sure they will for most other seasoned DBAs.  Ive lost more SAN arrays over the years to firmware flashes than I care to think about, so much so that I cringe when the SAN admin calls and even utters the word firmware.

Robert Davis (B | T) writes about backing up system configurations in the case of a complete server failure.  Good info in one place here about what you would loose if you lost one of the system databases.

Ricardo Leka (B | T) turns in his post letting us know that its important to have a backup plan but even more important to have a recovery plan! (his post was in portugese so if I’m way off I blame google translate!  Thanks for the post Ricardo)

Merrill Aldrich (B | T) reminds us to be aware of blind spots in the recovery scenario of our companies.  He shares some great info about cultures that can cause disasters to be unrecoverable.

Jack Vamvas (B) Shows us how he uses powershell to gather an inventory of SQL Server info that may be needed in the case of a disaster.

Mark Broadbent (B | T) Writes a post about how others mistakes can often become your problem when corruption lands in your lap.

Muthukkumaran Kaliyamoorthy (B) Goes over the various ways that you can build HA/DR system including Clusters, Mirroring, Replication, etc

Jason E Bacani (B | T) shows once again that backing up a database is important but making sure you are backing up what you think you are backing up is even more important

Bob Pusateri (B | T) recounts a story of a former employer and the resulting problems from having a “if it isn’t broken dont fix it attitude”

Chad Miller (B | T) writes about using powershell and CMS to inventory your SQL Servers

Ryan Adams (B | T) Writes some tips about using and configuring mirroring to prevent disasters

Gail Shaw (B | T) does her best to remind us that disasters arent just huge events in the world but rather most of them involve smaller more isolated events.  Id agree with her analysis and I live in the bullseye of hurricane country!

Nic Cain (B | T) writes about a full scale disaster at a former place of employment.  I see a running joke in these posts about san firmware upgrades being the cause of most DBA disasters.

Robert Pearl (B | T) shares his story of 9/11 and recovering from that disaster.  Things have certainly changed in the years since then.

Amit Banerjee (B | T) gives us 10 key points to keep in mind when thinking about disasters and how to best deal with them

Pinal Dave (B | T) recounts his early days as a DBA and 4 pieces of wisdom that he learned early on

Steve Jones (B | T) Writes about small disasters that arent natural disasters.  He’s right, these types disasters are considerably more likely than a massive natural disaster.

Thomas Rushton (B | T) Shared not one but two posts for this months edition of TSQLTuesda.  He reminds us to test our DR plans and recounts a story of what was likely someone updating every record in a database with the same value.  Which is a common disaster indeed.

Jason Brimhall (B | T) Shared a story of three personal disasters. included is a good tip about recovering the registered servers in ssms after a reinstall

Nick Haslam (B | T) wrote about an experience at a retail organization where a loss of power took out all of the systems.  Seems its often the small things that get overlooked (not that power is small but, often taken for granted)

John Samson (B | T) shared links to his prior posts about DBA responsibilities in planning for recoveries

Nancy Hidy Wilson (B | T) who lives just up the road from me in Houston recounts her own personal story from Hurricane Ike.  I learned I need a chainsaw and a tractor to recover from a hurricane.  Also I was reminded just how far our modern jobs have come in that we can personally experience disaster and move a few hundred miles away and continue to work our day jobs since their systems *should* be designed for uptime!

Thanks again to everyone who participated this month! 

Be on the watch for next months host and consider participating if you havent before!

Stepping out

For as long as I can remember Ive been involved in one way or another with the PASS Summit.. This simply means I have a short memory because I think I first started with the summit in 2006 or maybe it was 2005.  Either way its been a long run at making the premiere event for SQL Professionals a success. During the last PASS Board meeting I let it be known to the other directors that I would like to move on to other things within PASS.  I have spent a huge amount of energy over the last few years attempting to remake the processes around the summit education program.  Ive worked with a great set of volunteers over the years and together we have had some great success in making the processes better and more “translucent” as my favorite volunteer would say.  However, I think its time for a new challenge and Id like to put the same effort into one of the other PASS portfolios.  This will also give someone else with a renewed energy and vision a chance to make their mark on the Summit program.  This is where I’d like to start the handoff, unfortunately there isnt exactly a line at my door of people wanting to take it from me.  So, if you ever wanted to give up your life free time for the good of the SQL community feel free to drop me a line!

Stepping up

At the same board meeting, I stepped up and decided to put my name on the ballot for the PASS executive board.  There are lots of rules and regulations in the bylaws about who can run and when, I wont bore you with all the details but, the end result is out of those eligible to run: Myself, Tom LaRock, Rick Heiges and Douglas McDowell are running for the 2 vice president seats while Bill Graziano is running unopposed for President.  I admittedly didnt start that week thinking I was going to run for the executive board of PASS but, as it turns out it fits nicely with what I want to do for PASS going forward so I decided give it a shot and run.  Hopefully enough of the board will see that I have the right kind of energy to do good in one of the positions.

I appreciate the challenges and opportunity that have been given to me over the years in this community and hope this is merely the next logical step in my service to our community.

This is where the treasure is!

Ever find yourself with tons of extra time just looking for something to dig through?

yeah, me neither… But, I do make it a point to go out and read through lots of PASS documents regularly.  Sure, Some of those documents are not for public consumption but, a large portion of them are available for any PASS Member to view.  Almost all of them will require you to be logged in to the PASS site.

A good starting point is the PASS Governance Page <- lots of good stuff hides on this page, Im working on getting this page removed from behind the login wall

PASS BOD Meeting Minutes are posted on the left hand side

The Feb 2011 Minutes are here

• Good discussions in here about Globalization of PASS, especially revolving around events

The Jan 2011 Minutes are here

• This was an in-person meeting and there is a literal ton of info in here.  Highlights are  globalization, Summit 2011 Planning, Summit 2010 Post mortem, 5 Year plans, Bylaw Changes

PASS Monthly Reports are found in the middle on the left

These are gems that reveal the day to day inner workings of the BOD and HQ

The Feb report should be posted in the next day or 2

The Jan report however, is here

• In here You’ll find things about Chapters, IT Projects, Marketing initiatives, ERC info, Sponsorship Sales, Summit Program, SQLRally, Gloablization, etc

The Dec report is here

• This one contains things like Chapter info, HQ Finance, IT Projects, Marketing, Summit, Rally, 24hop, SQL Saturday,

The budget for PASS is included at the bottom of the governance page

2011 Budget is here

• Wanna know where the money is supposed to be coming from, and where its supposed to be going?  this is where to look.
• Side note: Im going to check into where the 2010 audited financials are, they should be available by now.

The SQL Rally has posted all of the planning meeting notes posted here

• There is tons of good stuff in here, its especially interesting to me to watch the minutes back and forth dealing with very familiar problems as what I’ve seen in the Summit program group.
• Wanna know how many attendees are registered so far for the Rally? yup its in there. Wanna know how many are in Precons?  yup its in there too

We (PASS Program) started posting meeting minutes near the lower left side of this page

• I have written about these minutes before
• Good information in here about many new changes that are being considered by the Program Committee
• Essentially It says that I’m not getting nearly enough done for the program committee lately.  I need to work on that!

PASS Blog

• Im including this here because lost of good stuff gets posted here but, for me I can only find it since its in my RSS Reader.

In Summary, PASS releases a ton of information about what its doing.  The problem with this is two-fold, one its a ton of information.  Two, the information is spread out all over the place and is often difficult to find on the site using conventional browsing methods so I hope this helps

Every successful business knows that its employees are its biggest asset.

As driven employees, we strive to be members of highly functioning teams.  Company management wants a highly effective team, everyone would seem to want the same thing yet it can be hard to achieve for a number of reasons.  Highly effective employees on great teams are the fundamental bedrock of all successful businesses.  But, what is the difference between an average team and an exceptional team?  A interesting discussion I had today about these differences prompted this post.

Are you an above average member of your team?

What makes someone a great technical team member, Is it deep knowledge of a technical subject?  Business application of said technology?  Ability to schmooze the boss?  Any of these things can lead a person to be known as a highly effective employee.  likewise inability to execute on any of these things (plus a host of others!) can lead someone to  be an average performer or worse an under performer.

What are you worth to your team?

Would you consider yourself 10% above average? 20%? , 50%?.  I’ve asked around and when most people are asked their self assessments are that they are at least 20% above average.  With that thought, as a business manager would you consider a 1% above average (what is average anyway ????) employee desirable?

Consider the math

As a team member at a reasonable sized organization lets say you are responsible for 50 Million$ in information (data for the DBA’s reading this).  If you are able to perform at a 1% higher level than average, you should be able to manage about 500K more than an average team member.  If somehow you managed the herculean task of being 20% better than average you’d be able to manage 10 Million$ more.  Likewise If you have a team member who is performing at a lower level , say 1% below average they’d only be able to manage 49.5 Million$.  20% below average and well, lets just say its a bad day to be responsible for 40 Million$.  If you extrapolate those numbers further and compound them yearly for a few years the 1% better employee Vs the 1% underperformer would look something like this

The 1% high performer nets $51,515,050

while the 1% underperforming employee nets $48,514,950

Thats a 3 Million dollar difference for a 1% above average performance. (not bad!!)

End result is the same

The next time you’re debating with your boss about the value you bring to your team, it can be helpful to point out the math especially if you can make a reasonable leap to huge numbers like billions of dollars in assets and 2% above average Vs 1% below.  1 great employee who brings tons to the table and say works at a 5% above average range is potentially worth more than the entire rest of an under-performing team.  Of course, asking your boss for a 500K raise might not go over well either but If you are lucky enough to be able to tie your work product to actual company revenue, this is a great way to show the leaders in your organization how important it is to hire good people, even if they are ONLY better than average by 1%

We published a set of changes to the PASS bylaws yesterday.  Seeing as how Bill Graziano already wrote everything possible and more about these changes and what they mean to the organization I figure Ill save the bits and say that he does a good job summing everything up nicely.  Since a lot of this was done before I joined the board, I wasnt involved in some of the discussion about these changes but, I will say that the discussions I was involved with were very detailed and often labored on the finer points of making sausage.  Needless to say, I agree 100% with these changes and think they make sense.

Whats next?

Ive got far too many PASS irons in the fire currently so I know I wont have time to do the necessary background work to push for a few additional changes but, at some point I’d like to see PASS go to a fully elected Executive committee.  Id like to see the executives be elected board members serving regular two year terms who are then elected by the board for the executive positions.  As part of that I also think Id also like to see a general election of our President.  How better for the membership to guide the organization than to have a direct say about who should be providing the overall direction.

What we started with was a sql 2005 sp3 server where 1 of the drives was apparently corrupted, So 2 SQL instances wouldnt start.  They were both erroring with the message :

Error: 9003, Severity: 20, State: 1.
The log scan number (23:5736:37) passed to log scan in database ‘master’ is not valid. This error may indicate data corruption or that the log file (.ldf) does not match the data file (.mdf). If this error occurred during replication, re-create the publication. Otherwise, restore from backup if the problem results in a failure during startup.

Using trace flag 3608 and startup parameters -c -m I set about to do a normal “disaster” recovery of our server

After rebuilding the master database, everything came online successfully.  Then master was recovered from the previous backup.  Once master was online I started getting the very same error message about the model database

Error: 9003, Severity: 20, State: 1.

The LSN (11:999:1) passed to log scan in database ‘model’ is invalid

This would prove to be a trying error!  it took about several iterations and quite a time to figure out exactly what was going on.

On this server after initial setup we had moved the system databases from the install drive to seperate drives for log and data.  When rebuilding master, the system db’s wind up back in the default directories but, after recovering master, the databases are pointed back to the original locations.

Once we got the server started the log scan error message for model showed up so, I began what I thought would be a normal restore of the model database.  Unfortunately, there was no way for model to be restored.  During the restore command, I got alternating messages that the model database log file was corrupted

Error: 3283, Severity: 16, State: 1.

The file “modellog” failed to initialize correctly. Examine the error logs for more detail

The Error 3283 Would be followed by

the database ‘model’ is marked RESTORING and is in a state that does not allow recovery to be run.

After trying various iterations of deleting the existing model log & database files, copying in the newly created ones and running restores, nothing was working.  I began to think the disks were actually having problems, or the backup was bad.  After verifying both the backup and the disk config I was left with only a hail mary –> sp_detach_db

After detaching model, I copied in the newly created model files (from the rebuild of master) and ran sp_attach_db on them.  Once the Model database was attached the instance started successfully!

After the instance started model was restored from the same backup and the instance restarted.  Finally, once the instance came online, it was a standard restore of all the user databases.

Im not sure what about the logscan error in model caused the errors I saw, but, both instances behaved exactly the same.  I had to detach and reattach a blank model to make the other instance work as well.

After going through this, I went back and tried to reproduce the problems by intentionally corrupting model and its transaction log in various ways.  Every corruption I could cause in model behaved as I expected and a simple restore statement worked.  Im still not sure WHY this happened but, hopefully it wont happen again and if it does there wont be so much testing to figure out how to get model online

<the views that follow are mine alone>

In a single sentence I think Id sum it up like this:

PASS is all about enabling our SQL Community to grow via education and networking.

Details details details

PASS has essentially one job, that job is to enable the many dedicated volunteers across the SQL community to execute on those core values we hear PASS throw about “Connect, Share, Learn”  Unfortunately, enabling these volunteers isnt always the most exciting work, and sometimes its down right benign.

PASS Enables this connecting, sharing and learning several different ways

• PASS Summit
• PASS SQL Rally
• 24 Hours of PASS
• SQL Saturday
• Virtual Chapters
• Regular Chapters

Every one of these has at its core a very strong group of (generally) under recognized volunteers working to make them happen.  PASS has varying levels of involvement in making these different “community events” happen.  Some like the Summit require an extreme level of involvement, others like chapters get very little help.  Sometimes the ratio of “glue” used to hold these things together is incorrect and we try to structure things too much in some places and too little in others.  I think this is a normal occurrence in most established organizations, the pendulum swings to far one way or the other and needs to be pushed back a bit.

PASS divided

PASS the organization is merely the infrastructure that holds all of these pieces together.  PASS the organization is nothing without the community volunteers that put their mark on these different community events.

PASS the community is all of the volunteers that support the organization from the BOD, Speakers, SQL Saturday leaders, chapter leaders, etc.  In many cases the same people serve many of these different roles.  PASS is not the SQL Community, its merely a single player in the community but, it requires a large part of the active members in the community to be successful.

What do you see PASS as?

Hopefully, Ive done a decent job with this quick explanation of what I see PASS as but, as always Id love to hear from you if you have different views of what PASS is, or more importantly where it should be headed

If you hadn’t guessed, today’s post is part of this months TSQL Tuesday.  This is an interesting topic for me since as a matter of principle I usually refuse to make resolutions and the like around the start of the new year.  I like to set goals, and work towards those goals but, I think “resolving” to do something has this nagging way of never turning out how I’d like.  It probably has something to do with the fact that I track goals but, typically only think about resolutions at a point in time.

So, this year Ill resolve to document a few of my goals for the year.

This year I only have a few professional goals.  Actually, quite a few less than usual.  I decided to trim down my professional goals this year to only a couple since they are quite large and very open ended.

1. Id like to make PASS as responsive as possible to the needs of our SQL Community.  This is simply to say that I plan to do what I feel I was elected to do.  Of all the directors I am as well positioned as anyone to make real change that can be seen to the average user of SQL Server.  I will need lots of help to make this happen, and I have no problem asking for that help (watch this space SOON for details)
2. I want to learn to be a better “manager/leader”  It takes a different set of skills to lead people than it does to be a DBA and do technical work.  I love the technical work, actually more than the management stuff but, my current roles are requiring more leadership and less technical.  I need to do better with the details of this and learn to inspire greatness in my teammates.

That’s it, 2 whole goals for the year, not much by count but, by effort I’d say these might be the some of the loftiest goals I’ve set in a long time…