Like most in the SQL community who are fortunate enough to attend the annual summit I am looking forward to next Sunday.  But, unlike most I am probably looking forward to this years summit for slightly different reasons. I’m definitely ready for the plethora of sessions, the great networking and all of the regular things everyone looks forward to (SQLKARAOKE!) but, when you are as heavily involved as I am in leading the Summit Program Committee, just getting it “started” is a huge relief.

For the last four years Ive looked forward to the Sunday before the Summit like ive looked forward to no other Sunday of the year.  However this year is going to be different!  This year will be my last year of leading the Summit “Program Committee”.  Its truly bittersweet for me as Ive really loved doing this but Its time for me to focus on doing other things for our great community.

I plan on handing the reigns of the summit over to _____ after the end of this years Summit.  I wish I knew who was taking over so I could tell you, but as it is no one has volunteered and PASS doesn’t exactly have a structure in place to allow for transnational hand off(more on this later).  I wish I could say it will be all Rainbows and Unicorns for the next person leading the program committee but, the reality is it is a lot of work.  The payoff is quite large though in knowing that you lead a group of 30ish volunteers to do a large portion of the work in making the summit a success but, even with a dedicated support staff (PASSHQ) its still a challenge.  After doing every job from the bottom to the top of this small piece of PASS I  feel like ive literally “Been there, done that, got the T-shirt”

I’d guess Ive spent at least 10 hours a week on average working on the Summit for the last several years and this year I added on regular PASS Board member tasks and am regularly exceeding those 10 hours a week, unfortunately, its just not sustainable for me any longer.  Im happy to spend the time for the community but, Its time for a new challenge and fortunately being on the PASS Board affords me all of the community challenges I could ever need (including figuring out to handle transition within PASS mentioned earlier). 

If you spot me at the Summit, feel free to stop over and introduce yourself, I promise not to try and convince you to volunteer to take my PASS job!

My favorite PASS memory isnt actually a single memory at all, its actually the conglomeration of all the SHARED experiences I have with all the hundreds(?) of people that I have gotten to know rather well over the years.  One of the most important things about our community of SQL professionals that I can never seem to put into words are those “Shared experiences”, they are what makes us unique.  Some first timers will listen to the conversations and think “what in the world is so funny” or “really, did that actually happen” or more likely, “what in the heck are they talking about” and I can see that there would be some intimidation factor with that but, the reality is you just have to jump in and start making those shared experiences of your own.  While some of my favorite shared experiences are from 2004-2006 that doesnt mean that I dont have some great ones from 2010 with people I had just met.  My point really is those experiences (which un/fortunately become memories) whether made at the Summit, a SQL Rally, SQL Saturday, or a local UG meeting are the things that actually tie us together as a group of professionals.  Sure, we all talk SQL and geek about it but, without the shared experience factor we’d all just be names behind posts on a Newsgroup/Forum which wouldnt be nearly as much fun!  

just choose one

If I had to list a single memory though it would be from 2004, where it all started for me.  You see I had just met Pat Wright, and Tom Larock and somehow they convinced me to attend the early morning (7AM If I recall) PASS volunteer session.  Im not an early morning person AT ALL so I definelty felt some peer pressure to go but, I can honestly say sitting in that session hosted by Wayne Snyder has probably changed my professional career as much or more than anything else.  Wayne is a very dynamic speaker and he had us all believing we could change the world, or at least the SQL portion of it.  I really miss those sessions, and wish we could find a way to ressurect them because this introvert would have never become involved with PASS had it not been for that 1 single session (and the encouragement of Tom and Pat) Thanks to the 3 of you for that!

I apologize for the tardiness of this post, its been a busy week with PASS finalizing the Summit Sessions.

As always, there were some awesome posts this month!  If youve ever wondered why you need to prepare to recover your databases, or your life for that matter I suggest reading through the huge amount of content below.

The good stuff

Rob Farley (B | T) Writes us a two part post with half being technical about migrations, downtime and high availability and the other half being personal with regards to dealing with and controlling life’s disasters.  Hats off to Rob for pouring it all out there.  (sometimes it just feels better to write it all down and put it in perspective)

Noel McKinney (B | T) recounts a bad situation where he played the part of message queue during a human disaster where a developers spouse unplugged the telephone in the middle of the night (surprising this didnt cost someone a job)

John Pertell (B | T) tells us about times where he learned lessons the hard way about backups and restores.  His stories hit home for me and im sure they will for most other seasoned DBAs.  Ive lost more SAN arrays over the years to firmware flashes than I care to think about, so much so that I cringe when the SAN admin calls and even utters the word firmware.

Robert Davis (B | T) writes about backing up system configurations in the case of a complete server failure.  Good info in one place here about what you would loose if you lost one of the system databases.

Ricardo Leka (B | T) turns in his post letting us know that its important to have a backup plan but even more important to have a recovery plan! (his post was in portugese so if I’m way off I blame google translate!  Thanks for the post Ricardo)

Merrill Aldrich (B | T) reminds us to be aware of blind spots in the recovery scenario of our companies.  He shares some great info about cultures that can cause disasters to be unrecoverable.

Jack Vamvas (B) Shows us how he uses powershell to gather an inventory of SQL Server info that may be needed in the case of a disaster.

Mark Broadbent (B | T) Writes a post about how others mistakes can often become your problem when corruption lands in your lap.

Muthukkumaran Kaliyamoorthy (B) Goes over the various ways that you can build HA/DR system including Clusters, Mirroring, Replication, etc

Jason E Bacani (B | T) shows once again that backing up a database is important but making sure you are backing up what you think you are backing up is even more important

Bob Pusateri (B | T) recounts a story of a former employer and the resulting problems from having a “if it isn’t broken dont fix it attitude”

Chad Miller (B | T) writes about using powershell and CMS to inventory your SQL Servers

Ryan Adams (B | T) Writes some tips about using and configuring mirroring to prevent disasters

Gail Shaw (B | T) does her best to remind us that disasters arent just huge events in the world but rather most of them involve smaller more isolated events.  Id agree with her analysis and I live in the bullseye of hurricane country!

Nic Cain (B | T) writes about a full scale disaster at a former place of employment.  I see a running joke in these posts about san firmware upgrades being the cause of most DBA disasters.

Robert Pearl (B | T) shares his story of 9/11 and recovering from that disaster.  Things have certainly changed in the years since then.

Amit Banerjee (B | T) gives us 10 key points to keep in mind when thinking about disasters and how to best deal with them

Pinal Dave (B | T) recounts his early days as a DBA and 4 pieces of wisdom that he learned early on

Steve Jones (B | T) Writes about small disasters that arent natural disasters.  He’s right, these types disasters are considerably more likely than a massive natural disaster.

Thomas Rushton (B | T) Shared not one but two posts for this months edition of TSQLTuesda.  He reminds us to test our DR plans and recounts a story of what was likely someone updating every record in a database with the same value.  Which is a common disaster indeed.

Jason Brimhall (B | T) Shared a story of three personal disasters. included is a good tip about recovering the registered servers in ssms after a reinstall

Nick Haslam (B | T) wrote about an experience at a retail organization where a loss of power took out all of the systems.  Seems its often the small things that get overlooked (not that power is small but, often taken for granted)

John Samson (B | T) shared links to his prior posts about DBA responsibilities in planning for recoveries

Nancy Hidy Wilson (B | T) who lives just up the road from me in Houston recounts her own personal story from Hurricane Ike.  I learned I need a chainsaw and a tractor to recover from a hurricane.  Also I was reminded just how far our modern jobs have come in that we can personally experience disaster and move a few hundred miles away and continue to work our day jobs since their systems *should* be designed for uptime!

Thanks again to everyone who participated this month! 

Be on the watch for next months host and consider participating if you havent before!

Its the first week of June and for those of us living along the Gulf and Atlantic coasts of the US, that brings the beginning of hurricane season.  It also means its time for this months installment of T-SQL Tuesday.

This Months Topic

There goes your weekend/month

Disaster Recovery.  This topic is very near and dear to me based on the fact that I live on a barrier island that was the site to the deadliest natural disaster in US history and more recently destroyed by the third costliest hurricane in history.  Needless to say preparing for disasters is nearly instinctive to me which might explain why I’m a DBA but I digress.  Anything you’d like to blog about related to preparing for or recovering from a disaster would be fair game, have a great tip you use to keep backups and recovers running smoothly, a horrific story of recovery gone wrong? or anything else related to keeping your systems online during calamity.  We want to hear it!

My street a month after Hurricane Ike

T-SQL Tuesday info

Originally an idea dreamed up by Adam Machanic (Blog|Twitter), it has become a monthly blog party where the host picks a topic and encourages anyone to write a post on that topic then a day or 3 later produces a roundup post of all the different perspectives from the community.

Rules

• Your post must be published between 00:00 GMT Tuesday June 14, 2011, and 00:00 GMT Wednesday June 15, 2011
• Your post must contain the T-SQL Tuesday logo from above and the image should link back to this blog post.
• Trackbacks should work, but if you don’t see one please link to your post in the comments section below so everyone can see your work

Nice to haves!

• include a reference to T-SQL Tuesday in the title of your post
• tweet about your post using the hash tag #TSQL2sDay
• consider hosting T-SQL Tuesday yourself. Adam Machanic keeps the list, if he let me do it you’re bound to qualify!

Check back in a few days to see the roundup post of all the great stories your peers shared

How many problems do we all see in our shops that arent related to disks?  Since disk related issues would normally be my #1-10 I guess Ill have to start at #11

.
11. ESX misconfigurations

12. Old DBMS Version requirements from vendors

13. Bad DB design

14. Active Directory

15. Network connectivity/configuration

16. JAVA

17. Security architecture & design

18. Vendor Code…

19. Overly complex solutions to satisfy edge cases

This is my list of 9, What are yours?

Ill go out on a limb and tag 3 people

Lori Edwards, you havent written a blog post lately, consider this your invitation

Pat Wright

Andy Leonard

Every successful business knows that its employees are its biggest asset.

As driven employees, we strive to be members of highly functioning teams.  Company management wants a highly effective team, everyone would seem to want the same thing yet it can be hard to achieve for a number of reasons.  Highly effective employees on great teams are the fundamental bedrock of all successful businesses.  But, what is the difference between an average team and an exceptional team?  A interesting discussion I had today about these differences prompted this post.

Are you an above average member of your team?

What makes someone a great technical team member, Is it deep knowledge of a technical subject?  Business application of said technology?  Ability to schmooze the boss?  Any of these things can lead a person to be known as a highly effective employee.  likewise inability to execute on any of these things (plus a host of others!) can lead someone to  be an average performer or worse an under performer.

What are you worth to your team?

Would you consider yourself 10% above average? 20%? , 50%?.  I’ve asked around and when most people are asked their self assessments are that they are at least 20% above average.  With that thought, as a business manager would you consider a 1% above average (what is average anyway ????) employee desirable?

Consider the math

As a team member at a reasonable sized organization lets say you are responsible for 50 Million$ in information (data for the DBA’s reading this).  If you are able to perform at a 1% higher level than average, you should be able to manage about 500K more than an average team member.  If somehow you managed the herculean task of being 20% better than average you’d be able to manage 10 Million$ more.  Likewise If you have a team member who is performing at a lower level , say 1% below average they’d only be able to manage 49.5 Million$.  20% below average and well, lets just say its a bad day to be responsible for 40 Million$.  If you extrapolate those numbers further and compound them yearly for a few years the 1% better employee Vs the 1% underperformer would look something like this

The 1% high performer nets $51,515,050

while the 1% underperforming employee nets $48,514,950

Thats a 3 Million dollar difference for a 1% above average performance. (not bad!!)

End result is the same

The next time you’re debating with your boss about the value you bring to your team, it can be helpful to point out the math especially if you can make a reasonable leap to huge numbers like billions of dollars in assets and 2% above average Vs 1% below.  1 great employee who brings tons to the table and say works at a 5% above average range is potentially worth more than the entire rest of an under-performing team.  Of course, asking your boss for a 500K raise might not go over well either but If you are lucky enough to be able to tie your work product to actual company revenue, this is a great way to show the leaders in your organization how important it is to hire good people, even if they are ONLY better than average by 1%

Reality intervenes

Ive come to the conclusion that there really is no one special thing or “skill” that its important for a DBA to know.  Sure, there are lots of qualities that tend to make a successful DBA.  Things like attention to detail, thirst for knowledge, and an uncanny love of BACON…  The problem is as huge as the SQL Server product is becoming it is also becoming increasingly difficult to be an expert in everything.  I am actually of the opinion that it is impossible to be an expert in everything SQL Server.

Knowing what you dont know

The most important thing for a DBA to know can be summed up in 3 simple words ” I Dont Know”  Why is that important?  Because, knowing what you don’t know, and being able to admit it to those around you is by far the most valuable skill that you can poses as a DBA at any skill level.

When in doubt its always good to remember that its ok to say “I don’t know” The Corollary to this statement is of course “Ill find out, or find someone who does know”.  Ive seen many small issues blow up into large problems over the years because someone didn’t really understand what was going on and they were afraid to let their coworkers (or boss) know that they don’t understand.  In many cases if we’ll just learn to say “I don’t know” some further pain can easily be avoided.

For this months entry I decided to keep it short and sweet, following in my Bits N Bytes theme.

The Meta Script

In the true sense of the word automation, this really doesn’t fit but, in the terms of quickly getting something done that would otherwise be a mundane repetitive task, this can save a world of time.

Lets say we have a list of objects in the Sales Schema and we have a request to grant Select and Insert access to a user for those objects.  There are two approaches, 1 is to grant select and insert to the actual schema like this

GRANT SELECT, INSERT ON SCHEMA::Sales TO BusinessUser 

However you might decide that you only want to grant direct SELECT and INSERT on the tables that exist in the DBO Schema today not those tables which may be created in the future (auditors love to make us do this)

A simple way to automate granting these rights is by writing a script that writes a script like so

    ON obj.schema_id = sch.schema_id

This should give you a result set that looks something like the following:

GRANT SELECT, INSERT ON Sales.People TO BusinessUser
GRANT SELECT, INSERT ON Sales.Sales TO BusinessUser

At this point, run the output in a separate command window and viola you’ve automated that grant of permissions

This may not be true “automation” in the sense that Pat was looking for but, perfecting the ability to write scripts that write scripts is a huge timesaver

What we started with was a sql 2005 sp3 server where 1 of the drives was apparently corrupted, So 2 SQL instances wouldnt start.  They were both erroring with the message :

Error: 9003, Severity: 20, State: 1.
The log scan number (23:5736:37) passed to log scan in database ‘master’ is not valid. This error may indicate data corruption or that the log file (.ldf) does not match the data file (.mdf). If this error occurred during replication, re-create the publication. Otherwise, restore from backup if the problem results in a failure during startup.

Using trace flag 3608 and startup parameters -c -m I set about to do a normal “disaster” recovery of our server

After rebuilding the master database, everything came online successfully.  Then master was recovered from the previous backup.  Once master was online I started getting the very same error message about the model database

Error: 9003, Severity: 20, State: 1.

The LSN (11:999:1) passed to log scan in database ‘model’ is invalid

This would prove to be a trying error!  it took about several iterations and quite a time to figure out exactly what was going on.

On this server after initial setup we had moved the system databases from the install drive to seperate drives for log and data.  When rebuilding master, the system db’s wind up back in the default directories but, after recovering master, the databases are pointed back to the original locations.

Once we got the server started the log scan error message for model showed up so, I began what I thought would be a normal restore of the model database.  Unfortunately, there was no way for model to be restored.  During the restore command, I got alternating messages that the model database log file was corrupted

Error: 3283, Severity: 16, State: 1.

The file “modellog” failed to initialize correctly. Examine the error logs for more detail

The Error 3283 Would be followed by

the database ‘model’ is marked RESTORING and is in a state that does not allow recovery to be run.

After trying various iterations of deleting the existing model log & database files, copying in the newly created ones and running restores, nothing was working.  I began to think the disks were actually having problems, or the backup was bad.  After verifying both the backup and the disk config I was left with only a hail mary –> sp_detach_db

After detaching model, I copied in the newly created model files (from the rebuild of master) and ran sp_attach_db on them.  Once the Model database was attached the instance started successfully!

After the instance started model was restored from the same backup and the instance restarted.  Finally, once the instance came online, it was a standard restore of all the user databases.

Im not sure what about the logscan error in model caused the errors I saw, but, both instances behaved exactly the same.  I had to detach and reattach a blank model to make the other instance work as well.

After going through this, I went back and tried to reproduce the problems by intentionally corrupting model and its transaction log in various ways.  Every corruption I could cause in model behaved as I expected and a simple restore statement worked.  Im still not sure WHY this happened but, hopefully it wont happen again and if it does there wont be so much testing to figure out how to get model online

Since SQL 2005 Alter Authorization has been the right way to change ownership of nearly anything

This is the most common use of alter authorization but there are many more.

sp_changeobjectowner would look like this

sp_changeobjectowner 'Products.Description','AllenK'

While the updated syntax would look like this

ALTER AUTHORIZATION ON OBJECT::[Products].[Description] TO [AllenK]

What really brought this post about was because I was working with certificates recently and at the time I needed to change who the owner and I just exported/imported & recreated them.  It was easy and what I knew would work.  After looking through the syntax for alter authorization I realized I could & probably should have done it this way.  I guess old habits die hard.

ALTER AUTHORIZATION ON CERTIFICATE::[MYCERTIFICATE] TO [CERTOWNER]

Some out there might question why I’m rehashing syntax changes that happened nearly 5 years ago, The reality is that even in 2011 a lot of the installations of SQL Server I look at are still 2005 and many of the people I run into still use old ways of doing things, this is for you (and me)!

ALTER AUTHORIZATION ON OBJECT::[Products].[Description] TO [AllenK]