Allen Kinsel - SQL DBA » SQL Server

Shadows Rock, Filtering Platform not so much!

Allen Kinsel — Mon, 16 Aug 2010 12:15:34 +0000

RDP remote control (shadowing) of multiple sessions is a great way to allow geographically separated teams to work on the same Server console. You can do this from task manager

Control your co-workers!

Today I had a new install of windows 2008 that was rejecting the attempts at remote control, the error was »remote control failed», nothing was logged in the System or Application event logs. In the Security event log was only one error: «The Windows Filtering Platform has blocked a bind to a local port»

After plenty of fiddling and making sure there was no »firewall» or reason for the filtering platform to be enabled, I came across this command I never knew existed «shadow»

Apparently whatever had the filtering platform angry and blocking access was ok with that simple command. So in this case going to a command window and running «shadow 3» worked perfectly, I could once again see both terminals and the windows filtering platform allowed me to actually work, instead of impeding me at every turn.

The Windows Filtering Platform on Server 2008 and 2008 R2 has been the culprit more times than I can count lately when the «gremlins» are inhabiting our servers, If only there were a way you could turn it off totally, but I guess that's sort of like Internet Explorer, it cant be unbundled from the OS.

Dont forget to vote for the PASS Summit Community Choice sessions

Allen Kinsel — Thu, 12 Aug 2010 15:01:46 +0000

If you are a PASS member, on August 3rd you should have gotten an email inviting you to vote on four of the sessions to be presented at the 2010 PASS Summit.

There are some great sessions up for selection, You can review them here.

If you havent voted yet, be on the lookout today for a reminder. You should get an email with personal links to vote, it will only take a few minutes of your time and your opinion counts!

If you've already voted, I want to thank you, if not please take this time to do so. If we can get enough community interest in this process I can see it growing and morphing into something more so while your opinion counts on the actual selections, your vote can also help shape the future way PASS does session selections.

Sql Server and SSPI handshake failed error hell

Allen Kinsel — Thu, 17 Jun 2010 15:18:00 +0000

The infamous SSPI Failed error strikes again!

One of our SQL servers was generating these errors for «some» Windows logins but not all.

Error: 17806, Severity: 20, State: 2.

SSPI handshake failed with error code 0x8009030c while establishing a connection with integrated security; the connection has been closed. [CLIENT: 192.168.1.1]

Error: 18452, Severity: 14, State: 1.

After exhausting all of the normal troubleshooting for this error (accounts locked, disabled, Sql Service accts, bad connection strings, SPN's, etc.) I spent the next few hours learning more about the way SQL handles authentication requests than I had ever wanted to know.

The Scenario –

A couple of separate individual Windows ID's started generating these errors while attempting connections, all other windows logins were working properly. The connections were initially happening through applications, but also occurred through sqlcmd. When logged in to the server locally with the offending ID's the connections to SQL would succeed.

The Troubleshooting process –

Check all the regular SSPI issues, I wont bore you with the details as they are easily searchable

A relatively easy way of checking the «easy» authentication issues If possible/appropriate is to log into the SQL Server locally with the offending ID and fire up sqlcmd and connect to the server via sqlcmd –Sservername,port –E (by specifying the port you force TCP/IP instead of LPC, thereby forcing the network into the equation)

Verify whether the login is trying to use NTLM or Kerberos (many ways to do this but simplest is to see if there are any other KERBEROS connections on the machine)

SELECT DISTINCT auth_scheme FROM sys.dm_exec_connections
If Kerberos is in use, there are a few additional things to verify related to SPN's, since only NTLM was in use on this server I skipped that

Determine if the accounts were excluded from connecting to the machine through the network through a group policy or some other AD setting

After all of these checked out OK, I began to try and figure out what the error code 0x8009030c meant, turns out, its fairly obvious what the description is : sec_e_logon_denied. This description was so helpful I thought about making this server into a boat anchor but, luckily for my employer the server room is located many miles away and has armed guards.

Since I knew we could logon locally to the SQL Server with the ID that SQL was rejecting with logon denied something else was trying to make my life miserable.

We didn't have logon failure security auditing turned on so, I had no way of getting a better error description, As luck would have it though this would prove instrumental in finding the root cause. To get a better error message, I found this handy KB article detailing steps needed to put net logon into debug mode.

Say hello to my new best friend! -- nltest.exe

After downloading nltest & using it to enable netlogon debugging on the SQL Server, I got this slightly better message in the netlogon.log file

06/15 14:15:39 [LOGON] SamLogon: Network logon of DOMAIN\USER from Laptop Entered

06/15 14:15:39 [CRITICAL] NlPrintRpcDebug: Couldn't get EEInfo for I_NetLogonSamLogonEx: 1761 (may be legitimate for 0xc0000064)

06/15 14:15:39 [LOGON] SamLogon: Network logon of DOMAIN\USER from Laptop Returns 0xC0000064

The error code 0XC0000064 maps to «NO_SUCH_USER»

Since I was currently logged in to the server with the ID that was returning no such user, something else was obviously wrong, and luckily at this point I knew it wasn't SQL.

Running «set log» on the server revealed that a local DC (call it DC1) was servicing the local logon request.

After asking our AD guys about DC1 and its synchronization status, as well as whether the user actually existed there, everything still looked OK.

After looking around a bit more I discovered this gem of a command for nltest to determine which DC will handle a logon request

C:\>nltest /whowill:Domain Account

[16:32:45] Mail message 0 sent successfully (\MAILSLOT\NET\GETDC579)
[16:32:45] Response 0: DC2 D:Domain A:Account (Act found)
The command completed successfully

Even though this command returned «act found» it was returning from DC2. (I dont exactly understand why the same account would authenticate against 2 different DC's based on a local desktop login or a SQL login but it apparently can)

After asking the AD guys about DC2 the light bulbs apparently went off for them as that server actually exists behind a different set of firewalls, in a totally different location. While DC2 would return a ping, the console wouldn't allow logons for some reason. After a quick reboot of DC2, and some magic AD pixie dust (I am not an AD admin, if it wasn't totally obvious from my newfound friend nltest) the windows Id's that were having trouble started authenticating against DC3 and our SSPI errors went away.

Interesting tidbit -- During troubleshooting, I found that this particular SQL Server was authenticating accounts against at least 5 different DC's. Some of this might be expected since there are different domains at play but, I haven't heard a final answer from the AD guys about whether it should work that way.

The solution

Reboot the misbehaving DC, of course there may be other ways to fix this by redirecting requests to a different DC without a reboot but, since it was misbehaving anyway, and the AD experts wanted to reboot so we went with that. A reboot of SQL would have likely solved this problem too but, I hate reboot fixes of issues, they always seem to come back!

Allowing effective developer access to SQL Server

Allen Kinsel — Thu, 29 Apr 2010 14:56:00 +0000

When creating a new application, after going through the entire business analysis & requirements gathering process, normally you wind up with a datamodel that includes many tables and relationships. By this time, depending on the size of the datamodel/system there has been considerable amounts of time invested on all sides. We need a way of preserving this investment of time while still allowing developers to do their thing!

Deploy

Most shops have policies in place for what level of access developers can have in each environment. In many places I've seen, developers are allowed DBO access in development, and some lesser access in the higher environments (read only usually).

After you've deployed the datamodel to the physical database in a development environment, before you grant the developer group dbo access consider all of the time/effort that has been spent making the datamodel what it is. In order to allow the developers to do their jobs but not allow them to modify the actual table/schema layout you can grant a combinations of privileges.

Grant Alter Schema on the schemas where the developers will need to modify database objects (for instance stored procedures and functions)

Grant db_datareader --to allow read access

Grant db_datawriter --to allow write access

Grant Create Procedure, Function, Default, Etc -- Allow developers to do whatever you are comfortable with

Deny Create Table in the database --This restricts all Table based DDL

Optional** Deny Create View, Function, Default, in the database -- Restrict any create/alter permissions as needed.

Important** Alter Schema permissions will allow Alter of ANY object type in the schema that you havent explicitly used a Deny on

Principle of least privilege

This method has proven effective to allow developers to write Stored procs, Functions & Views while still keeping the actual datamodel (tables and relationships usually) in pristine shape. You could also mix and match your own grants/denys on certain object types to allow for unlimited configuration without granting the almighty DBO. Yes, you might say that I'm a paranoid DBA who restricts permissions even in DEV! Of course my great developers would never change a modeled database thereby forcing my hand into figuring out this lockdown of privileges

Runaway System Cache Increase Kills SQL

Allen Kinsel — Wed, 21 Apr 2010 13:47:00 +0000

Ran into this a while back, and we finally found a root cause so, I thought Id put it out here in hopes that it saves at least 1 person the amount of head bashing I had with it

Environment

Windows 2003 Enterprise R2 SP2 w/32GB RAM

SQL Server 2005 standard ed SP3 64bit active/passive cluster

We started seeing this glorious message in the SQL Server Error log.

A significant part of sql server process memory has been paged out. This may result in performance degradation. Duration XX seconds. Working set (KB) XXX, committed (KB) XXX, memory utilization 0%

The message varied slightly but the essence was always the same.

This error message can be too common on systems where SQL memory is misconfigured or where something is unduly pressuring SQL for memory. In this case a quick verification of the settings showed that everything was in order. The first 2 times this happened it was the middle of the night during backups(in the SLA window), so no one really noticed a performance degradation. We didn't think much of it at the time but in hindsight, we should have.

The Failure

Monday morning 8 AM, developer makes a bad update to the database, No problem I say, Litespeed can rollback the transaction, So I start to copy the full db backup+tran logs off the server (~25gb) this is the way we process litespeed recoveries through the log reader. About 3 minutes later the server became totally unresponsive, and the error about paging the SQL process memory was logged. At the time I didn't put 2 and 2 together as this particular server runs a varied workload of about 1500 batches/sec and has anywhere from 1200-2500 connections open at a time, so It could have been anything! After some further digging I figured out that the file copies were causing the sql memory to get paged out. At the time I had never heard of a file copy causing an issue in SQL Server!

The experts weigh in

While looking at the issue 2 perfmon counters stuck out--> Memory\Cached Bytes and Memory\Avail MBytes. While file copies were happening the cache bytes counter would increase very quickly while the avail bytes counter would drop, once the available mbytes dropped to 0 sql server started to page memory out. After a bit of paging, the errors were logged that SQL had its memory paged out and SQL became unresponsive. Since this was a high priority system, I did what any good SQL Server DBA would do, I contacted a few people in my network who may have seen this before. Interestingly enough I got the exact same response from every one of them, «use lock pages in memory» and don't use windows explorer to do huge file copies as this is a known «problem».

Even though I trusted my sources of info, I had a hard time believing that file copies of sizes all the way down to 1GB would cause this sort of havoc without this being something Bing+Google would know about (different file sizes mattered, some sizes worked fine, some would cause the problem).

Workarounds not welcome

I had a valid workaround with lock pages in memory and not using explorer for file copies but, I don't normally like workarounds such as this on systems as important as this one is to us. After a few server rebuilds we finally figured out that we could reproduce this issue on any win2k3 R2 ent ed 64bit server, this would be the clue we finally needed to make a breakthrough. After rebuilding the systems from scratch and loading no drivers except SAN we noticed that we couldn't cause the error! So, after painstakingly adding each and every piece of our standard server build we realized that Symantec AV (10.1.9.9) was the cause. Yes, another file system filter driver was misbehaving.

In looking back through the change communication, we ID'd where a new version of AV was pushed out and we just didn't hit the error soon enough after the installation to put 2 & 2 together. Since disabling AV wasn't an option we started trying to find a setting that specifically caused the problem and happened across a change that could be made and allow AV to run and SQL to not get paged out. By unchecking the network scanning options, the windows cache no longer increases during a (network) file copy. problem solved!!

Some time in the future vendors are going to figure out how to write good file system filter drivers, or they are going to stop trying to use them! After fighting this issue for a few weeks (or was it months) I can only hope this happens sooner rather than later

April Houston SQL Server Users group meeting

Allen Kinsel — Tue, 13 Apr 2010 14:16:00 +0000

In Houston? Hungry for lunch? Want to learn SQL Server from an expert? Head over to the Microsoft office and learn the top SQL mistakes and how to avoid them from Kevin Kline

Not in Houston? or hungry? there's always the live meeting option!

Here are the details from the HASSUG Site:

Houston Area SQL Server User Group Monthly Meeting Reminder

When: Tuesday, April 13, 2010 – 11:30am-1:00pm

Where: Microsoft Houston Office

2000 W. Sam Houston Pkwy. S. #350

Houston, Texas 77042-3615

Pizza and drinks provided by Idera.

LiveMeeting Link:

https://www.livemeeting.com/cc/usergroups/join?id=HASSUG_APR2010&role=attend

Conference Call for audio – 1-888-320-3585 (passcode 76027128)

Topic: Top 10 Mistakes on SQL Server

Speaker: Kevin Kline, Technical Strategy Manager, Quest; Immediate Past President of PASS (blog | twitter)

Whose got my DAC?

Allen Kinsel — Thu, 04 Mar 2010 15:08:00 +0000

What is the DAC?

The Dedicated Admin Connection, Commonly called the DAC is used to manage SQL Server when a regular connection wont succeed. Here's what SQL Books Online (BOL) has to say about the DAC «This diagnostic connection allows an administrator to access SQL Server to execute diagnostic queries and troubleshoot problems even when SQL Server is not responding to standard connection requests.»

DAC Errors

Occasionally, while troubleshooting SQL servers in a large environment, especially one thats managed from many different geographic locations you could come up with this error, if more than 1 person is using the DAC. It should also be noted this only happens if you have remote DAC enabled in your environment

Error 17810

Could not connect because the maximum number of '1' dedicated administrator connections already exists. Before a new connection can be made, the existing dedicated administrator connection must be dropped, either by logging off or ending the process. [CLIENT: 127.0.0.1]

Since I could still connect with a regular connection currently, I set out looking for a query to determine who was using the DAC connection. I whipped this up, and since I couldnt find anything in search, I thought id blog it

select conn.session_id, sess.login_name, sess.nt_domain, sess.nt_user_name, conn.connect_time, conn.last_read, conn.last_write, sess.host_name, conn.client_net_address
    from sys.dm_exec_connections conn
    join sys.endpoints edp
        on conn.endpoint_id = edp.endpoint_id
    join sys.dm_exec_sessions sess
        on sess.session_id = conn.session_id
    where edp.is_admin_endpoint = 1

This should return everything you need to know about who is using your DAC connection so you can ask them to disconnect, or KILL their connection.

Convincing your boss to pay for your training

Allen Kinsel — Mon, 01 Mar 2010 14:11:00 +0000

Things are tight

It seems like budgets are being cut everywhere, and IT training seems to be one of the first places cut. In an earlier post I outlined where you could get a lot of valuable training for relatively little cost. In this post I will explore a few methods that have worked for me over the years to get your boss to say YES to paying for you to refresh your fountain of knowledge.

The direct approach

It may seem obvious but, if you don't ask your employer, they cant agree to send you to a SQLSaturday, or to the annual SQLPASS Summit. So sometimes its just as simple as broaching the subject with your boss. But before you try that continue reading!

Explain the ROI

Many times the people who make the final decisions about training forget what a struggle it can be to not only stay current on technology but to actually excel with it. Last year PASS put together a very good list of ROI points This could be used for any SQL event, make it work for you.

Volunteer

If you volunteer for SQLPASS (at the local or national level) you may be eligible for a discount admission to the annual summit. This discount can range anywhere from 100$ all the way up to a full comp (free) admission. Armed with the discount, and your ROI documentation it can be quite a bit easier to convince your boss to send you to a weeks worth of invaluable training that they perceive to be discounted.

Offer to partner

A few years back this might have been unheard of but, I've recently decided to start trying this method to get approval for some training. Try to combine the ROI documentation with an offer to pick up 1/2 the tab for the training. For instance, split the cost, ask them to pay the airfare to a SQL Saturday, while you pick up the hotel, food and ground transportation. Another idea, offer to pay the airfare/conference registration(Free if you volunteer enough) and let them pick up the rest to send you to the SQLPASS Summit. Either way its a win-win for both parties, you get to make a small investment in your career, and it shows your employer your serious about that career.

Mix and Match

Hopefully you can take a few of these ideas and mix-n-match them to reach the ultimate goal of getting your desired education.

Accessing Top Quality Training

Allen Kinsel — Wed, 24 Feb 2010 15:00:00 +0000

Where did the Training go?

With the economy being what it is, many companies are cutting back on training budgets yet our need for knowledge as database professionals never stops. Luckily the SQL Server community is one of the liveliest around and there are solutions.

Online

I'd estimate that almost everyday of the week you can find live webcasts about various subjects in the SQL Server arena, the problem here is that many of these top quality training events aren't well publicized. I never knew about them until joining Twitter a few years back. As it turns out, many SQL user groups around the world stream their meetings online via live meeting, there are PASS VC's webcasts, Vendor sponsored webcasts, and the list goes on and on.

Local Events

Local events usually put on by your Local PASS Chapter, these are usually monthly events put on by your local PASS Chapter. The added bonus is that these events provide networking opportunities.

Regional Events

For the time being these usually are SQL Saturdays. Held at various metropolitan areas across the US these are currently one of the best bets going for very cheap, very good, education. I would highly recommend that you consider traveling to one of these if they are within a reasonable drive, or better still, a cheap flight

National Events

Generally speaking these are SQLPASS and SQL Connections. I'll just say that these are topflight, all in wonderful opportunities, and if you have the chance Id highly recommend that you attend, everything I said about the other training events, is 10 fold when dealing with a week long immersive opportunity.

SQL/Windows Clustering Stupidity

Allen Kinsel — Thu, 18 Feb 2010 16:04:16 +0000

When is high availibility not?

When its any version of SQL running on a clustered Windows 2003 Server or sql 2005 running on clustered Windows 2008 Server. Seems that while you can dynamically add a disk and use it in SQL when your running a standalone server, if your running in a «highly availiable» cluster configuration you have to take SQL offline to add the disk as a dependency in order to be able to use it within SQL. Here's the proof right from SQL books online.

Im sure there was a good reason to require this at the time but clearly this wasnt thought out in the grand scheme of taking 0 downtime. Anytime a single server has a more highly availiable architecture than your system that is designed for high availibility -->youre doing it wrong, Sorry someone had to say it