Discovered someâŠinterestingâŠinformation about Maintenance Plans a few weeks ago when we had a DBA leave. I donât think it warrants being called a âbugâ, but it is behavior that I donât think is ideal, and it definitely can cause issues for us and others. The good news here is that I did get a workaround nailed down (thanks to the help of Random Internet Guy), although we havenât decided whether or not weâre going to use it.
How this all started
Recently, we had one of our DBAs leave. (Incidentally, this is the first time ever that Iâve had this happen to me and also the first time one has left this company.) After he left at the end of his last day, I pulled all of his access to the Production systems. We knew that any Agent jobs that he happened to own would start failing, and sure enough, there were a few failures that evening and over the course of the weekend.
One of these failures turned into a little more of an involved situation: The Maintenance Plan on the only production SQL 2008 box he set up failed. With us being used to SQL 2000, this was expected to be an easy fix and life would go on. Except, it wasnât.
Toto, Iâve a feeling this isnât SQL 2000 any more
First attempt to fix this was to change the owner of the SQL Agent Job to âsaâ. Heâs our standard job owner for consistency and because that makes the job impervious to people leaving. The job still failed, although the error was different this time. Obviously the Agent Job owner had some role in all of this, but it wasnât the whole story, as things werenât fixed.
After changing the Job owner, this is the error that occurred:
Executed as user: <SQL Service acct>, [blah blah blah] Started: 11:25:38 AM Error: 2010-10-06 11:25:39.76    Code: 0xC002F210    Source: <not sure if this GUID is secure or not> Execute SQL Task    Description: Executing the query “DECLARE @Guid UNIQUEIDENTIFIER     EXECUTE msdb..sp…” failed with the following error: “The EXECUTE permission was denied on the object ‘sp_maintplan_open_logentry’, database ‘msdb’, schema ‘dbo’.“. Possible failure reasons: Problems with the query, “ResultSet” property not set correctly, parameters not set correctly, or connection not established correctly. End Error Warning: 2010-10-06 11:25:39.79    Code: 0x80019002    Source: OnPreExecute     Description: SSIS Warning Code DTS_W_MAXIMUMERRORCOUNTREACHED. [blah blah blah] Error: 2010-10-06 11:25:40.58    Code: 0xC0024104    Source: Check Database Integrity Task     Description: The Execute method on the task returned error code 0x80131501 (An exception occurred while executing a Transact-SQL statement or batch.). The Execute method must succeed, and indicate the result using an “out” parameter. End Error Error: 2010-10-06 11:25:40.64    Code: 0xC0024104    Source: <another GUID>     Description: The Execute method on the task returned error code 0x80131501 (An exception occurred while executing a Transact-SQL statement or batch.). The Execute method must succeed, and indicate the result using an “out” parameter. End Error Warning: 2010-10-06 11:25:40.64    Code: 0x80019002    Source: OnPostExecute     Description: SSIS Warning Code DTS_W_MAXIMUMERRORCOUNTREACHED. [blah blah blah] End Warning DTExec: The package execution returned DTSER_FAILURE (1). Started: 11:25:38 AM Finished: 11:25:40 AM Elapsed: 2.64 seconds. The package execution failed. The step failed.
A Permissions error in MSDB? Huh?
I dug a little, and the short of it is that the Maintenance Planâs connection string was still set to the departed Adminâs sysadmin account (a SQL login). All of the work that the MP was trying to do, it was still attempting as the other DBA. This, of course, isnât going to get very far.
Maintenance Plans have connection strings?
This seems like a dumb question to ask, but itâs one of those things that I had glossed over in my head and not actually thought about. The answer: Sure they do! MPs are basically SSIS packages these days, and of course those have connection strings/settings. MPs are no different. Seems like a âDURRRâ now, but hindsight, yadda, yadda.
This connection string is controlled by the âManage Connectionsâ button in Maintenance Plan toolbar. If you never mess with this when you create an MP, it uses an auto-generated connection to the Local Server, using whatever name you used to connect to it (thereâs another topic right there, but someday Iâll talk about some quirks with Aliases Iâve run into, and Iâll address that then). This auto-genâd connection also uses the same auth credentials you are currently using. If you only use Windows Auth, you are stuck with that; if in Mixed Mode, a SQL auth account is an option, and you can set it to another account. Of course, you would need to know that accountâs password.
To fix our MPâs connection string, I could change it to my own credentials, but that of course doesnât actually fix anything, and will leave an even bigger mess for whoeverâs left if/when I leave or responsibilities change. I canât set it to Windows Auth, because when you select that, it validates the auth right then and there. Since we donât have Windows Auth SA accounts, that wonât work for us: the AD account that Iâm logged on with doesnât have enough rights. Something else needed to be done.
At this point I realized that this is a fair-sized problem and we canât be the first shop to run into this; I mean, SQL 2005/2008 have been out for a while, and Iâm sure lots of DBAs have left shops that heavily utilize MPs. Time to ask the Senior DBA.
Going nowhere fast
For as common as I expected this to be, I wasnât finding much helpful information. There were some references to this error message, and even someone in the same situation I was in, but most of what I found were dead-end threads.
At one point, I thought for sure that @SQLSarg had it. This blog post contains the revelation that MPs have Owners! His main point of changing the MP owner is to change the owner that the associated Agent Job(s) are set to when you save the MP. Even though I had already changed the job owner to a valid account, I hoped that the MPâs owner had something to do with the credentials that were used in the Planâs connection string.
Of course, it didnât. No behavior change.
Aside: Even though it didnât help with what I was trying to fix at the time, I believe that changing an MPâs owner to a shared or central service account is a good idea. Every time an MP gets modified, the Agent Jobâs owner is changed to the same account as the Planâs Owner. This might not even be you, and when that person leaves the company, you are guaranteed to have execution problems. Thereâs a Connect item open asking to make it easier to change the owner, so if you like Maintenance Plans and donât hate yourself, itâs probably a good idea to log in over there and mash the green arrowhead.
The more I dug, the more dead-end threads I read, the more almost-but-not-quite blog posts I ran across, the more I became reserved to one simple fact:
It is impossible to set the Connection String Credentials in a Maintenance Plan to an account that you donât know the password for.
As that started to sink in, I asked #SQLHelp if that were, in fact, true. I got a couple responses that yes, that was the case (I would be specific, but this was a number of weeks ago, and since Twitter sorta sucks for finding things like that and I canât remember who it was, weâre out of luck). Although not the news I was looking for, I was glad to know that the deal was pretty much sealedâwe were going to need to figure out something else to do.
One last gasp
Since I donât feel like Iâm doing everything I can while troubleshooting a problem if I donât grab for at least one straw, I decided to investigate something that was described in one of the dead-end threads that I had found while searching.
This thread on eggheadcafe is a conversation between âhowardwnospamnospaâ* and Aaron Bertrand, who we all know and love. This one âhowardwâ was in a pretty similar boat to what I was in, and was really close to getting it to work. He was only thwarted because of an apparent quirk when an Instance is in Windows Auth-only mode.
In the last post in the thread, he reports that when he assigns the Local System account (âNT AUTHORITY\SYSTEMâ) as owner of the MP, and sets the connection properties of the plan to use Windows Auth, the plan runs. To attempt to duplicate this, I started with an existing MP on a Dev instance that I had set up (therefore my SQL auth account was the MP owner and the user listed in the Connection Properties) and proceeded with the following steps:
- Using the statements in SQLSargâs post, I changed the owner of the MP to Local System
- I added my Domain account to the Instance with SA rights
- I opened the MP in question with SSMS, switched its connection to use Windows Auth (while logged in with the Domain account used in Step 2), and saved it.
- Checked the MPâs auto-genâd Agent Job to confirm that its owner was NT AUTHORITY\SYSTEM.
- Using a connection with my Domain account, I pulled my SQL accountâs SA rights. This would previously make the MP sub-plan fail.
I ran the job and, holy cow, it worked! đ
Thatâs good, right?
Ehhhh, maybe. Setting the owner to Local System seems pretty dirty. I havenât thought of a way that this would be a security problem, but it still feels like a stupid hoop to jump through. For us, it has the extra hoop of us having to add our Domain accounts to the instances every time we want to set up or change connection info on an MP (once the Windows Auth is set, it doesnât re-validate it when you save the plan, so you only have to do it that one time), then take it out when weâre done.
Those bits aside, this workaround (if you will let me get by without calling this a âhackâ) does allow setting up Maintenance Plans that will live through the DBAs that set them up leaving the company. By technicality, I achieved my goal.
Weâve talked about this setup very briefly at the office, but havenât made a final decision on whether or not to go down this road or to abandon MPs for DB maintenance altogether. Hopefully we will get that decision made before someone else leaves.
But at what cost?
The main cost here is a lot of destruction to my affinity for Maintenance Plans.
Using MPs in general, love them or hate them, is pretty much a religious debate. Usually I fall on the proponent side of those arguments. I think theyâre easy to set up, theyâre pretty flexible, they can dynamically pick up new DBs that get added to the system (not always an advantage), you can cook up some really crazy dependencies within them, and I like pictures & flowchart-y things :-).
After this whole ordeal? Pretty sure Iâve changed my mind. This whole default behavior of how they will stop working when the account that set them up initially is no longer an SA is way too much of a price to pay. What happens when youâre a single DBA shop and you get honked and bail one day? Near as I can tell, if your account gets restricted as it should, backups will stop working, but there wonât be anyone left to notice!
In lieu of MPs, thereâs at least one good option out there that has most all of the desired functionality. While this was going on, @TheSQLGuru posted a link to this set of SPs. I havenât had time yet to really dive into these, but on the surface, they look pretty good.
Final Comment(s)
This entire thing bugs me. Iâm almost certain Iâm missing something. This should be a huge problem for lots of people. I feel like Iâm missing a detail or itâs because we use SQL logins for the most part or some other thing. But⊠in all of the testing that Iâve done, bad things happen when the initial creating user gets SA pulled. Period. I canât get past the feeling that Iâm doing something wrong, but I havenât been able to figure it out yet.
In the meantime⊠I think Maintenance Plans are evil after allâŠ
* Apparently not to be confused with howardwSOLODRAKBANSOLODRAKBANSO[LODRA] (Eve-Online thing; itâs honestly better if you donât ask)