The Case of the Mysterious Failing Packages

Or, that time I pimp-slapped some GUIDs…

Let me set the stage for you. We’re less than a week out from a major production push. We move the SSIS (2008) packages over to the integration testing server to make sure everything is good to go. On each data load, something fails. Never in the same package, never in the same task, never from the same source data. We think it’s the server. We move the packages again. More failures.

I’m losing sleep, breaking out in a weird rash that may or may not be related to my panic, and my boss is at my cube every hour asking how things are going in his, “I’m trying to be supportive, but fix the damned thing already”, voice. I’m rethinking my Don’t Drink During Work policy. This is not good. I just can’t find anything wrong. I can run the packages independently and they’re all happy green boxes. I can run the whole load from the beginning and it’ll get past the point of the prior failure only to crap its pants further down the line. Now, full disclosure: I sort of suck at SSIS. It’s fairly new for me, and while I’m learning, I still have a long way to go. Let’s just say that my self-confidence was not at an, “I’m Okay, You’re Okay”, sort of level.

After the latest failure and an inappropriate-for-the-workplace, profanity-laden outburst from me where I may or may not have referred to SSIS as “the stinking offspring of a poxy two-dollar…”, oh, never mind… I hear a question from the next cube over. “Have you reset the GUIDs?” What? GUIDs? Uh, no. I stand up, plant myself in my co-worker Tom’s cube, and ask for an explanation. He explains that if packages are copied, sometimes the GUIDs can be the same in two packages. Then, if they end up running in parallel, it can screw up memory management and SSIS freaks out. And as we all know, when SSIS ain’t happy, ain’t nobody happy. We use a stock template for all of our packages that is copied and then saved off as the new package. And, since our dimension loads are pretty standard, if we need to add a new dimension, sometimes it’s easier to copy one load package and modify it rather than building one from scratch. Yes, I’ve definitely done some package copying in my short tenure as an SSIS developer.

Jeff wanders over, listens to my plight and confirms that I should definitely try resetting the GUIDs. These two guys, who have since been inducted into my Personal Heroes Hall of Fame, explain that with BIDS Helper (totally free download), I can reset the GUIDs on a package and hit every object within that package. A quick rundown of the feature and why it is useful is here. Luckily, I had already installed it for a couple of other features it offered, so I got to work. Can’t hurt, right? Our ETL process is fairly complex, and we have a lot of data moving around. Dimension loads, fact loads, auditing, staging, the works. It took about an hour between me and another co-worker, Tim, to step through all the packages, check them out of TFS (Team Foundation Server), reset the GUIDs, and check them back in. It was easy. Just right-click on the package, look down to the bottom of the menu, click Reset GUIDs, confirm, and you’re done.

We move the packages with their new and improved GUIDs back over to the integration testing server, and viola!, data’s flying around like Your Mama insults at a middle school. No failures. Let me reiterate: With no other changes, the sporadic failures stopped when we reset all the GUIDs.

Now, I can’t say with 100% certainty that resetting the GUIDs definitely fixed the problem. All I know is that we did it, and the failures stopped. That’s one hell of a coincidence if it wasn’t the issue. This is right up there with the legendary, “Try rebooting the server”, and his lesser known cousin, “Restart the service”. It is something to try when all else fails.

What did I learn? A few things:
1) SSIS does not have a personal vendetta against me. There was no ghost in the machine.
2) BIDS Helper is my friend. Just install it. You’ll love what it gives you.
3) If I copy a package, I make it part of my routine to reset the GUIDs.
4) When in doubt, curse loudly until someone comes up with an idea (just kidding). Seriously, this is a great example of why we should get up from our desks and enlist the ideas and knowledge of those around us. Turns out that I work with some pretty smart guys.

2 thoughts on “The Case of the Mysterious Failing Packages

  1. While not terribly helpful to the actual problem, curling up in a fetal position under your desk and sucking your thumb for a while sometiems makes you feel better. 🙂

    Seriously, great job of tracking that down. Especially the “ask the co-workers” part. Isn’t it wonderful to have people working with you who not only understand that you are talking about but can actually help solve it?

Leave a Reply