It was November 3rd, 2009 and after a long weekend at the office, I had unknowingly committed the single biggest and most public mistake of my career. A mistake that would impact thousands of customers and get prime coverage on a leading industry news site. To-date, this is the only time I ever honestly feared I’d be fired on the spot.
So what happened? What did I screw up so bad that it warranted press coverage?
I botched the launch of Dragon Age: Origins on Impulse. Customers who bought the game from us could launch the game, but couldn’t download any of the pre-order DLC content they were promised. This combined with the general struggles the game had those first few days meant our version of the game was the most problematic for PC gamers.
How did this happen? The short answer is the game did something exceedingly strange in its setup that I’ve only seen a few games do since: It installed a Windows service to handle authentication and downloading of the game’s DLC.
But the technical how didn’t matter one bit to our customers, or to Ars Technica when they posted a scathing news article advising gamers to avoid Impulse for purchasing the game. People bought a game, a hugely anticipated game that was getting rave reviews, and because I missed something, they couldn’t fully play it.
As I frantically tried to understand what happened, then develop and deploy a fix, one thought went through my head over and over again…
I’m so fired. I’m so fired. I’m so fired…
But regardless of what was going to happen to me, I was now determined to fix this problem. If this was going to be the last launch I worked on for Impulse, I was going to be sure I got it to a working state. For the next 48 hours, I poured over forums, ran diagnostic tools, spoke with developers both at Stardock and EA to figure out what was going on, and what I could do to make it better. With the help of the Impulse IT, QA & Customer Service teams, and some brilliant members of our forum community, we put together and tested an eventual solution to the problem.
As a gamer myself, I was very conscious of what I’d want if I were in this situation waiting for a game I bought to work correctly. So as we toiled to the eventual solution, I took to our forums, our Twitter & Facebook accounts to keep people up to date on what we were doing. As we solved pieces of the puzzle, I posted manual fixes people could apply themselves directly. And then as we worked to integrate and deploy the final fix into the Impulse game package, I posted the utility we built directly for customers to download themselves.
At the end of that 2-day period, we had a game that finally worked fully. Myself and my exhausted coworkers slumped in our chairs, the angry customer emails and phone calls finally winding down.
Then the owner of the company called me into his office.
This was it. I knew I was done. But I was too tired and wrung-out at that point to react. I sat there and probably gave Brad a completely blank zombie stare as I braced for the news. Would there be yelling, or would it be a short “Thanks for your work, but this was it…” talk?
What actually happened wasn’t what I was prepared for…
You did a great job with Dragon Age
Wait… what? I had cost us thousands of dollars in angry customer refunds, got us on the front page of Ars Technica with an article that basically said “DON’T BUY FROM IMPULSE” and it took me two days to fix the problem. What about all of that would constitute a “Great Job”?
It turns out that as we worked to figure out what went wrong, and then get a fix out, the senior folks at the company were keeping a very close eye on what was happening, but decided not to step in immediately to see if the Impulse team could figure it out. And as we learned what actually broke, as I worked late into the night with my colleagues and communicated the process every step of the way in great detail both internally and to our customers publicly, a few things became clear to Brad, my boss, and other directors at the company:
1. I wasn’t going to stop until I fixed the problem
2. I wasn’t shifting blame or trying to minimize the impact
3. No one was taking this harder than I was
4. There was no way I was ever going to be caught by a problem like this again
In short, I owned this problem fully. I owned it internally to my team. I owned it to management. I owned it publicly to all of our customers. This thing happened, and I put my name next to it, and then kept at it through to the end.
How I responded to the botched launch was far more important than botching the launch itself.
This was the greatest lesson I’ve ever had in the importance of ownership. To be honest about your mistakes, and to take responsibility for the resolution. Everyone will screw up at some point, but what separates people is how they react.
P.S. While I was by far the most public face of the whole ordeal at Impulse, and the one who caused it in the first place, I want to make sure I recognize the terrific coworkers who helped me diagnose, solve, test and finally deploy the DLC fix. Kirk Windisch & Laurence Parryfigured out how to set the correct Services permissions and build the installer for the fix.Curtis Hendzell for gathering essential troubleshooting data directly from our customers andJeff Sanders for helping test the dozens upon dozens of potential fixes before we finally got it right.
This is a re-post of an article I originally wrote on LinkedIn where I talk about what I call my Core Work Values. I first post these to LinkedIn and later bring them over to my personal site here. If you want to read them as they’re posted, please feel free to follow/connect with me on LinkedIn. You can find my profile here.