The best part of my job is release morning

I spent part of this weekend finishing up a presentation I’m giving to a new dev team about what a program manager does at Expedia, and how we work with the developers in particular. And I spent some time looking through my schedule at what I was doing, where my time was actually going, and what I really liked about my job.

Expedia shuts down when we do major releases (this is bad, yes, and also outside the scope here). We run a series of shifts on release nights: the A shift runs from before we start to roll servers out of rotation and ends at about seven in the morning, while B comes in at six (and C, about ten hours later, and so on). There’s an all-Expedia crew steering the ship, and then every team has its own mini-team (so there’s an overall release PM, tester, dev, as well as EU PM/test/dev).

It’s really nasty work. It’s bug investigation on extremely tight deadlines, it’s figuring out if the Dutch site is supposed to have insurance in the package path by default or not while the Dutch site is down… and with any roll-out the size of the Expedia code base, on a server farm the size of Expedia’s, strange things can happen where one server in the French cluster decides to serve bad pages one out of ten times.. and someone has to track that down.

You’d expect anyone with any sense would duck this — it’s so ugly, and all you get is comp time, so it’s entirely reasonable to think that everyone with an ounce of sense would sluff it off on the newbies, and as much as managers would like to have their best there, they can’t draft people.

No. Program managers who have projects in the release want to be around to troubleshoot and see their work go live, which means that the PMs on A/B are disproportionately PMs who regularly ship things, which somehow selects for an outstanding section of the PM corps. And the same’s true of the dev and test organizations.

On release morning, there’s a triage meeting where all the open bugs on the release are reviewed when both A and B shifts are there. It’s traditionally in a too-small room, with all of the finest people who sign up for those two worst duty shifts packed in, standing and sitting everywhere. The A-shift people are about to fall asleep on their feet and look disheveled, and the B-shift are smiling and eager, on their first cup of coffee, freshly-showered, often in clothes too casual for normal work days. Almost everyone knows everyone, and as the group goes through the bugs, teams doing badly have crap flipped at them by people they’ve worked with for years, and after they’re done, they’ll volunteer to help out.

I love that meeting. I love the camaraderie, the gallows humor, seeing everyone together with a single shared purpose, the constant pacing of release management as we go down the list of everything wrong in our world.

“Bug 77884, EU, French hotel path failing, what’s going on with that?”
“Ops didn’t copy all our files over to EU-FRA-06, we’re pulling it from rotation so you can do replication again.”
Much booing.
“Sorry about that. Bug 77886, APAC…”

That meeting is the heart of Expedia. The people who can’t stop going putting their hands up for the A/B shift after a couple years, who are shipping something each time there’s a release window, the senior individual contributors who want to read through error logs at 4am on a Friday because bookings are dropping… those are the people who make shit happen.

I love them. I love being one of them.