11.30.08
Posted in Uncategorized at 10:53 pm by DMZ
I spent part of this weekend finishing up a presentation I’m giving to a new dev team about what a program manager does at Expedia, and how we work with the developers in particular. And I spent some time looking through my schedule at what I was doing, where my time was actually going, and what I really liked about my job.
Expedia shuts down when we do major releases (this is bad, yes, and also outside the scope here). We run a series of shifts on release nights: the A shift runs from before we start to roll servers out of rotation and ends at about seven in the morning, while B comes in at six (and C, about ten hours later, and so on). There’s an all-Expedia crew steering the ship, and then every team has its own mini-team (so there’s an overall release PM, tester, dev, as well as EU PM/test/dev).
It’s really nasty work. It’s bug investigation on extremely tight deadlines, it’s figuring out if the Dutch site is supposed to have insurance in the package path by default or not while the Dutch site is down… and with any roll-out the size of the Expedia code base, on a server farm the size of Expedia’s, strange things can happen where one server in the French cluster decides to serve bad pages one out of ten times.. and someone has to track that down.
You’d expect anyone with any sense would duck this — it’s so ugly, and all you get is comp time, so it’s entirely reasonable to think that everyone with an ounce of sense would sluff it off on the newbies, and as much as managers would like to have their best there, they can’t draft people.
No. Program managers who have projects in the release want to be around to troubleshoot and see their work go live, which means that the PMs on A/B are disproportionately PMs who regularly ship things, which somehow selects for an outstanding section of the PM corps. And the same’s true of the dev and test organizations.
On release morning, there’s a triage meeting where all the open bugs on the release are reviewed when both A and B shifts are there. It’s traditionally in a too-small room, with all of the finest people who sign up for those two worst duty shifts packed in, standing and sitting everywhere. The A-shift people are about to fall asleep on their feet and look disheveled, and the B-shift are smiling and eager, on their first cup of coffee, freshly-showered, often in clothes too casual for normal work days. Almost everyone knows everyone, and as the group goes through the bugs, teams doing badly have crap flipped at them by people they’ve worked with for years, and after they’re done, they’ll volunteer to help out.
I love that meeting. I love the camaraderie, the gallows humor, seeing everyone together with a single shared purpose, the constant pacing of release management as we go down the list of everything wrong in our world.
“Bug 77884, EU, French hotel path failing, what’s going on with that?”
“Ops didn’t copy all our files over to EU-FRA-06, we’re pulling it from rotation so you can do replication again.”
Much booing.
“Sorry about that. Bug 77886, APAC…”
That meeting is the heart of Expedia. The people who can’t stop going putting their hands up for the A/B shift after a couple years, who are shipping something each time there’s a release window, the senior individual contributors who want to read through error logs at 4am on a Friday because bookings are dropping… those are the people who make shit happen.
I love them. I love being one of them.
Permalink
11.07.08
Posted in Uncategorized at 8:12 pm by DMZ
(throwing on the cloud for future generations)
I have a single-handle Moen kitchen faucet (it’s a Moen 87787) with sprayer which began to choke up a while ago, with low water pressure in both faucet and sprayer.
Here’s the faucet (which may not have the soap dispenser there)
(part of a Moen diagram)
This is frequently a problem with the diverter, which is the part that controls flow between the faucet and the sprayer. It lives under the spout and not the handle. If your house has old pipes that occasionally flake a bit of rust into the water or hard water that results in mineral or calcium build ups, that’ll clog the diverter and you’ll get no pressure to one, the other, and so on.
(from the Moen exploded parts diagram)
Here’s how you fix it. If you can swing a wrench and don’t have any access problems using it under the sink, it might take an hour to do this. The harder it is to disconnect and reconnect stuff under the sink, the tougher this will be.
Repair instructions
1. Remove the spout. You can do this two ways:
- reverse the installation instructions, leaving the spout body in place
- disconnect the hoses, undo the fastener piece, and pull the whole spout out with assembly and spout body. Then pull the spout assembly off the spout body by unscrewing it.
The first one was a major pain in the ass for me, so I went with the second and it worked well. Then
2. Now within the spout body
You’ll see a small plastic piece, white within the brass spout body. It’ll look like it wants to take a giant hex key.
(I believe that’s a Moen part drawing)
3. Remove the diverter
If you don’t have a giant Allen wrench, you can also take a pair of needle-nose pliers, grab an edge carefully, and rotate it out.
4. Clean the blockage
Examine it and the spout body. If the diverter is the problem, you should pretty clearly see the bits of rust or buildup blocking the holes and water path. Wash out the spout body, then take the diverter and do your worst. If it’s a calcium buildup, you can use vinegar (or whatever your fancy is). You can use water softener if that’s the problem. For me, it was the rust bits, and I just had to brush the whole thing out and replace it.
You can, if want to spend the money, just buy a new diverter from Moen or whoever and have it in hand for easy drop-in replacement.
Then it’s easy.
5. Put the diverter back in the spout body
6. Put the spout assembly back on the spout body
7. (if you took the whole thing out, put it back)
8. Reconnect hoses and fasteners.
Ta da!
Now turn the water back on and test it out. If it was the issue (and you’ll know that at step 4) you should now get the same water pressure as you had before, and the sprayer should work great again.
If it’s not, well, sorry. It’s probably the cartridge. But that’s it for me. Hopefully this will help future generations of people and save them from having to spend a ton of money on kitchen sink repair. And ideally, that particular part wouldn’t be so easy to clog, but I won’t hold my breath for design improvements.
Permalink
Posted in Uncategorized at 3:33 pm by DMZ
I can do plumbing, but it’s by far the least favorite home-improvement task. Wait… insulation… no. Plumbing combines all the worst elements: it’s often cold and wet, involves being sprayed in the face, awkward positions, there’s a lot of cuts and skinned knuckles, cramped working conditions, and a certain amount of incomprehensible results, where you can put together a beautiful set of connections and have it leak, take it apart, stare at them and see nothing wrong, and then put it all back together and have it not leak. Drive me batty.
I had the day off today (I have to work tomorrow) so I spent my time working on my kitchen faucet.
Digression: buying an entirely new faucet set is cheaper than calling a plumber to come out and see why your existing faucet doesn’t work unless you bought a really expensive faucet set. It’s almost cheaper to buy a new faucet set than it is to buy a replacement cartridge.
Anyway, so I get the whole thing apart again, figure out that the part I suspect is at fault is almost certainly at fault, and I decide to clean deposits off some of the parts by soaking them in a vinegar solution. Only to discover that in working under the sink, I had taken two cuts, one on the tip of my index finger and another on a knuckle… both of which I managed to briefly immerse in undiluted vinegar.
Fortunately I was the only one at home, so my cursing didn’t result in a citation for disturbing the peace. This is why you pay the plumber.
Permalink