I spent part of this weekend finishing up a presentation I’m giving to a new dev team about what a program manager does at Expedia, and how we work with the developers in particular. And I spent some time looking through my schedule at what I was doing, where my time was actually going, and what I really liked about my job.

Expedia shuts down when we do major releases (this is bad, yes, and also outside the scope here). We run a series of shifts on release nights: the A shift runs from before we start to roll servers out of rotation and ends at about seven in the morning, while B comes in at six (and C, about ten hours later, and so on). There’s an all-Expedia crew steering the ship, and then every team has its own mini-team (so there’s an overall release PM, tester, dev, as well as EU PM/test/dev).

It’s really nasty work. It’s bug investigation on extremely tight deadlines, it’s figuring out if the Dutch site is supposed to have insurance in the package path by default or not while the Dutch site is down… and with any roll-out the size of the Expedia code base, on a server farm the size of Expedia’s, strange things can happen where one server in the French cluster decides to serve bad pages one out of ten times.. and someone has to track that down.

You’d expect anyone with any sense would duck this — it’s so ugly, and all you get is comp time, so it’s entirely reasonable to think that everyone with an ounce of sense would sluff it off on the newbies, and as much as managers would like to have their best there, they can’t draft people.

No. Program managers who have projects in the release want to be around to troubleshoot and see their work go live, which means that the PMs on A/B are disproportionately PMs who regularly ship things, which somehow selects for an outstanding section of the PM corps. And the same’s true of the dev and test organizations.

On release morning, there’s a triage meeting where all the open bugs on the release are reviewed when both A and B shifts are there. It’s traditionally in a too-small room, with all of the finest people who sign up for those two worst duty shifts packed in, standing and sitting everywhere. The A-shift people are about to fall asleep on their feet and look disheveled, and the B-shift are smiling and eager, on their first cup of coffee, freshly-showered, often in clothes too casual for normal work days. Almost everyone knows everyone, and as the group goes through the bugs, teams doing badly have crap flipped at them by people they’ve worked with for years, and after they’re done, they’ll volunteer to help out.

I love that meeting. I love the camaraderie, the gallows humor, seeing everyone together with a single shared purpose, the constant pacing of release management as we go down the list of everything wrong in our world.

“Bug 77884, EU, French hotel path failing, what’s going on with that?”
“Ops didn’t copy all our files over to EU-FRA-06, we’re pulling it from rotation so you can do replication again.”
Much booing.
“Sorry about that. Bug 77886, APAC…”

That meeting is the heart of Expedia. The people who can’t stop going putting their hands up for the A/B shift after a couple years, who are shipping something each time there’s a release window, the senior individual contributors who want to read through error logs at 4am on a Friday because bookings are dropping… those are the people who make shit happen.

I love them. I love being one of them.

(throwing on the cloud for future generations)

I have a single-handle Moen kitchen faucet (it’s a Moen 87787) with sprayer which began to choke up a while ago, with low water pressure in both faucet and sprayer.

Here’s the faucet (which may not have the soap dispenser there)

(part of a Moen diagram)

This is frequently a problem with the diverter, which is the part that controls flow between the faucet and the sprayer. It lives under the spout and not the handle. If your house has old pipes that occasionally flake a bit of rust into the water or hard water that results in mineral or calcium build ups, that’ll clog the diverter and you’ll get no pressure to one, the other, and so on.

(from the Moen exploded parts diagram)

Here’s how you fix it. If you can swing a wrench and don’t have any access problems using it under the sink, it might take an hour to do this. The harder it is to disconnect and reconnect stuff under the sink, the tougher this will be.

Repair instructions
1. Remove the spout. You can do this two ways:
– reverse the installation instructions, leaving the spout body in place
– disconnect the hoses, undo the fastener piece, and pull the whole spout out with assembly and spout body. Then pull the spout assembly off the spout body by unscrewing it.

The first one was a major pain in the ass for me, so I went with the second and it worked well. Then

2. Now within the spout body

You’ll see a small plastic piece, white within the brass spout body. It’ll look like it wants to take a giant hex key.

(I believe that’s a Moen part drawing)

3. Remove the diverter
If you don’t have a giant Allen wrench, you can also take a pair of needle-nose pliers, grab an edge carefully, and rotate it out.

4. Clean the blockage
Examine it and the spout body. If the diverter is the problem, you should pretty clearly see the bits of rust or buildup blocking the holes and water path. Wash out the spout body, then take the diverter and do your worst. If it’s a calcium buildup, you can use vinegar (or whatever your fancy is). You can use water softener if that’s the problem. For me, it was the rust bits, and I just had to brush the whole thing out and replace it.

You can, if want to spend the money, just buy a new diverter from Moen or whoever and have it in hand for easy drop-in replacement.

Then it’s easy.

5. Put the diverter back in the spout body
6. Put the spout assembly back on the spout body
7. (if you took the whole thing out, put it back)
8. Reconnect hoses and fasteners.

Ta da!

Now turn the water back on and test it out. If it was the issue (and you’ll know that at step 4) you should now get the same water pressure as you had before, and the sprayer should work great again.

If it’s not, well, sorry. It’s probably the cartridge. But that’s it for me. Hopefully this will help future generations of people and save them from having to spend a ton of money on kitchen sink repair. And ideally, that particular part wouldn’t be so easy to clog, but I won’t hold my breath for design improvements.

I can do plumbing, but it’s by far the least favorite home-improvement task. Wait… insulation… no. Plumbing combines all the worst elements: it’s often cold and wet, involves being sprayed in the face, awkward positions, there’s a lot of cuts and skinned knuckles, cramped working conditions, and a certain amount of incomprehensible results, where you can put together a beautiful set of connections and have it leak, take it apart, stare at them and see nothing wrong, and then put it all back together and have it not leak. Drive me batty.

I had the day off today (I have to work tomorrow) so I spent my time working on my kitchen faucet.

Digression: buying an entirely new faucet set is cheaper than calling a plumber to come out and see why your existing faucet doesn’t work unless you bought a really expensive faucet set. It’s almost cheaper to buy a new faucet set than it is to buy a replacement cartridge.

Anyway, so I get the whole thing apart again, figure out that the part I suspect is at fault is almost certainly at fault, and I decide to clean deposits off some of the parts by soaking them in a vinegar solution. Only to discover that in working under the sink, I had taken two cuts, one on the tip of my index finger and another on a knuckle… both of which I managed to briefly immerse in undiluted vinegar.

Fortunately I was the only one at home, so my cursing didn’t result in a citation for disturbing the peace. This is why you pay the plumber.

The OB stands for “Oriental Brewery”. Another fine beer I picked up shopping at the “International” grocery store near my house. Has a nice blue can:

Beer can

The small text there reads, and this is exact:

Timeless enduring heritage, craftsmanship and new rice addition delivers refreshing smoothness and clean after taste, making OB the most drinkable beer

The can notes that Oriental Brewery’s been in business since 1933, so timeless… ummmm…

Anyway, it tastes like other rice beers. I’d have trouble picking it out from Bud in a lineup.

German in Germany
Fluency: long-broken college-level

There’s a comprehension line, where if they understood what I was saying, they’d just keep on going. If they didn’t, and they were young, they’d swap over to English. If they weren’t, and they didn’t, things got hairy. No one ever cared that I was speaking German either way.

Flemish in Belgium/Netherlands
Fluency: some traveler phrases, aided by knowledge of German

They were either delighted I tried or nice about figuring out what I meant. I frankly gave up, though, having already spent two weeks trying to speak German, after discovering that everyone spoke English.

French in France
Fluency: long, long-forgotten high school French, study

I was surprised at how nice Parisians were about French, to the point where they’d stop and talk to me about how to pronounce things. I felt like they were interested in having me speak French well.

…anyway.

I just got back from wandering around Munich, where I’d estimate I had several liters of delicious beer for approximately one euro, and on TV is the Simpsons… in German. It’s absolutely hilarious.

I’ll have a much longer account of this later, but I wanted to say this: I’m in Berlin, and I’ve been going through the layers of history: there’s a place I saw where Nazi basements were buried under the Berlin Wall, for instance.

Here’s my short point, though: I’ve read account after account about how the Nazis and then the Stasi harassed and tortured dissidents, and they’re things that the US currently employs. Against terrorism suspects, of course, but that was their justification too.

Germany, sixty years after the war, is still grappling with how to confront and acknowledge what happened during Nazi rule, and almost twenty years after East Germany essentially disintegrated, they’re openly stuggling with a set of much fresher wounds from a dictatorship of a different flavor.

We haven’t even stopped torturing people yet.

So, from Germany, I encourage everyone to go throw some money to stop this. Whatever your taste is: Amnesty International, Obama, whatever. This political season I’ve given until it hurts and every time I touch another memorial I feel like I didn’t give enough.

I made a clearly bad shot today, but before I tossed it in the sink before cleaning up to make a second, I tasted it. And for a second, I could taste all the subtle flavors culinary coffee people talk about: there was chocolate, a distinct berry taste, and a sweetness — then like I’d run into a wall, nothing but the most intense bitterness, enough that I spit into a sink and washed immediately.

My second shot was good, but I was thinking of that first taste of the first shot.

First, despite my complaining about their still-broken login page, I really do like eMusic. They’ve got a ton of stuff that I can’t find normally, and they way in front, selling DRM-free MP3s long before the big guys. But they really are acting badly here, and I want to call them out.

Here’s how this works: you sign up for a monthly plan.

Monthly, of course, means “calendar month” right? If I say “once a month” you know what that means. If I set up a monthly reminder — for something like, say, downloading my eMusic quota — that means once a month on, say, the 5th, I get a note to go make sure I get my music.

If you surveyed 100 people about what monthly means, all 100 would say a calendar month.

Not at eMusic.

Q: How does the eMusic subscription work?
A: Our subscription program is simple. For each 30 day period you receive a fixed number of downloads for one low price. Every 30 days your account is refreshed with the appropriate number of downloads. Downloads do not rollover from one period to the next.

30-day period. That’s… odd. Can that be right? It is:

Q: When will my downloads refresh?
A: Your downloads refresh every thirty days at the same time of day that you signed up. This means that your downloads may refresh on a different day each month. To find out when your downloads will refresh go to the Your Account page and look in the Download Summary area.

What happens here is that… say your subscription starts on Jan 1. You’ve got 30 days. Jan 31, your next period starts…. and expires in March. Through the course of the year, it’ll slip back and forth a little, and there’s no notification that your account’s refreshed its quota or that your downloads are about to expire.

eMusic profits in two ways: first and most obviously, instead of 12 monthly fees, they get thirteen in on a calendar year: 12 and the fractional. Or you can say they’re charging you overall for 5 days they shouldn’t — which amounts to a 1.4% hidden charge from the expectation they set in the signup and plan material and the actual implementation.

But moreover, by varying the dates, they get to catch late customers, like me, flat-footed and get a month of revenue where I don’t download anything because I logged in on the 5th to find out my 30d expired the month before. That’s 100% profit and one frustrated customer. That’s crazy! I wonder how many free months they manage to net over a year that way — it’s got to be substantial.

The obvious fix there is to either set up a recurring 30-day reminder, which is annoying and counter to the definition of the plan, or to make sure I check during the middle of the cycle, every cycle, to make sure the date variation doesn’t change up on me.

More than that, though, I’m really disappointed in them. Wasn’t there anyone there who said “hey, we’re setting customers up for frustration and confusion, we should really simplify this one way or the other” or “doesn’t monthly mean something different here?” It’s the revelation of a sudden disconnect between my like for eMusic, with editors I read and all the bands I like, and eMusic the business, which lifted a twenty off me because it could. That sucks.

“File not found.”

Found at Got Futurama