Solving Drobo random unmounting issues

(Writing this up for hopeful discovery of future Drobo owners)

My Drobo started acting up a while ago in an incredibly frustrating way:

  1. The Drobo would sometimes not show up, or not mount, requiring a dance of restarting it, restarting the computer, plugging, unplugging
  2. When it was mounted, you’d get a short while before the Drobo went unresponsive in the middle of an operation, and then it’d unmount (and OS X would throw a warning about dismounting drives improperly)
  3. Sometimes if you left it connected for long enough, it would show up again, hang around for a bit, and then disconnect.

Nothing worked: re-installing software, resets, the “remove drives, reboot the Drobo, wait, turn it off, put the drives back in…” And all the while status-light-wise, and Drobo Dashboard-wise, it reported everything was good

And unhappily, Drobo support costs money, and I’m cheap, so I wasted a ton of time troubleshooting it. As a bonus, their error logging and messaging is either unhelpful or encrypted.

(I feel like if you encrypt your device’s logs, you should offer free support at least for unencrypting the logs and letting the user know what’s up. I’m disappointed in them and will not be purchasing future Drobos. Or recommending them.)

Eventually I pulled each of the drives and checked their SMART status (OK status overall on all drives, though I also pulled the details and one of them had flags, but SMART’s not great (see: Backblaze’s blogs on this). So I cloned them sector-by-sector onto identically-sized drives. The drive with the odd SMART errors (but, again, overall OK status) made some really unsettling noises at a couple points during sustained reads, but the copy went off okay.

Fired it up, and it worked. Drobo came back on, mounted, works fine (for nowwwww….).

I spent some more time hunting around in the Drobo support forums looking for more information, and found someone reporting back on a similar issue said they’d had a drive go bad but the Drobo never reported any issues, and it wasn’t identified until support looked through the encrypted error logs and said “oh, drive number X is going bad, that’s causing your Drobo’s strange behavior.” Clearly, given my success, at least one of my drives was secretly bad and cloning and replacing was the solution

So! May writing this up help at least one future support-stranded Drobo owner: if your Drobo is unmounting randomly, not showing up in the Finder, throwing dismount errors, but the Drobo’s reporting that everything is hunky-dory, and you don’t want to pay for support and you’re willing to take advice from some random fellow owner on the Internet who may not even have the same issue… here’s one approach before you throw your malfunctioning Drobo out the window:

  1. Power it down and pull the drives
  2. Using whatever utility you like, check the high-level SMART status on the drives to see if something’s clearly screwed up
  3. (optional, if they’re all okay) look at the detailed SMART errors and see if any of the drives looks really wonky
  4. If any of them are bad, do a sector-by-sector clone of that drive, swap the clone in, power up the Drobo, see if that works. If yes: yay! If not —
  5. Clone & replace them all, see if that works.

May this work, and may the drive be in good enough shape to successfully clone.

I should also note that as much as I’m annoyed my Drobo was out of support, assuming they would have been able to tell me what was happening and which drive to clone and replace, it would have been worth it to pay for the per-incident to save myself the headache.

Promotions are recognition, not elevation

Or: the importance of good managers and 1-1s

When I was a Program Manager with no Senior title, I went through a period where I didn’t get promoted, not being promoted made me more and more impatient and even resentful, and that in turn prevented me from making progress towards being promoted.

I’ll paraphrase how I started one of my weekly 1-1s with my manager (Brian Keffeler!):

“Wahhhhhh! Why aren’t I a Senior Program Manager? Look at what I’m doing! It’s amazing! Look at these (n) people who are Senior Program Managers and they aren’t working on as big stuff or doing as well! Wah wah wah!”

And Brian, bless him, listened to me until I’d run out of rant and said:

“I’m not going to argue whether you’re doing better than (person) or (person). Set that aside for a second. None of that matters. You’re not going to be promoted because people look at you and think ‘he’s better than a couple of people who already have the title.'”

I thought “Fuck, he’s right.”

He kept on.

“If you want to be a leader, if you want to be promoted because you’re deserving, you need to stop comparing yourself to them. You need to be so good people assume you’re already in that role. You need people to be surprised to find out you’re not a Senior. When title reviews come up, you want everyone in the room to say ‘He’s not already a Senior? What the hell?’ Right? You want your promotion to be a recognition that you’re already successful operating at that level.”

It was one of the moments in my career where the skies parted, sun shone down on me, and trumpets sounded. I knew immediately that not only was he absolutely correct, that if I was ever to be promoted I needed to prove that demonstrating potential wasn’t enough — that I needed to be operating on this next level. But also, and just as importantly, that me being hung up in the petty bullshit of whether I was the best in my pay-grade and whether I was better than some people in the next pay-grade was fucking up my relationships and career, and that I needed to let go of it.

I might have spent years in that destructive spiral, burning myself out generating my own frustration, with a different manager, or if they’d delivered the message at a different time, or in a different manner.

So I went out and did great work, and people started to assume I was already a Senior Program Manager, and then I got promoted.

Brian’s awesome, and I owe him a great debt.

Honesty without obscenity

When I was a Program Manager at Expedia, and Aman Bhutani had just showed up to right the ship through by demonstrating the value of clear leadership, he started a regular “Big Boulders*” meeting with the Program Managers working on the most critical projects, like the giant re-platforming, or new shopping paths, or rethinking the checkout process.

He wanted to get direct feedback on what was going on, unfiltered, and to discover where he could help. We’d show up and give a high-level status using a standardized couple slides showing timelines and dependencies, and if Aman could help by raising an early point of emphasis to another of his peers about a cross-organizational dependency that had historically been trouble, we’d ask.

Aman built trust with us by delivering — if you brought something up that concerned you, and he said he’d go look after it, you could check it off your list of worries.

For us Program Managers, to have his ear and direct engagement was a huge step forward, though dangerous because we didn’t want to report status to him that we hadn’t already talked to our managers about (because at that point we hadn’t entirely recovered from the stabby years). And it was also pressure-filled. Not just because he was there, or because he’d ask amazingly insightful questions you wanted to be prepared for (and to which “I had not thought of that solution, wow” was a perfectly good answer). In front of a peer group of others trusted to deliver the most important projects, you wanted to have your shit together.

Some people didn’t deal with all of this well (each time starting with a forced grin and “It was another great week on the ____ team!”) but in general, Expedia’s Program Manager corps was a lot of no-credit-taking, jump-on-the-grenade, jaded leaders-through-delivering who’d kept at it through some dark years because they believed in the mission, and they’d be honest. But also, still, sometimes you left the door open knowing he’d ask a question, because you didn’t want to volunteer something you were worried about that your boss wasn’t, but it was keeping you up at night**, and you wanted him to know.

After the initial progress, Aman wasn’t satisfied with a true but also wary status report. So at one meeting, he challenged us. He wanted to hear the status with our insights, whatever they might be, into the present and future, no matter how dangerous the truth seemed.

I felt excited that for the first time someone way up the chain was not only recognizing the chain itself distorted and delayed truth, and he wanted to try and bridge that. And because we’d built so much trust, we were safe — it wasn’t a trap.

So off I went.

“We are so fucked,” I started, and I took off from there. “This org is fucking us, this other thing is fucked up, but this team is fucking amazing, totally saved our ass. This thing we bought from a vendor to help is a piece of shit…” I just went the fuck off, running down everything in terms that would have made a stub-toed sailor tell me to calm down.

Aman nodded through the whole thing, entirely even-keeled. When I was done, he said “So first, yes, that’s the transparency I’d like to see.” And then he paused for just a moment and said “But I’d suggest it’s possible for us to get that honesty without the obscenity.”

I felt relieved, and also like I could do better***.

He let that hang out there for a comfortable pause, thanked me, and then we moved to the next person.

It was an important step for me in how I expressed myself, taking this challenge to be concise, and true, and also not angry. Because I realized that while you get some truth in the emotion, you also lose clarity. “Fucked” expresses frustration, but does that express a need or problem to someone who might help you? And for many people, if you’re cursing like crazy, or you’re coming across angry, they’re not going to receive the message at all — and when you’re speaking, sometimes you can’t expect the audience to come to you, and if you want the right outcome, you’ve got to deliver in a way that’s most effective for them.

Afterwards, the Big Boulders meetings got way more raw, without the cursing, and we got to the next level of trust. And that led to things overall improving, and I felt like I’d contributed in some small way to taking that step forward. And taking Aman’s advice, I start trying to consistently hit that level of openness and honesty in all my communication, without the cursing.

DMZ

  • first you figure out the boulders, then you see what rocks you can cram in around them, and then you pour in sand until the container’s full

** if you’re sleeping well, you’re not paying enough attention to your project. It’s why we’re all such coffee fiends

*** the ability to support people while also helping them realize they can — and want to– do better being one of Aman’s super powers

My first job at Expedia, joining a small crack team who all seemed wildly smarter than me* my manager was Tim Besse**. Once I was stuck on a particularly thorny problem over a bit of UX, and he stopped by to help. We brainstormed, we drew all over the whiteboards in my office***, we argued, we revised and we came up with something that solved the tangled issues to everyone’s satisfaction.

Relived, I went to write the whole thing up. Tim, standing back from the whiteboard, shook his head and frowned.

“No,” he said. “This isn’t good enough. We can do better.”

I felt anger, frustration — we’d finally come up with a way out and he wanted to discard it? We both had a long list of other things we needed to figure out. Checking this off and moving on was a huge relief and a victory for everyone.

I looked at him in dismay while he stared at the diagrams. I took a couple deep breaths and let go of the frustration.

“Okay,” I said. “Where do we start?”

We began again. I remember it as taking twice as long before, in wavy boxes with my chicken-scratch handwriting everywhere, we’d found something wildly better in every way.

We looked at each other and smiled. I felt a sense of rightness and satisfaction I hadn’t touched in the previous one.

I’ve carried that with me since: that when you’ve arrived at something that’s good enough, push on it a little. As much as I pride myself on being pragmatic above all else, push on good enough. Does it rattle a little? Is there a little give? Do you feel like there’s a hidden switch that’ll rotate the whole thing?

Take the time. See if you can turn good enough into something amazing. Challenge others to do better.

And believe that when someone says “we can do better” they believe it, and that you can.

Thanks, Tim.

— DMZ

  • In the words of Isaac Jaffe:
    “If you’re dumb, surround yourself with smart people. If you’re smart, surround yourself with smart people who disagree with you.”

** Tim went on to co-found Glassdoor

*** shared. As a Microsoft spin-off, we were all about private & shared offices. It was great! Then they abandoned it and I’ve never since enjoyed such productive work spaces.

Flowcharting cheat sheet

How to go from sketching boxes to producing clear and consistently readable flowcharts, in under 500 words.

My team came across something like this online:

flowchart-no-no

It started a discussion on learning the most basic guidelines for making a good flowchart. I volunteered to write this and share it now in the hopes it’ll help future generations.

Using these will help not only make flowcharts more readable, by being consistent you’ll more easily find errors and things that are unclear in the flow you’re documenting.

Cultural note: this assumes you’re in a language/culture that reads left to right, top to bottom. Adjust as you see fit.

Direction matters

Overall, for chart flow

Reduce effort by flowing as your audience reads: left to right and then down. The chart as a whole all the way down to each object: arrows come in from the left and exit from top/bottom/right.

If you can’t do left-right for the chart (or an object’s connection), top to bottom’s 2nd-best.

dense L to R

Don’t go snake-style:

snake-style

Direction matters in decisions

Yes/No or True/False should go in the same direction each time they’re on the chart. Anything else creates confusion and possibly someone making the wrong choice.

Generally, I’ve found that the positive (“Yes”/”True”) is most easily read if they’re the up in up/down and right in left/right, but as long as you’re consistent it’ll be okay.

Sizing matters

Attempt wherever you can to keep the boxes a consistent size, unless the difference in sizing carries meaning.

Spacing matters

Keep the amount of space between symbols as consistent as you can. If you can, line up things of the same type, like decisions and conclusions, especially if they share something (for instance, they happen at the same time).

Decision boxes

Use them, they help immensely. Two ways to do this.

Recommended: diamond with annotated lines

diamond choice

If possible, put the labels right next to the decision — don’t make people search for what the decision is. They should at the decision point know the answer to the question and be able to immediately know which line to follow.

More readable for some people: diamond with answers. Requires the reader to scan all the landing points for the answer, and making the ‘answers’ obvious might require use of shapes and colors, resulting in more complexity. Still, if you prefer:

decisions as boxes

You will note that this is helped if you’ve already set the viewer’s expectations about which direction is which.

Okay, so let’s see this in practice

Take this:

flowchart-no-no

Applying only the suggestions here and a couple minutes of cleanup, and noting that there’s at least one problem in the flow there that’s concealed by it being a mess:

first pass cleaned up

If I put both of those in front of someone and asked them to follow through the decisions, it’s now much easier to read and figure out what to do.

Good flowing

Let me know if this helped, or if there’s more simple, easy-to-apply guidelines I should include.

DMZ PdM Bookshelf: “Good to Great” by Jim Collins

What I’m doing

As part of our hiring at Simple, there’s a little question at the end:

Please include a cover letter telling us something awesome about you and the projects you’ve worked on, along with the best product management book you’ve ever read, regardless of claimed subject. (You like “Design of Everyday Objects?” So do we.)

We ask because we’re curious about candidates, and we get often get more information in the cover letter than we do in the resume (for instance: why are they in product management? Do they read all of a job listing before applying?)

A surprising proportion don’t have a book. For those who did, I decided I’d read all the books that came up and do a write-up on their place on the Product Management Bookshelf.

Have a suggestion for something I should read? Nope! Gotta apply.

Today’s book tldr

“Good to Great” by Jim Collins

Is it worth reading? Yes but not for the reasons you’re told to read it.

Sarcastic summary: take lessons from several companies about to fall

Well go on then

“Good to Great” is a wonderful example of survivor bias, and a cautionary tale. If you go out and take a successful actor, for instance, you can go into excruciating detail about everything they did (she puts chia seeds in their smoothies!) and attribute their success to those details. But if you don’t look at it the other way — do actors who put chia seeds in their smoothies succeed more often? There may be millions who do and don’t succeed — chia seeds may make it less likely to succeed. Blind imitation doesn’t help.

Here, in selecting companies that seemed destined to rule forever, Collins demonstrates the fallibility of this approach.

He provides eleven examples, and compares them to someone else in their space.

One great company? Fannie Mae!

“For example, the Fannie Mae people were not passionate about the mechanical process of packaging mortgages into market securities. But they were terrifically motivated by the whole idea of helping people of all classes, backgrounds, and races realize the American dream of owning their home.” -p. 110

I’ll suggest that Collins bought into their line, but also that looking back at this, it’s strange to read that and think “perhaps they should have been more passionate about the mechanics of what they’re doing, as the mission relied on it.”

It becomes a cautionary tale that a great mission without sustainable economics is worthless and even dangerous.

“Good to Great” discusses Fannie Mae’s turnaround — seven years later, having brought about a market collapse, the U.S. Government placed them in receivership, at a cost of hundreds of billions of dollars. And they still haven’t spun them back out, because it’s unclear how they’d even do that without crashing the housing market again (and because as housing prices have risen, Fannie Mae’s profitable for the Feds to hold onto, a danger in itself).

This is not isolated. Circuit City went under eight years after the book.

“Let me put it this way: If you had to choose between $1 invested in Circuit City or $1 invested in General Electric on the day that the legendary Jack Welch took over GE in 1981 and held to January 1, 2000, you would have been better off with Circuit City — by six times.” — p. 33

After that through today, you’d have lost ~1/3rd of your GE money and all of your Circuit City money. GE’s market capitalization as I write this is $289 billion dollars.

On the other hand, Bethlehem Steel was gone in two years after the book and Nucor’s up 360%. Wells Fargo took a hit along with Bank of America and then has done wildly better. Kimberly-Clark bought their comparison point.

My point is this approach: asking why the successful are successful, is ultimately limited. You can ask a lucky gambler why they’re so hot at craps and odds are good they’ll have a convincing answer — or at least, one they’re convinced of.

No one at Fannie Mae was examining their success in 2001 and realizing they’d gone wrong. Examining failure, when assumptions are open to question and it’s easier to tease out luck’s role, is in general far more fruitful.

I would bet though that “Great to Gone” would not have sold nearly as well. We look for inspiration, not cautionary tales.

Takeaways for my fellow Product Managers

In hiring consider whether favoring people from high-profile successful companies is potentially survivor bias, and also, whether people from humbler backgrounds might have learned as much or more from their experiences, even failures.

Introducing the DMZ PDM Bookshelf: Horowitz’s “Hard Thing About Hard Things”

What I’m doing

As part of our hiring at Simple, there’s a little question at the end:

Please include a cover letter telling us something awesome about you and the projects you’ve worked on, along with the best product management book you’ve ever read, regardless of claimed subject. (You like “Design of Everyday Objects?” So do we.)

We ask because we’re curious about candidates, and we get often get more information in the cover letter than we do in the resume (for instance: why are they in product management? Do they read all of a job listing before applying?)

A surprising proportion don’t have a book. For those who did, I decided I’d read all the books that came up and do a little write-up on their place on the Product Management Bookshelf.

Have a suggestion for something I should read? Nope! Gotta apply.

Our first book

Today, a leading response in no small part by being massively popular when we opened our listing, Ben Horowitz’s “The Hard Thing About Hard Things”

Is it worth reading? No.

Sarcastic summary: successful wagon-train driver reminisces about how very sore his whip hand got driving those horses to death.

There are some good pieces of advice in here: the compressed “what a Product Manager should do” summary is a succinct and useful description of the job, and there are indeed bits worth reading.

And one must appreciate a business book that isn’t told in fable form.

But it’s not good. The only connecting thread in the book is that Ben Horowitz sure went through some tough times! Over and over, there’s a crisis, but he and his team had to make huge sacrifices!

There’s never a consideration of whether the sacrifices were worth it: he’ll mention the costs his teams took on as evidence of their heroic resolve. And because they came through, it’s all justified. The skies part, options vest, and there’s glory enough for everyone to bask in.

The result is if you see your leadership team reading this and talking about how much they’re inspired by it, you should be wary.

I’ll humbly submit as an outsider, as someone who has not accomplished what he has, that the truly difficult thing would have been to avoid the deathmarches entirely.

And his examples — if reading about one company’s dependence on one customer doesn’t set off all your Product Manager danger senses, you need a vacation.

Your data are racist

Say you’re a university loan administrator. You have one loan and two students, who anonymized seem to you in every way identical: same GPA, same payment history, all that good stuff. You can’t decide. You ask a data-driven startup to determine which one’s the greater risk to default or pay late. You have no idea how they do it, but it comes back —

The answer’s clear: Student A!

Congratulations, you’ve just perpetuated historical racism.

You didn’t know it. The startup didn’t know it: evaluated both students and found Student A’s social networks are stable, and their secondary characteristics are all strongly correlated with future prosperity and loan repayment. Student B scores less well on social network and their high school, home zip code, interests, and demographic data are associated with significantly higher rates of late payments and loan defaults.
From their perspective, they’re telling you — accurately, most likely — which of those two is the better loan risk.

No one at the startup may even know what the factors were. They may have grabbed all the financial data they could get from different sources, tied it to all the demographic data they could find, and set machine learning on the problem, creating a black box that they show is significantly better than a loan officer or risk analyst at guessing who’s going to default. No one

Machine learning goes like this: you feed the machine a sample of, say, 10,000 records, like

Record Foo

  • Zip Code: 98112
  • Married: Yes
  • Age: 24
  • Likes Corgis: Yes
  • Defaulted on a student loan: No
  • Made at least one payment more than 90 days late: No

Record Bar

  • Zip Code: 91121
  • Married: No
  • Age: 34
  • Likes Corgis: No
  • Defaulted on a student loan: Yes
  • Made at least one payment more than 90 days late: Yes

You set the tools on it and it’ll find characteristics and combinations of characteristics that it associates with the outcomes, so that if you hand it a new record: Angela, a 22-year old from Boston, unmarried, doesn’t like Corgis, and your black box says “I’m 95% sure they’ll default.”

It’s the ultimate in finding correlation and assuming causation.

You see how good it is by giving it sets of people where you know the outcome and see what the box predicts.

You don’t even want to know what the characteristics are, because you might dismiss something that turns out to be important (“People who buy riding lawnmowers buy black drip coffee at premium shops? What?”).

Because machine learning is trained up on the past, it means that it’s looking at what people did while being discriminated against, operating at a disadvantage, and so on.

For instance, say you take ZIP Codes as an input to your model. Makes sense, right? That’s a perfectly valid piece of data. It’s a great predictor of future prosperity and wealth. And you can see that people in certain areas are fired more from their jobs, and have a much harder time finding new ones, and so default on payments more often. Is it okay using that as a factor?

Because America spent so long segregating housing, and because those effects continue forward, using ZIP means that given ZIP X I’m 80% certain you’re white. Or 90% if you’re in 98110.

We don’t even have to know, as someone using the model, that someone is black. I just see that people in that zip code predict defaults, or being good on your loan. Or I might not even know that my trained black box loves ZIP Codes.

And if you can use address information to break it down to census tract and census block, you’re even better at making predictions that are about race.

This is true of so many other characteristics. Can I mine your social network and connect you directly to someone who’s been to jail? That’s probably predictive of credit suitability. Oh — black people are ~9 times more likely to be incarcerated.

Are your parents still married? Were they ever married? That’s — oh.

Oh no! You’ve been transported back in time. You’re in London. It’s 1066. William, Duke of Normandy, has just now been crowned. You have a sackful of gold you can loan out. Pretty much everyone wants it, at wildly varying interest rates. Where do you place your bets?

William, right? As a savvy person, you’re vaguely aware that England has a lot of troubles ahead, but generally speaking, you’re betting on those who hold wealth and power to continue to do so.

Good call!

What about say, 500 years later? Same place, 1566. Late-ish Tudor period. You’re putting your money on the Tudors, while probably being really careful to not actually reminding them that they’re Tudors.

Good call!

Betting on the established power structure is always a safe bet. But this means you’re also perpetuating that unjust power structure.

Two people want to start a business. They’re equally skilled. One gets a loan at 10% interest, the other at 3%. Which is more likely to succeed?

Now is the bank even to blame for making that reasonable business decision? After all, some people are worse credit risks that others. Is the bank to disregard a higher profit margin by being realistic about the higher barriers that minorities and women face? Don’t they have a responsibility to their shareholders to look at all the factors?

That’s seductively reasonable. Too see this at scale, look at America’s shameful history of housing discrimination. Blacks were systematically locked out of mortgages and house financing, made to pay extremely high rates without building mortgages, never building equity. At the same time, their white counterparts could buy houses, pay them off, and pass that wealth to their kids. Repeat over generations. Today, about a third of the wealth cap in families, where white families have over $100,000 in assets and minorities almost nothing, comes from the difference in home ownership.

When we evaluate risk based on factors that give us race, or class, we handicap those who have been handicapped generation after generation. We take the crimes of the past and ensure they are enforced forever.

There are things we must do.

First, know that your data will reflect the results of the discriminatory, prejudiced world that produced them. As much as possible, remove or adjust factors that reflect a history of discrimination. Don’t train on prejudice.

Second, know that you must test the application of the model: if you apply your model, are you effectively discriminating against minorities and women? If so, discard the model.

Third, recognize that a neutral, prejudice-free model might seem to test worse against past data than it will in the future, as you do things like make capital cheaper to those who have suffered in the past. Be willing to try and bet on a rosier future.

Citations on wealth disparity:

http://www.demos.org/blog/9/23/14/white-high-school-dropouts-have-more-wealth-black-and-hispanic-college-graduates

http://www.demos.org/publication/racial-wealth-gap-why-policy-matters

Learning Product Management from Examples: Marcos

I first encountered a true Product Manager a couple years in at Expedia, and the experience with good Product Managers led me to becoming one, years later. I’m going to write them up to talk about what makes a Product Manager effective to their team, as I saw them do.

Two brief paragraphs to explain a distinction: Expedia as a Microsoft spin-off had Program Managers, of which I was one, which if you haven’t met them, I’d recommend some of Joel Spolsky’s writing (for example, his definition of the role). You’re an individual contributor — no one reports to you — and responsible for taking varying amounts of business direction, a team of developers and testers, with some fraction of a designer’s time if you’re lucky, and charged with making great things happen. Responsibility without authority. As a bonus, if you’re really good at your job, your team gets the credit, and you take all the blame when things go wrong. Given a goal, a program manager figures out what’s possible, how to get there, and then leads the expedition.

A product manager at Expedia provides the “what we’re doing.” That can be stack-ranking improvements to a particular page or feature, or a whole shopping path, product, or site. They’re responsible for the spreadsheets and figuring out what the right features are.

(All of this changes as Expedia ages, but ignore that for now)

I was put to work on a response to the rising competitive threat of Booking.com. Expedia was an amazing shopping and ecommerce engine getting beaten up in organic search, because it had never been a consideration. We had all kinds of problems: content hidden behind javascript queries, nothing tagged correctly, no cross-linking of any worth, inconsistent URLs, URLs encoded with dozens of parameters…

Anyway. Enter Marcos. I love Marcos. Marcos knew how to do all of that stuff, he had a vision, and the powers that be gave him money to spin up a huge team.

As Expedia did, they assigned a Senior Program Manager for the overall thing, and then mid-line ones like me picked up individual pieces (I’d do, say, the re-jiggering of maps pages and related content).

Here’s what made Marcos special:

He set the vision. He took the time to say “here’s what’s happening in the wider world of search, here’s why this is important, here’s how we’re going to react now with the constraints we have.” When I and another Program Manager were still skeptical, he booked a huge chunk of time with us to just sit, listen to us, and either convince us or convince us to give it some time and see whether it worked out or not.

This made my work so much more effective. I’d be looking at something like the map widget and be able to ask “given this choice, which of these better fits what we’re trying to accomplish?” and almost always be able to make a choice and keep going. And that was true for everyone else – developers, testers, fractional designers.

Marcos made good, quick decisions. I often see Product Managers (and really, everyone) confronted with a question pause too long. More data needs to be gathered, research completed, stakeholders consulted, and all the while time’s a-wasting. Better a good decision immediately than a perfect decision long after the opportunity had passed.

When we’d hit an issue with serious business ramifications — or that looked like they’d be trading one project goal off in favor of another — we’d get a hold of Marcos and say “Vendor Foo is threatening to end-of-life the API we were using, I’ve got two alternatives but one’s massively expensive and the other will require us to leak business data to someone who long-term we know is going to be our enemy”

Marcos would listen intently, ask a couple questions (“Are they threatening because they want something, or are they really trying to end-of-life this?” “Will they help us move over?” then pause, and it felt like the number of seconds he spent approximated level of complexity of the decision, and then he’d say “stay with Vendor Foo. I think they’re bluffing and if they’re not, the cost to maintain support will be less painful than the other options.”

This was so great — we could then get back to making progress right away, that entire problem lifted from us.

Marcos admitted mistakes. Making good decisions quickly means sometimes you blow the decision, and if you’re not a strong product manager, that’s the worst. Marcos would start calls with what he’d gotten wrong. “Hey everyone, so it turned out we were the only person still on Vendor Foo’s old product, and they’re going to shut it off August 1st. We’re going to keep working on them to see if we can extend that, but we need to go with another option. I’m sorry about the wasted work.”

And because he was so honest about it, and because being quick in making the good decision so often saved us so much time, he’d built up an enormous well of goodwill with everyone. So what if Marcos made one bad decision out of twenty, or ten?

Marcos reported results. When we’d launch, he’d follow-up to talk to us about whether we were seeing progress, if we were getting closer to (or farther away from) our goals. We’d get updates on what our competition was doing, and what we might learn from their next steps. We’d get more background and raw education: for almost all of us, we were new to SEO entirely, and understanding concepts like how doing quality SEO also meant improving the user experience — and when they’d come into conflict.

He participated. He loved progress almost as much as I did, and he’d see the value in our increments and make course adjustments if the demos revealed something new. He was a Product Manager who’d see a sprint demo and want to ship it immediately because he’d realize it was a compelling value, and he’d see that what we’d thought would be release-quality would require more work.

It was my first time seeing someone take on and handle adeptly the “what’s the right thing to do?” role, freeing the team to find what was possible and attack the problems, and it made a lasting impression that would eventually lead to me putting on the cap myself.

Thanks Marcos.