Learning from uncooperative A/B testers

One of the joys of working at a tiny startup packed into an ill-equipped, too-small space was running an account at Khaladi Brothers, the coffee shop across the street, because all small meetings had to be done outside the office. As the top coffee nerd, I took on running fresh vacuum pots over (and yelling “Fresh pot!” as I entered) and exchanging the empty ones. When we moved to a newer, spacious, swanky, and quite expensive office space (hot tip to startups: don’t do this) with an actual kitchen and drip coffee maker, I was put in charge of deciding which coffee beans we’d order. We had many options and an office of high-volume consumers with strong opinions on everything, and needed to get down to one or two for the recurring bulk order.

Naturally, as a Product Manager, I decided to do selection through a series of A/B tests.

Must-have for the tests:

  • end up with a winner
  • clear methodology, publicly exposed
  • easy to participate — or not
  • takes as little as my time as possible, because this was amusing but everything’s on fire all the time
  • keep momentum out of it (so the first voter didn’t disproportionately determine the day’s winner)

I discarded forced-choice, so coffee drinkers didn’t have to vote if they didn’t feel like trying both or didn’t find a winner, I decided against setting up a dedicated “today’s test” table or doing “three-sample can you tell if one’s different” type advanced testing, I didn’t try to timestamp votes to determine if one did well fresh and one did well through the day… nope!

I went straight single-bracket, winner-advances, random seeding at each round. Every day I tried to get to the office before everyone, and made two giant pots of coffee labelled “A” and “B”. If someone wanted to vote for a winner, they could write it down and drop it in a tin, which I tallied at the end of the day. I will admit that having come out of Expedia, where our A/B tests were at colossal scale with live customers, this whole thing seemed trivial and I didn’t spend as much time as I might have.

You may already see where some of this is going. “I know! I too am from the future,” as Mike Birbiglia says.

It was not trivial, and I ended up learning from the experience.

Test assumptions, set baselines: I didn’t have 32 coffees, which was good, because some days I did an A/A test to see what the difference would be. I was surprised, on those days voting for winners was down, and results were remarkably close to 50%/50% — and the highest split was 58% (10/17), which was a vote off a straight split. 

Know that blind tests mean subjects may reject results or: Starbucks did really well. I don’t know what to say. I figured they’d barely beat out the clearly awful generic ones and Tully’s, but some of their whole beans did well got all the way to the semi-finals. Participants were not happy to learn this, came by to ask questions, and generally were reluctant to accept that they’d preferred it. If a Starbucks bean had won but it had made people unhappy, would I have gone through with ordering it? I’m glad I didn’t have to confront that.

Also… yeah, Seattleites have issues with Starbucks.

Consider the potential cost of testing itself. The relatively small amount of time I thought it would take each day turned into way more effort than I’d hoped. Doing testing in public is a colossal hassle. Even having told everyone how I was doing it, during the month this went on, there were those offering constant feedback:

  • it should be double-blind so I don’t know which pot is which
  • it should have three pots, and they might all be the same, or different
  • no, they’re wrong…
  • it’s easy to come in early and see which one I’m making

…and so on. By week two, getting up early to make two pots of coffee as someone offered methodological criticism was an A/B trial of my patience.

If testers can tamper, they will — how will you deal with it? For one example, I came into the kitchen one day to get a refill and a developer was telling everyone he knew what pot was which because he’d seen me brewing and had poured an early cup off that, and so knew the pot with the lower level indicator was that batch. He was clearly delighted to tell everyone drinking coffee which one they’d picked. I honored the day’s results anyway.

This kind of thing happened all the time. At one point I was making the coffee in a conference room to keep the day’s coffees concealed. In a conference room! Like a barbarian!

I was reminded of the perils of pricing A/B experiments, which Amazon was being called out for at the time — if customers know they might be part of a test and start clearing their browser cookies and trying to get into the right bucket, how does that skew the results? “People who reloaded the page over four times converted at a much higher rate… we should encourage refreshing!”

Think through potential “margin of error” decisions when structuring tests. There was a coffee I liked that dominated early rounds and then in the semi-finals lost by two votes to a coffee that had squeaked by in previous rounds by 1-2 votes each time. What should I have done in cases where the vote was so close? I’d decided the winner by any margin would advance, but was that the way it should have been? Should I have had a loser bracket?

In the end, we had a winner, and it was quite good — and far better than what the default choice would have been — but I was left unsatisfied. I’d met the requirements for the test, it’d been a pain in the ass for me but not taken that much time. I couldn’t help but think though that if I’d just set up a giant tasting session for anyone who cared, and let them vote all at once, I’d have saved everyone a lot of trouble and possibly had a better result.

But more importantly, like every other time I’ve done A/B testing in my product management career, the time I spent on the test and in thinking through its implications and the process helped me in every subsequent test, and was well worth it. I encourage everyone to find places to do this kind of lightweight learning. Surely there are dog owners out there wondering what treats are best, and dogs who would be happy to participate (and cheat, if you’re not wary).

Go forth!

Two dual-track agile attempts, and lessons for future generations

I’ve been through two tries at implementing dual-track agile, and I’d like to offer some perspective on the travails, the pros, the cons, and offer some advice to those who might attempt it.

What’s dual-track agile?

In short — you pull a product manager, a designer, and a developer with broad knowledge of the area you’re working in forward in the process, taking on scouting and evaluating work, which then drops into the backlog of whatever agile methodology you’re using.

This is intended to solve the problems of how you develop the stories that a team will work on — the design sprint, the reset sprint, the investigation card, or in carving capacity for those tasks out of the team’s capacity — which in themselves often provoke both a set of methodology challenges and a level of bikeshed argumentation about methodology that can be immensely draining.

Here’s Marty Cagan on this in 2012
And here’s this good Mind the Product article

Which includes this good gif:

Implementation one: everyone jump into the pool at once

At Simple, we’d been through a couple of massive crash re-platformings that hadn’t delivered new features to customers. We two product teams (one, mine, working on what would become Shared Accounts, and another working on long-neglected customer service features, so we were going after things that would create value for our customers, but we were still not shipping.

We brought in Silicon Valley Product Group to do an assessment and recommendation. One of their number came in, did interviews with key stakeholders, and then in a day-long session in which almost everyone was present (we had our recruiters there!), told us what they’d seen as problems and offered a set of prescriptions. The biggest and most systemic one was to go to dual track agile.

Our leadership then declared we’d adopt dual-track agile, and so it was.

It didn’t take. We adopted dual track in name, but in practice, we couldn’t get the developers with the required knowledge to participate, and so discovery withered. Ours could not move into doing that work, being continually pulled into on-call, supporting retiring the deep technical debt, and doing the architecture work that would keep a team working.

Without developer participation, the discovery track could evaluate whether work was valuable to the end user, and whether that value could be realized by customers, but it still meant that items about to go to the team did not have a ROI, because we still needed to figure out the I. And in turn, that meant that there still needed to be an additional step in what should be the “delivery” track to do even high level development investigation.

What could we have done?

  • no methodology exists in a vacuum — consider the people and teams that will be doing the work. The people may be willing, but if circumstance can’t permit, you’re set up to fail
  • if there are changes you can make that would allow you to use a methodology, you have to make the changes or wait — don’t do the reorg or change your approach and just hope
  • a top-down mass implementation wasn’t the way to go — ever the pragmatist, I’d have rather found a team where it was a particularly good fit, done it there, and then learned lessons that could spread. When we started doing Scrum on teams at Expedia, we got approval to try it in three teams with good conditions and very different problems (flight search, hotels, and packages) and were able to learn from each other and measure our success against other teams and spread the good word.

Two: easing into it

As Lead Product Manager at Manifold, I was able to drive methodology. I decided to follow my own advice and do it real subtle-like. We started to use Product Pad, doing more and more discovery activities… and my intention was to use more and more of the process until we’d be using dual-track across the teams without having to talk about it.

During this time, my boss, the VP of Eng, encouraged me to just do it! We’re small, you can do a lunch-and-learn and just go!

I, having been burned by the previous experience, and encouraged by success at gradual adoption of Scrum at Expedia, declined and decided to go ahead with moving forward. Let this be a lesson to us all:

  • If you have the opportunity to jump ahead, and the support to do so, maybe just do it

Dual-track took… kind of. Unfortunately for dual-track, the company made a hard business pivot and organized around long-term contractual obligations (so product teams organized around delivering promised functionality, rather than pursuing objectives of their own). There’s a little bit of work to be done within that, but you’re not going out to do basic user research, address problems, etc.

Fundamentally, dual-track exists to support testing ideas, learning, and exploration. If you’re not doing that as a business, don’t adopt a framework that supports it. It’s like the difference between the road-trip and doing a regular commute. One requires research, planning, friends, snacks, a good dog to hang out the window if you’re lucky, and the other requires you to figure out the vehicle and route once and then pay attention.

I also encountered more resistance in the form of nearly-endless tool and process questioning than I’d expected or was prepared for. I found myself answering questions like “What we do when robosaurs attack local food distribution centers is a good question, and we’d have to talk about what that would mean for ticket handling…””

Now, what was going on? There were two culture clashes with Manifold’s stated values of transparency and autonomy, where anyone should be able to do anything as long as there was an audit log and the action was reversible.

1: dual-track itself seemed to clash with culture: that there would be three people who, outside of a product team’s normal activities, would be making decisions on the direction of the team and what work there was to be done.
2: tooling: first, introducing another tool for people to use raised the “why not just do all this in (JIRA/github/___)?” and in particular, the tool I’d introduced, Product Pad, is great for a lot of things but has a ton of restrictions on roles (for instance, a normal user can’t go add stories to an idea filed by someone else) that rankled people. I had not done enough to consider this.

What happened?

We started, ideas and feedback go into the hopper, there are processes to do discovery, and it’s being used on some more-nebulous pieces to create a roadmap.

I feel like I should have abandoned it when we made our pivot — I should have considered that a change in direction and how we work as a company was worth wiping the white board clean and evaluating the process challenges against what the kind of work we’d be doing over the next 12-18 months.

Questions to ask and things to do as you weigh dual-track.

First, think about this:

  • What’s your elevator pitch for why the change is worth it — what will you gain, or what known harms would it have prevented?
  • What’s that pitch for each of the stakeholders in the process? What’s your pitch to your Marketing partners, for instance?
  • Who’s your champion at the exec level?
  • Consider the culture: will people react poorly to the appearance of a trio of people taking over the direction of the team where once it seemed open to all? What can you do in designing the process and communication to address those concerns?

And then, block out your calendar for a long time so you can concentrate, and

  • Map out the process from end-to-end before any public rollout or discussion.

How will an idea or customer request go through the process? What tools will be used? Consider the maximally complex cases —

  • A developer has an idea
  • We write up the idea — where? How is it tracked?
  • How do you decide that that idea is worth investigating out of all the other ideas? Who makes that decision? Where?
  • We need to do some user research on the idea — who does it? Where is that request tracked? How does the result get tracked?
  • We need to show a prototype to users, and it requires both design and a little bit of dev. How do you start and track that work? How do you record the work?
  • You now know that the idea has merit and can be made usable. How do you track that?
  • How do you cost out the work? How is that work assigned, and tracked?
  • How do you choose what goes onto the roadmap from all of the ideas that have cost/benefit ratings? How?
  • If you’re doing regular user research or usability testing, how will that fit? Where will those results go?
  • How will you communicate each decision out to everyone who is interested in the results?

Then for each person in the process, list how many tools they must use including any new ones, and how they’ll track their work at each stage. If they live in a world of GH issues and you now require them to put their head into Product Pad regularly, or participate in discussions there, you’re adding complexity into their life and they’ll have to see a lot of value come out of this extra effort — which may fall to you as a product manager.

Now you’ve got a rollout plan, of sorts, with an idea of the cost and what will need to happen for the process to be a success, and you can make an informed decision.

In summary, dual-track agile is a methodology of contrasts

Having been through two attempts to implement it, I’m much more likely to look at existing processes and find ways to build in things like regular user testing and fast feedback loops, and see if they improve things — or start to create the need for dedicated discovery process and activities.

I would love to hear other people’s success or failure stories implementing dual-track agile (or any of the latest hot methodologies) — please, drop me a line if you’ve got them.

Obsess over teams to drive organizational change

In “Stop Obsessing Over ‘The Teams
John Cutler argues “we shouldn’t focus on “The Teams”, until the conditions for effective teams are in place.”

Cutler begins:

Most Agile methodologies/frameworks focus myopically on “the teams” (front-line teams of individual contributors). Meanwhile, organizational dysfunctions (those causing 80% of the drag) persist.

And that you have to solve organizational problems first.

In a tweet, Melissa Perri adds

Companies usually ask me to “transform teams” but many of the problems they need fix stem from the org level. Local optimization over global rarely works. You have to work on both.

Everyone’s right — the trick is, sometimes it’s agile methodologies are what let you find the global problems, and make wide organizational change necessary.

Cutler continues:

The dreamland utopia of “The Business” and “The Developers / The Teams” just working it out over mugs of coffee … is ultimately unfeasible in any org — with >1 teams — that simply tries to graft something like Scrum on to their existing structures, dependencies, and project culture.

No.

When faced with something that seems intractable, you start with what you can control, improve what you can, and force the organizational change that seems unfeasible.

I worked at Expedia way back during a rough patch for the company, and if you’d asked “what’s going on here?” you’d get a litany of entirely different and well-supported answers from different organizations. Or teams. Or people at adjacent desks on the same team with different roles. Top-down “let’s unlock transformation” attempts went nowhere. The business deteriorated and everyone knew why, but no one could agree on the reasons, much less what to do.

Three of us Program Managers (Adam Cohn, Yarek Kowalik, me) got permission to turn our teams to Scrum from the Microsoft process Expedia kept after spin-off, which went:

1) Strategy comes from somewhere
2) Business crunches numbers, hands a direction to team in Business Requirements Document
3) Program Manager turns that into a full specification down to a really deep technical level
4) Spec is reviewed and iterated on until it gets stage-gate approvals
5) Development goes until it hits “Dev complete”
6) QA until it hits “QA complete”

…the whole thing.

There were all kinds of organizational and business problems, all the stuff Cutler points out. The teams weren’t really autonomous, the Program Managers were effectively Product Owners, Project Managers, and deputized Product Managers, it took forever to ship, what did ship didn’t do well — all of it.

We had to hand-lobby leadership and make ourselves pests until we got permission to try Scrum just on the team level, just on those teams. Adopting was rough, and there was a time where transparency and the ability to produce stats meant we were the only teams admitting we weren’t making progress.

But it worked. It transformed everything.

It started with the immediate team and radiated out:

  • showing the teams’ work created a sense of increasing success where there’d been frustration
  • seeing the effect of gates on productivity let us remove the “toss over the wall” steps and get everyone closer
  • reporting how having design do one set of “red-lines” in the spec and then only respond to critical issues delayed iteration got us more and better collaboration throughout
  • documenting the problems with releasing code helped drive efforts towards continual delivery

With the “engineering can’t ship” out as an excuse, it kicked off organizational change. Once, no one could agree on why things weren’t working in our organization. Now we could prove them, and when people agreed on the causes, we could solve them.

For instance, tracking the work done and the cost of it meant we could show executives the productivity-draining effect of business-level randomization. I could go and say “as the business, you told us to prioritize quick-wins in response to this metric, nothing happened and we made no progress on what you want for four weeks” and the leadership of that organization had to grapple with what that meant.

That started a conversation first about how much time we put into tech debt, and on-call, and soon it was a business-level reckoning of “are we doing the right things at all?”

Solving that drove change through the once removed business organizations: it required Product Managers who could act quickly, pragmatically, who were available and collaborative (I wrote about what I learned from working with one of them ). And for the first time, you could tell who was an effective Product Manager, and that organization transformed.

All of those changes were painful and it took way longer than anyone wanted. And if you’d started omniscent, knowing what all the barriers were going to be, sure, it’d have been way easier to make a list. But we’re not. Instead agile methodologies let you see and prove the barriers to the team’s work and force action, and then you get to see the next one.

When large organizations are faced with holistic questions, you don’t get progress. At best you get a weighty consultant-produced report on how you’re deviating from best practices, and each of the groups tasked with work finds some way to argue their bit of the findings don’t apply, or feet are dragged and no progress is ever made.

Agile methodologies allow you to flip that: to not have to solve the whole thing at once. To find the things that are small, and fixable, and get better as you build clear and incontrovertible cases for the next implementable organizational change. You don’t know what the effect of a set of wide-ranging best practice announcements will be, but you know having a designer embedded on product teams will make everyone more effective.

Obsess over teams. Not only is it the often the best way to create change, in large organizations beginning with with the team is the only place it can originate.

Review of Jason Calcanis’ Angel: Looking Down at You from a Useful Vantage

Calacanis’s Angel contains useful insight if you’re interested in dipping your toes in funding startups, if you’re thinking of starting one and want to know what the other side sees, and for those of us who are around or just interested in the world.

That insight costs putting up with Jason himself, and that’s enough to make this hard to recommend to anyone without this advice:

Do not under any circumstances read Chapter 1
Skip anything about how great Jason is or his history

The rest of the book is relatively tolerable, but I wrote “fuck you” in the margins repeatedly in this chapter. It kicks off some themes that keep coming up in the book that are problematic.

I’m going to entirely ignore Calacanis’s opinion of himself and his abilities, and how important he seems to find it that you too believe in him. I’m personally extremely skeptical of the kind of rabid, persistent self-promotion he engages in throughout. The book’s here, we can roll our eyes to the sky, and move forward. There are much larger issues here. You can check out this profile of him in the NYT titled “Should the Middle Class Invest in Risky Tech Start-Ups?”

Skip past this section if you’re already familiar with the problems with Calacanis (and much of Silicon Valley)’s attitude.

Jason thinks you’re pitiable moron if you for some inexplicable reason aren’t also a VC

First, privilege

Our parents and grandparents took factory and white-collar jobs and rode them for all they were worth in the last century. Now the robots — who never sleep and self-improve on an exponential curve — are taking these jobs. Meanwhile, we humans, with our nagging psychological and emotional needs, struggle to keep up.

Most of you are screwed.

But you’re here, so you’re clearly willing to learn and I can radically improve your odds if you do the work.

First, note his use of “you.” It’s “our” for the parents, but when it comes to who’s screwed, he’s clearly comfortable on his lifeboat seat.

This is important. In order to participate in this new economy, he suggests starting in a “syndicate” where someone like Jason finds investments, you and his crew of small investors jump in with him, and Jason takes a fee for making the match. To do this, you must be an accredited investor. Here’s the standard for that, which changed in 2011.

You must be:

  • worth more than $1 million dollars
  • your primary residence doesn’t count towards that

Angel was published this year. A million dollars is about 90th percentile in the net worth of US families (and I believe that counts home equity, but figure it doesn’t so we can forge ahead with even 10% qualifying). There’s an assumption throughout that you, the chump, can just start doing this. He refers to it as “taking the red pill” at one point (and let’s for a moment ignore that term’s association with some deeply toxic people).

For 90% of people, there’s no option on the investing path. Calacanis can’t “radically improve your odds” if you can’t participate. He has another option for those people, I’ll get there.

Second, that’s paired with contempt.

Calacanis goes back and forth in the book from being insufferable to being able to look at his own history and even at times in his own luck to heaping shit on people who aren’t in the tech industry.

Cafe X and other startups will also eliminate millions of jobs in which humans get paid to stand behind a counter and repeat back your seven precious little instructions on how to prepare your morning libation, before pressing one button and masturbating a milk-frothing pitcher for two minutes.

I’m not going to argue even that he’s wrong that baristas could be endangered. His tone is fucking unacceptable though. Think about someone you know that has a “service” job and does it amazingly well. Doesn’t even have to be a barista, though it would warm my heart if it was entirely counter to Calacanis’s example. It’s almost certain that what they do beyond the core and potentially mechanizable task is what makes them great.

What does a great city bus driver, for instance, do beyond drive the bus? They’re getting people in wheelchairs on or off and secured, they’re doing customer service like helping people navigate a public transit system, and infrequently but critically they’re defusing dangerous situations or dealing with crimes as they happen. Why shit on them for not having a million dollars and investing in startups that will put their fellow co-workers on the street?

Now, to his credit, Chapter 8 is “How to be an Angel Investor with Little or No Money” and then 9 is “The Pros and Cons of Advising” — but this chapter’s about signing on as an advisor, a probably unpaid position in which you work for equity. Which, and again, Calacanis doesn’t seem to realize this — requires you both have skills useful to a startup, like his internet marketing expertise, which very very few of the 90% of people without a million dollars will, and also the connections to the startup, which that 90% won’t have, and time to make that happen, which the 90% are probably not going to have — particularly since the jobs being created by Calacanis & Co. are frequently high-effort, no benefit gigs that I’m sure Calacanis would express contempt for if he thought to (“…eliminate millions of jobs where humans get paid to pick up your overly-precious sushi order and ride their overly-burdened bikes in their ridiculous helmets, weaving to dodge potholes and confusing my autonomous car’s prediction sensors causing it to slow and making me several moments late for my next pitch meeting…”)

This is — and despite being written in first-person, it’s possible he did not write this — a stark contrast to what the book promises, that if you’re a “mom in the bookstore with screaming kids, the sales executive in the airport exhausted from layovers, or the kid graduating from college wondering, “What now?”” that you can be like Calacanis and “live in a big house with a bunch of Teslas in the driveway and an ATM balance receipt that makes me smile from ear to ear every time I see it.”

They can’t. That college student’s almost certainly graduating with crippling debt and joining an economy with little to offer them. The mom with the screaming kids? Is she going to take them to the pitch meetings? Tech bros don’t need daycare to meet with founders. And — yeah.

What’s good here?

Once we’re well into it, there’s some legitimately worthwhile stuff regardless of Calacanis himself.

Chapter 11 starts into how you can start by making tiny deals in angel syndicates, and use that to gain experience in how to evaluate founders and make connections with other investors and people in the community, and how you can help your founders.

The sections on evaluating startup pitches and what you should do to prepare for and how to be in those meetings are great.

Chapter 22, “Why Angels Should Write Deal Memos” is great advice for almost any kind of decision like investing in a startup (investing generally, for instance) — that you need to create a clear case for why you’re acting not only as a clarifying thought exercise but so that you can look back at it in the future and compare it to where you are, offering a valuable reality check.

Then the walk through of what it’s like to be an investor of a startup in those early years, and how it will be challenging, are all excellent advice.

This is also where Calacainis’ experiences can make the story — when you hear about him getting screwed over in a deal, or acting poorly, it’s easy to sympathize and learn from him. When he’s rude in response to something, you get it, and there are lessons there both in how to conduct yourself as an investor but in what to watch for to avoid being in that situation.

For founders, you’ll be able to see it all through the investor’s eyes, and walking through the pitfalls of investing (in particular, that the money is not going to get you where you think it will and everyone outside the startup knows this). Knowing the expectations of a good investor in both communication and what they want to hear so they can try to help is excellent to set yourself up well.

On balance

I was well into this book still angry about its opening attitude and considering a pointed email to Calacanis. Once I took a walk and could set it aside, I returned to get into the nuts-and-bolts, and it was entirely worthwhile.

If you take my advice and skip that first chapter and then gloss over any of his “I’m amazing” self-reassurance, you’ll have a much more pleasant experience in getting all the good stuff.

A long review of Phil Town’s 2006 Rule One Investing

Who’s the author and do we care?

Phil Town is the author of two books, 2006’s Rule #1 and Payback Time

Town’s origin story is that casting about after serving in Vietnam as part of the Green Berets, he worked as a river guide in the Grand Canyon, where an older investor he met offered to teach him the ways of the stock market.[1] He then claims to have turned $1,000 into $1,450,000 in five years and became a full-time investor[2].

Now, he’s got a popular podcast where he’s teaching his daughter Danielle Town (who is so great as a foil and voice of the listener), and wants you to attend his seminars and subscribe to his stock research tools from him.

Should I read this or not?

Totally worth reading if you don’t already have a valuation strategy or process you’re into, with the caveat that theres’s some confusing additional material that might make you scratch your head.

If you do, or you’re more generally familiar with value investing (why it can work, particularly), there may not be a lot here.

The meat of the book: how to find good businesses on sale

Distilled, Rule One tells us to invest in good stocks that are 50% off. He’s got the “Four Ms” for you

  • Meaning
  • Moat
  • Management
  • Margin of Safety

And then walks you through each one. Some are easy to understand: for “Moat” he talks about competitive advantages you can count on over time. For “Management” you’re looking for some key metrics, and there are good explanations of why you want to look at each one.

“Meaning” isn’t quite the right term (and to his credit, on his podcast Town’s joked about how the neccessities of book marketing forced the names so there would be a catchy “Four Ms”) — but here, do you understand what the business is doing, and would you buy the whole business?

Where the real meat is for people will be in how you value and price stocks, looking for the discount. And here’s where a lot of math comes in, and I respect that: this isn’t Greenblatt’s “Magic Formula” where you just rank all the stocks. For every stock, Town walks through how to estimate the future value of the stock, and compare it to alternatives, like investing in U.S. Treasury bills, or against expectations.

I dug this a lot, but I also had to break out the spreadsheets, and it’s way more of a pain to get all the data together (this as much a problem with the web in 2017 though). Handy solution though — you can just subscribe to his website and get all those numbers in one place.

You end up with what Town calls the “sticker price” for a stock, the current fair value for a company given expectations for its growth. The explanation for the concept is excellent — that price should be a starting point, and you want a sale.

Running those numbers was an eye-opener for me and taught me a lot about how to think about stocks as both being a company’s value in the present but an expectation of future growth as well. For instance, I’d been griping that one company seemed wildly under-valued based on their price/earnings ratio. When I ran through the process here, I understood that it was priced perfectly reasonably.

Once you can calculate the sticker price, you want stocks that are at least 50% off. For this to be true, either the company is deeply out of fashion or something has gone horribly wrong and the stock’s being beaten up over bad news. As I write this, the CAPE (Cyclically Adjusted PE Ratio) is at 30.62, which is historically very high, and you’re not going to find many stocks on sale for half off.

Going off the rails: add technical analysis

Chapter 12 is “The Three Tools” and here Town talks about “red light” and “green light” indicators for whether to buy a stock. Here we diverge entirely from the value of a stock and get into some entirely different territory, and where I think the book loses people. Buying quality businesses on sale is simple, and while it takes a little work, it makes sense as you’re doing it.

In Chapter 12, suddenly we’re not just looking at business, we’re now trying to figure out what everyone else is doing, and things get way harder to understand.

The three tools:

  1. Moving average convergence divergence. As a measure of whether there’s “pressure” pushing the stock up or down
  2. Stochastics. Measures if something is “overbought” or “oversold” — and the explanation of this one is not great
  3. Moving average.

I’m going to say up front I didn’t get this chapter at all. I feel like I’m missing the point, so this will be a bit more uncertain.

This chapter confuses the issue by . For instance, you’re supposed to look at Stochastics for crossings of the 14-day and 5-day lines. For one, most places you look you don’t get a ‘buy’ and a ‘sell’ line as the book describes them, you get a “%k” and “%d” and it’s confusing. When I got through the instructions for setting one up using Town’s settings for time frames, I looked up some stocks, and it was a coin flip on whether the stock went in any discernable direction.

I don’t know if I was just unlucky or if I’m even using the tools right. My point is that if you’ve gone through the book and you’re working with pretty clear, understandable numbers (earnings per share is pretty much earnings per share), this shift to difficult to understand technical indicators is confusing and exaperating.

Moreover, there’s no convincing why here. The explanation of what we’re doing in this chapter doesn’t make it clear why it’s important to pay attention to it. It seems like the fact that you’ve found a company that’s on sale for 50% off should far outweigh whether others are getting into or out of the stock at the moment.

I was left thinking there are important signals I shoud pay attention to in addition to basic valuation, but unsure how I’d do that.

This is also where the book falls down on the cover promise of “successful investing in only 15 minutes a week!” Even if you’re only monitoring the few stocks you’ve bought, using these three tools once a day to see if you should bail on a stock, you’ve burned your 15 minutes squinting at whether one line crossed another.

It all feels like Town, as a long-time experienced investor familiar with tools like these, was unable to explain these or their importance in a way that made sense to those without the same experience, which is unfortunate because I got a lot out of the valuation sections.

And from there, much good is undone

Chapter 13 then walks through a possible example as a couple considers buying The Cheesecake Factory, Inc. The valuation all makes sense, and they buy. A chart of spectacular returns is shown — CAKE goes from $18.90 to ~$36 in two years, an 90% gain. Nice. The couple’s example $20,000 is now $38,000. Except… here’s what Town writes:

By getting in just below $19, and then moving in out 11 times in two years with the big guys, and by adding $500 a month they were saving, by July 2005, CAKE gives Doug and Susan a nice compounded rate of return of 56% and their $20,000 is now worth $78,000

He’s claiming twice that return rate.

Nope. Nope nope nope nope. Just… not true. At all.

1) Is the regular additional contribution counted as part of the return? It’s unclear. But why mention it otherwise? You can’t have them put more cash in and count that as what it’s worth now. I could have an $100 investment that’s totally flat and add $1m and wow, I beat the stock market over two years because now I have $124. This is, at best, confusing. At worst, he’s counting over $10,000 and associated gains in value as part of the return that shouldn’t be there.
2) They moved in and out 11 times? How do we know that? Does this assume that they perfectly timed each of those? When were those 11 moves? What did the technical indicators say at the time?
3) They’re going to get creamed on short-term taxes. The investor who held for more than a year pays way less.

Investor A: bought at the same time and held. +$18,000, may be paying zero in taxes
Investor B: bought, as a beginner times moving in and out 11 times, up $58,000, might be $40,000 after taxes.

If you think Town is counting the additional contributions in taxes, we’re down to around $30,000 for Investor B.

It’s not okay that this important calculation is so unclear, and the handling of the additional contributions makes me extremely suspicious. It doesn’t make sense as written, and I don’t understand what the point is. 89% over two years is great! Why confuse things?

This raises a larger question: does this all work? We’re presented with a couple examples of Company A compared to Company B, walked through the Cheesecake Factory example, but beyond those hand-picked examples, all we have to rely on is Town’s accounting of his track record. I’d have felt much better about the whole thing given a wider accounting of his trades, or of studies done on historical data — like we have for Greeblatt’s Magic Formula book.

Put it all together, what’s it mean

I dug reading it, and particularly found it useful in making me do real work putting things together and looking at companies. But I also found the technical tools section confusing and hard to understand, and the example math particularly worrying. It occupies a weird place in an investment bookshelf: I can see recommending it to someone who is interested in Warren Buffet’s investment philosophy and wants to know how to crunch the numbers to find those great companies at attractive prices.

[1] The story, as told, is so perfectly mythic that it’ll probably raise at least one eyebrow if you read it. But whatever.

[2] (I spent a little time looking at this, but besides him raising money in 2013 for Rule One Capital, I couldn’t find anything on early investment proof or his returns as a manager).

[3] Similarly, Town claims early that as a Vietnam veteran, on his last day in uniform he was at the Sea-Tac International airport when someone spat on him and ran away. This… I don’t know. That spitting at veterans happened at all is disputed: there’s a whole book on this, Spitting Image by Lembcke, discussing how no incident like this has ever been documented as part of a larger case that antiwar activists and veterans were allied far more than is now generally believed.

I thought about looking into this in greater detail (particularly, why would he be at Seatac on his last day in uniform, when it seems like he’d have flown into McChord Air Force base?), and stopped. This has been hashed out many other places, and it doesn’t seem relevant. Let’s just grant Town this.

Solving Drobo random unmounting issues

(Writing this up for hopeful discovery of future Drobo owners)

My Drobo started acting up a while ago in an incredibly frustrating way:

  1. The Drobo would sometimes not show up, or not mount, requiring a dance of restarting it, restarting the computer, plugging, unplugging
  2. When it was mounted, you’d get a short while before the Drobo went unresponsive in the middle of an operation, and then it’d unmount (and OS X would throw a warning about dismounting drives improperly)
  3. Sometimes if you left it connected for long enough, it would show up again, hang around for a bit, and then disconnect.

Nothing worked: re-installing software, resets, the “remove drives, reboot the Drobo, wait, turn it off, put the drives back in…” And all the while status-light-wise, and Drobo Dashboard-wise, it reported everything was good

And unhappily, Drobo support costs money, and I’m cheap, so I wasted a ton of time troubleshooting it. As a bonus, their error logging and messaging is either unhelpful or encrypted.

(I feel like if you encrypt your device’s logs, you should offer free support at least for unencrypting the logs and letting the user know what’s up. I’m disappointed in them and will not be purchasing future Drobos. Or recommending them.)

Eventually I pulled each of the drives and checked their SMART status (OK status overall on all drives, though I also pulled the details and one of them had flags, but SMART’s not great (see: Backblaze’s blogs on this). So I cloned them sector-by-sector onto identically-sized drives. The drive with the odd SMART errors (but, again, overall OK status) made some really unsettling noises at a couple points during sustained reads, but the copy went off okay.

Fired it up, and it worked. Drobo came back on, mounted, works fine (for nowwwww….).

I spent some more time hunting around in the Drobo support forums looking for more information, and found someone reporting back on a similar issue said they’d had a drive go bad but the Drobo never reported any issues, and it wasn’t identified until support looked through the encrypted error logs and said “oh, drive number X is going bad, that’s causing your Drobo’s strange behavior.” Clearly, given my success, at least one of my drives was secretly bad and cloning and replacing was the solution

So! May writing this up help at least one future support-stranded Drobo owner: if your Drobo is unmounting randomly, not showing up in the Finder, throwing dismount errors, but the Drobo’s reporting that everything is hunky-dory, and you don’t want to pay for support and you’re willing to take advice from some random fellow owner on the Internet who may not even have the same issue… here’s one approach before you throw your malfunctioning Drobo out the window:

  1. Power it down and pull the drives
  2. Using whatever utility you like, check the high-level SMART status on the drives to see if something’s clearly screwed up
  3. (optional, if they’re all okay) look at the detailed SMART errors and see if any of the drives looks really wonky
  4. If any of them are bad, do a sector-by-sector clone of that drive, swap the clone in, power up the Drobo, see if that works. If yes: yay! If not —
  5. Clone & replace them all, see if that works.

May this work, and may the drive be in good enough shape to successfully clone.

I should also note that as much as I’m annoyed my Drobo was out of support, assuming they would have been able to tell me what was happening and which drive to clone and replace, it would have been worth it to pay for the per-incident to save myself the headache.

Promotions are recognition, not elevation

Or: the importance of good managers and 1-1s

When I was a Program Manager with no Senior title, I went through a period where I didn’t get promoted, not being promoted made me more and more impatient and even resentful, and that in turn prevented me from making progress towards being promoted.

I’ll paraphrase how I started one of my weekly 1-1s with my manager (Brian Keffeler!):

“Wahhhhhh! Why aren’t I a Senior Program Manager? Look at what I’m doing! It’s amazing! Look at these (n) people who are Senior Program Managers and they aren’t working on as big stuff or doing as well! Wah wah wah!”

And Brian, bless him, listened to me until I’d run out of rant and said:

“I’m not going to argue whether you’re doing better than (person) or (person). Set that aside for a second. None of that matters. You’re not going to be promoted because people look at you and think ‘he’s better than a couple of people who already have the title.'”

I thought “Fuck, he’s right.”

He kept on.

“If you want to be a leader, if you want to be promoted because you’re deserving, you need to stop comparing yourself to them. You need to be so good people assume you’re already in that role. You need people to be surprised to find out you’re not a Senior. When title reviews come up, you want everyone in the room to say ‘He’s not already a Senior? What the hell?’ Right? You want your promotion to be a recognition that you’re already successful operating at that level.”

It was one of the moments in my career where the skies parted, sun shone down on me, and trumpets sounded. I knew immediately that not only was he absolutely correct, that if I was ever to be promoted I needed to prove that demonstrating potential wasn’t enough — that I needed to be operating on this next level. But also, and just as importantly, that me being hung up in the petty bullshit of whether I was the best in my pay-grade and whether I was better than some people in the next pay-grade was fucking up my relationships and career, and that I needed to let go of it.

I might have spent years in that destructive spiral, burning myself out generating my own frustration, with a different manager, or if they’d delivered the message at a different time, or in a different manner.

So I went out and did great work, and people started to assume I was already a Senior Program Manager, and then I got promoted.

Brian’s awesome, and I owe him a great debt.

Honesty without obscenity

When I was a Program Manager at Expedia, and Aman Bhutani had just showed up to right the ship through by demonstrating the value of clear leadership, he started a regular “Big Boulders*” meeting with the Program Managers working on the most critical projects, like the giant re-platforming, or new shopping paths, or rethinking the checkout process.

He wanted to get direct feedback on what was going on, unfiltered, and to discover where he could help. We’d show up and give a high-level status using a standardized couple slides showing timelines and dependencies, and if Aman could help by raising an early point of emphasis to another of his peers about a cross-organizational dependency that had historically been trouble, we’d ask.

Aman built trust with us by delivering — if you brought something up that concerned you, and he said he’d go look after it, you could check it off your list of worries.

For us Program Managers, to have his ear and direct engagement was a huge step forward, though dangerous because we didn’t want to report status to him that we hadn’t already talked to our managers about (because at that point we hadn’t entirely recovered from the stabby years). And it was also pressure-filled. Not just because he was there, or because he’d ask amazingly insightful questions you wanted to be prepared for (and to which “I had not thought of that solution, wow” was a perfectly good answer). In front of a peer group of others trusted to deliver the most important projects, you wanted to have your shit together.

Some people didn’t deal with all of this well (each time starting with a forced grin and “It was another great week on the ____ team!”) but in general, Expedia’s Program Manager corps was a lot of no-credit-taking, jump-on-the-grenade, jaded leaders-through-delivering who’d kept at it through some dark years because they believed in the mission, and they’d be honest. But also, still, sometimes you left the door open knowing he’d ask a question, because you didn’t want to volunteer something you were worried about that your boss wasn’t, but it was keeping you up at night**, and you wanted him to know.

After the initial progress, Aman wasn’t satisfied with a true but also wary status report. So at one meeting, he challenged us. He wanted to hear the status with our insights, whatever they might be, into the present and future, no matter how dangerous the truth seemed.

I felt excited that for the first time someone way up the chain was not only recognizing the chain itself distorted and delayed truth, and he wanted to try and bridge that. And because we’d built so much trust, we were safe — it wasn’t a trap.

So off I went.

“We are so fucked,” I started, and I took off from there. “This org is fucking us, this other thing is fucked up, but this team is fucking amazing, totally saved our ass. This thing we bought from a vendor to help is a piece of shit…” I just went the fuck off, running down everything in terms that would have made a stub-toed sailor tell me to calm down.

Aman nodded through the whole thing, entirely even-keeled. When I was done, he said “So first, yes, that’s the transparency I’d like to see.” And then he paused for just a moment and said “But I’d suggest it’s possible for us to get that honesty without the obscenity.”

I felt relieved, and also like I could do better***.

He let that hang out there for a comfortable pause, thanked me, and then we moved to the next person.

It was an important step for me in how I expressed myself, taking this challenge to be concise, and true, and also not angry. Because I realized that while you get some truth in the emotion, you also lose clarity. “Fucked” expresses frustration, but does that express a need or problem to someone who might help you? And for many people, if you’re cursing like crazy, or you’re coming across angry, they’re not going to receive the message at all — and when you’re speaking, sometimes you can’t expect the audience to come to you, and if you want the right outcome, you’ve got to deliver in a way that’s most effective for them.

Afterwards, the Big Boulders meetings got way more raw, without the cursing, and we got to the next level of trust. And that led to things overall improving, and I felt like I’d contributed in some small way to taking that step forward. And taking Aman’s advice, I start trying to consistently hit that level of openness and honesty in all my communication, without the cursing.

DMZ

  • first you figure out the boulders, then you see what rocks you can cram in around them, and then you pour in sand until the container’s full

** if you’re sleeping well, you’re not paying enough attention to your project. It’s why we’re all such coffee fiends

*** the ability to support people while also helping them realize they can — and want to– do better being one of Aman’s super powers

My first job at Expedia, joining a small crack team who all seemed wildly smarter than me* my manager was Tim Besse**. Once I was stuck on a particularly thorny problem over a bit of UX, and he stopped by to help. We brainstormed, we drew all over the whiteboards in my office***, we argued, we revised and we came up with something that solved the tangled issues to everyone’s satisfaction.

Relived, I went to write the whole thing up. Tim, standing back from the whiteboard, shook his head and frowned.

“No,” he said. “This isn’t good enough. We can do better.”

I felt anger, frustration — we’d finally come up with a way out and he wanted to discard it? We both had a long list of other things we needed to figure out. Checking this off and moving on was a huge relief and a victory for everyone.

I looked at him in dismay while he stared at the diagrams. I took a couple deep breaths and let go of the frustration.

“Okay,” I said. “Where do we start?”

We began again. I remember it as taking twice as long before, in wavy boxes with my chicken-scratch handwriting everywhere, we’d found something wildly better in every way.

We looked at each other and smiled. I felt a sense of rightness and satisfaction I hadn’t touched in the previous one.

I’ve carried that with me since: that when you’ve arrived at something that’s good enough, push on it a little. As much as I pride myself on being pragmatic above all else, push on good enough. Does it rattle a little? Is there a little give? Do you feel like there’s a hidden switch that’ll rotate the whole thing?

Take the time. See if you can turn good enough into something amazing. Challenge others to do better.

And believe that when someone says “we can do better” they believe it, and that you can.

Thanks, Tim.

— DMZ

  • In the words of Isaac Jaffe:
    “If you’re dumb, surround yourself with smart people. If you’re smart, surround yourself with smart people who disagree with you.”

** Tim went on to co-found Glassdoor

*** shared. As a Microsoft spin-off, we were all about private & shared offices. It was great! Then they abandoned it and I’ve never since enjoyed such productive work spaces.