Towards better agile status reports

“It is important to communicate to stakeholders that early calculations using velocity will be particularly suspect. For example, after a team has established some historical, data, it will be useful to say things like, “This team has an average velocity of 20, with a likely range of 15 to 25.” These numbers can then be compared to a total estimate of project size to arrive at a likely duration range for a project. A project comprising a total of 150 story points, for example, may be thought of as taking from 6 to 10 sprints if velocity historically rangers from 15 to 25.”

– Mike Cohn, Succeeding with Agile 2010.

This requires that you have the project’s scope and cost in story points clear at that point, which you won’t have. Story point estimation takes time to achieve any useful consistency, and especially to do the kind of learning (“last time we took something like this on, it was an 8, not a 3″) that gets you to numbers you might want to rely on.

It depends on scope being set — and how often is that true, particularly on agile projects where you’re able to show users the product at the end of each sprint and make adjustments?

More importantly, say these teams are relying on the release date of a project:

  • customer support, which needs to update all the help documentation and set up training with the people who pay you money
  • marketing, which needs to create materials based on final screenshots and capabilities
  • other teams, which will use the delivered work to create new projects

Where if you deliver early, that’s great, they have more time to rehearse, edit, and control over whether they want to release early.

If you’re late, they need to know as soon as possible if they need to start cutting schedule or scope, or spend more to keep those constant and meet a new deadline.

Then take Cohn’s example. If my company is lining everything up behind a date, and my status email is

“We have 150 story points remaining, and the team’s current velocity is 20 per two-week sprint, with a range of 15-25, so we should deliver between 20 weeks from now and 12 weeks, and probably around 16 weeks, which is on time.”

People would rightly stand up from their desks and hurl the nearest team-building trophy at me.

At the same time, putting more information about the probabilities and how you calculated them isn’t going to help. You can’t talk about the power law or the Hurst Exponent here. The question people want answered is: do I need to take action?

You’re going to end up reconciling this to some organizational standard, like:

  • Green: On target
  • Yellow: Risks but on target
  • Red: Will miss date unless changes are made

Which actually means for PMs:

  • Green: I’m not paying attention
  • Yellow: I’m on top of things
  • Red: Please have your VP come by my desk immediately and yell at me

And for your customers, eventually means:

  • Green: the PM isn’t paying attention so I won’t either
  • Yellow: something meh whatever
  • Red: aieeeeeeeeeeee

Then what do you say in your one-sentence “summary” at the top of your status report, the one thing people will read if they open it at all?

My best results came from straight team votes at the end of the sprint: ask “Do you feel confident we’ll hit our date?”

80% or more thumbs up? Green.

50%? Yellow.

Under 50%? Red.

This requires the team trusts they can give their honest opinion and not be pressured. If that’s not the case, do the rest of the PM world a favor and leave our ranks.

Ask, get the number, and nod. Don’t argue, don’t cheer, just nod, and use that. Then in your status report, you write

“We’re green.” If you want, offer “(87% of team members believe we will meet our target date)”

Now, you do want to express the uncertainty, but in a way that people can use. Think of yourself as a weather forecaster. Do you tell people to bring umbrellas? Put on sunscreen?

Hurricane forecasting has a useful graphic for this:

Hurricane forecast example

Where on the calendar does your project land, and with what certainty? Assume that you ship every week on Friday. Why not

Status forecasting

Distance from the calendar is how far out you are, and the shading indicates likeliness. Mark the release deadline with red (or a target, or…).

Daniel Worthington offered this:

Screen Shot 2014-08-18 at 11.57.58 AM

Which is even better.

So two questions for you — 1) How do you effectively measure uncertainty on a project’s delivery date? 2) How do you convey that so it’s simple and easy to act on?

The Side: Everything fails

In which I whine about tools.

I’m sure for developers who work with Ruby the maze of tools, installs, and dependencies is like swimming in water to a fish. Except just from work, the fishes complain a lot about it too. For me, though, it’s so fucked up I want to throw something at it.

Let’s say I want to install Ruby and Rails on my Mac.

% brew install ruby
(vwhoosh! works)
% gem install rails
(bombs on crazy UnknownHost error)
% gem install rails
(vwhoosh! installs a ton of stuff)
% rails –version
Rails is not currently installed on this system. To get the latest version, simply type:

$ sudo gem install rails

You can then rerun your “rails” command.

(cursing)

Okay, so let’s figure out how to do that. Stack Overflow answer says this is a command line tools issue. Do I have the command line tools? I’ll check — Crap, it returns a “Can’t install software because it is not currently available in the Software Update server” error. That makes no goddamn sense.

Back to Stack Overflow… ah! That’s a bug. I guess I’m actually fine. BLARGH.

Except rails will install and then tells me it didn’t. Still haven’t figured that one out.

And hey — as in Python, there’s a huge set of version differences!

Q: Hey, I’m a normal dude trying to get this thing up, which version of Ruby do I –
A: (shrug)
Q: Thanks for that.

To deal with all these dependencies, I can use rvm which… I don’t even know! Woo!

This seriously makes me want to go back to wrestling with Python. Or set fire to my computer.

The side: they don’t make this easy

Here’s my weekend progress.

Business checking: I can’t find a non-horrible bank that has a decent product.

Design: reached out to someone recommended to me. I’d like to have other options, and I have frankly no idea how much to pay.

The code: I’ve got nothing. I was learning Python so I could work with my dad on it, but frankly web2py frightens me and I feel like if I’m going to do any kind of lifting on this, it’ll have to be Ruby, which would have the recommendation of being what work uses, something I have some experience writing in.

There’s a potential disadvantage of having a side project prejudiced by people and decisions at work.

And I’m worried I should be learning Python. Just generally.

But of course, questions like “I’m building a website, what’s the easiest way to get there” are hilariously dumb. I might as well start with Ruby and work from there.

Review skimming escalation

Dungeon Keeper’s catching flak for its terribleness, and today’s story is their new implementation of review skimming.

Check out their writeup.

Here’s their picture of where this goes beyond anything I’d seen before:

Where, say, OkCupid asks “love us?” and “leave feedback” to try and skim off only the people who’ll leave good reviews, Dungeon Keeper’s explicitly asking you what your star rating would be, as if you’re rating it right then (in a window labelled “Rate Your Experience”)(!).

Logical next step would be to present a fake Android rating screen that discards 1-4 star reviews and then submits 5-star reviews on the user’s behalf.

How skimming reviews makes apps twice as slimy

I read the fine “Choices and Consequences” (which responds to John Gruber on one-starring apps that obnoxiously beg for feedback).

Daniel:

At the other end, you find blatant harassment and tricky language meant to confuse users into capitulating. Modal alert panels might interrupt a user’s workflow at inopportune times, demanding that they either leave a review now or be reminded later to do so.

Part of the frustration is much deeper than that, and goes to a deeply scummy tactic Apple’s let proliferate. I’m going to call this skimming reviews: you pop up a request for a dialogue, but in a way that encourages only people who are going to leave good ones to do it. OkCupid’s app is the clearest example of this:

photo

Check out the levels of ridiculousness:

  1. They’re first trying to make users self-select. Do you love the app? Well, then…
  2. The call to review is not “review us” it’s “Rate us highly.” What if you don’t want to review them highly? Presumably you look for another option, which…
  3. “Send feedback” keeps them out of the review ecosystem

Right? As much as the cumulative annoyance of “rate us” requests might grind us down, this kind of thing is way more toxic and makes the whole experience seem dirty to me, because it’s not even a sincere request for feedback, but an attempt to turn users into positive reviews.

I don’t understand why this passes App Store review.

Mavericks power savings on a macro level

I did some napkin math on the overall effect of the new version of Mavericks, and wanted to share. Standard caveats apply: as you’ll see, I’m doing a lot of hand-waving.

Starting assumptions:

  • Adoption of Mavericks will equal Mountain Lion* for ~30 million users
  • 75% of Mac sales are laptops
  • I’m using the Mac Mini for desktop power consumption to concentrate on processing/disk tasks
  • I’m also using it for the under-load draw for laptops

So! Thirty million Macs, idling, draw 195 megawatts. Under load, they draw 2,380 MW.

Assume Mavericks gets you a 15% savings in power both at idle and under load.*** If all Macs are at idle, you’re saving 30MW. If all the Macs are running at the same time, you’re saving 357MW.

How much is that?

  • The smallest nuclear reactor in the US generates 438 megawatts
  • An average coal plant generates about 500 megawatts

Now, you don’t of course get all the savings at once, and I’m totally omitting how Mavericks affects machines staying in minimal power draw states longer instead of waking, working, sleeping, waking…

I welcome thoughts on how to improve my napkin math and get to a better number.

Even as a rough guess, 80% of a coal plant… that’s pretty awesome.

* I assume this will be low, as new Macs will come with Mavericks installed and replace ones that don’t, and also because with power-saving features, there’s a huge incentive for laptop users to upgrade if they are able but haven’t. Also, it’s free.
* nuke cite http://www.eia.gov/tools/faqs/faq.cfm?id=104&t=3
*** I’m guessing, based on anecdotal reports of battery life improvements, mostly during betas

My AT&T Customer Service Nightmare

First, I worked for AT&T Wireless for five years, and more generally, I know how at the ground level, you can make a mistake or be limited in your ability to act by policies.

And second. I entirely acknowledge that in normal circumstances, I’d have looked over my bill at some point before I did.

So!

At the end of November, I was getting divorced and called AT&T to de-activate the phone my wife used. The rep said she’d de-activated it, and it would stop working at the end of that billing cycle (December 22nd).

What actually happened: the rep did not cancel that account, and somehow after that call I had *two* accounts: one, for my phone, and a second, for hers (at $120/mo+tax = $140/month). That there’s two plans is actually kind of important, because it meant that checking on my phone showed everything was cool, and there was the old phone showing up on its separate tab, and the only way I’d have figured it out was to say “hey, the total billed amount isn’t what my phone plan is.”

My wife calls them a week later to confirm with them that they’re canceling the line. They say “yup!”

Then because I was getting divorced, I go deal with the other stuff in my life for months, totally ignoring all my auto-billing like rent, auto insurance, and phone. I finally notice at the start of this month what’s screwed up. I call in, I get a friendly rep, he’s apologetic, puts in a case (CM20130805-70519441), cancels the phone, and says I should expect a credit of about $971.91 (there’s some tax uncertainty).

What actually happened: he submitted the case, but the second phone/plan was not cancelled.

At the end of this billing cycle, I check the bill, see it’s not cancelled, see I’ve only received a small credit.

The math: AT&T owes me ~$979.91 + $140 another month they billed me after not canceling again (and let’s assume it’s all set up so that’s the end of that).

They’d refunded $423.83 of the $979.91. No one ever contacted me to talk about what happened.

I call back in. I talk to a guy who seems more stymied than anything.

He reports the “back room team” took the case, looked at the account and said that because there was usage on it for some of those months, they weren’t going to refund those months.

Now, when you look at the usage, it’s clearly accidental. In one month, In another, they forwarded a phone call to her new phone. For the April-May billing cycle, the statement says 4mb of data used, and when you look at it, it’s 0kb. For May-June, someone sent her a text message, and there’s another couple random phone-dialed in usages.

None of which should have happened, if they’d cancelled the account. I just don’t understand this part.

I went around and around with that rep, in which he repeated the reason why they’d denied my refund, I said “look, I understand that’s not you, but this is crazy” and he’d say there was nothing to be done about it. I asked about appealing, he said I couldn’t, there was no way to escalate, if I filed another case about those three months it would just be denied. I asked him what I should do about this and he says “there’s nothing you can do”. Eventually, after about an hour, he told me he’d set me up with a supervisor callback.

His supervisor calls back! He wants to know about the call, if there’s anything (rep name) was unable to help me with. I say “no” and say it’s not really his fault, he’s clearly constrained, so I’m glad he escalated it.

The supervisor’s a little baffled. It seems like he’s calling because he saw his rep was on that call for so long, not because the rep put me in for a callback.

I don’t know if that’s a mixup or the rep told me he’d have someone call me back and then just dumped me, but if it’s the latter, that’s reaaaally frustrating.

So he walks through the whole thing with me, does a lot of math, credits me for what I’m being billed this month, puts a thing in for next month (since we’re now on another billing cycle and I still have both plans attached).

At one point, he talks to his supervisor, and comes back with a partial solution. It sounds like the supervisor’s unwilling to go back farther than three months. Through some customer service foo, he refunds ten things in varying amounts that add up to $369.50 and after talking to me for a minute about the gap, credits me $150.

I’m entirely uncertain of whether that happened or not, because when I look at my account, it doesn’t show the credits he talked about. But it’s not super-easy to figure out credits and adjustments on the online site.

Here’s where I am, then.

Best case: the zombie plan is cancelled. Next month’s bill will show up with a credit for the zombie plan. There will appear on my account a $200 credit towards this month’s bill and a $150 credit that don’t show up on my AT&T account on the site.

Math as it shows on my bill right now
~971.91 owed
+140 charged after that
-$423.83 credited by secret back room team
-$369.50 by supervisor ninja service against past bills
= I’m still out ~$319 because AT&T didn’t cancel the account back at the end of November

And next month, assuming the supervisor’s right, I get billed again for the zombie plan, but also credited for it.

Now, if you assume that the credit the supervisor yesterday is happening but just not there yet, I’m actually to the good. But you’ll forgive me if at this point having someone tell me they’re crediting $350 to this month’s bill but where that does not actually show up on my bill, I’m pretty skeptical that
a) there actually was a credit or
b) there will be a credit

Much less that zombie phone/plan is actually cancelled, or that I’m not going to be having this exact same conversation next month.

And at this point, it’s been five calls and hours on hours on the phone being frustrated, and having guarantees that something will be done turn out to almost always be entirely untrue.

The most galling thing is the back-room team dismissal based on trivial usage. Who gets a ticket like that, where it’s clearly entirely their company’s fault, and looks for the slightest pretense to deny it? I don’t even understand why, given that their error allowed that trivial usage, it’s even a pretense — is there a huge fraud problem with people who cancel, wait to see if it’s not cancelled, and then send one text message?

But yeah. From looking at my bill, AT&T’s got me for $300+.

Sigh.

Call for a price on how to improve calls to call for pricing

My new employer, Simply Measured, offers free tools and an enterprise plan for large companies. Which means that if you want to know how much it would cost your business to use our excellent products, a conversation needs to occur.

Which is funny, because I’d always pretty much believed what Joel about this in an excellent piece on pricing:

Bad Idea #2: How Much Money Do You Have? Pricing.

This is the kind used by software startups founded by ex-Oracle salesmen where the price isn’t on the website anywhere. No matter how much you search to find the price, all you get is a form to provide your name, address, phone number, and fax number, for some reason, not that they’re ever going to fax you anything.

It’s pretty obvious here that the plan is to have a salesman call you up and figure out how much you’re worth, and then charge you that much.

Which cracks me up. In our case, to briefly attempt a defense of our company, is that because we’re not selling a widget, and instead we’re providing a massive number-crunching, data-heavy process, our costs per customer are high, recurring, and very hugely by what they’re asking for.

It’s fine if you don’t take my word for it.

The fascinating problem I’ve been having is that there’s essentially no widely available information on how the industry does this. At Expedia, when I was working on the package path and trying to figure out if making something clearer was going to help, and how much it could help, there was all kinds of great research into ecommerce websites, case studies, and I could talk to people I knew who worked at other companies.

I have no idea. Let’s say our conversion rate in the trial is 0%. Or 50%. Or 97%. The only comparison point I have to go on is that it’s not as high as I want it to be. We have to build our own metrics, benchmarks, and methods as we go.

In absolutely delicious irony, I have been able to find a ton of information on enterprise trial paths, people with good advice, and ways other huge companies have made improvements selling expensive things to other huge companies.

At consulting companies, who want me to call them to talk about pricing.

“I’ve never really listened to my voice and gone, ‘That is a quality instrument.’ It’s more like, ‘Okay, that’s good and fucking loud.’ I’m kind of the horn section of any band I’m in.”
– Neko Case