The Average That Hid the Business

Situation. A single adoption number stood in for the whole of a B2B commerce platform carrying $4B+ in annual orders across 50,000 dealer accounts, and the roadmap was being argued from it.

Decision. Check the instrument before the story. Stop arguing from the number, and build the view that shows the population underneath it.

Scope. The measurement path behind one headline metric, the population underneath it, and the roadmap conversation both were feeding.

What changed. The question moved from how to raise the rate to which cohort, and what that cohort actually needs. The hypothesis the roadmap rested on did not survive the cut, and so did a second one engineering was building against.

The situation

Every platform ends up with one number that stands in for the whole thing.

On a B2B ordering platform carrying $4B+ in annual orders across 50,000 dealer accounts, that number was the share of ordering that happened without a person in the middle of it. It was easy to read, it charted well, and it had appeared in enough reviews that nobody went back to ask what it was counting.

It was also the number the roadmap was being argued from. If the platform sat somewhere in the middle, then the obvious job was to move the middle: find the accounts that were partway digital and carry them the rest of the way.

That framing had produced a backlog, and a target.

What the system said

The metric said the platform was mid-adoption. The story that grew on top of the metric said the middle was the opportunity.

Those are the same mistake made twice, and it is the one I write about most: an average is the arithmetic result of combining populations that behave differently, and the strategic question is almost never how to move it.

I did not start there. I started with a smaller and more boring suspicion, which was that I did not know how the number was produced.

What I found

Two things, and the order they came in is the part worth keeping.

The first was about the instrument. Identity fields were arriving empty on a portion of commerce events, so activity that did have a person attached to it could not be resolved to one. Internal traffic was being counted as customer traffic. And the pipeline was truncated before its last step, so a slice of activity never reached the layer the number was computed from.

None of that was anyone’s failure. It was ordinary. A tracking implementation gets extended a few times, each extension is reasonable on its own, and the result is a measurement path that nobody has read end to end in a while.

Figure 1. The measurement path behind the reported number, and the three points where activity was lost or misattributed before reaching it. Schematic. No volumes, rates, or system names are shown.

The second thing was about the population, and I only got to it because the first had made me stop trusting the cut.

There was no middle to move. There was a cohort that ordered digitally as a matter of course, a cohort that had never done so at all, and a space between them that was thinner than the average implied. The number in the deck was sitting in that thin space, describing a group that was mostly an artifact of adding the other two together.

Figure 2. The shape the average concealed: two poles and a middle thinner than a single rate implies. Drawn to describe form only. No values, counts, or axis quantities are published.

What I decided

Three decisions came out of it, and I made all three.

I stopped arguing from the number, and I stopped letting the roadmap assume it. The figure itself is still produced, because a metric that has already travelled does not disappear when one person stops citing it, and I am not going to claim otherwise. What I could change was what I said in the rooms where it was used, and what I built next to it: a view of the same activity by cohort, so that anyone reading the aggregate can see the population underneath it in the same place. That work is in progress.

I replaced the argument with direction and shape. Two poles and a thin middle is a statement anyone can act on, and it does not pretend to a precision the instrument cannot support. It is also the version that survives the next pipeline change, which a decimal place would not.

And I changed the question the roadmap was asking. Not how to raise the rate, which had produced a backlog aimed at a group that was largely an artifact, but which cohort, and what that cohort actually needs. Those are different products. The never digital cohort is not a slower version of the fully digital one.

What changed

Nothing about the business changed the day I finished. That matters, and I am not going to dress it up: this was a measurement finding, and measurement findings change what gets asked before they change what gets built.

What changed is what I was willing to say, and what the roadmap conversation was allowed to assume.

The framing the organization had walked in with also died. The middle was supposed to be the addressable group, and the target had been set on that basis. It did not survive the population cut. Saying so cost something, because the number was already in circulation and the people who had built on it were in the room, and the only version of that conversation that works is the one where the reasoning is shown rather than the conclusion asserted.

A second hypothesis went the same way. A gap that had been read as a yield-management problem, and that engineering proposals were anchored on, turned out to be produced mostly by how one class of order entered the data. Once I separated sample orders from product orders, the gap that remained was small enough that the proposals built on it no longer made sense, and the capacity moved to work that could.

What it cost to learn

The expensive part was not the analysis. It was that the number had already travelled.

Challenging a figure the organization has already built on is harder than finding a new one, because you have to go back to the rooms it was carried into. There is no version of that conversation that does not cost something. The alternative costs more, and it compounds, because every quarter the figure stays in circulation is another quarter of decisions made against it.

The habit that came out of it is the one I now apply everywhere, including to research I run myself where nobody else would check the instrument. Read the measurement path before reading the measurement. If you cannot say how a number was produced, you do not yet know what it is a number about, and any strategy built on it is a strategy about something you have not identified.

That is not a sophisticated idea. It is just one that is easy to skip when the number is already on the slide.