What I Got Wrong

This site is built on finding the gaps in other people’s numbers. That is not a position I can hold honestly without keeping the same record about my own, so here it is.

The pipeline ran clean, which I read as correct

What I believed. The Behavior Intelligence pipeline I built was returning a full month of activity, and I believed that because it ran start to finish without errors.

What the evidence showed. It was silently returning a fixed slice of the period rather than the whole of it, and it was counting internal staff activity as customer engagement, so the numbers I was working from had the same class of problem I was auditing other people’s numbers for.

What changed in the model. A pipeline that returns cleanly is unverified until its counts reconcile against the source, and staff traffic is separated out before anything is called engagement.

A yield gap that was an artifact of how orders were sampled

What I believed. A gap in the data looked like a yield-management problem, which is to say a real behavior worth pointing the roadmap at.

What the evidence showed. The pattern was produced by how a particular class of order entered the data rather than by anything the customers were doing, so the gap described the instrumentation and not the business.

What changed in the model. I stopped treating an unexplained gap as a finding until the path that produced the data has been traced end to end.

The average was the thing to explain, not the thing to move

What I believed. A platform-wide adoption rate that sat in the middle of its range described a population in the middle of a transition, and the work was to move the number.

What the evidence showed. The population underneath it was splitting into a cohort that had fully adopted and a cohort that never had, with a middle far thinner than the average implied, so moving the average would have meant optimizing for customers who were not there.

What changed in the model. I now treat a stable aggregate over a large population as a question rather than as a status, and the first move is always to ask which sub-population the strategy is actually addressing.

A measured rate is a snapshot, and I published it as a constant

What I believed. A figure that came out of an instrument I built and checked was a fact about the world. Once it was measured, cleared, and written into the registry, the work was done and the number could sit there.

What the evidence showed. The corpus the figure was measured over kept growing, and it grew unevenly. Sites discovered later pass at a higher rate than sites discovered earlier, so every new batch pulled the published rate upward without anything about the measurement changing. Re-auditing the original population returned the same answer it had always returned. The instrument was right, the figure was right on the day it was taken, and the claim still drifted past what it said.

What changed in the model. A rate over a population that is still being collected is a snapshot with a date attached, and publishing one without either a recheck cadence or a round number wide enough to absorb the drift is publishing something that quietly expires. Rates now ship with one or the other, and the registry row carries the date the figure is next checked rather than only the date it was taken.