Measurement
Attribution reality check
Attribution reality check is licensed CC BY 4.0. Attribution: Katafacts (katafacts.com).
Customise with AISkill file ↓1 · What it is
What it is
An attribution reality check computes how much credit for one outcome shifts across three deliberately simple, honest models — first-attributed, last-attributed, and an even split across every contributor — and flags which contributors are robustly credited (all three models roughly agree) versus contested (the models disagree sharply). It doesn't resolve which model is correct — no spreadsheet can, without a real controlled experiment — it stops a single-model number from quietly passing as settled fact when it's really just one convenient choice among several equally defensible ones.
2 · When to use it
When to use it — and when not to
Use it when
- Multiple changes or contributors could plausibly be credited for one observed outcome, and nothing about the timing alone proves which one actually caused it.
- You want to know honestly how much your current crediting convention would change if you picked a different, equally reasonable model — before betting a bigger decision on it.
- A follow-up plan (like rolling a fix out to a second area) is about to assume one specific contributor gets the credit, and you want that assumption pressure-tested first.
Not when
- You already ran a real controlled experiment (a holdout, an A/B test) that isolated the actual cause — trust that over any of this tool's three models.
- There's genuinely only one contributor in play — there's no real attribution question to reality-check.
3 · How to fill it in
How to fill it in
- Scope area
- What outcome, and what period, does this reality check cover?
- Current model
- How is credit actually assigned today? Name the real convention in use, even if it's informal or nobody deliberately chose it — not the model you wish you had.
- Cases
- For each outcome: its real value, and every contributor with the real order it occurred in — your own record, never guessed. Credit by model and the swing between them are computed for you.
- Narrative
- Drafted from the computed swing, naming which contributors are robust and which are contested.
4 · What good looks like
What good looks like
The example below continues the Line 2 storyline running through this catalogue's A3, scoping canvas, FMEA, and metric definition sheet examples — it's directly grounded in the A3's own two countermeasures, rolled out on the same date, and honestly names a real confound (a seasonal order-volume dip) rather than assuming the sequencing rule alone gets the credit its own follow-up plan already takes for granted.
Same example, as a downloadable xlsx workbook.
Download .xlsxAttribution reality check
Line 2 — on-time shipment rate attribution reality check
Line 2 on-time shipment rate recovery, Feb–Mar 2026
Priya Nair, Line Supervisor · 2026-03-17
Scope area
Line 2 on-time shipment rate recovery, Feb–Mar 2026
Current model
Whoever presents at the follow-up review
There's no deliberate model today. Informally, credit for the recovery tends to go to whichever countermeasure owner is presenting when the metric comes up in the follow-up review — not a real calculation of which change actually moved the number.
Cases
On-time shipment rate recovery — Line 2, Feb–Mar 2026 — outcome value 16
Seasonal dip in small-batch order volume → Visual queue board at paint booth → Fixed small-batch sequencing rule
Credit by model
Narrative
The three models disagree sharply about who deserves credit. First-attributed hands full credit to a seasonal dip in small-batch order volume — a real factor, but not a deliberate countermeasure, and an uncomfortable result if it's the honest one. Last-attributed hands full credit to the sequencing rule instead, since it took effect latest. The visual queue board gets zero credit under either edge model, even though it was installed specifically to make the rule stick — a real contributor getting shortchanged by both single-touch views, the same known bias multi-touch marketing attribution research already documents for assist channels. Today's actual convention (whoever presents at the review) is closer to a last-touch habit than anyone realizes. Before betting Line 1's rollout on "the sequencing rule worked," it's worth being honest that this reality check can't settle which of the three actually moved the number — only a controlled trial on Line 1 itself could.
5 · Common mistakes
Common mistakes
Treating whichever model the org already uses as though it were validated, just because it's familiar.
A familiar model is not the same as a checked one — the whole point of this tool is to see how much the number would change under an equally reasonable alternative.
Picking the model that happens to support the story you already wanted to tell.
If a swing exists, it exists regardless of which model you'd prefer to be true — naming the contested contributor plainly is what actually protects the next decision built on it.
Treating a low swing on every contributor as proof the outcome is fully understood.
Agreement across three simple models is reassuring, but it's still not a controlled experiment — it just means these three particular ways of looking at it happen to agree.
6 · What it connects to
What it connects to
upstream
Metric definition sheet
A metric definition sheet is where an unresolved crediting question first gets named as a real gap; this reality check is where that gap actually gets pressure-tested against alternative models.
downstream
7 · Where AI helps
Where AI helps
Judgement — stays yours
- Deciding whether a swing is genuinely worth flagging or within normal noise
- Deciding what a real controlled experiment to settle a contested contributor would need to look like
Analysis — AI helps
- Tightening the current-model description from raw notes into specific, checkable language
- Drafting the narrative naming which contributors are robust and which are contested, from the computed swing
Drudgery — automated
- Computing credit under all three models and the swing between them
- Exporting to xlsx in the house format
9 · Rate this kata