Test 08
The Forecast Value Test
Does your managers' forecast judgment beat a model that applies no judgment at all, and does it predict value or only closure?
What everyone does instead
Forecast accuracy is measured by comparing the final roll-up to the actual number. That tells you whether the total was right. It does not tell you whether any individual judgment helped, and it is scored with a rule that rewards confidence rather than honesty. Almost no organisation has tested manager judgment against a no-judgment baseline, and none scores it against what the deals turned out to be worth.
What to compute
- A stage-rank baseline: historical win rate by stage, segment, size band and age, estimated out of sample.
- Manager forecast categories calibrated to their own realised rates, then scored against the baseline with a Brier decomposition.
- The skill correlation of manager adjustments, computed twice, against closure, and against realised 24-month value.
- The break-even adjustment size for this organisation, and the ceiling on achievable improvement.
The arithmetic
Scored against closure Skill correlation 0.21 Break-even adjustment size 0.174 Scored against realised 24-month value Skill correlation 0.13 Break-even adjustment size 0.108 Optimal adjustment magnitude at 0.21 0.087 Actual adjustment magnitude 0.190 0.190 / 0.087 2.19 times oversized Ceiling on improvement from human judgment 0.21 x 0.21 0.044 4.4% of baseline error
Worked example
In one organisation, skill correlation was 0.21 against closure and 0.13 against realised value, while managers adjusted at 2.19 times the size their measured skill supports. Maximum achievable improvement from human judgment is the square of the correlation, 4.4% of baseline error, even with perfectly sized adjustments.
Illustrative figures. The relationship shown is the hypothesis the test exists to check.
What changes if this is true
The ceiling on forecast improvement from human judgment is the square of the skill correlation, which means the weekly ritual cannot pay for itself on accuracy however well it is run. That is not an argument for abandoning it. It is an argument for knowing what it is actually buying, and for looking for the return in capacity and allocation instead.
What you need and how long
Weekly forecast snapshots, four quarters
- Where it lives
- CRM field history or warehouse
- Days
- 1
Renewal and expansion outcomes, older cohort
- Where it lives
- Billing
- Days
- 1
Baseline, calibration, scoring, null simulation
- Where it lives
- Analysis
- Days
- 3
Every test runs on the same two extracts, forecast history from the CRM and contract history from billing, pulled by a RevOps analyst in about an hour. Everything is measured against your own outcomes: no benchmark, no composite, no assumed uplift.