Part 3 · Operate
Learning
Hypotheses, outcomes including silence, and the benchmark against dumb baselines.
01What it is
What the system believes about who buys, what happened when it acted on those beliefs, and how it compares against three simple baselines.
02Why it exists
Any system can claim it is learning. The benchmark is what makes the claim checkable: three baseline arms run alongside the real one, and a reserved share of budget keeps them running so the comparison stays honest.
Negative outcomes count. Ignored, bounced, hostile and wrong-person all feed allocation. Discarding them would make every hypothesis look better than it is.
03How to use it
- 01Read the hypothesis list and its posterior rates.
- 02Compare the system arm against the baselines. If it is not beating them, that is the finding.
- 03Look at what is being killed and why. Nothing is killed before enough resolved actions.
- 04Act on it: pause a hypothesis, change a segment, or change the message.
04What good looks like
- You give it
- Three hypotheses with 60 resolved actions each.
- You get
- Posterior rates with intervals, a leader, and a kill recommendation on the arm sitting far below it.
- Why
- The minimum sample floor has been passed, so the comparison means something.
Derived from the rules in the code
05What weak looks like, and what it costs
- You give it
- Two hypotheses with eight resolved actions.
- You get
- Wide intervals and no kill recommendation.
- Why
- Nothing is killed below the minimum sample, because eight results cannot separate two rates.
Derived from the rules in the code