Part 3 · Operate
Learning
Ideas being tested, outcomes including silence, and how you compare against dumb baselines.
01What it is
What the system believes about who buys, what happened when it acted on those beliefs, and how it compares against three simple baselines.
02Why it exists
Any system can claim it is learning. Comparing against baselines is what makes the claim checkable: three baseline versions run alongside the real one, and a reserved share of budget keeps them running so the comparison stays honest.
Negative outcomes count. Ignored, bounced, hostile and wrong-person all feed allocation. Discarding them would make every idea being tested look better than it is.
03How to use it
- 01Read the list of ideas being tested and their posterior rates.
- 02Compare the system’s version against the baselines. If it is not beating them, that is the finding.
- 03Look at what is being killed and why. Nothing is killed before enough resolved actions.
- 04Act on it: pause an idea being tested, change a segment, or change the message.
04What good looks like
- You give it
- Three ideas being tested with 60 resolved actions each.
- You get
- Posterior rates with intervals, a leader, and a kill recommendation on the version sitting far below it.
- Why
- The minimum sample floor has been passed, so the comparison means something.
Derived from the rules in the code
05What weak looks like, and what it costs
- You give it
- Two ideas being tested with eight resolved actions.
- You get
- Wide intervals and no kill recommendation.
- Why
- Nothing is killed below the minimum sample, because eight results cannot separate two rates.
Derived from the rules in the code