Reviews
Part of Reading campaign measurement with a sceptical eye
Campaign measurement practices: the metric contract to write before launch
Campaign measurement practices for the contract stage: how to fix the metric, source, identity, cost and maturity definitions before a campaign launches.
What to take away
- Write the metric contract before launch, once the question and the design are settled.
- Five definitions carry itmetric, source, identity, cost and maturity.
- Each definition names a source system and an owner, so two reports cannot quietly use two meanings.
- A changed definition goes in the change log, with the date and the reason.
- Thresholds come from your own economics, not from a peer figure.
Campaign measurement best practices make evidence easier to reproduce and harder to overstate. The practice that does the most to prevent disputes is the contract: a short written list of what every number in the report means.
Why the contract comes before the data
Write the contract before launch, because after the numbers arrive every definition becomes negotiable. A team that fixes its meanings first can say later which result was planned and which was found along the way.
Specify and design come first. Specify names the decision, outcome, population, period and primary question. NIST's handbook section on selecting an experimental design explains that objectives, factors, resources and desired information shape design choice. It is not advertising guidance, but it supports choosing a design after the question is clear.
The five definitions a contract fixes
A contract has five fields. Each one answers a question that a reviewer will otherwise ask after the fact, and each one needs wording specific enough that a second analyst would reach the same number.
| Field | The question it settles | Example wording |
|---|---|---|
| Metric | What exactly is counted? | Subscriptions started by new customers |
| Source | Which system is the record? | The billing system, not the ad platform |
| Identity | What counts as one person? | One account, however many devices |
| Cost | Which spend belongs to the campaign? | Media spend and fees for the same dates |
| Maturity | When is the outcome settled? | After the first renewal window has closed |
The example wording is illustrative, not a standard. Replace each line with the choice your own campaign makes, and keep the field names, because reviewers will look for them.
Metric and source: one count, one record
Define the metric as an event, a population and a period. "Conversions" is not a definition. "Subscriptions started by new customers in the campaign window" is one, because another analyst can reproduce it from the same records.
Name the source system beside it. When two systems report the same event, say which one wins and why. The contract should also record how the number is extracted and which calculation version produced it, so a later rerun matches the original or explains the gap. A written policy has little value when access changes, source defects and disputed conclusions leave no trace that another reviewer can follow.
Identity: who counts as one person
Identity decides how many people a campaign appears to reach. A device, an account and a household can each be the unit, and moving from one to another changes the count without any change in behavior.
State the unit, and state what happens when identity is missing or cannot be matched across sources. Credit rules sit on top of identity, so the attribution method you use inherits whatever the contract says here. If the unit changes mid-campaign, report the two periods separately.
Cost and maturity: what was spent, when it counts
Cost and maturity are the two fields people leave loose, and both change the verdict. Cost says which spend belongs to the campaign. Maturity says when the outcome is settled, rather than merely early.
Write the cost definition so that finance and the analyst add up the same items. Write the maturity rule so an early signal is labeled early. Ask later which early metric predicted the mature outcome, and keep the answer for the next contract.
Thresholds belong here too. They are easier to defend when local economics set them rather than a peer figure, which is the argument made in campaign measurement benchmarks.
Keeping the contract honest
A contract only helps if it is controlled. Control means reconciliation against the source, quality gates before a number is published, access limits on who may change a definition, and a change log that records every edit.
The U.S. Census Bureau publishes statistical quality standards for its own information products, covering work from planning through dissemination and correction. Those federal requirements do not govern a private campaign, but they provide a concrete example of quality discipline.
Turn the contract into a review with the campaign measurement checklist, which sets out the items and the evidence for each one. The failures that keep coming back are collected in campaign measurement mistakes.
Keep the contract beside the decision it supported.
