Careers
Part of Customer analytics: what to keep and what to drop
What vendor-published customer analytics case studies actually prove
Vendor-published customer analytics case studies show workflows worth testing, once you check the unit, source, decision, evidence and result behind each claim.
What to take away
- A vendor-published case study documents one customer account, chosen by the vendor. Treat it as a lead, not a benchmark.
- Check the customer unit, data source, decision, evidence type and result denominator before the headline number.
- Recalculate every quoted percentage from its numerator and denominator. If either is missing, the figure proves nothing.
- No vendor ranking appears in this review. Two cases cannot order a market.
- Transfer the workflow, then re-test it on your own traffic before copying a result.
What a published case can and cannot prove
A vendor case study proves that a named customer used a product and agreed to say so. It rarely proves the product caused the reported change. Most are descriptive accounts written by an interested party.
What it supports
- A named customer used the product
- Adoption happened
- A percentage improvement
- A change in that account
- A reported time or cost saving
- The customer's own account
- A commissioned interview
- A testimonial exists
What it does not support
- A named customer used the product
- That adoption caused the result
- A percentage improvement
- That your account will match it
- A reported time or cost saving
- That the method transfers
- A commissioned interview
- Independent verification
Choose the unit of analysis before you read the claim. A customer analytics scope settles which fields belong in the case file and which you can drop.
Example: PepsiCo and consumer-centered marketing data
Google Marketing Platform published a PepsiCo account built on consumer-centered marketing, an internal capability and selected platform use.
The measured item is adoption of that capability, and the claim is marketing organized around consumer data. Missing: a baseline, a control group and an independent review of the sales outcome.
Read it as a program description, not proof of incremental revenue.
Ask of every case: what changed, against which baseline, across which population, and who verified it?
Example: 412 Food Rescue and reporting time
Google also published an account of 412 Food Rescue changing reporting work with Google Analytics. Staff time on reporting is the measured outcome. The claim is a workflow improvement, not program impact. Missing: hours before and after, reports counted, and whether the saved time was reinvested.
The arithmetic is short. Illustrative, not from the case: a 40 percent cut in a weekly task of 10 hours saves 4 hours a week. Without the starting hours, the percentage cannot be tested.
Verify reporting-time claims
- Verify who supplied each number
- Separate implementation time from recurring effort
- Check outcome and population definitions
- Look for excluded costs and failed cases
- Ask whether counterfactual supports effect claim
- Contact reference with permission when stakes justify
Reporting-time claims belong with customer analytics benchmarks, where the comparison has to be fair before the number means anything. A faster dashboard says nothing about retention or revenue.
What the FTC rules let you challenge
The FTC advertising substantiation policy requires a reasonable basis for objective advertising claims before they are disseminated. That standard gives you grounds to ask vendors for the evidence behind a published performance figure. It sets a bar, not a method, so route legal questions to counsel.
The FTC guidance on soliciting and paying for reviews covers fake feedback, selective solicitation, conditioned incentives, hidden relationships and paid placement. Demand answers whenever a roundup names providers. Inclusion is not proof of quality.
A six-field review checklist
- Customer unitnamed account, sector, size, and whether it resembles yours.
- Data sourcewhich system produced the numbers, and what it excluded.
- Decisionwhat the customer changed, and when.
- Evidence typedescriptive account, before-and-after comparison, held-out test, or independent audit.
- Numerator and denominatorthe counts behind every percentage.
- Cost and limitationfull cost to run, plus what the case admits it did not measure.
Run these fields against both cases above, then file the outcome beside the decision. The customer analytics audit checklist lists the questions to settle before a pilot starts, so they stop resurfacing mid-project.
Case review fields
- Customer unitperson, household, segment, market?
- Sourcefirst-party, platform, research, modeled?
- Decisionplanning, creative, audience, product, budget?
- Evidencedescriptive, predictive, experimental, attributed?
- Resultmetric, denominator, period, baseline, uncertainty?
- Transferwhich process works without copying number?
Rebuild the claim on your own data
- Turn one material claim from the case into a testable sentence.
- Fix the customer unit and the outcome window before you look at results.
- Run a holdout or switchback so the comparison has a control. A/B testing design covers assignment, sample size and the limits of a single test.
- Build the full cost model, then compare cost per outcome instead of percentage change.
- Write down what you will do if the result comes back flat.
Recalculate each percentage from its own numerator and denominator. Check that dates, customers, channels and outcome windows match before you compare anything.
A case study earns its place when a reviewer can rebuild your reasoning without asking you. Set the next review date and name the change that would trigger an earlier look.
Two published write-ups get this treatment in campaign measurement case studies, which is worth reading for a second worked example of the method.
Common questions
Do vendor case studies prove anything at all?
They prove a named customer used the product and agreed to be quoted. They rarely prove causation, and they never show your account will repeat the result.
What can I safely transfer from one?
The questions the customer asked, the controls they built, and the workflow they changed. Test each locally before adopting it.
Should a case study claim causation?
Only when the design, assignment and stated uncertainty support it. Most published accounts are descriptive, so test the effect yourself.
How many cases justify ranking vendors?
More than any vendor publishes, and only under identical acceptance tests. Until then, compare contract terms rather than case counts.
