Martech, explained · AI and agents in martech · 1:46
How to measure what an AI agent does
An agent will report its own success. Check it the way you'd check someone new: the work, the rules, and the result
Embed this film
Paste this where the film should play. It links back here.
The short answer
Check an AI agent like someone new on your team. The work: read a regular sample of what it did. The rules: an action log, and a count of the limits it hit and should have hit. The result: a holdout, a share of customers the agent doesn't touch; a before-and-after absorbs the season and the ad platforms.
Questions this answers
Why can't I trust the lift an AI agent reports?
Because it reports its own success. A twelve percent lift raises two questions: twelve percent of what, and compared with what? Measuring it is your part, not the agent's.
How do I check the work an AI agent does?
Take a regular sample of what it did this week and read it, as someone who knows the brand. Would you have signed off on it? Not every decision, a sample.
What should an AI agent's action log contain?
Every action: what it did, when, and why. Then count the times it hit a limit, and the times it should have. A limit nobody ever hits may not be wired in at all.
Why is a before-and-after the wrong way to measure an AI agent?
It absorbs the season, the price change, and whatever the ad platforms did that month. The fix is a holdout: a share of customers the agent doesn't touch. The difference between the two groups is what the agent added.
What does an AI agent cost to run?
The credits, and the hours your team spends checking it. And a holdout costs you something too: a small group left untouched is the price of telling a real lift from a good story.
Transcript
Every chapter as text. Click a time to play the film from there.
0:00 A twelve percent lift
Your agent reports a twelve percent lift.
Twelve percent of what? And compared with what?
An agent will happily report its own success. Measuring it is your part.
Check it the way you'd check someone new on your team: the work, the rules, and the result.
0:22 The work
First, the work. Take a sample of what it did this week, and read it. Would you have signed off on it?
Not every decision. A regular sample, read by someone who knows the brand.
0:35 The rules
Second, the rules. Every action should go into a log: what it did, when, and why.
Count the times it hit a limit, and the times it should have. A limit nobody ever hits may not be wired in at all.
0:51 The result
Third, the result. Most reports go wrong here, without meaning to.
A before-and-after comparison absorbs the season, the price change, and whatever the ad platforms did that month.
The fix is a holdout: a share of customers the agent doesn't touch.
It works like the control group in an A/B test.
Compare the two groups. The difference is what the agent added.
Yes, a holdout costs you something. That's the price of knowing.
And count what it costs to run: the credits, and the hours your team spends checking it.
1:28 Start the holdout
If you only start one thing, start the holdout.
A small group, left untouched, is what tells a real lift from a good story.
Learn more at Martech Stack Builder, or build your own.