Skip to content
NEW: Martech, explained →

Martech, explained · AI and agents in martech · 1:46

How to measure what an AI agent does

An agent will report its own success. Check it the way you'd check someone new: the work, the rules, and the result

Embed this film

Paste this where the film should play. It links back here.

The short answer

Check an AI agent like someone new on your team. The work: read a regular sample of what it did. The rules: an action log, and a count of the limits it hit and should have hit. The result: a holdout, a share of customers the agent doesn't touch; a before-and-after absorbs the season and the ad platforms.

Questions this answers

Why can't I trust the lift an AI agent reports?

Because it reports its own success. A twelve percent lift raises two questions: twelve percent of what, and compared with what? Measuring it is your part, not the agent's.

How do I check the work an AI agent does?

Take a regular sample of what it did this week and read it, as someone who knows the brand. Would you have signed off on it? Not every decision, a sample.

What should an AI agent's action log contain?

Every action: what it did, when, and why. Then count the times it hit a limit, and the times it should have. A limit nobody ever hits may not be wired in at all.

Why is a before-and-after the wrong way to measure an AI agent?

It absorbs the season, the price change, and whatever the ad platforms did that month. The fix is a holdout: a share of customers the agent doesn't touch. The difference between the two groups is what the agent added.

What does an AI agent cost to run?

The credits, and the hours your team spends checking it. And a holdout costs you something too: a small group left untouched is the price of telling a real lift from a good story.

Transcript

Every chapter as text. Click a time to play the film from there.

0:00 A twelve percent lift

Your agent reports a twelve percent lift.

Twelve percent of what? And compared with what?

An agent will happily report its own success. Measuring it is your part.

Check it the way you'd check someone new on your team: the work, the rules, and the result.

0:22 The work

First, the work. Take a sample of what it did this week, and read it. Would you have signed off on it?

Not every decision. A regular sample, read by someone who knows the brand.

0:35 The rules

Second, the rules. Every action should go into a log: what it did, when, and why.

Count the times it hit a limit, and the times it should have. A limit nobody ever hits may not be wired in at all.

0:51 The result

Third, the result. Most reports go wrong here, without meaning to.

A before-and-after comparison absorbs the season, the price change, and whatever the ad platforms did that month.

The fix is a holdout: a share of customers the agent doesn't touch.

It works like the control group in an A/B test.

Compare the two groups. The difference is what the agent added.

Yes, a holdout costs you something. That's the price of knowing.

And count what it costs to run: the credits, and the hours your team spends checking it.

1:28 Start the holdout

If you only start one thing, start the holdout.

A small group, left untouched, is what tells a real lift from a good story.

Learn more at Martech Stack Builder, or build your own.