How we check vendor capabilities
Method as of 11 November 2026
Our catalog records what each martech product does: 12,074 capability grades across 175 capabilities. 11,994 of them (99%) link the page on the vendor’s own site the grade was read from. This page explains how a grade gets there, how it can change, and how often we measure it to be right.
Two grades, and no grade
- Core: the capability is a named module in the product’s navigation or pricing, or the main job of one.
- Supported: the product genuinely offers it, documented on its own site, but it is not what the product is for.
- No grade means one of two things, and we keep them apart. Where we checked and did not find it, comparison pages say “Not offered” and link the page we read. Where nobody has checked, the cell is blank.
A written definition for each capability
Each capability has a definition that every grade is judged against, shown at the top of its capability page. 34 of them also say what does not count, because those were the ones where careful readers of the same page disagreed. Changing a definition is a reviewed change to the catalog, and the grades it touches are checked again.
Evidence from the vendor’s own pages
- Research runs read the vendor’s documentation, product and pricing pages. Review sites, analyst reports and partner pages are not evidence for a grade.
- A grade records the page it came from and a one-line note of what that page shows. Capability pages print both, so you can check us.
- Open-source products are graded on what the project itself ships. A third-party module or plugin does not count as a capability of the product.
- Each grade carries the date it was last checked.
The cited pages are re-checked every month
Once a month a script fetches every page a grade cites, without AI, and checks that the page is still there and still says what we quoted. Where we hold only a summary of the page, it checks that the page has not changed since the last good check. A page that has gone, moved or changed is queued for review; the grade itself is not changed by the check. Vendor pages show both dates: when a grade was last verified and when its source was last checked. Pages that block scripts or only render in a browser keep their earlier check date. The script identifies itself as MSBSourceCheck and follows each site’s robots.txt.
How a grade changes
AI research is not perfectly repeatable: two runs over the same pages can disagree. So one run never removes or lowers a grade on its own. When a new run disagrees with what we hold, the disagreement goes to a separate review that sees both sides’ evidence and the definition. To keep a grade, that review has to quote a page on the vendor’s own site that it actually opened. Where the evidence is ambiguous, the grade stays as it is and is marked for a person to look at.
How accurate it is
We keep a set of 50 grades checked by hand against the vendor’s own pages by two people who did not see the AI’s answer. These were deliberately the hard cases: rows where two AI runs had disagreed. Every change to our research method is scored against this set before it touches the catalog. Last measured 11 November 2026:
- 94%
- agreed on whether the product offers it (46 of 49)
- 96%
- of removals confirmed (25 of 26)
- 2 of 6
- “core” calls confirmed in disputed cases
The last figure is the weak spot, and we say so. AI research tends to call something core when it is only supported. So when a disputed grade is “core” and a person has not confirmed it, we record it as supported. Treat the core and supported split as a strong hint, and the yes or no as the firmer part of the record.
Changes are public
Changes to the catalog are listed in what changed in martech, with an RSS feed.
If you think a grade is wrong, use Suggest a correction next to it, on any capability page or in compare. You don’t need an account. Say what is wrong and link the page on the vendor’s own site that shows it; the link is required, because it is what we check. An email address is optional, and we use it only to tell you the outcome.
How a correction is settled:
- The evidence decides, not who asked. A vendor’s own suggestion is settled the same way as anyone else’s.
- The page you linked is read and judged against the capability’s written definition, by the same review that settles any other disagreement. Review sites and partner pages do not count.
- If the page shows the change, the grade changes and the change appears in the change feed. If it does not, the grade stays. If it is ambiguous, a person looks at it.
Vendors can also claim their listing and correct it directly.