Reading an Ecommerce Vendor Case Study Critically

Most DTC operators read vendor case studies to make a purchasing decision, not to admire a piece of marketing writing, and that distinction matters because the format is built for the opposite purpose. A case study is a document produced by a vendor's marketing team, built from data the vendor alone controls, to support a conclusion the vendor already needs to reach. No individual has to act dishonestly for that structure to produce a misleading number. The vendor decides what gets measured, when measurement starts, which customers appear in the result, and whether any baseline is shown at all, and the reader only ever sees the output of those decisions, never the decisions themselves.

This gap between what gets reported and what actually happened is not a fringe concern limited to disreputable vendors. Research into AI agent purchasing behavior has found that even signals that look objective, such as rankings, endorsement tags, and sponsored labels, produce outcomes that are model-dependent and unstable: the same storefront produces different results depending on which AI model is doing the buying. Vendor dashboards built on top of these systems never surface that variance to the people reading them. The same structural opacity runs through AI-tool case studies more broadly. A vendor defines the cohort, picks the time window, chooses the metric, and publishes a result, all without showing the reader a single one of the choices that produced it.

None of this means a given case study is fraudulent. It means the format rewards the best-looking number over the most accurate one, and a reader who wants an accurate picture has to go looking for the specific gaps that structure tends to leave. The sections that follow name those gaps one at a time: cohort selection, attribution, missing baselines, and a new layer of AI-specific metrics that are harder to verify than anything that came before them. Each comes with a direct question an operator can ask before a contract gets signed.

Cherry-picked cohorts inflate the headline number before any other manipulation occurs

The simplest way to inflate a result is to measure only the customers who were likely to succeed regardless of the tool, and not disclose that the group was selected that way. This happens before any other distortion gets layered on top, and it is the first thing worth checking in any case study, because a cohort that was chosen to win will win no matter how carefully everything downstream is measured.

The most common version compares customers who actively engaged with a tool, by clicking, querying, or interacting with it, against all other visitors to the site. A shopper who chooses to open a chat window or step into an AI-assisted flow has already shown more purchase intent than an average visitor who scrolled past. The tool then gets credit for intent the shopper brought with them before the interaction ever started. Studies of AI agent behavior conducted under controlled conditions use randomized experimental designs for exactly this reason: randomization is what separates the intent a customer already had from whatever marginal difference the tool actually made. Vendor case studies almost never use that design; instead they compare engaged users against everyone else on the site.

Watch for a few recurring patterns. A claim that "customers who used Feature X converted Y% better" is nearly always comparing engaged users to all users, not engaged users to a matched control group that didn't use the feature. Results reported only for a brand's top-performing SKU category, or only for a single strong geography, tell you the vendor picked the segment that would produce the best number rather than the segment that represents the brand's business as a whole. A result window that opens at a product launch or a seasonal peak and closes before the novelty wears off is measuring a moment, not a trend.

The question to ask before accepting any headline figure: who exactly sits in the numerator and who sits in the denominator, and was that definition set before the vendor saw the data, or after?

Attribution inflation gives the tool credit for revenue it did not cause

A fairly defined cohort can still produce a misleading result if the case study assigns the tool credit for revenue it did not actually generate. This is a separate failure from cohort selection, and it persists even after a vendor has, to its credit, built a reasonably matched comparison group.

Most vendor reporting relies on last-touch attribution, which assigns full credit for a sale to whichever tool the customer interacted with last before converting. Consider a shopper who researches a product through an AI channel, receives an email promotion a few days later, and then interacts with an on-site AI tool immediately before buying. Under last-touch logic, the on-site tool claims the entire sale, even though the email offer or the earlier research may have been what actually decided the purchase. Research on AI agent purchasing behavior shows that agents respond differently to platform endorsement tags than to organic positioning. An attribution model that ignores the full channel mix a shopper passed through before landing on-site is missing a meaningful part of the picture by design, not by accident.

This AI-specific problem appears constantly in vendor claims about revenue lift. A vendor reports a lift during a period when the brand also ran a promotion, changed pricing, expanded its paid acquisition spend, or simply benefited from a seasonal demand spike, and the case study makes no attempt to separate the tool's contribution from any of those concurrent changes. Genuine isolation requires a holdout group: customers who did not have access to the tool, matched on the characteristics that predict whether someone converts, measured over the same time window, exposed to the same promotions. Without that holdout, the number a vendor reports only shows that something happened compared to nothing happening. It does not show what happened with the tool compared to without it, and those two comparisons are not interchangeable.

The question to ask here: what would have happened to this same group of customers without the tool, and the case study should show evidence that someone actually measured that outcome directly.

The missing baseline is the most reliable signal that a case study should not be trusted

Every inflated number traced through the two sections above eventually leads back to a single missing piece of information: the baseline. A baseline is what the measured metric looked like before the tool was introduced, for the same customers, under comparable conditions. Its absence is the fastest and most reliable signal that a case study cannot be trusted, because it reveals whether the cohort was selected to win and whether the tool's contribution was isolated from other causes, in a single check.

A percentage improvement with no stated starting value carries no real information. A small conversion rate that doubles and a much larger conversion rate that also doubles both get reported as the same "doubled," yet the two outcomes describe entirely different businesses. The vendor chooses which framing to put in the headline, and a reader who never sees the starting number has no way to tell which story they're actually looking at. Missing baselines take a few recognizable forms: a percentage improvement with no starting value attached ("conversion rate improved dramatically"); a comparison to an industry average rather than the brand's own prior performance, where that average aggregates across businesses with radically different traffic quality, price points, and categories; a before-and-after comparison with no date attached to the "before" period, making it impossible to check whether that period was representative or hand-picked; and results reported only as a relative multiplier, with no absolute numbers appearing anywhere in the document.

A trustworthy baseline looks specific. It names a client, names a metric, states an exact date range for the pre-period, and reports the actual number, not a percentage change calculated from a figure the reader never sees. The question to carry into every case study: what did the metric look like before the tool was deployed, expressed as an absolute number, for the same customer group, over a time window comparable to the one being used to claim success?

AI-specific metrics create a new category of vanity number

AI commerce vendors have introduced a layer of metrics, such as citation share, AI visibility scores, and recommendation frequency, that are easier to inflate than conventional ecommerce metrics for a simple reason: no standardized definition exists for any of them, and most DTC operators have no independent way to verify what's being reported.

The gap between citation and recommendation shows how a metric can look impressive while contributing nothing to revenue. A brand that an AI model mentions frequently across responses has reach, but reach is not the same as being recommended in a context where a shopper is actually ready to buy. Citation share measures exposure. Recommendation frequency is closer to a conversion-adjacent measure. Treating the two as interchangeable is the AI equivalent of reporting impressions as though they were sales. A vendor selling an "AI visibility score" that is never connected to actual recommendation frequency, or to measurable on-site traffic, is selling a number with no established relationship to revenue.

The instability runs deeper than definitional looseness. Research on AI agent purchasing behavior shows that agent preferences are model-dependent and shift across model versions, so an AI visibility score measured against one model may look entirely different after that model gets updated. A case study that doesn't specify which model and which version a metric was measured against is not something a reader can reproduce or check later. The same research documents that agents tend to concentrate demand on a small number of "modal" products, and that model updates can reshuffle which products hold that position. A case study showing strong AI-channel performance at a single point in time may be capturing a temporary preference baked into one model version rather than a durable advantage the tool actually created.

Three questions separate a defensible AI metric from a vanity one. Which specific AI model and version was the metric measured against? Is the reported figure connected to on-site traffic, conversion, or revenue, or does it stop at the AI surface and never touch a number the business can bank? And has the metric been measured more than once, across more than one model update, to show the result holds up rather than reflecting a single model's temporary quirk?

Traits of a credible case study

A credible case study is identified by whether a reader can trace its headline number back to a clearly defined group, a measured baseline, an isolated variable, and a verified outcome, and most vendor case studies fail at least one of those four conditions.

A DTC operator evaluating a vendor's claims should apply a short checklist before trusting any of them. The case study should name an actual client and a contact a reader could, in principle, reach to verify the result. It should state a specific pre-period baseline as an absolute number, anchored to a date range. It should explain how the test group was defined, and whether a holdout or control group existed against which the test group was compared. It should account for anything else that changed during the measurement period, such as pricing, promotions, or acquisition spend. Its central metric should connect directly to revenue, or to a leading indicator with a documented relationship to revenue, rather than a proprietary visibility index invented by the vendor itself. For any AI-specific claim, it should state the model, the version, and the measurement date, and show evidence the result has been checked again across at least one subsequent model update.

The posture a DTC operator should expect from any AI commerce vendor follows directly from that checklist: engaged-shopper cohorts measured against the brand's own historical baseline instead of an industry average, incrementality framing that openly accounts for what else was happening during the measurement window, and absolute numbers presented alongside any percentage claim. Research on agentic e-commerce argues that in a market where buyer agents can investigate a claim as exhaustively as they choose, verified, trustworthy information becomes the scarce resource. The same standard applies to a vendor's own case studies. A willingness to expose the methodology behind a claimed result is itself a signal about the quality of what's being sold.

A vendor that cannot, or will not, answer the questions raised across cohort definition, attribution, baseline, and AI-specific measurement is making a choice about what to leave out of the conversation. That choice is itself the most useful piece of information a DTC operator can gather before signing anything. Asked directly and in sequence, these questions produce one of two outcomes, and both are useful: a vendor with a defensible result will answer directly and the case study holds up, while a vendor without one will hedge, deflect, or produce a new metric mid-conversation, telling an operator everything they need to know before a dollar changes hands.