Incrementality Testing for On-Site Conversion Tools

Brands have spent the last several years learning that platform-reported ROAS is a story, not a measurement. Meta and Google overstate measured incremental ROAS by as much as 3x, with the gap widest on brand search and retargeting, which happen to be the placements that already look best on the dashboard. That same distortion is built into how on-site conversion tools report their own performance, and the industry has not caught up to the problem yet.

On-site conversion tools and the attribution fiction paid media created

AI chatbots, recommendation engines, quizzes, and popups all report lift the way Meta and Google do: by comparing people who converted after touching the tool against some other, non-randomized group of shoppers. The comparison looks like evidence because it comes with a number attached, but the number describes who used the tool, not what the tool caused. A visitor who opens a product recommendation quiz or starts a conversation with a chatbot has already signaled something that most browsers never signal: active intent to buy. Measuring that visitor's conversion rate against the site average answers a question nobody asked. It does not say whether the tool moved that sale from "no" to "yes." It says that interested people convert more often than uninterested people, which was true before the tool existed.

Feastables' chatbot, FeastyBot, built with Certainly, reported a meaningfully higher conversion rate among its users along with a meaningful lift in average order value. None of that is a mark against the brand or the vendor. It is a measurement design gap, the same gap that let Meta and Google report inflated ROAS for years before brands started demanding holdout tests to check the math. The chatbot may well be driving real incremental revenue for Feastables. The reported figures cannot distinguish that outcome from simple self-selection, and until a holdout exists, nobody, including the brand, actually knows which story is true.

How incrementality testing works and its difference from attribution

Attribution assigns credit to touchpoints based on what happened before a purchase: a click, an impression, a session. Incrementality asks a different question entirely: would the sale have happened without that touchpoint? Attribution works from correlation. Incrementality tries to isolate causation by comparing a treated group against a group that never saw the ad, the email, or the tool. The counterfactual is the whole exercise: what happens in a world where the marketing never ran. That number is almost always lower than whatever the dashboard shows, because dashboards are built to count everything that happened near a touchpoint, not just what the touchpoint caused.

Three test designs cover most of what brands need. Synthetic control methods, such as Meta's GeoLift and Google's CausalImpact, build an artificial counterfactual out of a weighted blend of untreated markets, which helps when a brand can't find two markets clean enough to match directly.

None of these designs produce a permanent answer. A single test ages out in roughly 90 days, as creative rotates, seasonality shifts, and competitors change their own spending. Incrementality is a habit, not a one-time project, and that fact matters just as much once the subject turns from ad channels to the tools sitting directly on a brand's own site.

Adapting the paid-media holdout design for on-site tools

Geo holdouts work because channel spend can be turned off in one market and left on in another. An on-site tool doesn't have that luxury: it lives on one domain, serving every visitor who lands there, so there's no geographic lever to pull. The treatment unit has to change from a market to a visitor session. Randomization has to happen at the level of the individual session or user cohort, not at the level of a DMA pair, because every visitor is reachable by the same tool at the same time.

The holdout mechanism has to change to match. Instead of withholding ads from a market, the test withholds the tool itself from a randomly assigned slice of otherwise-qualifying visitors, a structure closer to a PSA holdout, where a control group sees a placeholder message instead of the real creative, than to a DMA split. The success metric has to change as well. Paid-media incrementality targets incremental ROAS, a spend-to-revenue ratio that makes sense when the variable being tested is a media budget. On-site tools don't have a spend line to measure against, so the right metrics are incremental order rate and incremental revenue per session, compared across the full holdout and treatment cohorts, not just among the subset of users who happened to engage.

That last distinction is where the self-selection trap does its damage. Platforms like Kinect that deploy brand-trained AI sales reps on-site have an incentive to test incrementally rather than lean on engagement metrics, since measuring lift against a genuine holdout cohort is the only way to show the tool generated revenue that would not have happened anyway.

Budget thresholds carry over too, in a different unit. Geo-holdout tests need enough order volume per region to detect a real difference. Brands under roughly $30,000 to $50,000 in monthly spend often lack the data density for a clean geo test. On-site experiments hit the same wall measured in session volume per day rather than ad spend per month: a tool triggering on too few qualifying sessions daily won't produce a statistically meaningful read no matter how long the test runs.

Honest lift numbers from a correctly designed holdout

A properly randomized holdout almost always produces a smaller number than the vendor dashboard, for a simple reason: the vendor-reported baseline is built from self-selected high-intent users, while the holdout baseline is built from every qualifying visitor, engaged or not. The gap between those two baselines is the same gap that separates platform ROAS from measured iROAS in paid media, and it tends to run in the same direction and at a similar scale.

Northbeam's Black Friday planning work shows how large that gap can be once cross-channel effects are measured properly. Last-click attribution credited Meta heavily for the holiday period, but media mix modeling, which accounts for how channels interact rather than crediting whoever touched the sale last, found that Google Search was actually driving a larger share of incremental revenue. Shifting budget to match that finding increased revenue without increasing total spend. The lesson transfers directly to on-site tools: the number that matters is lift measured across the full treated cohort, not lift measured only among the people who chose to engage. The first number is the honest one. The second number is a description of who shows up, dressed up as a description of what the tool did.

Haus's public case studies, covering Bombas, True Classic, and Liquid Death, put the paid-media overstatement at up to 3x. A smaller, holdout-verified lift number is a number a finance team can actually build a budget around, which is worth more than a flattering number nobody can defend in a board meeting.

Where on-site measurement meets AI-referred traffic and agent-based discovery

The measurement gap is about to widen, because a new category of visitor is arriving on brand sites with a different intent profile than anything attribution models were built to handle. AI-referred shoppers reach a site after an external agent, ChatGPT, Perplexity, Gemini, has already recommended it to them. That's a structurally different kind of arrival than retargeting, where the visitor was already inside the brand's funnel before the ad ever ran. A net-new buyer sent by ChatGPT is a much harder case to wave off as "would have bought anyway" than a cart-abandoner who saw a retargeting ad the next day. AI-referred traffic, as a category, is inherently more incremental than retargeting traffic, and brands that lump the two together in a single attribution model will misread the signal in both directions.

The scale of this shift is no longer theoretical. Shopify turned on Agentic Storefronts for all eligible US merchants on March 24, 2026, and a large number of Shopify merchants, including Etsy, are now live on ChatGPT Shopping, with Glossier, SKIMS, Spanx, and Vuori announced as early partners though not yet confirmed live. That's a new, measurable traffic source that existing attribution infrastructure was never designed to parse, arriving at meaningful scale inside a single calendar year.

On-site data quality now has consequences that extend beyond the storefront itself. Pages with complete Product schema are more likely to be cited by AI systems in the first place, and brand mentions inside AI answers are heavily sourced from third-party pages, including community platforms and review sites. A brand's own product data quality affects whether it gets recommended by an agent at all, long before any visitor lands on the site. And once that visitor does land, AI chat sessions running on the brand's own storefront generate a kind of data no attribution model has ever produced: which specific objection actually stopped a purchase, which compatibility question went unanswered on a product page, where a high-intent buyer gave up. That data has no equivalent in a last-click report, and brands that treat it as just another conversion-rate input are leaving the most useful part of it on the table.

Determining what would have happened if the tool had not been there gets harder for a tool that runs continuously on every session rather than flighting on and off like an ad campaign. That's why geo-holdout and synthetic-control thinking has to be rebuilt around user-level randomization for on-site tools. Without that adaptation, brands fall back on vendor-reported metrics that conflate correlation with causation the way platform-reported ROAS did in paid media, at the moment that AI-referred traffic is making intent more consequential to measure correctly, not less.

Criteria for tools that support on-site incrementality testing

The criteria that matter here are different from the criteria that matter for paid-media holdouts: session-level randomization rather than geo-level, the ability to suppress a feature for a control cohort rather than an entire geography, and a success metric defined across all treated sessions rather than limited to engagers alone.

Kinect, which deploys brand-trained AI sales reps directly on-site, fits this need because its self-selection problem is the sharpest version of what this piece has described throughout: a visitor who starts talking to an AI rep has already shown more intent than one who doesn't, so measuring the rep's impact honestly requires randomizing at the session level, exposing some qualifying visitors to the rep and withholding it from a control slice, and then comparing order rate across the entire cohort rather than just among the people who chose to chat.

Haus received the highest assigned score for experiment recommendations and insights in a September 2026 Sellforte evaluation, a self-published assessment rather than an independent third-party review, with full marks for features including AI-generated summaries and conversational AI for experiments. Haus is strongest for geo experimentation and causal MMM, and it suits brands with enough volume to run continuous, always-on geo testing, though that strength sits closer to paid-media measurement than to the session-level design on-site tools need.

Brands choosing between self-serve and managed approaches face a real trade-off, not a false one. Neither choice is wrong; the choice should follow from what the team can actually execute and defend.

Running a defensible on-site holdout test in practice

The entire exercise comes down to one counterfactual question: if the tool had never been deployed, how many of the orders in the treated group would have happened anyway? Everything in the test design exists to answer that question honestly rather than flatteringly.

The first step is defining the qualifying visitor pool correctly. Every session that meets the tool's trigger conditions, a product page visit, a cart add, an exit-intent signal, becomes eligible for assignment to the test, not just the sessions where the visitor actually chose to engage. Narrowing the pool to engagers before randomizing reintroduces the self-selection bias the whole test exists to remove.

The second step is randomizing at the session level. A random slice of qualifying sessions goes to control, where the tool is suppressed, and the remainder goes to treatment, where the tool runs normally. The assignment should stay stable per visitor across repeat sessions, since a visitor who bounces between groups contaminates the comparison and makes the eventual lift number unreliable. This is the same principle behind isolating an on-site agent's incremental lift across the entire visitor cohort, whether the agent in question is Kinect's or any other brand-trained AI rep: the order-rate comparison only means something if the two groups were split cleanly before anyone engaged with anything.

The third step is measuring order rate, and revenue per session, across every session in each group, not conversion rate among the people who actually used the tool. The denominator is the full qualifying pool in both control and treatment, never just the subset who interacted. A test that quietly narrows its denominator to engagers has stopped measuring incrementality and started measuring the same self-selection bias the holdout was built to remove.

The fourth step is running the test long enough to clear seasonality noise, long enough that a single promotional week, a competitor's sale, or a short-term traffic spike doesn't get mistaken for the tool's effect. A test that ends the moment the numbers look good is a test that hasn't actually finished running. Brands that hold the line on all four steps get a lift number built on the same logic that paid-media teams fought for against Meta and Google: a number that survives scrutiny because it was never designed to flatter anyone.