GEO-Based Holdout Tests for Shopify Brands

Every paid media dashboard overclaims revenue, and it does so in a single, predictable direction: too high. Meta, Google, and the multi-touch attribution layers built on top of them measure credit, not causation. They answer a different question than the one brands actually need answered. Meta's attribution window, a seven-day click, a one-day engage-through, and a one-day view, catches customers who were already going to buy and hands the campaign credit for a sale it did not create. None of these models asks the only question that matters: would this customer have purchased without ever seeing the ad?

Shopiator's independent review states the scale of the distortion: a campaign reporting a high ROAS in Ads Manager can be running at a small fraction of that in incremental ROAS once you strip out the brand search clicks and warm retargeting hits that were converting on their own. A typical Meta prospecting campaign shows reported revenue well above what it actually generated, because the majority of the sales it claims credit for were going to happen regardless of whether the ad ran. Running Meta, Google, Klaviyo, and an on-site chatbot at the same time, each logging its own credit for the same purchase, inflates the combined revenue these tools report beyond what the store actually sold.

The eBay test remains one of the cleanest demonstrations of this on record. When eBay turned off all non-branded paid search across a set of US test markets, the resulting sales lift measured at exactly zero percent. The dollars attributed to those campaigns did not represent incremental revenue at all; they represented searches eBay would have captured organically either way. Meta's own reporting changes make a related point from the other direction. In March 2026, Meta redefined what counts as a click-through, removing engagement actions like likes, shares, and saves from the calculation, and reported ROAS for many advertisers fell by roughly thirteen percent overnight. The underlying sales did not move. Only the counting did.

Why geography works as the experimental unit

A geo-holdout test measures what advertising actually caused, not what a platform's model decided to credit it for. The method solves a problem that attribution can never solve on its own terms: you cannot show a given customer an ad and withhold the same ad from that same customer at the same moment, so there is no way to directly observe what that person would have done otherwise. Attribution models paper over that gap with assumptions about credit. A geo-holdout sidesteps it by changing the unit of analysis from the individual customer to the market.

Pausing spend in Denver and Kansas City while keeping it running in markets with comparable buying patterns turns the paused cities into a live control group for the entire marketing mix, not just one channel. Whatever would have happened in Denver without the pause gets approximated by what actually happened in the markets where spend continued. That is the whole logic: geography lets you build the counterfactual that a single customer can never provide.

The comparison to other measurement methods sharpens why this matters. An A/B test checks which creative performs better within an audience that is already being advertised to, which tells a brand how to run ads, not whether running them does anything. Platform lift studies, including Meta Conversion Lift and Google Ads Conversion Lift, randomize at the user level, but they do it using the platform's own conversion tracking. The platform is grading its own homework. A geo-holdout randomizes at the market level and reads its results from revenue data sitting outside any platform's walls, so it picks up effects a platform-run study structurally cannot see: if pausing Meta spend shifts how people search organically, or changes email conversion rates in those same markets, the holdout captures all of it as part of the real effect of turning the channel off.

The most serious objection to this method is also the most obvious one: markets differ. Population, seasonality, local competition, and customer mix all vary by region, and if the treatment and control markets aren't genuinely comparable, the test produces a number that looks precise and means nothing. Adlibrary's 2026 holdout design guide states the stakes directly: the methodology is only as good as the comparability of the markets chosen. It is a design requirement, and it is the first thing the next section works through in detail.

None of this requires new infrastructure. A Shopify order file already has a shipping address on every row, which is everything needed to read a holdout's result. No iOS privacy update, no cookie deprecation, and no platform policy change can take that data away, since it was never built on a platform's pixel to begin with.

The four design decisions that determine whether a test produces a trustworthy result

A geo-holdout test comes down to four decisions made in sequence: the question, the markets, the duration, and the method used to read the result. Get any one of them wrong and the test still produces a number, but not one a brand can act on.

The first decision is the question, and it has to be written down in a single sentence with a number attached to it. "Does Meta work?" is not a testable question because nothing about it specifies what would count as an answer. "Does branded search spend drive any incremental revenue, or does it harvest sales we would get anyway?" is testable, because it names one channel and one measurable outcome. Test one channel at a time. Running multiple changes at once makes it impossible to tell which one caused the difference in the data. Lydec's geo-holdout guide recommends branded search as the best channel to test first for most DTC brands, since it is the one most likely to be claiming credit for demand that would have converted anyway. Retargeting is a strong second choice for the same reason: of all channel categories, it tends to carry the widest gap between what a dashboard reports and what a holdout actually measures.

The second decision is market selection, and it has to be done with data rather than hunches about which cities feel similar. Pull around a year of weekly revenue by region from Shopify order history, then pair up markets whose revenue curves have been moving together over that stretch. The bar for a usable pair is a correlation above 0.8 on that weekly revenue series; anything below that shouldn't be treated as a matched pair. Holding out somewhere between twenty and thirty percent of revenue-weighted markets tends to be the workable range: go below fifteen percent and the signal gets lost in noise, go above forty percent and the media budget lost to the experiment starts to cost more than the test is worth. Leave the one or two largest markets, typically New York and Los Angeles for US brands, out of the holdout pool entirely, since nothing else in the dataset moves similarly enough to predict them. Brands running this in Australia face a different map but the same logic: eCommerce Circle's Shopify guide points to state-level holdouts as the cleanest unit there, with Western Australia and South Australia together usually making up eighteen to twenty-two percent of national e-commerce revenue, a workable size for a holdout group, while smaller spenders can build the same structure out of postcode clusters instead of states. If no single pair clears the 0.8 bar, group several smaller markets together until the combined curve tracks its counterpart. Miami and Minneapolis, pulling in opposite directions seasonally, make a bad pair. Denver and Salt Lake City, rising and falling together across the year, make a good one.

The third decision is runtime, and four to eight weeks is the range that holds up. That span covers at least one full purchase-consideration cycle with room to spare, because the effect of turning ad spend off doesn't vanish the instant the campaign goes dark. It decays over days and weeks. Set the end date before the test starts and write it down, because the single most common way a real experiment turns back into a guess is ending it early once the numbers start looking favorable. Nothing else about the business should move during that window either. No promotions that run in test markets but not control markets, no site redesign, no new channel launch, no creative overhaul in one region but not another. Any change that happens has to happen everywhere at once, or the comparison between holdout and control stops meaning anything.

The fourth decision is how results get read: Shopify order data, not the platform dashboard. For each matched pair of markets, compute the revenue ratio between them across the twelve months before the test started, apply that ratio to the control market's actual revenue during the test, and that gives a projection of what the holdout market would have made without the pause. Sum those projections across every pair, compare the total to what the holdout markets actually generated, and the gap between projected and actual revenue is the effect of the channel. Divide that gap by the amount of spend paused in those markets to get incremental ROAS. Two sanity checks catch most false reads before they do damage. First, plot each holdout market against its projection week by week: a genuine effect appears within the first two weeks and holds, while a gap that only appears in week five, or flips sign from one week to the next, is noise rather than signal. Second, run the same math on two control markets where no effect should exist. If the placebo produces a gap anyway, the market pairing is off and needs to be rebuilt before trusting anything else the test produced.

Lydec's guide notes that holdout-measured returns commonly run thirty to sixty percent of what the platform dashboard claims. That is not a reason to distrust the holdout method. The gap between the dashboard number and the holdout number is the finding the test exists to produce.

Spend thresholds and the tools that match each budget level

The right tool for this kind of testing depends on monthly spend, not on preference, and the threshold for a full geo-holdout is lower than most brands assume. Running a clean geo-holdout across channels calls for roughly $20,000 a month in spend to avoid results too noisy to trust; below that level, the platform's own built-in tools are the more sensible starting point. The simple on/off test, running ads for a stretch and then stopping to see what changes, should be ruled out regardless of budget. Seasonality, ad fatigue, and ordinary organic momentum all bleed into that kind of comparison, so it works only as a rough sanity check before a budget conversation.

Google Ads Conversion Lift sits inside Google Ads at no added cost, and it's a reasonable place to start for any advertiser already spending there. It needs a minimum campaign budget and enough observed conversions to produce a usable sample, and it measures lift from Google exposure alone, so it won't capture what happens to sales through other channels. It is a single-channel read, useful for deciding whether a channel deserves the investment of a full geo-holdout.

A manual, spreadsheet-based geo-holdout is the right starting point for brands still deciding whether platform spend is worth the money. It costs only the internal time it takes to build. It carries real risk if run carelessly, though: there's no built-in statistical significance testing, so the placebo check and the week-by-week plot described in the design section aren't optional extras here, they're what keeps a manual test from producing a confident, wrong answer. Jordan Glickman's holdout framework makes the underlying point well: a geo-holdout is the most accessible incrementality test available to a brand without a data science team, since it needs no platform configuration beyond excluding certain zip codes or DMAs from campaign targeting.

Brands running incrementality testing as an ongoing practice rather than a one-time project tend to outgrow the spreadsheet and move to purpose-built platforms. Haus was built primarily for geo-experimentation and has since added Causal Attribution and Causal MMM as native products, turning it into a broader measurement platform; its feature set is narrower than a full attribution suite, its methodology is purpose-built for this kind of test, and it does not replace day-to-day attribution reporting. Recast fits brands that want marketing mix modeling paired with incrementality testing and want to rely less on pixel-level tracking; MMM output is only as reliable as the historical spend data behind it, so it needs real history across channels to work well. Prescient AI suits brands that want budget allocation recommendations built on incrementality data rather than a test result alone; its predictive layer sits on top of an MMM core and adds a degree of black-box reasoning, though that core model can be calibrated using holdout data. All three price by custom quote rather than a published rate card.

Synthetic control methods address the hardest part of market selection directly: finding a single market that cleanly matches another. Instead of pairing one holdout region with one control region, a synthetic control builds a weighted composite out of many similar regions, which cuts down on noise and false positives substantially. The technique traces back to Abadie and Gardeazabal's 2003 work and has been used in marketing contexts since at least 2021, and it matters most for brands whose geography doesn't offer obvious, clean pairs to work with. Google has previewed Meridian GeoX, a sign that this approach is moving from academic method toward standard practice in marketing measurement.