Demand Experimentation Maturity Snapshot 2026

The honest answer, 2026 edition

Most “A/B tests” in B2B demand gen are launches with a dashboard. Someone ships a landing-page change, the dashboard goes green, a Slack screenshot circulates, and the variant goes live. That is not an experiment; it is a measurement ritual attached to a foregone conclusion.

Real maturity in 2026 is a stack, not a dashboard. Five things, in this order, none optional:

1. A pre-registered hypothesis with a falsifiable claim.
2. A fixed sample size computed from a minimum detectable effect (MDE) you actually believe in.
3. A single primary metric, with guardrails listed.
4. A documented ship/kill rule committed to before exposure.
5. A holdout running long enough to catch effects the in-test metrics can’t see.

If any of those is missing, the result is closer to astrology than analytics — and even with them all, the math is unforgiving. The industry has known this for decades. What is new in 2026 is that the evidence base has tightened around the failure modes.

The 2026 evidence shake-up: most “trustworthy” A/B patterns don’t replicate

Two 2026 publications deserve to change how B2B teams size tests and what they call a “win.”

Kohavi, Linowski, Vermeer, Andreev, Dodin, and Furuseth — Trustworthy A/B Patterns and the Winner’s Curse: Lessons from Eight Large-Scale Replications — published at KDD 2026 (ACM DL). They ran eight pre-registered replications of widely cited UX patterns (rounded buttons, page performance, coupon-code field, sticky CTA) at a median of 2.4M users per experiment, designed at 80% power for MDEs of 0.3%–2.2%. The result: only two of eight showed a statistically significant effect in the direction claimed; one was significant in the opposite direction; effects across the board were substantially smaller than the original claims. The authors attribute the gap largely to the original studies being underpowered (often below 50%), which systematically inflates observed effect size — the winner’s curse the title calls out.

A companion paper — Kohavi et al., Power Analysis is Essential: High-Powered Tests Suggest Minimal to No Effect of Rounded Shapes on Click-Through Rates, Econ Journal Watch 23(1), March 2026 — is the academic write-up of the rounded-corners replication. Biswas, Abell & Chacko’s 2024 Journal of Consumer Research paper had claimed curved digital marketing designs lift click-through. Underpowered replication, no effect. The lesson generalizes: most B2B “design lift” claims on landing pages are made from underpowered tests and overstate the true effect.

Practical consequence for B2B demand teams:

• Don’t import a “20% lift on testimonial above fold” headline into your roadmap. Treat every prior claim as a hypothesis to pre-register and re-test.
• For marketing-copy and design tests, an MDE of 5%–10% relative is no longer defensible as your lower bound. Kohavi’s own LinkedIn post “Why 5% should be the upper bound of your MDE” makes the same point.
• Replicate before you scale. The KDD 2026 paper explicitly recommends replication as the cure for the winner’s curse.

Decision rules and pre-registration — what they look like in 2026

A working template, adapted from Statsig PM Paul Morrill’s What’s the point of hypothesizing? (July 2026, Statsig blog):

We believe that [change], for [target users], will [direction] the [success metric] from baseline X% to at least Y%, based on [evidence]. We will measure over [timeframe] with an MDE of Z%, and monitor [guardrails] to ensure no harm. Decision: ship if p<0.05 and effect ≥ MDE; kill if p>0.20 at planned sample size; otherwise extend.

Three things that make this a real rule and not a paragraph of prose:

• The decision is committed before exposure.
• MDE is in the doc, not inferred after the fact.
• There is an explicit kill condition for the no-effect case.

Statsig’s Hypothesis Advisor (described in the same post and in their January 2026 AI features release, Statsig blog) auto-flags missing fields — change, users, metric, MDE, guardrail, decision rule. Eppo’s Drew Harry published a parallel case for Make Decisions Before Experimenting on the Eppo blog, January 2025, and Microsoft’s ExP group’s “Three Key Checklists and Remedies for Trustworthy Analysis of Online Controlled Experiments at Scale” (Fabijan et al., ICSE-SEIP 2019) is still the engineering checklist most teams should be running against.

A pre-registration also means the decision survives the test. Not the result — the decision.

Sample size, power, and what to actually put in the calculator

Three levers, all with trade-offs, all pulled deliberately:

• Baseline conversion rate. Use the trailing 4–8 weeks of traffic exposed to the surface you’re testing, not the all-product number. Statsig’s Akhil Prakash, Running faster tests: The basics (Part 1) (July 2026), makes the same point: divide by users exposed per day, not total DAU.
• MDE. Bigger MDE → smaller sample → coarser answer. The Kohavi 2026 replications make the practical floor obvious: 5% relative is a sane default for product tests, 10–20% for marketing copy and design, and anything finer requires either huge traffic or variance reduction.
• Error rates. α=0.05, β=0.20 (80% power) remains the industry default, dating back to Kohavi’s “Seven rules of thumb for web site experimenters” (KDD 2014, ACM). Evan Miller’s sample size calculator is still the cleanest interactive tool for binomial tests. Plugged in August 2026, baseline 10.2% → MDE 13.2% (10.2% absolute = 30% relative), it returns 2,545 per variation. That is a 4–6 week test on a mid-traffic B2B SaaS surface and an afternoon for Booking.com.

If your calculator says the test needs more than ~6 weeks, you have a design problem, not a patience problem. Fix the targeting, the metric, or the MDE — in that order.

Peeking, sequential testing, and Bayesian engines — when each actually pays off in B2B

The peeking problem hasn’t moved. Evan Miller’s 2010 simulation still holds: peek after every observation on a coin flip and your actual false-positive rate climbs from a nominal 5% to 26.1%, more than five times what the dashboard claims (Evan Miller, How Not To Run an A/B Test). The cure is one of three:

• Fix the sample size in advance and don’t peek. Still the default for high-stakes B2B product tests.
• Use a sequential engine. Always-valid inference — Johari, Pekelis, McQueen and others, Peeking at A/B Tests KDD 2017 and Always Valid Inference: Continuous Monitoring of A/B Tests Operations Research 70(3), 2022 (ACM). The gambler’s-ruin approximation Miller published in 2015 gives you a usable threshold on the cheap: at α=0.05 (two-sided), stop when |T−C| ≥ 2.25·√N or when T+C ≥ N. Microsoft ExP’s On Post-selection Inference in A/B Testing (Deng, Li, Lu, Ramamurthy, KDD 2021) is the deeper treatment.
• Use a Bayesian engine. GrowthBook’s Bayesian default (GrowthBook stats docs) returns a “Chance to Win” probability and a posterior distribution; Eppo/Datadog (Eppo statistics blog index) does the same with optional proper priors. A non-informative prior doesn’t make Bayesian peeking safe — Tyler’s Buffington’s The Illusion of Safety (Eppo blog, April 2025) and Demetri Pananos’s Bayes vs. Frequentism Can Be a Big Deal (Eppo blog, March 2025) lay out why.

Practical rule for B2B in 2026: fixed-sample for anything quarterly-board-reportable; sequential or Bayesian for copy/design iterations where you want to ship winners fast. Both sequential and Bayesian let you peek without paying the peeking tax, but neither lets you pretend you can ship losers.

Variance reduction: CUPED and what it actually buys

CUPED (Deng, Xu, Kohavi, Tang, Improving the Sensitivity of Online Controlled Experiments by Utilizing Pre-Experiment Data, WSDM 2013, ACM) is now table stakes. Microsoft has published that for one product team CUPED is “akin to adding 20% more traffic” for a majority of metrics (Microsoft ExP, Deep Dive into Variance Reduction). Netflix reported ~40% variance reduction on key engagement metrics in a 2016 KDD paper (Xie & Aurisset). GrowthBook ships CUPED plus post-stratification as CUPEDps (GrowthBook docs); Statsig and Eppo/Datadog ship the same underlying regression adjustment on top of pre-exposure covariates.

CUPED is not magic. It tightens the variance on the same true effect; it does not let you detect effects smaller than the raw data supports. Two common misuses: using a covariate that is itself affected by the experiment (post-exposure leakage), and assuming CUPED lets you drop MDE — it doesn’t, it just shrinks the noise around the same MDE.

B2B-specific patterns: geo tests, holdouts, and the interference problem

Three patterns that look different in B2B than in B2C.

Geo tests for marketing. When you can’t randomize by user — paid media, brand campaigns, offline channels — switch to geo. Eppo/Datadog’s Geolift (Eppo blog) compares synthetic-control and matched-market methods; Microsoft’s switchback / time-of-day designs are useful when geo randomization is too coarse. The cleanest reference is Eppo’s Eppo Geolift vs. CausalImpact and Matched Markets benchmark (April 2025).

Holdouts. A 5%–10% always-on holdout lets you see effects — good and bad — on metrics the in-test window couldn’t measure: 30-day retention, 90-day pipeline, multi-touch attribution. Eppo’s Giorgio Martini wrote the clearest 2023 practitioner case in Holdouts: Measuring Experiment Impact Accurately (Eppo blog). Microsoft’s Pitfalls of Long-Term Online Controlled Experiments (Dmitriev, Frasca, Gupta, Kohavi, Vaz, IEEE Big Data 2016) is still the warning shot.

Interference / network effects. In B2B, two users from the same account colliding in different arms breaks SUTVA. Cluster at the account level, not the user level. Eppo’s Introducing Switchback Testing on Eppo (March 2025) and GrowthBook’s cluster-experiments documentation (GrowthBook docs) cover the design patterns; the canonical interference reading is still Saveski et al.’s work on network-aware experiments at LinkedIn and Airbnb’s Overlapping Experiment Infrastructure (Tang, Chen, et al., KDD 2010).

Low traffic. B2B SaaS funnels convert at 1%–5%, with maybe a few hundred MQLs per month on a single page. Two options: lower the MDE bar (and accept larger relative errors), or move to a Bayesian engine with a proper prior. The third option — running the test for six months — is the worst option and the most common.

The 2026 vendor landscape

The platform market consolidated hard in 2025–2026. What each vendor actually sells:

• Statsig (Bellevue, WA; now part of Amplitude per their May 2026 announcement, Statsig + Amplitude: Phase 1). Unified experimentation + feature flags + analytics + warehouse-native. 2026 additions: AI-assisted hypothesis review (Jan 2026), a Knowledge Graph tying gates, experiments, and metrics (Jan 2026), and an MCP server for headless agent workflows (Statsig MCP, July 2026). Named customers include Notion, OpenAI, Brex.
• Eppo → Datadog Experiments. Acquired by Datadog in May 2025 (Eppo is now part of Datadog). Warehouse-native, fixed + sequential + Bayesian in one engine, CUPED, contextual bandits for personalization, Geolift for marketing incrementality, switchback for marketplaces. Coinbase reports a 40% reduction in experiment analysis time (Coinbase case study).
• GrowthBook (open source, Apache 2.0; cloud option). Stats engine is public (GitHub). Bayesian by default, optional frequentist; CUPEDps; bandits; cluster experiments; holdouts; sticky bucketing.
• LaunchDarkly. Repositioned around CodeControl (runtime code release) and AgentControl (LLM-agent behavior in production, LaunchDarkly AgentControl). A/B and multi-armed bandit experiments sit on top of the same flag infra. Cited customers include Hireology, Relay Network, Poka, Dior, Paramount.
• Optimizely (Stockholm / New York). Pivoted hard into Opal AI agents — directory of pre-built agents including a Variation Development agent, a GA4 Web Traffic Report agent, and a Page Builder agent (Optimizely agents directory). The original Stats Engine white paper (Pekelis/Walsh/Johari, the SPRT basis for sequential testing) is still hosted on the site.
• AB Tasty and Kameleoon remain relevant for non-engineering marketing/UX teams; they have not broken into the agent narrative in 2026.

The honest way to choose: warehouse-native vs. integrated-product, frequentist vs. Bayesian default, agentic workflow support, governance and SRM checks. UI is a tiebreaker.

Agentic experimentation: where AI helps, where it creates new failure modes

The 2026 storyline is that every credible platform now ships some form of AI-in-the-loop. Microsoft’s ExP group brands the mission: “Empower every AI builder at Microsoft to accelerate innovation through trusted, low-friction experimentation” (Microsoft Research ExP). The vendor lines:

• Statsig MCP Server. Lets an AI agent configure and read out experiments through the Model Context Protocol. The post emphasizes that MCP enforces the same governance and review policies as the Console UI — “the agent went through governance instead of routing around it” (Statsig blog, July 2026).
• Optimizely’s Opal agents. Pre-built agents for variation generation, content, and analytics (optimizely.com/agents).
• LaunchDarkly’s AgentControl. Self-healing prompt/config routing with offline evals before promotion, online evals after, and multi-armed bandit optimization on top (LaunchDarkly AgentControl).
• Microsoft’s own ExP. Embeds AI into experiment creation, monitoring, and interpretation.

The academic literature caught up. Lu et al., Agent A/B: Automated and Scalable A/B Testing on Live Websites with Interactive LLM Agents, CHI Extended Abstracts 2026, describes an LLM agent that proposes, runs, and reads out web experiments.

Where AI genuinely helps: drafting hypothesis templates, flagging missing decision-rule fields, reading out segment results, summarizing long-running tests. Where it creates new failure modes: an LLM can generate 50 candidate copy variants in an afternoon and the team can ship them sequentially with no peeking protection. “Agentic” becomes “uncontrolled.” The peeking problem is now an agent problem.

Audit checklist (a 60-minute working session)

• Hypothesis. Written and falsifiable. (If “we want to try X,” stop.)
• Decision rule. Ship/kill threshold written before exposure.
• Sample size and MDE. Calculator output vs. actual runtime. Are they consistent?
• Primary metric. Exactly one. Guardrails listed.
• SRM check. Did the randomization ratio come out as expected? Fabijan et al., Diagnosing Sample Ratio Mismatch in Online Controlled Experiments (KDD 2019), is the rules-of-thumb reference. Eppo’s What to Do When You Encounter Sample Ratio Mismatch (March 2025) is the 2020s practitioner rewrite.
• Peeking discipline. Sequential or Bayesian engine, or fixed-sample only? If fixed-sample only, was there any premature stop?
• Segmentation. Slice by tenure, channel, and account. Any Simpson’s-paradox reversal?
• Novelty window. Did the test run long enough to wash out day-1 novelty? Microsoft’s Dmitriev et al., Pitfalls of Long-Term Online Controlled Experiments (IEEE Big Data 2016) says plan for ~2× the minimum runtime on B2B surfaces.
• Holdout. Is a long-term holdout running? Are you reading it?
• Replication. Has the result been replicated at least once before broad rollout? Kohavi’s 2026 KDD paper argues this is the single most underused discipline.

A “yes” to the first five and at least a partial yes to the next five puts you at Level 4. Replication is what separates Level 4 from Level 5.

The bottom line

In 2026 the tooling has caught up. Sequential engines, Bayesian defaults, CUPEDps, switchback, geo tests, bandit routing, MCP-driven agentic workflows — all shipped, all credible. The discipline hasn’t moved: pre-register, fix the sample size, commit to a decision rule, hold out, replicate. The Kohavi 2026 replications are the corrective: most published A/B lift claims in B2B marketing are exaggerated by an order of magnitude because the original tests were underpowered. The way to be a mature experimentation team in 2026 is not to buy more tools. It is to pre-register, to size honestly, and to ship the loser as often as the winner.