Back to Blog Homepage
    Deploy Blog
    Nathan Chick

    Native A/B Testing Arrives on Shopify: What You Can (and Can't) Test Without an App

    Shopify added native A/B testing via Rollouts in June 2026, letting you split-test themes, checkout and customer accounts from admin. Here's what it replaces, and where Horizon fits in.

    Native A/B Testing Arrives on Shopify: What You Can (and Can't) Test Without an App

    A quick correction to how this one's often reported: Horizon isn't a new Spring '26 announcement. Shopify's block-based theme has been the default for new stores since it launched in May 2025, and it's had a steady stream of updates since, including the 250+ enhancements called out in the Winter '26 Editions. What Spring '26 actually added is native A/B testing, through a feature called Rollouts, and that's genuinely new, and genuinely useful.

    What's new: native A/B testing via Rollouts

    As of 5 June 2026, Rollouts lets you run an A/B test on a whole theme, or on your checkout and customer-account configuration, directly from the Shopify admin. No third-party testing app required. At the theme level, that means you can split traffic between, say, Dawn and Horizon and measure the difference in real conversion data rather than switching cold and hoping. At the checkout/account level, it extends to configuration changes that previously needed a dedicated experimentation tool to test safely. Worth checking before you plan around it: full checkout and customer-account testing sits above entry-level plans, so confirm it's actually included on your current plan rather than assuming parity with theme-level testing.

    This is a meaningful gap closed. Plenty of merchants have historically avoided testing theme or checkout changes altogether because the tooling friction (and cost) of a third-party CRO platform wasn't worth it for an occasional test. Native, built-in testing removes that excuse.

    Where Horizon actually fits

    Horizon is built on "theme blocks", a newer architecture that supports up to eight levels of nested, reusable blocks across the whole store, compared with Dawn's two levels of fixed section blocks. Practically, that means far more layout flexibility without custom code, and (on Shopify Plus specifically) support for Combined Listings, which Dawn doesn't offer.

    Dawn hasn't been deprecated: it's still actively maintained (version 16.0.0 shipped in August 2026) and existing stores aren't required to move. Horizon is simply the default for new stores and the theme Shopify is clearly investing in longer-term.

    Native testing vs. a dedicated CRO tool

    For a lot of common experiments (does this theme convert better than that one, does this checkout tweak help or hurt), native Rollouts testing is genuinely sufficient, and it's free where a third-party platform wasn't. Where a dedicated CRO tool still earns its cost: more advanced statistical methods (sequential testing, Bayesian approaches), multi-page funnel tests that span beyond a single theme/checkout swap, and integration with a broader experimentation programme that spans email, ads, and on-site tests together.

    Should you move to Horizon now?

    If you're already comfortable on Dawn or a mature custom/paid theme and it's performing well, there's no forced deadline to move. If you're evaluating a rebuild anyway, Horizon's nested-block architecture is worth strong consideration, particularly if your merchandising needs more layout flexibility than Dawn's fixed sections allow. Before moving, check: whether any custom theme code or app embeds you rely on have confirmed Horizon compatibility, and whether the specific Plus features you use (like Combined Listings) change your calculus either way.

    This question (theme-based flexibility vs. building something more bespoke) comes up a lot in our client conversations; if you want the fuller version of that argument, see our piece on Hydrogen vs. theme-based Shopify.


    Weighing up a Horizon migration, or want help setting up your first native A/B test properly? Talk to Deploy: we build on Shopify alongside Magento and can help you avoid the common experimentation mistakes (not running tests long enough, testing too many variables at once).

    Need Help with Your eCommerce Project?

    Get expert advice from our Magento specialists and marketing professionals.