Chrome extension A/B testing: run experiments in MV3

Chrome extension A/B testing sounds like it should be easy, and then you try a website testing tool. A developer summed up the problem in a Stack Overflow question: the tool "is asking for a url", and "How am I supposed to do an A/B in a chrome extension given that there's no url for them." Extension UI lives in popups, side panels and content scripts, every code change waits for store review, and Manifest V3 forbids loading code from a server.

This guide shows a setup that works within those limits: a remote configuration file that only switches between variants already in your package, a stable assignment per install, an exposure event, and a plain statistical check at the end. It includes the exact Chrome Web Store policy text that allows it, a tested experiments.js module, a sample size table and the code to compare results.

Key takeaways

• The Chrome Web Store explicitly allows "Fetching a remote configuration file for A/B testing", as long as all the logic is in the extension package. Ship every variant in the build and let the config choose.
• Assign each install to a variant with a hash of a random install ID, store the result, and enroll only installs that started after the test began.
• Log an exposure event when a user actually sees the variant, and pick one primary metric before you start.
• To detect a lift from 40% to 45% activation, you need about 1,534 users per variant. Small changes on low conversion rates need thousands more.
• The store listing itself cannot be split-tested on the Chrome Web Store, unlike Google Play and the App Store.

What you can A/B test in a Chrome extension#

Almost anything that runs inside the extension can be tested, because you control the code:

• Onboarding: a welcome page with a checklist versus a single call to action.
• Popup and side panel copy: button labels, empty states, the order of features.
• Defaults: a setting on or off for new installs.
• Upgrade prompts: when a paywall appears, how the plan is described, trial length if your billing supports it.

Two things are harder. The first is the store listing. Google Play has store listing experiments and the App Store has product page optimization, but the Chrome Web Store has no built-in way to show two versions of a listing to different visitors. You can test the pages that send traffic to the listing, such as your landing page, and measure the listing with the store's own funnel; the guide to Chrome Web Store conversion rate shows how to read impressions, page views and installs by source.

The second is anything that needs new code. A variant that is not in the package cannot be turned on remotely, so every new idea still goes through a release and review.

What Chrome Web Store policy allows#

The rule that matters is in the Additional Requirements for Manifest V3. It says "the full functionality of an extension must be easily discernible from its submitted code", and that external resources "must not contain any logic." Then it lists what is still allowed, including:

Fetching a remote configuration file for A/B testing or determining enabled features, where all logic for the functionality is contained within the extension package

The same page lists the violations to avoid. Besides remote <script> tags and eval() of fetched strings, it names "Building an interpreter to run complex commands fetched from a remote source, even if those commands are fetched as data." That last line is the trap for ambitious experiment frameworks. A config that says "onboarding_checklist": { "active": true, "weights": [50, 50] } is data. A config that describes which elements to inject, in what order, with what behavior, starts to look like a program.

Firefox has the same principle. Mozilla's add-on policies say "Add-ons must be self-contained and not load remote code for execution." A config-driven design that is safe for the Chrome Web Store works on AMO and Edge Add-ons too.

The practical rule: the config chooses, the package decides. Every variant's code ships in the build, and the config only selects which one runs and for how many users.

Step 1: Give every install a stable ID#

Assignment needs an identifier that stays the same for one install and says nothing about the person. A random UUID created on first install is enough. Record the install time too, so you can enroll only installs that started after an experiment began:

// service-worker.js
chrome.runtime.onInstalled.addListener(async ({ reason }) => {
  if (reason !== 'install') return;
  await chrome.storage.local.set({ installId: crypto.randomUUID(), installedAt: Date.now() });
});

chrome.storage.local survives browser restarts and extension updates. The chrome.runtime.onInstalled guide covers the update and migration cases, such as creating the ID for existing users who installed before this code shipped.

Step 2: Assign variants from a remote config#

Host a small JSON file on your own server:

{
  "onboarding_checklist": { "active": true, "weights": [50, 50], "startedAt": "2026-10-20T00:00:00Z" },
  "popup_upgrade_copy": { "active": false }
}

Then add a module that reads it, caches it and assigns variants. The list of experiments and their variants lives in the code, so the config cannot add behavior:

// experiments.js: assign A/B variants from a remote config. Every variant's code ships in the package.
const CONFIG_URL = 'https://api.example.com/extension/experiments.json';
const CONFIG_MAX_AGE_MS = 6 * 60 * 60 * 1000;

// The only experiments and variants this version knows. The remote config can switch
// them on or off and set weights; it cannot add behavior.
export const EXPERIMENTS = {
  onboarding_checklist: ['control', 'checklist'],
  popup_upgrade_copy: ['control', 'benefit_first'],
};

async function loadConfig() {
  const { experimentConfig: cached } = await chrome.storage.local.get('experimentConfig');
  if (cached && Date.now() - cached.fetchedAt < CONFIG_MAX_AGE_MS) return cached.data;
  try {
    const res = await fetch(CONFIG_URL, { signal: AbortSignal.timeout(5000) });
    if (!res.ok) throw new Error(`HTTP ${res.status}`);
    const data = await res.json();
    await chrome.storage.local.set({ experimentConfig: { data, fetchedAt: Date.now() } });
    return data;
  } catch {
    return cached?.data ?? {}; // offline or server down: keep the last config, or run nothing
  }
}

// A stable 0-99 bucket per install and experiment.
async function bucket(experiment, installId) {
  const bytes = new TextEncoder().encode(`${experiment}:${installId}`);
  const digest = await crypto.subtle.digest('SHA-256', bytes);
  return new DataView(digest).getUint32(0) % 100;
}

export async function getVariant(experiment) {
  const variants = EXPERIMENTS[experiment];
  const config = (await loadConfig())[experiment];
  if (!variants || !config?.active) return 'control';

  const { installId, installedAt, assignments = {} } = await chrome.storage.local.get([
    'installId',
    'installedAt',
    'assignments',
  ]);
  if (!installId) return 'control';
  if (variants.includes(assignments[experiment])) return assignments[experiment];
  // Enroll only people who installed after the experiment started.
  if (config.startedAt && (installedAt ?? 0) < Date.parse(config.startedAt)) return 'control';

  const weights = config.weights ?? variants.map(() => 100 / variants.length);
  let point = await bucket(experiment, installId);
  let variant = variants[0];
  for (let i = 0; i < variants.length; i++) {
    if (point < weights[i]) {
      variant = variants[i];
      break;
    }
    point -= weights[i];
  }
  assignments[experiment] = variant;
  await chrome.storage.local.set({ assignments });
  return variant;
}

A few design choices are worth knowing:

• Hashing the experiment name with the install ID gives each experiment an independent split, so the users in the checklist group are not automatically the users who see the new upgrade copy.
• The stored assignment wins. If you later change the weights from 50/50 to 90/10, people already enrolled keep their variant, which keeps your results clean.
• Turning an experiment off is the kill switch. With "active": false, getVariant returns control for everyone as soon as the cached config expires, even for people previously assigned to the variant.
• Failure means control. No install ID, no network and no config all fall back to the current behavior.

We tested this module in Node.js with a mocked chrome.storage. Across 20,000 random install IDs a 50/50 config produced 9,997 control and 10,003 checklist assignments, and a 90/10 config produced 18,037 and 1,963. Repeated calls for the same install always returned the same variant, installs from before startedAt stayed in control, and switching active off moved a previously assigned install back to control.

The fetch call needs your server in host_permissions, or a server that answers with CORS headers for your extension's origin. Keep the config tiny and cacheable; the service worker may be woken many times a day.

Step 3: Render the variant and log exposure#

Use the variant where the UI decides what to show, and record an exposure event the first time the user actually sees it:

// welcome.js (the onboarding page)
import { getVariant } from './experiments.js';
import { track } from './analytics.js';

const variant = await getVariant('onboarding_checklist');
document.body.dataset.onboarding = variant; // both layouts are in welcome.html and welcome.css
track('experiment_exposure', { experiment: 'onboarding_checklist', variant });

Exposure matters because assignment is not the same as seeing. If you assign at install but only 70% of new users ever open the welcome page, analyzing everyone assigned dilutes the effect with people who never saw either version. Analyze exposed users, and check that exposure rates are similar in both groups; a big difference usually means a bug in one variant.

Send the event through whatever analytics you already use. The PostHog setup for Manifest V3 and the Google Analytics 4 guide for Manifest V3 both show how to send events from an extension without remote code. Feature-flag products work too, as long as their SDK is bundled in the package and the flags only choose between code paths you shipped.

Step 4: Measure one primary metric#

Decide before you start which number decides the test, and write it down. Pick it from the Chrome extension metrics you already review every week, so the test moves something you track anyway. Good primary metrics for extensions are:

Then add guardrails, numbers that must not get worse even if the primary metric improves. Uninstall rate is the most important one. Chrome's chrome.runtime.setUninstallURL lets you open a page when someone uninstalls, and the runtime API reference notes it can be used to "do analytics" with a URL of up to 1023 characters. Adding the assignment to that URL lets you count uninstalls per variant:

const { assignments = {} } = await chrome.storage.local.get('assignments');
const tags = Object.entries(assignments).map(([key, value]) => `${key}.${value}`).join(',');
await chrome.runtime.setUninstallURL(`https://example.com/uninstalled?exp=${encodeURIComponent(tags)}`);

Call it again whenever an assignment changes. For more on measuring retention and uninstalls without collecting more than you need, see how to measure extension retention and uninstall rate.

Store signals are the other guardrail. A variant that annoys people shows up in reviews and the rating before it shows up in a significance test, so watch new reviews during the test window.

How many users a Chrome extension A/B test needs#

Small extensions run out of users before they run out of ideas, so size the test before you start. The standard formula for comparing two proportions at 95% confidence and 80% power gives these numbers per variant:

Divide by your daily new installs to get a duration. With 200 new installs a day split 50/50, the first test needs about 16 days of enrollment, and a day 7 retention metric needs another 7 days after the last enrollment before the result is complete. If the numbers come out at months, test a bigger change; a new onboarding flow can move activation by ten points, a button color will not.

Here is the calculation, plus the comparison you run at the end, with no dependencies:

// ab-stats.mjs: sample size and a two-proportion z-test.
const Z_95 = 1.959964; // two-sided alpha = 0.05
const Z_80 = 0.841621; // power = 0.8

// Users needed in EACH variant to detect a change from p1 to p2.
export function sampleSizePerVariant(p1, p2) {
  const pBar = (p1 + p2) / 2;
  const a = Z_95 * Math.sqrt(2 * pBar * (1 - pBar));
  const b = Z_80 * Math.sqrt(p1 * (1 - p1) + p2 * (1 - p2));
  return Math.ceil((a + b) ** 2 / (p2 - p1) ** 2);
}

// Standard normal CDF (Abramowitz and Stegun 7.1.26).
function normalCdf(z) {
  const t = 1 / (1 + (0.3275911 * Math.abs(z)) / Math.SQRT2);
  const poly = t * (0.254829592 + t * (-0.284496736 + t * (1.421413741 + t * (-1.453152027 + t * 1.061405429))));
  const erf = 1 - poly * Math.exp(-(z * z) / 2);
  return z >= 0 ? (1 + erf) / 2 : (1 - erf) / 2;
}

export function compare(control, variant) {
  const p1 = control.converted / control.users;
  const p2 = variant.converted / variant.users;
  const pooled = (control.converted + variant.converted) / (control.users + variant.users);
  const z = (p2 - p1) / Math.sqrt(pooled * (1 - pooled) * (1 / control.users + 1 / variant.users));
  const se = Math.sqrt((p1 * (1 - p1)) / control.users + (p2 * (1 - p2)) / variant.users);
  return {
    lift: p2 - p1,
    ci95: [p2 - p1 - Z_95 * se, p2 - p1 + Z_95 * se],
    pValue: 2 * (1 - normalCdf(Math.abs(z))),
  };
}

We checked both functions against a Python calculation using the standard library's normal distribution; the sample sizes in the table above come from this code.

Read the results#

Two examples show how to read the output. Suppose the checklist onboarding finished with 701 activated out of 1,532 exposed, against 612 out of 1,540 in control. compare() reports a lift of 6.0 points, a 95% confidence interval from 2.5 to 9.5 points, and a p-value of 0.0008. That is a clear win.

Now suppose a smaller test ended at 610 out of 1,500 against 560 out of 1,500. The lift is 3.3 points, but the confidence interval runs from -0.2 to 6.8 points and the p-value is 0.06. The honest reading is "probably positive, not proven." You can ship it as a judgment call, but do not report it as a measured win.

Three habits keep results trustworthy:

• Do not stop early because the numbers look good. Checking every day and stopping at the first p-value under 0.05 inflates false positives. Run to the planned sample size.
• Watch for novelty. A new popup layout can get more clicks in week one simply because it is new. Retention metrics are less prone to this than click metrics.
• Keep releases out of the window. If you ship a new version halfway through, compare results before and after it; what to watch after an extension release lists the signals that move after an update.

Roll out the winner and clean up#

When a variant wins, set its weight to 100 in the config so every new install gets it immediately, then make it the default in the next release and delete the losing code path and the experiment entry. Old experiments left in the code become dead branches that confuse the next person who reads it, and reviewers who see unused variants have more to understand.

Keep a simple log for each test: hypothesis, primary metric, start and end dates, sample sizes, result and decision. After a few tests it tells you which kinds of changes actually move your numbers.

Store-level numbers close the loop. A winning onboarding variant should eventually show up as better weekly user growth and fewer uninstalls on the listing. Extenify records users, ratings, reviews and versions every day across the Chrome Web Store, Edge Add-ons and Firefox Add-ons, so you can check that an experiment's win holds up in your Chrome extension analytics a month later, next to competitors. Every plan starts with a 7-day free trial, no credit card required; see pricing.

Chrome extension A/B testing checklist#

• Every variant's code is in the package; the config only switches and weights.
• A random install ID and install time are stored on first install.
• Assignment is hashed per experiment and stored, and only new installs are enrolled.
• An exposure event fires when the variant is actually shown.
• One primary metric and at least one guardrail (uninstall rate) are written down before launch.
• The sample size and planned end date are calculated before launch.
• Your privacy policy and store privacy disclosures cover the usage events you collect.
• The losing variant is removed in the next release.

Frequently asked questions#

Can I A/B test a Chrome extension without publishing an update?#

Only between variants that are already in the published package. A remote config can turn them on, off or change their weights at any time. A new variant needs new code, which means a new version and a store review.

Is remote config allowed in Manifest V3?#

Yes. The Chrome Web Store's Manifest V3 requirements list "Fetching a remote configuration file for A/B testing or determining enabled features" as allowed, "where all logic for the functionality is contained within the extension package." Fetching and running code is not allowed.

Can I A/B test my Chrome Web Store listing?#

Not inside the store. The Chrome Web Store has no listing experiments feature like Google Play's store listing experiments or Apple's product page optimization. You can test your own landing page and the traffic you send to the listing, and compare listing changes over time with the store's analytics.

Which tool should I use for A/B testing in an extension?#

A small module like the one above is enough for a few experiments a quarter. If you already use a product analytics or feature-flag service, its SDK can work in an extension as long as it is bundled and only returns flag values. There is also an open-source package, ext-ab-testing, built for Manifest V3 extensions.

How long should an A/B test run in a Chrome extension?#

Long enough to reach the sample size you calculated, and at least one full week so that weekday and weekend users are both included. Add the length of your metric window, for example 7 days for day 7 retention, after enrollment ends.

Do I need user consent to run experiments?#

Assigning a variant needs no personal data, but measuring outcomes means collecting usage events, which must be covered by your privacy policy and the store's privacy disclosures. In Firefox, also declare the collection in the manifest; the guide to Firefox data collection consent explains how.

Summary#

Chrome extension A/B testing works when you separate choosing from building. Ship every variant in the package, fetch a small remote config that only switches and weights them, which the Chrome Web Store explicitly allows, and assign each new install with a stored, hashed bucket. Log exposure, pick one primary metric and an uninstall guardrail, and calculate the sample size before you start, because small extensions often need weeks to detect anything smaller than a large change. Then run to the planned end, read the confidence interval rather than just the p-value, ship the winner and delete the loser.

Image credits#

• Cover photo: Two paths covered in snow by Nathan Wrigley on the WordPress Photo Directory, CC0.