{"id":1053,"date":"2026-05-11T17:00:00","date_gmt":"2026-05-11T22:00:00","guid":{"rendered":"https:\/\/tolinku.com\/blog\/?p=1053"},"modified":"2026-03-07T03:34:28","modified_gmt":"2026-03-07T08:34:28","slug":"ab-testing-sample-size","status":"publish","type":"post","link":"https:\/\/tolinku.com\/blog\/ab-testing-sample-size\/","title":{"rendered":"A\/B Testing Sample Size Calculator for Deep Links"},"content":{"rendered":"\n

Running an A\/B test without calculating sample size first is like measuring with a broken ruler. You'll get a number, but it won't mean anything. Most deep link A\/B tests require more traffic than teams expect, and stopping early leads to false conclusions. This guide explains how to calculate the right sample size, what affects it, and how to work around low-traffic situations.<\/p>\n\n\n\n

For understanding statistical significance, see Statistical Significance for A\/B Tests: What It Means<\/a>. For measuring A\/B test results, see Measuring A\/B Test Results for Deep Link Campaigns<\/a>.<\/p>\n\n\n\n

\"Tolinku\nThe A\/B tests list page showing test names, status, types, and variant counts.<\/em><\/p>\n\n\n\n

The Formula<\/h2>\n\n\n\n

Standard Sample Size Calculation<\/h3>\n\n\n\n

For a two-proportion z-test (comparing conversion rates between two variants):<\/p>\n\n\n\n

function calculateSampleSize(baselineRate, minimumDetectableEffect, significance, power) {\n  \/\/ Default: 95% significance (alpha = 0.05), 80% power (beta = 0.20)\n  const alpha = significance || 0.05;\n  const beta = 1 - (power || 0.80);\n\n  \/\/ Z-scores\n  const zAlpha = getZScore(1 - alpha \/ 2); \/\/ 1.96 for 95%\n  const zBeta = getZScore(1 - beta);       \/\/ 0.84 for 80%\n\n  const p1 = baselineRate;\n  const p2 = baselineRate * (1 + minimumDetectableEffect);\n  const pAvg = (p1 + p2) \/ 2;\n\n  const numerator = Math.pow(\n    zAlpha * Math.sqrt(2 * pAvg * (1 - pAvg)) +\n    zBeta * Math.sqrt(p1 * (1 - p1) + p2 * (1 - p2)),\n    2\n  );\n  const denominator = Math.pow(p2 - p1, 2);\n\n  return Math.ceil(numerator \/ denominator);\n}\n\nfunction getZScore(percentile) {\n  \/\/ Approximation using the inverse error function\n  if (percentile === 0.975) return 1.96;\n  if (percentile === 0.80) return 0.842;\n  if (percentile === 0.90) return 1.282;\n  if (percentile === 0.95) return 1.645;\n  if (percentile === 0.995) return 2.576;\n  \/\/ For other values, use a lookup table or library\n  return 1.96;\n}\n<\/code><\/pre>\n\n\n\n
What the Variables Mean<\/h3>\n\n\n\n
\n\n\n\n\n\n\n
Variable<\/th>\nWhat It Is<\/th>\nTypical Value<\/th>\n<\/tr>\n<\/thead>\n
Baseline rate<\/td>\nYour current conversion rate<\/td>\nVaries (e.g., 5% CTR)<\/td>\n<\/tr>\n
Minimum detectable effect (MDE)<\/td>\nThe smallest improvement worth detecting<\/td>\n10-20% relative lift<\/td>\n<\/tr>\n
Significance level (alpha)<\/td>\nProbability of a false positive<\/td>\n0.05 (95% confidence)<\/td>\n<\/tr>\n
Power (1 – beta)<\/td>\nProbability of detecting a real effect<\/td>\n0.80 (80% power)<\/td>\n<\/tr>\n<\/tbody><\/table><\/figure>\n\n\n\n
Sample Size Reference Table<\/h2>\n\n\n\n
Per Variant (Not Total)<\/h3>\n\n\n\n
These are samples needed per variant<\/strong>, not total. For a 2-variant test, double the number.<\/p>\n\n\n\n
\n\n\n\n\n\n\n\n\n\n
Baseline Rate<\/th>\n5% MDE<\/th>\n10% MDE<\/th>\n15% MDE<\/th>\n20% MDE<\/th>\n30% MDE<\/th>\n<\/tr>\n<\/thead>\n
1%<\/td>\n3.1M<\/td>\n780K<\/td>\n347K<\/td>\n196K<\/td>\n87K<\/td>\n<\/tr>\n
2%<\/td>\n1.5M<\/td>\n383K<\/td>\n171K<\/td>\n96K<\/td>\n43K<\/td>\n<\/tr>\n
5%<\/td>\n592K<\/td>\n149K<\/td>\n66K<\/td>\n38K<\/td>\n17K<\/td>\n<\/tr>\n
10%<\/td>\n281K<\/td>\n71K<\/td>\n32K<\/td>\n18K<\/td>\n8K<\/td>\n<\/tr>\n
20%<\/td>\n125K<\/td>\n32K<\/td>\n14K<\/td>\n8K<\/td>\n3.6K<\/td>\n<\/tr>\n
30%<\/td>\n73K<\/td>\n19K<\/td>\n8K<\/td>\n5K<\/td>\n2.2K<\/td>\n<\/tr>\n
50%<\/td>\n31K<\/td>\n8K<\/td>\n3.6K<\/td>\n2K<\/td>\n900<\/td>\n<\/tr>\n<\/tbody><\/table><\/figure>\n\n\n\n
Key takeaway: lower baseline conversion rates require dramatically more traffic.<\/p>\n\n\n\n
Deep Link Scenario Examples<\/h3>\n\n\n\n
\n\n\n\n\n\n\n\n
Scenario<\/th>\nBaseline Rate<\/th>\nMDE<\/th>\nSample Per Variant<\/th>\nAt 1K clicks\/day<\/th>\n<\/tr>\n<\/thead>\n
Smart banner tap rate<\/td>\n3%<\/td>\n15%<\/td>\n~140K<\/td>\n140 days<\/td>\n<\/tr>\n
Landing page install rate<\/td>\n8%<\/td>\n10%<\/td>\n~45K<\/td>\n45 days<\/td>\n<\/tr>\n
Email deep link CTR<\/td>\n15%<\/td>\n10%<\/td>\n~20K<\/td>\n20 days<\/td>\n<\/tr>\n
CTA button click rate<\/td>\n25%<\/td>\n15%<\/td>\n~5K<\/td>\n5 days<\/td>\n<\/tr>\n
In-app upgrade prompt<\/td>\n2%<\/td>\n20%<\/td>\n~95K<\/td>\n95 days<\/td>\n<\/tr>\n<\/tbody><\/table><\/figure>\n\n\n\n
Common Mistakes<\/h2>\n\n\n\n
1. Stopping Too Early<\/h3>\n\n\n\n
The most common mistake. You see one variant "winning" after 3 days and declare victory.<\/p>\n\n\n\n
\/\/ BAD: Checking results daily and stopping when significant\nfunction checkResultsDaily(experiment) {\n  const results = getResults(experiment.id);\n  if (results.pValue < 0.05) {\n    declareWinner(results.leadingVariant); \/\/ Don't do this\n  }\n}\n\n\/\/ GOOD: Pre-calculate when to check\nfunction setupExperiment(experiment) {\n  const sampleSize = calculateSampleSize(\n    experiment.baselineRate,\n    experiment.mde,\n    0.05,\n    0.80\n  );\n\n  return {\n    ...experiment,\n    requiredSamplePerVariant: sampleSize,\n    estimatedDays: Math.ceil(sampleSize * experiment.variants.length \/ experiment.dailyTraffic),\n    checkpoints: [0.5, 0.75, 1.0].map(p => Math.ceil(sampleSize * p)),\n  };\n}\n<\/code><\/pre>\n\n\n\n
Why early stopping is dangerous: with enough "peeking," random fluctuations will appear significant. If you check results after every 100 visitors, there's up to a 30% chance of a false positive (not 5%).<\/p>\n\n\n\n
2. Not Accounting for Multiple Variants<\/h3>\n\n\n\n
Testing 4 variants instead of 2 requires a correction:<\/p>\n\n\n\n
function adjustedSampleSize(baseSize, numVariants) {\n  \/\/ Bonferroni correction: divide alpha by number of comparisons\n  const numComparisons = numVariants * (numVariants - 1) \/ 2;\n  const adjustedAlpha = 0.05 \/ numComparisons;\n\n  \/\/ Recalculate with stricter alpha\n  \/\/ This typically increases sample size by 20-50%\n  return Math.ceil(baseSize * Math.log(1 \/ adjustedAlpha) \/ Math.log(1 \/ 0.05));\n}\n<\/code><\/pre>\n\n\n\n
\n\n\n\n\n\n\n
Variants<\/th>\nComparisons<\/th>\nSample Size Multiplier<\/th>\n<\/tr>\n<\/thead>\n
2<\/td>\n1<\/td>\n1.0x<\/td>\n<\/tr>\n
3<\/td>\n3<\/td>\n~1.3x<\/td>\n<\/tr>\n
4<\/td>\n6<\/td>\n~1.5x<\/td>\n<\/tr>\n
5<\/td>\n10<\/td>\n~1.6x<\/td>\n<\/tr>\n<\/tbody><\/table><\/figure>\n\n\n\n
3. Using the Wrong Baseline Rate<\/h3>\n\n\n\n
Your baseline should come from at least 2-4 weeks of data. Don't use a single day's data or an estimate from a different product.<\/p>\n\n\n\n
async function getReliableBaseline(metric, minDays = 14) {\n  const data = await getMetricData(metric, { days: minDays });\n\n  return {\n    rate: data.totalConversions \/ data.totalImpressions,\n    sampleSize: data.totalImpressions,\n    dateRange: data.dateRange,\n    dayOfWeekVariation: data.stdDevByDayOfWeek,\n    isReliable: data.totalImpressions > 1000 && data.days >= minDays,\n  };\n}\n<\/code><\/pre>\n\n\n\n
4. Ignoring Day-of-Week Effects<\/h3>\n\n\n\n
Deep link traffic patterns differ between weekdays and weekends. Always run tests for full weeks:<\/p>\n\n\n\n
function calculateMinimumRuntime(dailyTraffic, requiredSample, variants) {\n  const totalRequired = requiredSample * variants;\n  const daysNeeded = Math.ceil(totalRequired \/ dailyTraffic);\n\n  \/\/ Round up to complete weeks\n  const weeksNeeded = Math.ceil(daysNeeded \/ 7);\n  const adjustedDays = weeksNeeded * 7;\n\n  \/\/ Minimum 14 days regardless\n  return Math.max(adjustedDays, 14);\n}\n<\/code><\/pre>\n\n\n\n
Working with Low Traffic<\/h2>\n\n\n\n
Reduce the Number of Variants<\/h3>\n\n\n\n
Fewer variants means faster tests:<\/p>\n\n\n\n
\/\/ Instead of testing 4 CTA variants at once\nconst slowTest = {\n  variants: ['Download Free', 'Get Started', 'Try It Now', 'Install'],\n  samplePerVariant: 38000,\n  totalRequired: 152000, \/\/ At 500\/day = 304 days\n};\n\n\/\/ Test 2 at a time, iterate faster\nconst fastTest = {\n  round1: { variants: ['Download Free', 'Get Started'], totalRequired: 76000 }, \/\/ 152 days\n  round2: { variants: ['round1_winner', 'Try It Now'], totalRequired: 76000 },\n  \/\/ Total time is longer but you get actionable results sooner\n};\n<\/code><\/pre>\n\n\n\n
Increase the MDE<\/h3>\n\n\n\n
If you only care about large improvements (20%+ lift), you need fewer samples:<\/p>\n\n\n\n
\/\/ Detecting a 10% lift on 5% baseline: 149K per variant\nconst conservativeTest = calculateSampleSize(0.05, 0.10, 0.05, 0.80);\n\n\/\/ Detecting a 20% lift on 5% baseline: 38K per variant\nconst aggressiveTest = calculateSampleSize(0.05, 0.20, 0.05, 0.80);\n\n\/\/ Detecting a 30% lift on 5% baseline: 17K per variant\nconst largeEffectTest = calculateSampleSize(0.05, 0.30, 0.05, 0.80);\n<\/code><\/pre>\n\n\n\n
The tradeoff: you might miss real improvements of 10-19% because you weren't powered to detect them.<\/p>\n\n\n\n
Lower Confidence Level<\/h3>\n\n\n\n
Use 90% confidence instead of 95% for exploratory tests:<\/p>\n\n\n\n
\/\/ 95% confidence: 149K per variant\nconst standard = calculateSampleSize(0.05, 0.10, 0.05, 0.80);\n\n\/\/ 90% confidence: 109K per variant (27% fewer)\nconst exploratory = calculateSampleSize(0.05, 0.10, 0.10, 0.80);\n<\/code><\/pre>\n\n\n\n
Use 90% for initial discovery (which variants are worth testing further) and 95% for final decisions.<\/p>\n\n\n\n
Combine Small Tests into Larger Ones<\/h3>\n\n\n\n
If you have multiple low-traffic deep link campaigns, pool them:<\/p>\n\n\n\n
\/\/ Bad: Run separate tests on 5 campaigns with 200 clicks\/day each\n\/\/ 38K \/ 200 = 190 days per test\n\n\/\/ Better: Run one test across all campaigns with 1000 clicks\/day total\n\/\/ 38K \/ 1000 = 38 days\n\/\/ But only if the effect is expected to be consistent across campaigns\n<\/code><\/pre>\n\n\n\n
Pre-Test Checklist<\/h2>\n\n\n\n
Before starting any A\/B test:<\/p>\n\n\n\n
function preTestChecklist(config) {\n  const checks = [];\n\n  \/\/ 1. Baseline rate available?\n  const baseline = getBaseline(config.metric, 14);\n  checks.push({\n    name: 'Reliable baseline',\n    pass: baseline.isReliable,\n    value: baseline.rate,\n  });\n\n  \/\/ 2. Sample size calculated?\n  const sampleSize = calculateSampleSize(baseline.rate, config.mde, 0.05, 0.80);\n  checks.push({\n    name: 'Sample size calculated',\n    pass: true,\n    value: sampleSize,\n  });\n\n  \/\/ 3. Enough traffic?\n  const dailyTraffic = getDailyTraffic(config.route);\n  const daysNeeded = Math.ceil(sampleSize * config.variants.length \/ dailyTraffic);\n  checks.push({\n    name: 'Runtime feasible',\n    pass: daysNeeded <= 60,\n    value: `${daysNeeded} days at ${dailyTraffic}\/day`,\n  });\n\n  \/\/ 4. Full weeks?\n  const runtime = Math.max(Math.ceil(daysNeeded \/ 7) * 7, 14);\n  checks.push({\n    name: 'Full week runtime',\n    pass: true,\n    value: `${runtime} days (${runtime \/ 7} weeks)`,\n  });\n\n  \/\/ 5. No conflicting tests?\n  const conflicts = getConflictingExperiments(config.route);\n  checks.push({\n    name: 'No conflicts',\n    pass: conflicts.length === 0,\n    value: conflicts.length === 0 ? 'None' : conflicts.map(c => c.id).join(', '),\n  });\n\n  return checks;\n}\n<\/code><\/pre>\n\n\n\n
Runtime Calculator<\/h2>\n\n\n\n
function estimateRuntime(params) {\n  const {\n    baselineRate,\n    minimumDetectableEffect,\n    dailyTraffic,\n    numVariants = 2,\n    significance = 0.05,\n    power = 0.80,\n  } = params;\n\n  const samplePerVariant = calculateSampleSize(\n    baselineRate, minimumDetectableEffect, significance, power\n  );\n  const totalSample = samplePerVariant * numVariants;\n  const rawDays = Math.ceil(totalSample \/ dailyTraffic);\n  const fullWeeks = Math.max(Math.ceil(rawDays \/ 7) * 7, 14);\n\n  return {\n    samplePerVariant,\n    totalSample,\n    rawDays,\n    recommendedDays: fullWeeks,\n    recommendedWeeks: fullWeeks \/ 7,\n  };\n}\n\n\/\/ Example: Testing smart banner tap rate\nconst estimate = estimateRuntime({\n  baselineRate: 0.03,       \/\/ 3% tap rate\n  minimumDetectableEffect: 0.15, \/\/ Detect 15% relative lift\n  dailyTraffic: 2000,       \/\/ 2000 banner impressions\/day\n  numVariants: 2,\n});\n\nconsole.log(estimate);\n\/\/ {\n\/\/   samplePerVariant: ~140000,\n\/\/   totalSample: ~280000,\n\/\/   rawDays: 140,\n\/\/   recommendedDays: 140,\n\/\/   recommendedWeeks: 20,\n\/\/ }\n<\/code><\/pre>\n\n\n\n
When Not to Use Sample Size Calculations<\/h2>\n\n\n\n
Bandit Algorithms<\/h3>\n\n\n\n
For optimization (not measurement), multi-armed bandit<\/a> algorithms don't need fixed sample sizes. They continuously allocate more traffic to better-performing variants:<\/p>\n\n\n\n
function banditAllocation(variants) {\n  const total = variants.reduce((sum, v) => sum + v.successes + v.failures, 0);\n\n  return variants.map(v => {\n    const successes = v.successes + 1; \/\/ Beta prior\n    const failures = v.failures + 1;\n    const mean = successes \/ (successes + failures);\n    const variance = (successes * failures) \/ (Math.pow(successes + failures, 2) * (successes + failures + 1));\n\n    return {\n      variantId: v.id,\n      weight: mean + Math.sqrt(2 * Math.log(total) \/ (successes + failures)), \/\/ UCB1\n    };\n  });\n}\n<\/code><\/pre>\n\n\n\n
Use bandits when you want to minimize regret (lost conversions during the test) rather than measure a precise lift.<\/p>\n\n\n\n
Very High Traffic<\/h3>\n\n\n\n
If you have millions of daily impressions, any reasonable test will reach significance quickly. Focus on preventing false positives from multiple comparisons rather than worrying about sample size.<\/p>\n\n\n\n
Best Practices<\/h2>\n\n\n\n
    \n
  1. Calculate sample size before starting<\/strong>: Never run a test without knowing how long it needs to run.<\/li>\n
  2. Use your own baseline data<\/strong>: Don't use industry benchmarks. Measure your actual conversion rate for 2+ weeks.<\/li>\n
  3. Plan for realistic MDE<\/strong>: If a 5% lift wouldn't change your business, don't power for it. Use 10-20% MDE for most deep link tests.<\/li>\n
  4. Run for full weeks<\/strong>: Day-of-week effects are real. Always complete full 7-day cycles.<\/li>\n
  5. Don't peek constantly<\/strong>: Pre-set checkpoints at 50%, 75%, and 100% of required sample. Only evaluate at those points.<\/li>\n
  6. Document everything<\/strong>: Record your baseline, MDE, sample size, and runtime before the test starts.<\/li>\n<\/ol>\n\n\n\n
    For A\/B testing features, see Tolinku A\/B testing<\/a>. For understanding test results, see the A\/B testing results documentation<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":"
    Calculate the right sample size for deep link A\/B tests. Avoid false positives with proper statistical planning for link experiments.<\/p>\n","protected":false},"author":2,"featured_media":1052,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"rank_math_title":"A\/B Testing Sample Size Calculator for Deep Links","rank_math_description":"Calculate the right sample size for deep link A\/B tests. Avoid false positives with proper statistical planning for link experiments.","rank_math_focus_keyword":"A\/B test sample size","rank_math_canonical_url":"","rank_math_facebook_title":"","rank_math_facebook_description":"","rank_math_facebook_image":"https:\/\/tolinku.com\/blog\/wp-content\/uploads\/2026\/03\/og-ab-testing-sample-size.png","rank_math_facebook_image_id":"","rank_math_twitter_title":"","rank_math_twitter_description":"","rank_math_twitter_image":"https:\/\/tolinku.com\/blog\/wp-content\/uploads\/2026\/03\/og-ab-testing-sample-size.png","footnotes":""},"categories":[13],"tags":[60,37,191,20,225,69,256,258],"class_list":["post-1053","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-growth","tag-ab-testing","tag-analytics","tag-conversions","tag-deep-linking","tag-experimentation","tag-mobile-development","tag-optimization","tag-statistics"],"_links":{"self":[{"href":"https:\/\/tolinku.com\/blog\/wp-json\/wp\/v2\/posts\/1053","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/tolinku.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/tolinku.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/tolinku.com\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/tolinku.com\/blog\/wp-json\/wp\/v2\/comments?post=1053"}],"version-history":[{"count":2,"href":"https:\/\/tolinku.com\/blog\/wp-json\/wp\/v2\/posts\/1053\/revisions"}],"predecessor-version":[{"id":2229,"href":"https:\/\/tolinku.com\/blog\/wp-json\/wp\/v2\/posts\/1053\/revisions\/2229"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/tolinku.com\/blog\/wp-json\/wp\/v2\/media\/1052"}],"wp:attachment":[{"href":"https:\/\/tolinku.com\/blog\/wp-json\/wp\/v2\/media?parent=1053"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/tolinku.com\/blog\/wp-json\/wp\/v2\/categories?post=1053"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/tolinku.com\/blog\/wp-json\/wp\/v2\/tags?post=1053"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}