Attribution tells you who converted after seeing your ad. Incrementality testing tells you who converted because of your ad. The difference is critical: if 60% of the users your campaign "converted" would have converted anyway, your true ROAS is 40% of what your attribution dashboard shows.<\/p>\n\n\n\n
This guide covers how to run incrementality tests for mobile marketing. For A\/B testing deep links, see A\/B testing deep links and landing pages<\/a>. For attribution models, see last-click vs multi-touch attribution<\/a>.<\/p>\n\n\n\n "If I had not run this campaign, how many conversions would I have lost?"<\/p>\n\n\n\n The incrementality rate is 40% (4,000 \/ 10,000). The true CPI is 2.5x higher than the attributed CPI.<\/p>\n\n\n\n The most common incrementality test for mobile:<\/p>\n\n\n\n Instead of simply not showing ads to the holdout group, show them a public service announcement (PSA) or charity ad. This controls for the "any ad exposure" effect:<\/p>\n\n\n\n Instead of user-level randomization, exclude entire geographic regions:<\/p>\n\n\n\n Geographic holdouts are easier to implement (no need for user-level targeting exclusions) but require more cities to achieve statistical significance.<\/p>\n\n\n\nWhat Incrementality Measures<\/h2>\n\n\n\n
The Core Question<\/h3>\n\n\n\n
\n
Test Design<\/h2>\n\n\n\n
Intent-to-Treat (ITT) Test<\/h3>\n\n\n\n
\n
Test Group (80%): Sees ads \u2192 2.5% install rate\nHoldout Group (20%): No ads \u2192 1.5% install rate\n\nIncremental lift: 2.5% - 1.5% = 1.0 percentage points\nIncrementality rate: 1.0% \/ 2.5% = 40%\n<\/code><\/pre>\n\n\n\nGhost Ads \/ PSA Test<\/h3>\n\n\n\n
Test Group: Sees your ad \u2192 2.5% install rate\nPSA Group: Sees PSA ad \u2192 1.6% install rate\n\nIncremental lift: 2.5% - 1.6% = 0.9 percentage points\n<\/code><\/pre>\n\n\n\nGeographic Holdout<\/h3>\n\n\n\n
Test markets (80% of cities): Run campaigns normally\nHoldout markets (20% of cities): No campaigns\n\nCompare install rates in test vs holdout markets\n<\/code><\/pre>\n\n\n\nRunning the Test<\/h2>\n\n\n\n
Step 1: Define Hypotheses<\/h3>\n\n\n\n
H0 (null): The campaign has no incremental impact on installs\nH1 (alternative): The campaign drives incremental installs\n\nTarget: 95% confidence level\nMinimum detectable effect: 20% lift\n<\/code><\/pre>\n\n\n\nStep 2: Calculate Sample Size<\/h3>\n\n\n\n
from scipy import stats\nimport math\n\ndef required_sample_size(\n baseline_rate: float,\n minimum_detectable_effect: float,\n confidence: float = 0.95,\n power: float = 0.80\n) -> int:\n alpha = 1 - confidence\n z_alpha = stats.norm.ppf(1 - alpha \/ 2)\n z_beta = stats.norm.ppf(power)\n\n p1 = baseline_rate\n p2 = baseline_rate * (1 + minimum_detectable_effect)\n\n n = ((z_alpha * math.sqrt(2 * p1 * (1 - p1)) +\n z_beta * math.sqrt(p1 * (1 - p1) + p2 * (1 - p2))) \/\n (p2 - p1)) ** 2\n\n return math.ceil(n)\n\n# Example: 1.5% baseline, detect 20% lift, 95% confidence\nsample_size = required_sample_size(0.015, 0.20)\n# Result: ~85,000 users per group\n<\/code><\/pre>\n\n\n\nStep 3: Run the Test<\/h3>\n\n\n\n
\n
Step 4: Analyze Results<\/h3>\n\n\n\n
def analyze_incrementality(\n test_conversions: int, test_size: int,\n holdout_conversions: int, holdout_size: int\n):\n test_rate = test_conversions \/ test_size\n holdout_rate = holdout_conversions \/ holdout_size\n\n lift = test_rate - holdout_rate\n relative_lift = lift \/ holdout_rate if holdout_rate > 0 else float('inf')\n incrementality_rate = lift \/ test_rate if test_rate > 0 else 0\n\n # Statistical significance (chi-squared test)\n chi2, p_value = stats.chi2_contingency([\n [test_conversions, test_size - test_conversions],\n [holdout_conversions, holdout_size - holdout_conversions]\n ])[:2]\n\n return {\n 'test_rate': test_rate,\n 'holdout_rate': holdout_rate,\n 'absolute_lift': lift,\n 'relative_lift': relative_lift,\n 'incrementality_rate': incrementality_rate,\n 'p_value': p_value,\n 'significant': p_value < 0.05\n }\n<\/code><\/pre>\n\n\n\nInterpreting Results<\/h2>\n\n\n\n
Scenario Analysis<\/h3>\n\n\n\n