Incrementality Testing for Mobile Marketing

Attribution tells you who converted after seeing your ad. Incrementality testing tells you who converted because of your ad. The difference is critical: if 60% of the users your campaign "converted" would have converted anyway, your true ROAS is 40% of what your attribution dashboard shows.

This guide covers how to run incrementality tests for mobile marketing. For A/B testing deep links, see A/B testing deep links and landing pages. For attribution models, see last-click vs multi-touch attribution.

What Incrementality Measures

The Core Question

"If I had not run this campaign, how many conversions would I have lost?"

• Attributed conversions: 10,000 installs credited to a Facebook campaign.
• Incremental conversions: 4,000 installs that would not have happened without the campaign.
• Non-incremental: 6,000 installs that would have happened anyway (organic users who also saw an ad).

The incrementality rate is 40% (4,000 / 10,000). The true CPI is 2.5x higher than the attributed CPI.

Test Design

Intent-to-Treat (ITT) Test

The most common incrementality test for mobile:

1. Define the target audience (e.g., all users in the US who match your targeting criteria).
2. Randomly split into a test group (80%) and a holdout group (20%).
3. The test group sees your ads.
4. The holdout group does not see your ads (they are excluded from the ad campaign).
5. After the test period, compare conversion rates between the two groups.

Test Group (80%):     Sees ads → 2.5% install rate
Holdout Group (20%):  No ads   → 1.5% install rate

Incremental lift: 2.5% - 1.5% = 1.0 percentage points
Incrementality rate: 1.0% / 2.5% = 40%

Ghost Ads / PSA Test

Instead of simply not showing ads to the holdout group, show them a public service announcement (PSA) or charity ad. This controls for the "any ad exposure" effect:

Test Group:    Sees your ad  → 2.5% install rate
PSA Group:     Sees PSA ad   → 1.6% install rate

Incremental lift: 2.5% - 1.6% = 0.9 percentage points

Geographic Holdout

Instead of user-level randomization, exclude entire geographic regions:

Test markets (80% of cities):    Run campaigns normally
Holdout markets (20% of cities): No campaigns

Compare install rates in test vs holdout markets

Geographic holdouts are easier to implement (no need for user-level targeting exclusions) but require more cities to achieve statistical significance.

Running the Test

Step 1: Define Hypotheses

H0 (null): The campaign has no incremental impact on installs
H1 (alternative): The campaign drives incremental installs

Target: 95% confidence level
Minimum detectable effect: 20% lift

Step 2: Calculate Sample Size

from scipy import stats
import math

def required_sample_size(
    baseline_rate: float,
    minimum_detectable_effect: float,
    confidence: float = 0.95,
    power: float = 0.80
) -> int:
    alpha = 1 - confidence
    z_alpha = stats.norm.ppf(1 - alpha / 2)
    z_beta = stats.norm.ppf(power)

    p1 = baseline_rate
    p2 = baseline_rate * (1 + minimum_detectable_effect)

    n = ((z_alpha * math.sqrt(2 * p1 * (1 - p1)) +
          z_beta * math.sqrt(p1 * (1 - p1) + p2 * (1 - p2))) /
         (p2 - p1)) ** 2

    return math.ceil(n)

# Example: 1.5% baseline, detect 20% lift, 95% confidence
sample_size = required_sample_size(0.015, 0.20)
# Result: ~85,000 users per group

Step 3: Run the Test

• Duration: 2-4 weeks minimum (longer for lower-frequency events).
• Monitor for contamination (holdout users seeing ads through shared devices).
• Do not peek at results before the planned end date (p-hacking risk).

Step 4: Analyze Results

def analyze_incrementality(
    test_conversions: int, test_size: int,
    holdout_conversions: int, holdout_size: int
):
    test_rate = test_conversions / test_size
    holdout_rate = holdout_conversions / holdout_size

    lift = test_rate - holdout_rate
    relative_lift = lift / holdout_rate if holdout_rate > 0 else float('inf')
    incrementality_rate = lift / test_rate if test_rate > 0 else 0

    # Statistical significance (chi-squared test)
    chi2, p_value = stats.chi2_contingency([
        [test_conversions, test_size - test_conversions],
        [holdout_conversions, holdout_size - holdout_conversions]
    ])[:2]

    return {
        'test_rate': test_rate,
        'holdout_rate': holdout_rate,
        'absolute_lift': lift,
        'relative_lift': relative_lift,
        'incrementality_rate': incrementality_rate,
        'p_value': p_value,
        'significant': p_value < 0.05
    }

Interpreting Results

Scenario Analysis

Scenario	Test Rate	Holdout Rate	Lift	Action
High incrementality	3.0%	1.0%	2.0pp	Scale the campaign
Moderate incrementality	2.5%	1.5%	1.0pp	Continue, optimize targeting
Low incrementality	2.0%	1.8%	0.2pp	Reduce spend, retest
No incrementality	2.0%	2.0%	0.0pp	Stop the campaign
Negative impact	1.5%	2.0%	-0.5pp	Stop immediately (ad fatigue)

Calculating True ROAS

Campaign spend: $50,000
Attributed installs: 10,000
Attributed CPI: $5.00

Incrementality rate: 40%
Incremental installs: 4,000
True incremental CPI: $12.50

If LTV per user is $15:
  Attributed ROAS: $15 / $5 = 300% (looks great)
  True incremental ROAS: $15 / $12.50 = 120% (barely profitable)

Testing Frequency

Campaign Type	Test Frequency	Duration
Always-on campaigns	Quarterly	4 weeks
New channel launch	Before scaling	2-4 weeks
Major creative change	After launch	2 weeks
Seasonal campaigns	Before and during	1-2 weeks
Retargeting	Bi-annually	4 weeks

Retargeting campaigns often have the lowest incrementality because they target users who are already likely to convert.

Common Pitfalls

1. Testing too briefly. Underpowered tests produce unreliable results. Run until you reach the required sample size.
2. Contamination. If holdout users see your ads through another channel (e.g., a friend's shared link), the test is contaminated.
3. Seasonality. Running a test during Black Friday will produce different results than running it in January.
4. Selection bias. The test and holdout groups must be truly random. If the holdout group is systematically different, results are invalid.
5. One-time tests. Incrementality changes over time. A campaign that was incremental 6 months ago may not be incremental today.

Tolinku for Experimentation

Tolinku supports A/B testing for deep links, which can be used as part of incrementality testing. Route different user segments to different experiences and measure conversion differences. Configure experiments in the Tolinku dashboard.

For attribution models, see last-click vs multi-touch attribution. For mobile attribution, see mobile attribution: a developer's guide.