Can LLMs Replace Humans in A/B Tests? Only If You Accept These Two Assumptions

The Pitch, and the Hole in It

Everyone wants to skip the traffic split. Run the experiment on a model, get answers in hours, ship faster. It sounds like free velocity.

But most proposals quietly skip the statistical question that makes an experiment valid in the first place: under what conditions does this procedure identify the treatment effect you actually care about?

Randomized experiments are the gold standard because they identify causal effects by design. Swap real user responses for LLM predictions and that guarantee evaporates. Identification now holds by assumption — and nobody is spelling out which assumptions.

A recent paper formalizes exactly that, borrowing surrogate endpoint theory from biostatistics. The logic is clean: if a surrogate outcome captures everything about a treatment that matters for the real outcome, then experimenting on the surrogate gives you the right answer. Clinical trials do this with lab biomarkers all the time. The question is whether LLM predictions qualify as a valid biomarker for user behavior. See the underlying research for the full formalization.

The Empirical Reality: Raw Predictions Are Biased, Not Just Noisy

The team tested this on the Upworthy Research Archive — the largest open-access A/B test dataset available, containing click-through rates for thousands of news headline variants. They prompted gpt-4o-mini to predict CTR for each headline, separately for treatment and control, then ran a standard experimental analysis.

Result: raw LLM predictions recovered only 39% of the observed human treatment effect.

# Simplified illustration of the attenuation problem
# (실제 재현 코드가 아닌 개념 설명용)

import numpy as np
from sklearn.ensemble import RandomForestRegressor
from sklearn.linear_model import LinearRegression

# y_human: 실제 사용자 클릭률, y_llm: LLM 예측 클릭률
tau_human = y_human_treated.mean() - y_human_control.mean()
tau_llm_raw = y_llm_treated.mean() - y_llm_control.mean()

print(f"Human ATE:        {tau_human:.4f}")
print(f"Raw LLM ATE:      {tau_llm_raw:.4f}")
print(f"Recovery ratio:   {tau_llm_raw / tau_human:.2%}")  # ~39%

# OLS 보정: 너무 경직되어 실패 (인간 벤치마크 대비 3.8 표준오차 이탈)
ols = LinearRegression().fit(y_llm_train.reshape(-1, 1), y_human_train)

# 비선형 ML 보정: 표본 오차 범위 내로 들어옴
rf = RandomForestRegressor(n_estimators=500).fit(X_train, y_human_train)

The bias is systematic and directional: LLM outcomes attenuate treatment effects toward zero. Treatments look weaker than they are. Scale that across a product org and you systematically undervalue what you ship — and make bad launch calls.

Two Assumptions That Make It Work

1. Surrogacy — the LLM output must fully mediate the treatment effect on the human outcome. After conditioning on the LLM prediction and baseline covariates, treatment assignment tells you nothing extra about user behavior. In plain terms: the LLM captures everything about the treatment that matters.
2. Comparability — the calibration function (the mapping from LLM predictions to human outcomes) must be stable between your historical training data and the new experiment. If that mapping shifts when the treatment changes, calibration breaks.

When both hold, calibrating LLM outputs against real A/B data recovers the human effect. When either fails, you're not estimating the human effect at all — you're estimating the effect on the LLM. Infinite LLM samples won't save you, because the procedure is identifying a different quantity.

Calibration Method Matters More Than You'd Think

• OLS linear calibration: failed the falsification test, landing 3.8 standard errors from the human benchmark. Too rigid for the nonlinear LLM→human mapping.
• Random forest / gradient-boosted trees: flexible enough to learn the nonlinearity; calibrated estimates fell within sampling error of the human effect.
• Sampling temperature noise: a single LLM prediction is noisy. Drawing multiple outputs per unit and averaging them (measurement error theory) reduces attenuation and variance.

Where This Breaks: The Future-Intervention Problem

Here's the uncomfortable core: surrogacy and comparability can be partially checked on historical data, but can never be proven for a treatment you've never run before.

The further a new treatment departs from past experiments — a new UI paradigm, a new pricing model, a feature unlike anything shipped before — the weaker the basis for trusting LLM outputs. For genuinely novel interventions, the assumptions are inherently untestable.

Which leads to the punchline: LLMs offer the most benefit precisely where they are least likely to work. Human experiments remain indispensable for real product innovation.

A Cautionary Note on the Test Case Itself

The Upworthy dataset is close to a best-case scenario: binary outcome, text-based linguistically similar treatments, and LLMs trained on oceans of text about what makes headlines clickable. Change the layout, the algorithm, or the price, and the conditions get much harder to justify. There is no empirical evidence the assumptions hold in general across a company's innovation portfolio — which is exactly what "use it at scale" would require.

The Data You Were Trying to Avoid

Calibration needs historical user experiments. Most product teams don't have thousands of clean, homogeneous experiments — they have a messy mix across surfaces, with inconsistent logging. The framework doesn't eliminate user experiments; it lets you reduce them only when new experiments resemble old ones. The upfront investment in real user responses isn't optional — it's what makes the whole thing trustworthy.

And LLM version churn adds another layer: a calibration function fit to a specific model at a specific time may be invalid six months later. New calibration ideally needs new user experiments, or it's temporally biased. Better LLMs, prompting, or fine-tuning don't remove the need for human validation.

If you're thinking about where surrogate metrics have already paid off in production systems, the Netflix case study on cutting recommendation CPU from 7.5% to 1% with the JDK Vector API is a good companion read on measuring real wins rigorously.

The Takeaway

LLM predictions can make human experiments better — filtering weak ideas before they consume an experiment slot, or serving as covariates for variance reduction. That's a real win when you have strong historical data and the new treatment resembles past ones.

Substituting them for human outcomes is a different move entirely. It trades identification by design for identification by assumption. Don't give that up casually.

Limitations to keep front of mind:

• Assumptions are unverifiable for novel treatments — the highest-value use case.
• Calibration is model-version-specific and decays over time.
• Raw predictions attenuate effects toward zero; naive use systematically undervalues features.

Next steps if you want to explore this:

1. Read up on surrogate endpoint theory in biostatistics — it's the rigorous backbone here.
2. Start with LLM predictions as a pre-filter for experiment prioritization, not as a replacement for the experiment.
3. Audit your historical experiment corpus for homogeneity before assuming calibration will transfer.

Security and platform changes can also invalidate assumptions overnight — if you're running React infrastructure, the critical React Server Components RCE vulnerability guide is worth a look before your next experiment cycle.