Evaluation environment · July 2026
Confounded churn: what adoption actually causes
The naive read says high-adoption accounts churn 30.4 percentage points less. Adjusting for account size puts the true effect at 11.7 points. The correction removes about 88% of the confounding bias, and the generator knows the right answer, so the correction is checkable.
Why the finding is trustworthy
- Synthetic data generation of 500 accounts over 24 months with a known churn process and a deliberate confounder (account size), so the true causal effect is known before any model runs.
- XGBoost on a strict temporal train and test split, with leakage guards under test.
- LLM intent labeling scored against a frozen 100-ticket gold set.
- The full pipeline is deterministic and runs offline in about 30 seconds with no API keys, under 55 pytest tests.
Results
- Naive (confounded) effect of adoption on churn
- 30.4 pp lower churn
- True effect, known from the generator
- 11.7 pp
- Confounding bias removed by adjustment
- about 88%
- XGBoost on holdout
- 0.806 ROC-AUC, 0.624 PR-AUC
- LLM intent labeling against the gold set
- Cohen's kappa 0.80
- Churn AUC lift from intent features
- 0.735 to 0.814
- Cost per account scored
- about $0.0035