The question.
Teams make decisions from noisy tests, inconsistent exposure logs, and convenient metrics without checking whether the underlying comparison is valid.
The workbench makes assignment, exposure, eligibility, and outcome definitions explicit before statistical analysis begins.
The proposed approach.
- Separate assignment from actual exposure and instrument both
- Choose estimands, power assumptions, and guardrail metrics before launch
- Use suitable fixed-horizon or sequential methods consistently
- Document identification assumptions for observational comparisons
Target architecture.
Experiment definitionAssignment and exposureMetric computationAnalysisSensitivity checksDecision record
The difficult parts.
- Sample-ratio mismatch and missing exposure events
- Repeated peeking and multiple comparisons
- Interference between experimental units
How to evaluate it.
- Type-I error and power under simulated experiments
- Robustness to missingness and logging failures
- Clarity and reproducibility of final decision records
Planned deliverables.
- An experiment contract and metric library
- A simulation-based method comparison
- A reproducible analysis and decision template
Design principles.
State the causal question before selecting the estimator.
A reliable experiment starts with reliable exposure data.