Understand Experiment Results
The experiment page, Analytics -> Experiments, and experiment summaries in Analytics dashboards use the same versioned server result. Dashboard charts can still provide directional breakdowns, but they are not a separate source of experiment truth.
Read the populations first
| Population | What it proves |
|---|---|
| Assigned | A canonical contact was deterministically randomized into A or B |
| Agent reached | The assigned immutable agent version started a live provider |
| Target reached | The declared Flow node began executing from that pinned version |
Target reach is retry-safe per node execution. The summary reports unique assigned contacts that reached the target so it remains comparable with the Contact analysis unit.
Outcome metrics currently use Assigned contacts from experiment-stamped analytics rows. They do not yet restrict outcomes to the target-reached ledger. The result includes this population label so assignment, execution, and outcome attribution are not confused.
Check traffic health before outcomes
The canonical result combines routing diagnostics with the metric readout. Review:
- intended versus actual assigned share for each variant;
- contaminated assignments;
- assignment sample-ratio mismatch;
- revision-exposure sample-ratio mismatch; and
- assigned contacts that never reached an agent provider.
Traffic needs attention means outcomes should not be interpreted yet. Traffic is being watched means the split is not known to be invalid, but some assigned contacts have not reached the provider. A small sample without a warning is not proof that routing is correct.
Read metric values
For each available metric, the server returns:
- A and B numerator and denominator samples;
- an explicit unit: percent, ratio, or number;
- the observed value for each variant; and
- the observed difference from control.
A percent difference is shown in percentage points. For example, a move from 32% to 39% is +7 percentage points, not a seven-percent relative lift.
If the analytics query or metric definition is unavailable, the result keeps the traffic-integrity section and explains why that metric cannot be shown. It does not silently turn missing data into zero.
Read readiness conservatively
Readiness can indicate that the experiment has not started, has no traffic, has a traffic-integrity problem, lacks an available primary metric, lacks samples for every variant, is collecting, or is ready for human review.
It does not declare a statistically significant winner. No automatic decision threshold or practical-effect policy is configured in this release.
Safety metrics
Safety metrics are required at launch and appear beside the primary and supporting outcomes. Thresholds and intended breach actions can be stored, but Applied does not yet evaluate them automatically, stop traffic, or roll back a variant. A human owner must monitor and act on the predefined safety plan.
Known limitations
- Knowledge, Procedure, whole-Flow, and state-less operational target-execution hooks are not included in the first Flow-node release.
- Outcome attribution is assignment-based rather than target-reached-based.
- Metric definitions are reusable and may be edited; the result is generated from the definition available at read time.
- Pause, winner promotion, Keep A, and Inconclusive decisions are deferred.
- Routing supports new web chat, email, and SMS conversations, not voice or social-channel experiments.
For creation, launch, identity, and shutdown guidance, see Create and Run an Experiment.