Deployment and experiments
Use experiments to compare an immutable control with one bounded treatment for new web chat, email, and SMS conversations.
Current scope: The simplified in-context workflow is available for Flows, Procedures, and Responses. Knowledge creation still uses the advanced revision workflow. Pause, automatic guardrail actions, winner promotion, and outcome decisions are not yet available.
Start from the behavior you want to test
Start from the behavior you want to change:
- Open a live Flow, Procedure, or Response.
- Select Experiment on a Procedure or Response. On a Flow, open Experiments in the Flow sidebar and select New.
- Edit Variant B. For a Flow, save changes to one or more nodes in the same local draft. Control A is locked to the current live version.
- Choose an audience, traffic percentage, primary outcome, and safety metric.
- Select Verify and launch.
Applied creates the treatment version, experiment, and traffic rule together. The server verifies the complete launch configuration before any traffic is routed. The selected behavior shows its active A/B status and links back to the experiment.
What stays controlled
Variants are still complete immutable agent versions at runtime. A scoped experiment also records its target, base version, target hash, and allowed change paths. Launch verification rejects a treatment if anything outside the declared target changed.
This provides both properties needed for reliable execution:
- the runtime receives a complete, immutable configuration; and
- the experiment can prove that only the selected behavior differs.
Assignment and reach
Eligible contacts are assigned deterministically and remain on the same variant through their canonical contact identity. Applied reports three different populations:
| Population | Meaning |
|---|---|
| Assigned | Canonical contacts randomized into A or B |
| Agent reached | Assigned contacts whose experiment version started an agent provider |
| Target reached | Assigned contacts whose selected Flow node began executing from the pinned version |
These counts should not be treated as interchangeable. Outcome metrics are currently attributed from assignment-stamped analytics rows; target-specific outcome attribution is not yet available.
Results
The experiment page and Analytics -> Experiments use the same canonical server result. It shows intended versus actual allocation, assignment and reach counts, traffic-integrity warnings, metric values for A and B, and the observed difference from control. Analytics dashboards use this same summary when an experiment is selected; dashboard charts remain directional breakdowns.
The result does not declare a winner. Review traffic health, the observation window, primary outcome, and safety metrics before changing production behavior.
Advanced revision workflow
Use Deploy -> Experiments when creating a whole-agent experiment, testing a Knowledge item, or when the in-context editor does not support the field you need. The advanced workflow still exposes saved versions and traffic rules directly.
See: