Create and Run an Experiment

This guide covers the simplified behavior-scoped workflow and the advanced revision-level workflow.

Before you launch: Experiments route eligible new web chat, email, and SMS conversations immediately. Applied does not currently schedule a start, pause a running test, automatically act on a safety metric, or promote a winner.

Before you start

Prepare:

• one live behavior to change;
• a falsifiable hypothesis;
• one executable Contact-attributed primary metric;
• one executable Contact-attributed safety metric; and
• an observation window, minimum sample, and human owner.

Configure and launch in context

1. Select the target

Open the live behavior you want to test:

• On a Flow, open Experiments in the Flow sidebar and select New. The sidebar also shows current and past experiments for that Flow.
• On a Procedure, select Experiment in the editor header.
• On a Response, select the flask action in the list or Experiment in the editor header.

The selected Flow, node, Procedure, or Response must exist in the agent’s current live version. Applied uses that immutable version as Control A and creates Variant B from a full clone.

2. Name the experiment and define success

Add an experiment name and hypothesis. Choose one primary outcome and one different safety metric. Only active, executable, Contact-attributed metrics are eligible. You can create a metric from the composer if the required event definition does not exist yet.

For a rate, the result event is the result being counted and the population event identifies every conversation that had a chance to produce that result. The population event affects measurement only; experiment audience and allocation are configured separately in the Traffic step.

Safety settings are required for launch, but threshold evaluation and automatic intervention remain manual.

3. Edit Variant B

Control A is read-only. The full-screen comparison uses the same editor as the selected behavior: Procedure mentions remain readable action chips and Responses keep their rich text editor.

For a Flow, choose a node, edit Variant B beside its deployed control, and select Save node changes. You can then choose another node and repeat. All saved node edits stay in the local experiment draft until you review and launch the complete Flow treatment.

The composer shows the exact controlled fields. The server independently validates the treatment and rejects it if anything outside the selected target differs from the base version.

4. Configure the audience and split

Choose Everyone eligible for this agent or a contact group. Set the share of experiment traffic sent to Variant B; the remaining share stays on Control A. Each variant must receive between 10% and 90% of eligible traffic.

Applied creates and manages the underlying experiment traffic rule. You do not need to create a separate Traffic Control rule for this path.

5. Verify and launch

Select Verify and launch. Applied atomically creates the treatment version, experiment variants, metrics, and routing rule, then requests authoritative server preflight. Traffic starts only if every server check passes.

If verification fails, the Draft is preserved and the experiment page shows the checks that need attention.

Monitor the experiment

Open the experiment or Analytics -> Experiments. Review, in this order:

1. intended versus assigned traffic;
2. contamination, sample-ratio, and exposure-loss warnings;
3. assigned, agent-reached, and target-reached counts;
4. primary and supporting outcomes; and
5. safety metrics.

Target reach is recorded only when the selected behavior begins execution from the conversation’s pinned immutable version. State-less operational Flow runs that execute live database nodes are deliberately excluded because they cannot prove which experiment version ran.

Outcome values currently use assignment-stamped analytics rows. They are not restricted to contacts that reached the selected node, and the product states that limitation directly.

Advanced whole-agent workflow

For whole-agent tests, Knowledge tests, or unsupported scoped fields:

1. Create distinct immutable versions under Deploy -> Versions.
2. Create a Draft under Deploy -> Experiments.
3. Select exactly one control, configure allocations totaling 100%, and add a primary plus safety metric.
4. Create an active Traffic Control rule targeting the experiment.
5. Return to the experiment and use the server-owned launch checklist.
6. Start only when the checklist is ready.

Traffic rules are first-match-wins. Place narrow experiment audiences above broad rules. An active rule targeting a Draft does not route experiment traffic until the experiment starts.

Contact identity and sticky assignment

Applied assigns on the canonical Contact, so aliases and recognized identifiers reuse the same experiment assignment. Public chat resolves identity through its signed session, contact context, and ContactIdentifier path.

A raw contact_id payload is accepted only for authenticated, same-shop test traffic with is_test=true; it is not a public production identity mechanism.

Stop and deploy manually

Pause, Promote B, Keep A, and Inconclusive are deferred. To end a test today:

1. record the final observation window and reviewed result;
2. deploy the chosen version through the normal approval path;
3. archive or update the experiment traffic rule; and
4. select Stop experiment.

Stopping completes the experiment. It does not deploy a variant or clean up routing automatically.

Troubleshooting

Launch is disabled

Read the server launch checklist on the experiment page. Common causes are a missing executable safety metric, duplicate variants, allocations that do not sum to 100%, an unreachable traffic rule, a treatment outside the declared target, or another running experiment on the same target.

Target reached stays at zero

Confirm the selected node actually executed in a new eligible conversation and that the execution used the conversation’s pinned agent version. Assignment or agent reach alone does not count as target reach.

Outcomes are empty

Confirm numerator and denominator events are emitted in production and visible in source analytics. An unavailable event stream is instrumentation work, not a zero result.

The split differs from the allocation

Inspect the traffic-integrity section. Small samples vary naturally, but contamination, unaccounted variants, or a reported sample-ratio mismatch require investigation before outcomes are interpreted.