Running a containment pilot: a 30-day plan with exit criteria

A pilot with no exit criteria ends when the licence trial does. Decide what pass and fail look like before you connect anything, and put the criteria in writing where the vendor can see them.

Good criteria are numbers and yes-or-no checks:

• Median and 90th percentile time to contain, by action type, against your baseline.
• Wrong actions: the count of containing actions that should not have run. The target for auto-run categories is zero in the final two weeks.
• Rollback: every undo tested, and every undo recorded.
• Record: an outside reader can answer the five audit questions from an exported record.
• Coverage: the share of the actions on your catalogue the platform can run through a documented integration.
• Cost: the monthly cost at your alert volume, with the unit stated.

Hold the vendor to the same list. A vendor that wriggles on a criterion has told you what you need to know.

Trial terms vary, and several vendors publish little.

• Splunk's SOAR page offers a free trial with no credit card (read 2026-11-03).
• Tines publishes a free edition with 3 live workflows and a one-time USD 50 AI allowance (read 2026-11-03). That is enough to prototype a few actions and not enough to run a production load.
• Prophet Security's AI SOC Analyst page offers account creation and a free proof of value (read 2026-11-03).
• D3 Security's pricing page names a migration program for customers leaving Cortex XSOAR, Splunk SOAR, Swimlane, Torq or Tines as free (read 2026-11-03), a vendor claim.

For the rest, there is no published trial term on the pages read. Ask for a written proof-of-value scope with a start date, an end date and the exit criteria above.

Spend the first week on measurement, not on automation.

1. Pull 30 to 60 recent incidents where a containing action ran, and compute your current median and tail from the six timestamps in our time to contain guide.
2. Connect the platform in read-only mode to the detection sources and the action targets.
3. Turn on shadow mode if the vendor offers it. Prophet says remediation is previewed before it runs (AI SOC Analyst page, read 2026-11-03), which is a preview and not a full shadow period. In shadow mode the platform proposes actions and runs none, so you can compare its proposals with what your analysts did.
4. Agree the incident replay set: a mix of phishing, credential abuse, malware on an endpoint and a cloud key exposure, taken from your real history.

Use a test copy of your estate where you can. Google's SOAR documentation describes exporting and importing playbooks as ZIP files between staging and production and testing them in a Playbook Simulator (read 2026-11-04). Microsoft's docs say Sentinel automation rules can be exported as ARM JSON for CI/CD (read 2026-11-03). Either lets you promote an automation from a test workspace to production with a recorded change.

The platform proposes, an analyst approves. Prophet says remediation is previewed before it runs (read 2026-11-03). Google's Response Agent defaults to manual approval (docs, read 2026-11-03). Whatever the vendor's wording, the behaviour you want is the same.

Record, for each proposal:

• Would the analyst have taken the same action?
• Was the evidence cited enough to approve without a second look?
• How long did the approval take, and who approved?

This week produces your first view of the platform's error rate and your approvers' latency. If approvals sit for hours, the bottleneck is your on-call process and not the product. Fix it before you automate.

Pick the first three actions from our guide on which containment actions to automate first: session revocation, message purge and indicator blocks. Turn on auto-run for those only, on a defined user group, with a numeric cap per run.

Set these guard rails before enabling:

• An owner who reviews every automated action each morning.
• A kill switch that reverts every automated category to approval-required, tested once.
• Exclusions for privileged accounts, service accounts and production asset tags.
• A short list of conditions under which the pilot stops, such as two wrong actions in a week.

Run at least one planned test of rollback. Revoke a test session, then restore it. Isolate a test host, then release it. File the records.

Take the criteria from day one and score each one with evidence.

• Compute the median and tail again from the pilot's records, and compare them with the week 1 baseline on the same incident mix.
• Count the wrong actions, with the cause of each.
• Have a person outside the SOC read two exported records and answer the five audit questions.
• Recompute coverage against your action catalogue from the vendor's own documentation, not from the sales deck.
• Ask for the price at your volume in writing, with the unit and the term.

Write one page per vendor: pass, fail or partial on each criterion, with the evidence. Share it with the vendor. They may dispute a finding, and that is useful, because disputes produce documentation.

A pilot covers a slice. Record what it did not cover: the asset groups you excluded, the actions you left on approval, the integrations you never connected. Those become the first items in the production rollout plan and the first items in the contract schedule.

If two vendors pass, choose on the criterion your organisation is least able to build around. For a regulated team that is usually the record. For a lean team it is often the effort to maintain the integrations.

A 30-day pilot cannot show rare events, such as a sophisticated intrusion in the first week, nor behaviour across a model or version change. Ask the vendor how it announces changes to model, policy or action behaviour, and what notice you get. Keep the shadow or recommend-only mode available as the fallback you can return to in a minute.

• Exit criteria are written, numeric where possible, and shared with the vendor before the pilot starts.
• A baseline of 30 to 60 past incidents has been computed from source timestamps.
• Shadow or recommend-only mode runs for at least the first two weeks.
• Auto-run starts with session revocation, message purge and indicator blocks only, each with a numeric cap.
• Privileged accounts, service accounts and production tags are excluded from auto-run.
• A kill switch exists and has been tested once.
• Every action's undo has been run at least once and recorded.
• A reviewer outside the SOC has read two exported records.
• Price at your volume, with unit and term, is in writing.
• Untested scope is listed and carried into the rollout plan.