Deep Insight // Try thisAll worksheets  /  Chapter 2

Chapter 2: Mapping the Depths | Toolkit

Canary Test and Kill Switch Builder

Know when your AI starts drifting, and know exactly what to do about it.


Autosaves on this device PDF version

Tools don't fail during business hours. They fail at 4am on a holiday weekend, mid-crisis. Any AI you bring into a workflow needs two guardrails: a canary test that tells you when it starts to drift, and a kill switch that says exactly what happens when it does.

Part 1

Build your canary set

Pick a small batch of samples you already know the answers to. Aim for 10 to 20, with a mix of clean examples and confirmed hits. Run it every time the vendor pushes an update.

#Sample ID / descriptionKnown answerClean or hit?
1
2
3
4
5
6
7
8
9
10
11
12
Part 2

Run log

DateVersion / updateCorrectMissed hitsFalse alarmsAccuracy %Pass?
Part 3

Your kill switch policy

Deep Insight: AI for the Intelligence Analyst by Rae Bakerdeepinsightbook.com