Treat system messages as executable policy
When Legal asks what you instructed the bot to do, pointing at the vendor default is a governance failure.
Anthony Hildoer
June 6, 2026 · 6 min read
When your member-facing bot gives billing advice it should not, Legal asks what you instructed the system to do. A shrug and a link to a vendor default means you have a hole in the System Message layer: the behavior layer where brand, ethics, and clinical boundaries meet code.
Prompts are not "just implementation"
The usual plan: sign the vendor contract, let engineers tune prompts in sprint zero, move on. Prompts get filed next to CSS.
That plan breaks the moment a journalist screenshots the wrong answer. Standing instructions are executable policy: role, scope, prohibitions, escalation. If those rules live on one engineer's laptop, you have prompt debt instead of governance.
Healthcare already governs words that change behavior. Formularies have authors, reviewers, effective dates, and sunset clauses. System messages need the same policy release train.
Rules are separate from facts
Strategy separates System Message from Knowledge:
- System Message: how to behave, what is forbidden, when to hand off
- Knowledge: what is true in your organization this quarter, retrieved and cited
"Do not diagnose" is policy. "This payer requires step therapy for drug X" is knowledge. One blob guarantees you will retrain when you should have edited a corpus, or edit a corpus when you should have changed policy.
Who signs this?
If RACI is missing on a whiteboard, you are early for patient-facing AI:
| Role | Owns |
|---|---|
| Clinical / medical leadership | clinical boundaries, escalation triggers |
| Legal / compliance | prohibited claims, documentation standards |
| Product / operations | scope of service, tone, handoff UX |
| Engineering | implements versions; does not invent policy alone |
The artifact is a system message spec: role, scope, prohibitions, escalation paths, locale rules, and explicit silence behaviors (when the model must refuse rather than guess).
One bad headline costs more than a quarter of careful governance. Executable policy is cheaper than executable damage.
The velocity counterargument
"We will slow innovation." Ungoverned prompts already slowed you: rework, incident response, clinicians who will not touch the tool again. Versioned policy enables speed inside guardrails.
Delegate deliverable
A one-page governance charter: naming convention (sm-member-faq-v2.1), review cadence, emergency rollback owner, and the prohibition list Legal already wants in writing.
Engineering will eventually need version-and-test discipline; the executive failure mode is shipping with no job description for the bot. If it governs behavior, it is policy.