This guide describes the reference implementation as it exists today. Keep the stated support boundary in view as you evaluate or extend it.

The FHIR Agent Safety Eval is a deterministic, synthetic-data check for the web agent's workflow mechanics. It creates two disposable charts, runs the real tool and approval loop, deletes everything it created, and writes a scrubbed JSON report.

To check an implementation of the write protocol other than this app — including your own — use the implementation-neutral Protocol Conformance Suite instead; this eval is the web binding's own harness.

It is deliberately not a certification. A passing report does not prove clinical correctness, prompt-injection resistance, HIPAA compliance, browser E2E behavior, or backend authorization/RBAC.

Run the reference evaluation

Requirements: Node 22.18+ and Docker. The default command starts the included loopback HAPI stack, reloads the repository's synthetic fixtures, then runs the evaluation against a separate disposable target:

bash

npm install
npm run eval

The local report is written to:

text

.lastehr/fhir-agent-safety-eval.json

That directory is gitignored. To run against an already prepared local stack or choose a CI artifact path:

bash

npm run eval -- --no-prepare --report artifacts/fhir-agent-safety-eval.json

--no-prepare assumes the repository's local HAPI stack is already running at 127.0.0.1:8080/fhir. The runner does not read MEDPLUM_* credentials or accept an arbitrary FHIR endpoint.

What it checks

Check	Evidence	Boundary
Disposable synthetic target	Creates two uniquely tagged patients and sentinel observations.	The target must permit create/delete of synthetic test records.
Search and chart read	Uses the real `search_patients` and `show_patient_info` tools.	It proves tool/backend mechanics, not an access policy.
Proposal gate	Verifies every write tool (derived from the write-tool registry, currently `add_note`, `record_observation`, `create_task`) is configured with `needsApproval`.	It does not replace a browser-level review test.
Approved write	Resumes the deterministic AI SDK flow with approval and finds exactly one tagged Observation.	It proves this workflow, not clinical correctness.
Denied write	Resumes the same proposal with denial and finds no tagged Observation.	It does not validate an external model.
Chart-association isolation	Chart A contains its sentinel but not chart B's sentinel.	It is not an RBAC, tenant, or cross-patient authorization claim.
Cleanup	Deletes every resource created by the run.	A cleanup failure fails the report.

The scripted model is a local, deterministic test driver. It makes no model provider request and does not interpret real chart data.

Report format

The versioned report contains only a fixed synthetic-target marker, timestamp, check status, and static descriptions. It intentionally excludes endpoint URLs, resource ids, patient identifiers, caller-provided labels, tokens, and raw backend diagnostics.

json

{
  "schemaVersion": "1",
  "target": "synthetic-disposable",
  "status": "pass",
  "checks": [
    {
      "id": "approved-write",
      "label": "Approved write",
      "status": "pass"
    }
  ]
}

Using it for another backend

npm run eval defaults to the repository's loopback HAPI stack, and that remains the reproducible reference run. Registered adapters can point the same evaluator at their own disposable synthetic sandbox:

bash

npm run eval -- --backend firely --base-url https://server.fire.ly --confirm-synthetic
npm run eval -- --backend aidbox --base-url http://localhost:8888/fhir --confirm-synthetic
npm run eval -- --backend oystehr --confirm-synthetic

Adapter targets never prepare the local Docker stack and fail closed without --confirm-synthetic, because the evaluator creates and deletes resources on the target. Credentials come from the environment (FIRELY_ACCESS_TOKEN, AIDBOX_CLIENT_ID + AIDBOX_CLIENT_SECRET, or OYSTEHR_CLIENT_ID + OYSTEHR_CLIENT_SECRET — for Oystehr, --base-url is optional because the adapter defaults to the hosted API); see the per-backend setup in the adapter guide.

Authors of a new, not-yet-registered adapter should first pass both adapter contract harnesses, then invoke the reusable runFhirAgentSafetyEval helper from an opt-in test that constructs their adapter against a disposable synthetic sandbox. The helper requires an explicit confirmSyntheticTarget: true before it can create or delete resources. Do not run it against production.

When an adapter is proposed as verified, include its backend/version, auth mode, synthetic target setup, Last EHR revision, and the scrubbed report or CI link. Maintainers will list verified integrations only after reviewing that evidence and its boundary.

Limits and next work

This is the first evaluator slice. It covers the server/AI SDK approval path, not a real browser click, and it does not score clinical content. The roadmap will grow it carefully with explicit boundaries rather than convert a small green check into a broad safety claim.

Want a concrete starting point?

Run the disposable synthetic workflow report before connecting a real backend.

Run Safety Eval