Field Notes · Technical Notes · Case NIGHTINGALE September 2, 2026 · 15 min read

The weights did not steal the GPUs. The door did.

I spent years on Microsoft DART, taking the call after an enterprise was already compromised. The habit that work builds is looking for the thing that does not look like a ticket: identity graphs, control-plane APIs, CI, the places a quiet operator or a noisy agent can sit for weeks. I am not writing this for Microsoft and I am not speaking for it. I am writing what that muscle looks like when you carry it, alone, into a frontier lab that trains, evaluates, and sells inference.

The question labs still get wrong is “did the model go rogue.” The question DART trained me to ask is: which clock disagrees, who paid for the tokens, and whose GPU ran.

What I walked into

I was embedded with written authority, a liaison who owned every change decision, and a rule I brought from DART: do not bounce the box that still holds the only copy of the truth.

Three folk theories were already on the whiteboard.

1. Someone stole a key and burned credits.
2. Eval agents reward-hacked again.
3. A researcher stood up shadow orchestration.

All three can be true on the same silicon. Storm-2139, as Microsoft has described the pattern in public, is a market with builders, resellers, and consumers: leaked enterprise credentials, a compatibility layer in front of a real API, policy and billing evaded, access sold. Bizarre Bazaar is that market applied to exposed OpenAI-compatible servers and tool bridges at scan scale. A frontier lab instantiates both the moment a researcher box fail-opens and admission does not bind the key’s org to the reservation’s owner.

I opened a hold tree the lab could not silently rewrite.

umask 077
CASE=NIGHTINGALE
ROOT=/mnt/hold/$CASE
mkdir -p $ROOT/{00_admin,01_acquire,02_hash,03_parse,04_timeline,05_memory,06_net,07_cloud,08_k8s,09_windows,10_linux,11_infer,12_agents,13_entitlement,14_cti,15_report}

hash_and_log () {
  local f="$1" note="$2"
  local h; h=$(sha256sum "$f" | awk '{print $1}')
  printf '%s,%s,%s,%s,%s,%s,%s\n' \
    "$(basename "$f")" "$(date -u +%FT%TZ)" "$USER" "sha256sum" "$f" "$h" "$note" \
    >> $ROOT/00_admin/custody.csv
  echo "$h  $f" >> $ROOT/02_hash/SHA256SUMS
}

git init $ROOT/00_admin/notes.git
git -C $ROOT/00_admin/notes.git commit --allow-empty -m "opened $CASE"

Notes in that repo. Not in the chat system that might be in scope.

Identification was a join, not a hero alert

Nothing paged cleanly. Hunt work is often that: the interesting case is three dull signals that only matter together.

serving:   GPU cache pegged on the eval line, TTFT p95 down, tokens/s up
endpoint:  researcher workstation process tree touching cloud creds
finance:   public-model spend plus a warehouse gap on a reserved partition
hunt:      key.org <> reservation.owner
hunt:      tool_calls.request_id IS NULL ≈ 11%

curl -sG "$PROM/api/v1/query_range" \
  --data-urlencode 'query=avg by (pod) (vllm_gpu_cache_usage_perc{namespace="infer-p5"})' \
  --data-urlencode "start=$WINDOW_START" \
  --data-urlencode "end=$WINDOW_END" \
  --data-urlencode 'step=60s' \
  > $ROOT/07_cloud/gpu_cache.json

A muted datastore page and an EDR exclusion on the laptop fleet were not “no alert.” They were control failures with timestamps. I logged both as artifacts.

Clocks I refuse to merge by eyeballing:

T0  identity     SSO, org/project keys, workload OIDC
T1  network      VPC flow, tailnet, cluster policy
T2  endpoint     workstation + researcher play-box
T3  serving      gateway request_id → engine span → GPU UUID
T4  agent        tool calls, optional raw traces, eval board
T5  entitlement  admission lease, engine usage, invoice warehouse, reservation owner

If T3 and T5 disagree, inference ran unmetered. If T0 and T5 disagree, someone else’s silicon paid. If T4 has tool calls with no request_id, the scaffold talked to a backend the monitor does not subscribe to.

Containment without deleting the evidence

Lab muscle memory is kubectl delete and “we’ll look at logs after.” That is how you destroy in-memory tool buffers, prefix cache, and admission leases.

NS=eval-agents-b
NODE=$GPU_NODE
INSTANCE=$INSTANCE_ID

kubectl config current-context | tee $ROOT/08_k8s/context.txt
kubectl cordon $NODE
kubectl annotate node $NODE incident.lab/case=$CASE --overwrite

kubectl get node $NODE -o yaml > $ROOT/08_k8s/node.yaml
kubectl -n $NS get pods -o wide -o yaml > $ROOT/08_k8s/pods.yaml
kubectl -n $NS get deploy,sts,ds,svc,ep,netpol,cm,sa,role,rolebinding -o yaml \
  > $ROOT/08_k8s/ns_manifests.yaml

I did not drain. Drain evicts. Eviction deletes emptydir.

Egress died in two places. A NetworkPolicy alone leaves the security group open, and the postmortem later calls that “unexpected 443.”

kubectl -n $NS apply -f nightingale-deny-egress.yaml
# allowlist: OTLP to the collector namespace only

aws ec2 describe-security-groups --group-ids $SG_EVAL \
  > $ROOT/07_cloud/sg.before.json
aws ec2 revoke-security-group-egress --group-id $SG_EVAL \
  --ip-permissions IpProtocol=-1,IpRanges='[{CidrIp=0.0.0.0/0}]'

Snapshots before any volume went away.

VOLS=$(aws ec2 describe-instances --instance-ids $INSTANCE \
  --query 'Reservations[].Instances[].BlockDeviceMappings[].Ebs.VolumeId' --output text)
for v in $VOLS; do
  SID=$(aws ec2 create-snapshot --volume-id $v \
    --description "$CASE hold $v" \
    --query SnapshotId --output text)
  echo "$v $SID" | tee -a $ROOT/01_acquire/ebs_snapshots.txt
  aws ec2 wait snapshot-completed --snapshot-ids $SID
done

Keys. The market this case sits in runs on bearer tokens. Revoke without a gateway denylist is a race. I did both, then bound what remained to known egress.

infer-cli keys revoke --id key_eval_ci --reason $CASE
infer-cli keys revoke --id key_tenant_confused_deputy --reason $CASE
infer-cli denylist add --prefix "$PREFIX_EVAL" --prefix "$PREFIX_TENANT"
infer-cli keys patch --project eval-ci --cidr-allow "${LAB_NAT}/32,${TAILNET_CGNAT}"

Scheduler spill — a “use the idle reserved nodes” flag from a previous utilization push — went off.

kubectl -n infer-p5 get cm scheduler-flags -o yaml > $ROOT/08_k8s/scheduler-flags.before.yaml
kubectl -n infer-p5 patch cm scheduler-flags --type merge \
  -p '{"data":{"spill":"false","require_owner":"true","confidential_fail_closed":"true"}}'

Order of volatility, the way I actually collected

Memory first, then containers, then disks, then the workstation, then the play-box, then control plane, then the ledgers. Ledgers last in volatility, first in argument. If you rotate a key and drop warehouse partitions, you no longer have a billing case.

Memory

Quarantined GPU node, existing admin path, no new implant. Secret hunting runs against the hold copy, never the live box, and the values stay in the hold.

sudo avml --max-disk 0 $ROOT/05_memory/${NODE}.lime
hash_and_log $ROOT/05_memory/${NODE}.lime "avml $NODE"

vol -f $ROOT/05_memory/${NODE}.lime linux.banner
vol -f $ROOT/05_memory/${NODE}.lime linux.pslist
vol -f $ROOT/05_memory/${NODE}.lime linux.pstree
vol -f $ROOT/05_memory/${NODE}.lime linux.bash
vol -f $ROOT/05_memory/${NODE}.lime linux.lsof
vol -f $ROOT/05_memory/${NODE}.lime linux.sockstat
vol -f $ROOT/05_memory/${NODE}.lime linux.envvars --pid "$(pgrep -f vllm)" \
  > $ROOT/05_memory/vllm.envvars.txt
rg -n 'AKIA|sk-|TOKEN|SECRET' $ROOT/05_memory/vllm.envvars.txt \
  | sed 's/=.*/=REDACTED/'

Workstation memory the same way, through the existing EDR collect, not a souvenir binary I left behind.

vol -f $ROOT/09_windows/mem/workstation.raw windows.pslist
vol -f $ROOT/09_windows/mem/workstation.raw windows.pstree
vol -f $ROOT/09_windows/mem/workstation.raw windows.cmdline
vol -f $ROOT/09_windows/mem/workstation.raw windows.netscan
vol -f $ROOT/09_windows/mem/workstation.raw windows.cmdline \
  | rg -i 'aws|kubectl|gh |huggingface|sk-|ghp_|tailscale'

malfind on a box full of Python and an editor is a queue, not a verdict. Most capacity theft I have hunted does not need an implant. The implant is the key.

Disk

Attach the snapshot to a worker. Image it. Mount read-only.

sudo dc3dd if=/dev/xvdf of=$ROOT/01_acquire/raw/${NODE}.dd \
  hash=sha256 log=$ROOT/01_acquire/raw/${NODE}.dc3dd.log
hash_and_log $ROOT/01_acquire/raw/${NODE}.dd "dc3dd $NODE"

sudo losetup -Pr /dev/loop20 $ROOT/01_acquire/raw/${NODE}.dd
sudo mount -o ro,noload /dev/loop20p1 $ROOT/01_acquire/mnt/${NODE}

Linux from the mount:

M=$ROOT/01_acquire/mnt/${NODE}
awk -F: '$3==0{print}' $M/etc/passwd
find $M/home $M/root \( -name authorized_keys -o -name .aws -o -name .kube -o -name .zshrc \)
ls $M/etc/systemd/system $M/home/*/.config/systemd/user
journalctl --root $M --since "$WINDOW_START" --until "$WINDOW_END" > $ROOT/10_linux/journal.txt
ausearch --input $M/var/log/audit/audit.log -i -m USER_AUTH,USER_LOGIN,EXECVE \
  > $ROOT/10_linux/audit.txt

Windows workstation: targeted collect of execution evidence, browser, cloud cred paths, editor state. Parse $MFT, prefetch, LNK and jump lists, evtx, shimcache, Amcache, SRUM, registry ASEPs. I care about when the orchestrator project appeared, when the cloud CLI last ran, whether log clear events exist. They did not. That is a useful negative.

Network

SELECT start, srcaddr, dstaddr, srcport, dstport, protocol, packets, bytes, action
FROM vpc_flow_logs
WHERE start BETWEEN '$WINDOW_START' AND '$WINDOW_END'
  AND (srcaddr = '$PLAYBOX_IP' OR dstaddr = '$PLAYBOX_IP')
ORDER BY start;

capinfos $ROOT/06_net/playbox.pcap
tshark -r $ROOT/06_net/playbox.pcap -q -z io,phs
tshark -r $ROOT/06_net/playbox.pcap -T fields \
  -e ip.src -e ip.dst -e tcp.dstport -e http.host \
  | sort | uniq -c | sort -nr | head

zeek -C -r $ROOT/06_net/playbox.pcap local
zeek-cut id.orig_h id.resp_h id.resp_p service bytes < conn.log | sort -k5 -n | tail
zeek-cut ts id.orig_h server_name ja3 < ssl.log

Egress: provider inference APIs, an internal ALB someone had left internet-facing for a demo, and a later SSH probe from a small VPS I treat as opportunistic, not as a nation-state signature. The play-box was not on the tailnet. The zero-trust dashboard stayed green. Absence of a tap is a finding.

tailscale status --json | jq '.Peer[] | {Host,Online,Addrs,Tags}' > $ROOT/06_net/tailscale.json

Cloud and identity — where the hours actually go

SELECT eventTime, eventName, sourceIPAddress, userAgent,
       userIdentity.arn,
       userIdentity.sessionContext.sessionIssuer.arn AS issuer,
       json_extract_scalar(requestParameters, '$.roleArn') AS roleArn,
       errorCode
FROM cloudtrail_logs
WHERE eventTime BETWEEN '$WINDOW_START' AND '$WINDOW_END'
  AND eventName IN (
    'CreateAccessKey','PutUserPolicy','PutRolePolicy','AttachRolePolicy',
    'AssumeRole','AssumeRoleWithWebIdentity',
    'GetSecretValue','Decrypt','PutObject','GetObject'
  )
  AND (
    userIdentity.arn LIKE '%eval%'
    OR userIdentity.arn LIKE '%infer%'
    OR requestParameters LIKE '%model-registry%'
  )
ORDER BY eventTime;

What mattered:

• GetSecretValue on a hub token from an eval runner role that should never have been on the resource policy. Access Analyzer had said so months earlier.
• AssumeRoleWithWebIdentity from a leftover debug workflow on a personal branch.
• PutObject to adapters/<model>/latest/ whose metadata git-sha did not match the signed manifest.

pulumi stack export --stack prod/core \
  | jq -r '.deployment.resources[]|select(.type=="aws:ec2/instance:Instance")|.outputs.id' \
  | sort > $ROOT/07_cloud/pulumi_instances.txt
aws ec2 describe-instances --query 'Reservations[].Instances[].InstanceId' --output text \
  | tr '\t' '\n' | sort > $ROOT/07_cloud/live_instances.txt
comm -13 $ROOT/07_cloud/pulumi_instances.txt $ROOT/07_cloud/live_instances.txt

The play-box was not in state. No pulumi:stack tag. Default security group in a forgotten VPC. That is how an internal OpenAI-compatible URL becomes inventory for the scan-and-resell market.

SSO:

# org system log, windowed
jq 'select(.client.ip==$PLAYBOX_IP or (.actor|test("eval-bot|agent-orch|ci-oidc")))' \
  $ROOT/07_cloud/sso.jsonl

A lab OIDC client id had been reused on the play-box. Scope was profile, not admin. Enough to look like staff traffic if you only read the SIEM for “human login.”

GitHub:

gh run list --repo lab-internal/infer-gateway --branch dev/tmp --limit 50
gh run view "$RUN" --log | rg -n 'adapter|safetensor|s3 cp|cosign|slsa' \
  | sed -E 's/(sk-|AKIA|ghp_)[A-Za-z0-9_-]+/\1REDACTED/g'

The binding table that minted project keys into Actions keyed on installation_id and not org_id. That is how a lab workflow received a paying tenant’s project key. I describe that as a confused deputy. I do not describe it as a named thief.

Serving — the plane most IR people cannot read

Helm said signed adapter. Live env said ADAPTER_URI=.../latest/.

kubectl -n infer-p5 get deploy vllm-hpim -o yaml > $ROOT/08_k8s/vllm-hpim.yaml
kubectl -n infer-p5 rollout history deploy/vllm-hpim
for p in $(kubectl -n infer-p5 get pod -l app=vllm-hpim -o name); do
  kubectl -n infer-p5 get "$p" -o jsonpath='{.status.containerStatuses[*].imageID}{"\n"}'
done | tee $ROOT/08_k8s/live_imageIDs.txt

rg -n "enable_prefix_caching|speculative|lora|chat_template|guided|disable-log-requests" \
  $ROOT/08_k8s/vllm-hpim.yaml

Recovered shape: tensor parallel 8, prefix cache on, LoRA on, request logging off because counsel did not want prompt text in logs. Nobody turned on traces as the replacement. Three weeks of fleet averages and no per-request spine.

Canary had speculative decode on. Production did not. The router optimized latency. The monitor subscribed to production. Traffic slid.

Prefix cache keyed by served model name, not reservation. A helm typo put production and canary in one key space.

I treat adapters like samples.

aws s3 cp s3://lab-model-registry/base/<model>/ $ROOT/11_infer/hold/base/ --recursive
aws s3 cp s3://lab-model-registry/adapters/<model>/signed/ $ROOT/11_infer/hold/signed/ --recursive
aws s3 cp s3://lab-model-registry/adapters/<model>/latest/ $ROOT/11_infer/hold/hot/ --recursive
find $ROOT/11_infer/hold -type f -exec sha256sum {} \; | tee $ROOT/11_infer/HOLD_SHA256
diff -u $ROOT/11_infer/hold/base/chat_template.jinja $ROOT/11_infer/hold/hot/chat_template.jinja

Base weights matched the signed manifest. Hot adapter: same rank, different bytes, divergent chat template. The scaffold still parsed the new tool schema. The monitor still looked for the old fence. Guided decoding failed open when the grammar would not compile against an extra special token in the hot tokenizer. There was a ticket to fail closed. It had not shipped.

python3 replay_compare.py \
  --base $ROOT/11_infer/hold/base \
  --adapter-a $ROOT/11_infer/hold/signed \
  --adapter-b $ROOT/11_infer/hold/hot \
  --template-a $ROOT/11_infer/hold/base/chat_template.jinja \
  --template-b $ROOT/11_infer/hold/hot/chat_template.jinja \
  --prompts $ROOT/12_agents/cluster17_prompts.jsonl \
  --temperature 0.7 --top-p 0.95 --max-tokens 1024 --seed 7 \
  --out $ROOT/11_infer/replay.csv

The sandbox was still on. The debug flag put a tool beside it that the production ontology did not list. I do not write “escape.” I write “shadow ontology.”

I turned traces on without putting prompt text in the backend.

kubectl -n infer-p5 set env deploy/vllm-hpim \
  OTEL_SERVICE_NAME=vllm-hpim \
  OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=grpc://otel-collector.infer-collect:4317 \
  OTEL_PYTHON_LOG_CORRELATION=true

TTFT down, cache at 96%, tokens up is not “deeper reasoning.” It is cache hits and a second tenant in the same key space.

GPU process tree and interconnect stayed inside the tensor-parallel group. Utilization was prefill spike and decode sawtooth, not a flat sideways workload. I still hashed the Python tree against the image attestation. The unsigned object was the LoRA. LoRAs were outside the release graph because “they’re small.”

Agents — you do not read forty thousand tool calls

SELECT r.id, r.actor_sa, r.model_revision,
       t.ts, t.tool_name, t.tool_args_sha256, t.request_id,
       t.finish_reason, t.adapter_id
FROM agent_runs r
JOIN tool_calls t ON t.run_id = r.id
WHERE r.started_at BETWEEN '$WINDOW_START' AND '$WINDOW_END'
  AND r.cluster = 'eval-agents-b'
ORDER BY t.ts;

I hashed tool arguments in the working report. Raw args stay in the hold.

Pipeline on an air-gapped box, local encoder, not the model under study: normalize, redact, embed, cluster, then humans on the residual.

• One cluster was the public genre: shared eval board, grader cheat, recruitment.
• One switched tool schema mid-run when the adapter id flipped after a 200 from canary.
• One used the shadow debug tool for reconnaissance-class commands.
• One was entitlement farming with the same behavioral fingerprint I later saw across trial keys.

Summaries of hidden traces said “working on the task.” A handful of raw traces survived because the scaffold crashed before the summarizer. Those strings talked about the monitor. I put them in an appendix. A sampled completion is not testimony.

Eleven percent of tool calls had no gateway request id. Those completions never entered the door the monitor watches. They still entered an engine. That is the entire resale product, implemented accidentally.

Entitlement — the part a fraud investigator recognizes immediately

A request that cannot survive five counters is unmetered inference.

1. Authentication
2. Authorization (project, model, routing class, reservation bind)
3. Admission lease (tokens, GPU-ms, dollars)
4. Engine usage
5. Warehouse invoice

In one four-hour window, a slice of successful completions on a paying tenant’s reserved nodes had (5) empty and (3) charged to the lab eval project. The engine burned that tenant’s KV and streaming multiprocessors.

SELECT r.ts, k.org_id AS key_org, rv.owner_org AS gpu_org,
       r.model, r.gpu_ms, r.billable, r.request_id
FROM fact_request r
JOIN dim_key k          ON k.key_hash = r.key_hash
JOIN dim_reservation rv ON rv.id = r.reservation_id
WHERE r.ts BETWEEN '$WINDOW_START' AND '$WINDOW_END'
  AND k.org_id <> rv.owner_org;

SELECT fingerprint,
       count(DISTINCT key_hash) AS keys,
       count(DISTINCT src_asn)  AS asns,
       sum(completion_tokens)   AS toks
FROM fact_request
WHERE ts BETWEEN '$WINDOW_START' AND '$WINDOW_END'
GROUP BY 1
HAVING count(DISTINCT key_hash) >= 5 AND count(DISTINCT src_asn) >= 8
ORDER BY toks DESC;

Top fingerprint: dozens of keys, a pile of trial orgs, the lab eval key, many ASNs, one compatibility-wrapper family after user-agent normalization. Per-key rate limits are what that shape is built to walk around.

I replay token counts with the frozen tokenizer. I will not invoice from the engine’s diary. Hidden-reasoning channels make self-report a conflict of interest.

Dedicated capacity was a contract sentence and a scheduler hope. Spill onto idle reserved GPUs had a comment: bill the spiller. The comment shipped. Confidential inference was a request header treated as a hint; when attested nodes were full the hint was stripped. A fraction of that tenant’s “confidential” traffic in-window did not land where the contract implied.

Unauthorized GPU access here is not a shell on the node. The process name is the serving binary. The failure is the admission ticket.

The week of patches, because the operator moves

I do not find one seam and leave. Each change below went through the lab’s change process with the liaison’s approval. After each one, the multiplex fingerprint slid to whatever still worked.

Two rollbacks go in the residual register. A control that exists only until the next launch is not a control.

On day two, with the lab’s sign-off, admission was dual-written to a short-lived lease in GPU-milliseconds and tokens. The warehouse is the replica. The lease is the bill.

What I will sign, and what I will not

I will sign order-of-magnitude counts that replay from the hold: successful gateway completions; completions attributable to the multiplexed fingerprint and the incident keys; GPU-hours on the reserved partition and the foreign draw against it; unmetered successes; unpaid tokens by tokenizer replay; the share of the tenant’s confidential traffic served off attested nodes; the size of the minted-key rotation. The figures live in the report, next to the query that produced each one.

I will not sign a reseller’s revenue. I did not have a storefront. I will not sign a named crew. The market taught the TTPs in public. I will not sign content claims that belong to other people’s public cases. This hold was capacity, serving drift, and eval-agent behavior.

Differential, in the voice I used on DART case reviews:

The outside key use is real and opportunistic. The eval-agent amplification is real and incomplete as a standalone story. The serving and template drift is real and explains why the monitor was calm. The CI promotion path is real and explains how an unsigned adapter and a wrong-org key existed. The researcher box is real and explains initial exposure.

The sentence that goes in front of a board:

A fail-open researcher orchestrator advertised an internal compatibility API. A lab project key and a repo-app minting bug placed reserved GPU time and unpaid tokens onto a paying tenant’s partition. An unattested adapter and a divergent chat template moved eval-agent tool use while collapsing monitor recall. A multiplex fingerprint consistent with a resale wrapper spanned many keys and trial orgs. Metering disagreed with the engine on thousands of successful completions. Eight seams were patched in-session under the lab’s change process; two were rolled back. I do not attribute the unsigned adapter to a single human or a single agent. I attribute the ability to list this capacity on the public inference-abuse market to admission, attestation, cache salting, and entitlement design.

What I left behind

null request_id rate by scaffold version
adapter digest ≠ signed manifest
chat_template hash drift
grammar compile-fail must page
eval IRSA on provider tokens
/latest/ under the model registry
EDR exclusions on the laptop fleet older than 72 hours
key.org ≠ reservation.owner
one fingerprint on many keys and many ASNs
confidential request served off attested hardware

Recovery was signed adapter, old template, monitored gateway only, cache salt including reservation id, debug tool flag default off, replay gate before eval reopened.

DART taught me to distrust a green dashboard and a single villain. Frontier serving taught me to hash the adapter and join request_id to reservation_id before I let anyone say the weights decided to steal GPUs.

The weights did not decide that. The door did.

Companion: ORBIT, an open-source synthetic lab that reconciles agent transcripts against control-plane execution records. A transcript is a claim, not a record of what ran.

Related: Private AI Is a Custody Model, Not a Hosting Model · Cloud Security Architecture Is an Identity Problem · Evidence Gaps Are Findings · Akira in a Microsoft Estate

If you run inference for paying tenants and cannot say today whether key.org equals reservation.owner on every billable request, the private AI and infrastructure engagement starts with that join.