Research · Ongoing
The Private AI Exposure Index
Measuring how private AI systems are actually exposed, and whether they can be investigated when something goes wrong.
The Private AI Exposure Index is an ongoing research project examining the security exposure and forensic defensibility of real-world private and self-hosted AI systems.
The study looks beyond whether an inference server is reachable from the internet. It examines the surrounding system: identity, models, data, retrieval, agents, tools, credentials, logging, provenance, containment, and recovery.
The methodology is being published before the final results. Observations will remain source-backed. Unknowns will remain unknown. Results will be published only when the evidence supports them.
Research status
Methodology development and data collection in progress.
01 · Why
Private AI changes the incident.
Traditional incident response often reconstructs a short chain of relationships. A private AI system can introduce a much larger evidence chain, and determining what happened may require evidence across every layer of it.
The Index is designed to measure not only whether a system can be reached or abused, but whether an investigator could reconstruct the event afterward.
Conventional incident
- User
- Process
- File
- Network connection
- Destination
Private AI incident
- User
- Application
- System instructions
- Conversation state
- Retrieved data
- Model and model version
- Inference configuration
- Generated output
- Agent decision
- Tool invocation
- Credential
- External system
- Resulting action
02 · Measures
What the Index measures.
The research currently treats three measurements separately. Security exposure and forensic defensibility are different properties. A system can have meaningful exposure while maintaining excellent evidence and response capability. Another can be network-restricted while providing almost no reliable way to reconstruct an AI-related incident.
1 · Exposure
How much consequential attack surface appears to be exposed by the deployment architecture and configuration.
- Ingress and reachability
- Identity and authorization
- Execution and tool privilege
- Data and retrieval
- Model and software supply chain
- Secrets and external dependencies
2 · Forensic defensibility
Whether an organization could investigate and respond to an AI-related incident using reliable evidence. Based on a modified incident-response model derived from PICERL.
3 · Evidence coverage
How much of the deployment can actually be established from available evidence.
Missing information is not automatically treated as a security failure. Unknown is recorded as unknown.
03 · Framework
PICERL-AI
A DFIR-derived model for private AI incident response.
The research uses a modified PICERL sequence as an investigative framework because AI environments create evidence and recovery problems that differ from conventional endpoint investigations. The framework is not presented as a replacement for current incident-response standards. It is an analytical instrument for this research.
Preparation
Can the organization establish what constitutes the AI system before an incident occurs?
Potential evidence includes model inventory, model versions and hashes, runtime versions, containers and images, system instructions, RAG sources, vector stores, agents and tools, identities and credentials, logging configuration, and architecture and network boundaries.
Identification
Can investigators determine what actually happened?
Which user or service initiated the request? Which model actually ran, and what exact version? What instructions were active? What context entered the model? What documents were retrieved? What tools were invoked? What credentials were used? What output was generated? What downstream actions occurred?
Containment
Can the organization stop harmful activity without unnecessarily destroying evidence?
Isolate inference services. Revoke agent credentials. Disable tool access. Freeze model changes. Quarantine retrieval sources. Preserve volatile evidence. Prevent additional downstream actions.
Eradication
Can the organization find and remove the state responsible for the incident?
A malicious or altered model. A compromised container. An unauthorized adapter. Poisoned RAG content. Changed system instructions. A compromised credential. A malicious tool or MCP component. An unauthorized configuration.
Recovery
Can the environment be restored to a known-good and defensible state?
Verified models. Known-good configuration. A clean retrieval corpus. Rebuilt vector indexes. Rotated credentials. Restored service. Validated integrations.
Lessons learned
Can the organization produce a defensible explanation?
What happened. When it happened. Why it happened. What evidence supports the conclusion. What remains unknown. What changed afterward.
04 · Evidence
Evidence before conclusions.
Observations in the Index may receive an evidence grade. The grade records how the observation is known, not how alarming it sounds.
Reported
A person or source states that a configuration, architecture, or behavior exists.
Demonstrated
Configuration, code, architecture, screenshots, or other technical material supports the claim.
Observed
Logs, outputs, artifacts, or other direct evidence demonstrates the behavior.
Reproduced
The behavior can be independently reproduced or directly verified under controlled conditions.
Unknown ≠ failure
If a source does not establish whether authentication is enabled, authentication is recorded as unknown. It is not automatically scored as absent.
This evidence model is preliminary and may change before the first formal Index release. Changes to the methodology will be documented.
05 · Scoring
Scoring model under development.
Candidate observations may eventually use an ordinal severity scale:
No overall Index number exists yet, and none is displayed here. Weights, aggregation rules, and thresholds will not be finalized until the underlying observations have been examined for bias, coverage, and repeatability.
06 · Sources
Where the evidence comes from.
Five source classes are planned or in use. None is a completed dataset.
Public deployment discussions
Technical discussions where operators describe real architectures, configurations, failures, and operational decisions: Hacker News, Ask HN and Show HN threads, Reddit communities such as r/LocalLLaMA, r/sysadmin, and r/selfhosted, runtime-specific communities, technical blogs, and engineering writeups. These sources are self-selected and cannot automatically be treated as representative of all private AI deployments.
Technical issue reports
GitHub issues and discussions, bug reports, deployment troubleshooting, and configuration examples. These can provide unusually detailed evidence about runtime versions, commands, environment variables, proxy design, authentication, containers, and failure modes.
Passive internet measurement
Publicly available measurement datasets and search platforms may be used to characterize observable internet-facing AI infrastructure. The project prefers passive or previously published measurement where possible. This research does not attack or interact with third-party systems.
Published security research
Vulnerabilities, incident reports, forensic research, runtime security research, and supply-chain research. Software vulnerabilities are kept separate from deployment and configuration failures.
Controlled laboratory testing
Original experimental work in purpose-built reference environments, described below.
07 · The Lab
The Private AI DFIR Lab.
Public observations can show what people deploy. Controlled experiments are required to determine what evidence those systems actually leave behind when something goes wrong.
Planned reference environments include Ollama, Ollama with a web interface, a llama.cpp server, vLLM, containerized inference, RAG and vector retrieval, agentic systems, and MCP or comparable tool integrations. Not every environment is built yet.
What we intend to break.
Planned controlled scenarios may include:
- Unauthorized inference
- Model substitution
- Altered model provenance
- Configuration changes
- Compromised serving container
- RAG poisoning
- Malicious or incorrect document ingestion
- Changed system instructions
- Agent credential misuse
- Unexpected tool invocation
- Secret exposure through context
- Unauthorized downstream action
- Failed update and rollback
- Recovery from known-good state
For each scenario, the research asks: Can the incident be identified? Can its scope be established? Can evidence be preserved? Can the harmful state be contained? Can the cause be removed? Can the system be restored to a known-good state? Can the resulting conclusion be defended from evidence?
08 · Method Example
Method example · Not a published finding
RAG poisoning, walked through the framework.
Scenario: a document containing misleading or malicious information is introduced into a retrieval corpus.
Is the source corpus inventoried and versioned?
Can investigators determine which document was retrieved for the affected response?
Can the source be quarantined while preserving evidence?
Can derived embeddings, indexes, or cached artifacts associated with it be identified and removed?
Can the retrieval system be rebuilt from a known-good corpus?
Can investigators determine when the document entered the system, who introduced it, which responses used it, and what changed afterward?
A conventional vulnerability checklist would not surface most of these questions. That gap is what the Index measures.
09 · Reproducibility
Reproducibility.
Where practical and legally appropriate, the project commits to a public methodology, versioned methodology changes, published field definitions, source provenance, explicit unknown values, reproducible scoring rules, human verification of observations, controlled lab configurations, publication of the code used for analysis, and a clear separation between observation and inference.
LLMs may assist with discovery and preliminary extraction, but they are not the final authority for published observations or scores. Published observations are human-verified.
10 · Limitations
Limitations.
- Public discussions are self-selected.
- Internet-visible systems are not representative of all private deployments.
- Configuration descriptions may be incomplete.
- Software and deployment practices change quickly.
- Different runtimes produce different forensic artifacts.
- An unknown control cannot be assumed absent.
- Observed exposure does not prove exploitation.
- A software vulnerability does not automatically mean a deployment is vulnerable.
- Laboratory results do not automatically generalize to every production environment.
- The scoring model is still under development.
11 · Principles
Research principles.
Evidence before inference.
Unknown before assumption.
Deployment before product label.
Reproduction before certainty.
Security exposure and forensic readiness are measured separately.
A dramatic result is not more valuable than a defensible one.
12 · Status
Current status.
13 · Contribute
Have a private AI deployment worth studying?
If you operate a private or self-hosted AI environment and are willing to contribute architecture or operational information to the research, I would like to hear from you. Useful contributions include anonymized architecture, deployment configuration, incident lessons, logging approaches, forensic artifacts, failure modes, and recovery experience.
Do not submit confidential material through the public form. NDA and secure channels are available where appropriate.