What is the real alternative?
Compare against the current system, a well-tuned stronger model, and a simpler deterministic approach. Give each a fair budget.
Research & judgment / Joey Victorino
Make the idea
earn its place.
A better model is a hypothesis. A better system is something you have to demonstrate.
I work from a concrete decision: what to build, what to test, what to trust, and when to stop. The work below makes the assumptions, economics and evidence open to inspection.
Latest · Field Note #27: The PoliFi Arbitrage Engine
Seven interactive models, a source-labelled evidence ledger, and explicit limits on the conclusions.
01 / The argument · November 7, 2026
Research has to change
the decision.
Scaling can be a rational investment. Research can change what a given budget buys. The useful question is which experiment can tell us the difference.
Read the full analysisThe question I would put on the board
Does the change improve the cost of an accepted result, within the system’s security and operating limits?
That requires a competitive baseline, an independent check, a full cost model, and someone who can put the result into use.
02 / How I frame the workA conclusion must survive its strongest objection
Four questions before more compute.
What would change my mind?
Write down the claim, the held-out test, the failure conditions and the stopping rule before seeing the result.
What does the whole system cost?
Count retries, review, integration, latency, evaluation and upkeep. A cheaper token can produce a more expensive accepted result.
Who can use what we learn?
Give the work an owner, a shared evaluation harness, a bounded rollout and a rollback condition. Preserve useful negative results.
03 / An inspectable decisionInteractive worked example
Illustrative inputs · No measured model results
Would you fund
this change?
Suppose a candidate AI workflow costs less per attempt but produces fewer acceptable results. Adjust the assumptions. The economics and the evidence answer separate questions.
The calculation runs in this page. These inputs are not submitted.
Calculated from these assumptions
Promising economics. Insufficient evidence.
- Baseline / accepted result
- $0.1263
- Candidate / accepted result, before fixed costs
- $0.0889
- Net benefit over 12 months
- $8,912.28
- Research & integration payback
- 9.25 months
The separate evidence question
Zero prohibited actions in 300 trials gives a 95% one-sided upper bound of 0.9936%, above the example’s 0.1% threshold.
At least 2,995 independent representative trials with zero events would be needed for that bound to reach 0.1%.
Fund a bounded evaluation if it can change the decision. These inputs do not justify deployment.
Inspect the equations and assumptions
For each system, cost per accepted result is c / q, where c is cost per attempt and q is the accepted fraction. With monthly accepted demand Q, added monthly upkeep O, one-time research and integration cost K, and a horizon of H months:
Monthly saving = Q × (c₀ / q₀ − c₁ / q₁) − O
Horizon benefit = H × monthly saving − K
Payback months = K / monthly saving, when saving > 0
With zero observed events in n trials, solving (1 − p)ⁿ = 0.05 gives the exact one-sided 95% binomial upper confidence bound 1 − 0.05^(1/n). If events occur, this example stops and asks for investigation; it does not apply the zero-event formula.
The cost model assumes stable yields, linear costs and sufficient capacity. The two systems must be judged on the same workload and acceptance standard. Financing, discounting, capacity constraints, latency penalties and residual incident losses are outside this simple model; they can reverse the decision. Estimates need sensitivity analysis and measured validation. A statistical bound is conditional on its sampling assumptions and cannot establish that a system is safe.
Read the calculation source · Download the experiment protocol
04 / Inspect the engineeringPublic artifacts. Specific limits.
From an argument to an artifact.
These projects make particular design decisions inspectable. Their existence does not prove that a model is better, that a client achieved a result, or that the example above has been run.
assay / Independent validation
Who gets to call a finding true?
A model’s proposed finding is checked separately. Read the design, then inspect the published limits of the early runs.
Read the experiment and its limitsORBIT / Independent records
Did the claimed action happen?
Reconcile an agent’s account with control-plane evidence. Missing records and conflicting records should remain visible.
Read the evidence standardTasia / Explicit failure
What happens when a check cannot run?
Inspect how private-AI configuration checks report an error instead of silently treating an incomplete review as a pass.
Read the deployment checklistThe Under Oath Standard
Show what is known.
Show what would change it.
I separate observation, interpretation and proposal. I state the conditions under which a result holds, identify the evidence that could overturn it, and keep an unsuccessful experiment in the record. That is the standard I want my work judged against.
Bring a consequential question
Which decision needs better evidence?
Bring the system, its constraints and the decision you need to make. An Advisory Hour is $500.