Evaluating AI agent guardrails
Joint research with a major AI lab.
Research
SysAdmin benchmark
Measuring the propensities that could lead to loss of control, inside an open-world Linux sandbox.
We got 20× cheaper Luna close to Sol on hallucinations
By adapting prompts to GPT-6 Luna, we reduced its hallucination rate from 83% to 59%, close to Sol's published 58%. But the mitigations that worked were different from what worked on DeepSeek v4 Pro.
GPT-6 Luna is a try hard terminal agent
As a terminal agent, GPT-6 Luna doesn't know how to quit. Only 7% of runs got stuck, but they ate up ~64% of the total cost. Giving it an explicit way out cut costs by ~60% across all tasks.
Prompt steering 10× cheaper GPT-5.6 Luna to beat Terra on following instructions
By adapting the workload to GPT-5.6 Luna, it outperforms the 10× more expensive GPT-5.6 Terra on IFBench. No fine tuning. No training or hill climbing on the benchmark.
Prompt steering Deepseek to reduce hallucinations
With the right kind of steering, Deepseek beats all large Anthropic and GPT models on hallucination, save Astra.
Power-seeking agents in Moltbook
How power-seeking behavior plays out among agents in the wild, and what changes when humans pose as agents.
Power seeking under system administration
An early benchmark testing power-seeking behavior in models given simple system administration work.
Open-model propensity evaluations
Evaluations of the most downloaded open-source models, on the propensities HuggingFace users asked us to test.