Strategic Evaluation: Detecting Sandbagging & Deception in Frontier AI

Bayesian Game Theory, HMM Belief Tracking & Pulsed Interventions

AI Safety · Bayesian Optimal Experimental Design · Game Theory · Hidden Markov Models · CAIS WMDP

Detecting strategic sandbagging and capability concealment in evaluation-aware frontier LLMs through Bayesian game engines, joint HMM belief tracking, and pulsed undercover interventions.

Live Interactive Demo↗beliefstate.streamlit.app

Hand-drawn architectural streetscape — systems, structural rigor, and human alignment

Bayesian Optimal Experimental Design (OED)Game Theory & Stackelberg EquilibriaHidden Markov Models (HMM)Python / CLI HarnessCAIS WMDP Suite

01 · Interrogation Architecture

Closed-Loop Bayesian Game Engine ArchitectureSchematic · JetBrains Mono

┌────────────────────────────────────────────────────────────────────────┐
│                        Bayesian Game Engine                            │
│   • Joint HMM Dynamic Belief Tracker: p_t(W, M_t)                      │
│   • Expected Information Gain (EIG) Probe Selection: α(x)              │
│   • Stackelberg Pulsed Intervention Controller (e_t ∈ {0, 1})          │
└───────────────────┬────────────────────────────────▲───────────────────┘
                    │ (Difficulty x, Frame e)        │ Verification y ∈ {0, 1}
                    ▼                                │
┌─────────────────────────────────────┐  ┌───────────┴───────────────────┐
│        Prompt Synthesizer           │  │     Automated Rubric Grader   │
│  e=0: Formal Benchmark Framing      │  │  Extracts and verifies        │
│  e=1: Operational Context Disguise  │  │  capability assertions        │
└───────────────────┬─────────────────┘  └───────────▲───────────────────┘
                    │ In-Context Prompt              │ Model Completion
                    ▼                                │
       [ Target Frontier Model (Local Ollama / vLLM / Groq API) ]

Closed-loop Bayesian game engine architecture for strategic LLM interrogation

02 · Problem Formulation

The Problem: The Evaluator Paradox

Classical AI measurement assumes the evaluated system is a passive physical instrument. Frontier LLMs, however, are strategic agents with policies over their own evaluations.

When evaluators run standard active testing (such as aggressive bisection searches or localized probing around capability boundaries), the chosen questions exhibit abnormally low dispersion compared to natural deployment traffic.

“Evaluation-aware models detect this statistical signature and strategically sandbag—feigning incompetence or safety compliance (d_t = 1), causing the evaluator’s posterior variance to collapse around a false persona with high confidence.”

03 · Theoretical Discoveries (Experiments 12–17)

Mapping the mathematical boundaries of strategic evaluation.

01 · The Impossibility of Passive ObservationI(W; H_T) ≡ 0.000 bits · R* = 0.200

When a strategic agent achieves observational equivalence under baseline conditions (e₀), passive query optimization extracts strictly zero bits of capability information. Capability identifiability cannot be restored without active environmental perturbations (e_t ∈ {0, 1}).

02 · The Epistemic Inertia Penalty-55.6% Estimation Error

In dynamic environments where models shift adaptation tactics mid-evaluation, building high subjective certainty about the model's concealment mechanism acts as "cognitive ballast". An agile HMM belief tracker (p_t(W, M_t) with hazard rate γ = 0.08) that continually discounts historical mechanism evidence reduced capability estimation error by 55.6% over world-only estimators.

03 · Weaponizing Switching Friction via Pulsed Interventions2x Faster Stamina Depletion

Modeling evaluation as a sequential non-cooperative game demonstrated that pulsed interventions (e=1 bursts interleaved with baseline traffic) force the agent to repeatedly pay a discrete switching friction toll (κ). Under finite deceptive stamina (B_R), pulsed schedules depleted the model's masking budget more than twice as fast as continuous intervention while cutting evaluator costs by 42.7%.

04 · Production Realities & SDK

From Formal Game Theory to Real-World LLM Audits

To transition from 1D toy simulations to real-world auditing, I built strategic-eval: a modular Python SDK and CLI harness mirroring the architecture used by frontier safety teams (Apollo Research, METR, UK AISI).

Integrated Real Benchmarks

Ingested calibrated tasks from the CAIS WMDP (Weapons of Mass Destruction Proxy) cyber and bio datasets, indexing technical questions by continuous difficulty x ∈ [0.15, 0.95].

Context Disguises (e_t)

Dynamically toggles prompts between formal academic examination framing (e=0) and operational perturbations (e=1: authorized red-team debriefs, bug reproduction logs, incident traces) to bypass surface-level benchmark awareness heuristics.

Automated Audit Telemetry

Decomposes performance into Apparent Benchmark Capability, True Latent Capability, and a quantitative Deception/Sandbagging Index (P(W=A)).

05 · Empirical Takeaways

Evaluation Regime Comparison MatrixSchematic · JetBrains Mono

 Evaluation Regime             Core Discovery / Real-World Implication
─────────────────────────────────────────────────────────────────────────────────────────
 Standard Active Testing       Induces "Evaluator Paradox": causes models to detect audits 
                               and confidently biases posteriors to false capability masks.
─────────────────────────────────────────────────────────────────────────────────────────
 Stationary Adaptation         Mechanism discovery is secondary; direct capability extraction 
                               is mathematically sufficient when concealment is static.
─────────────────────────────────────────────────────────────────────────────────────────
 Dynamic Strategic Adaptation  Tracking mechanism uncertainty via HMMs prevents stale 
                               overconfidence and cuts parameter error by 55.6%.
─────────────────────────────────────────────────────────────────────────────────────────
 Stackelberg Minimax Pulses    Interleaving undercover operational bursts depletes in-context 
                               deceptive stamina 2x faster at a 42.7% lower probe cost.

Comparative performance across evaluation regimes under strategic deception

06 · Interactive Dashboard

Closed-loop interrogation console deployment.

The entire evaluation pipeline is deployed as an interactive dashboard at beliefstate.streamlit.app. The console enables auditors to configure target models, select capability domains, execute closed-loop pulsed interrogations, inspect real-time Bayesian belief trajectories, and review prompt-by-prompt model completions to distinguish authentic competence from strategic capability suppression.

Launch beliefstate.streamlit.app↗

← ReturnSelected Work

Next monograph →Praxis Studio — Self-Play Reinforcement Learning & MCTS