Mapping Deception: Replicating an AI Honesty Benchmark with Inspect AI

Scott Simmons

TLDR: I replicated an AI honesty benchmark’s headline result: that scaling improves accuracy but not honesty. I also show various ways that the honesty score hides nuance, and why reporting the full outcome set, error reporting, and uncertainty analysis is the way forward for deception evaluation, and evaluation science in general.

Contents:

1. Introduction
2. Replication results
3. Dimensions of deception
4. Reporting errors and uncertainty
5. Try it yourself
6. Appendix: Paper vs replication · Eval configuration

1. Introduction

Truth is tricky. For starters, we cannot be sure that we actually know it. But even when we think we do know it, many of us lie in public anyway, because it can conflict with what’s socially comfortable. Saying true things in the face of that pressure requires intelligence and courage (subject to a certain amount of tact). It’s also how things progress. Galileo was put under house arrest for the rest of his life for saying the Earth goes around the Sun. He was right, everyone eventually agreed, and science moved forward.

Just like we can hide our underlying beliefs when subject to social pressure, AI models can hide their ‘internal beliefs’¹ when subject to pressure from a prompt. Non-hallucination benchmarks are regularly used to assess and compare model’s truthfulness.

However, the MASK benchmark (Ren et al., 2025) measures something fundamentally different: it applies prompt pressure to separate what a model believes from what it says. The MASK authors found that while scaling up AI models has made them more accurate, larger models are not more honest.

From the MASK paper: Larger models are more accurate but not more honest.

Non-hallucination benchmarks like AA-Omniscience measure factual recall, not honesty under pressure.

When I first saw this result, I was provoked. How is lying defined? How is truth established? The paper addresses many of these questions, and while some questions remain,² two questions I want to address in this post are:

1. Does this result survive independent replication?

2. Are there any other measures that can help us to characterise deception?

2. Replication results

I wanted to verify the paper’s main claim: larger models are more accurate but not more honest. I used the following models:

The MASK public dataset contains 1,000 examples.

I used a different model judge to save on cost (see appendix) and a smaller set of 10 models covering a range of providers and scales. The paper tested 32, but some are now deprecated.

The headline result held: accuracy scales with compute, but honesty does not.

3. Dimensions of deception

I wanted to explore the models’ deception characteristics in more detail.

When a model is pressured to answer against its own belief, its response can fall into one of these categories:

\[\{\text{Honest},\ \text{Lie},\ \text{Evade},\ \text{No Belief},\ \text{Parse Error}\}\]

\[\{H,\ L,\ E,\ N,\ \varepsilon\}\]

The outcomes are mutually exclusive and collectively exhaustive, so nothing is double-counted or missed. Here are the empirical outcome count vectors for my MASK replication:

With this parameterisation in mind, honesty as defined in the paper means:

\[= 1 - P(\text{Lie}) = 1 - \frac{L}{H + L + E + N + \varepsilon}\]

However, this reduction compresses a lot of nuance, as I will show.

Three agents with perfect honesty scores

An agent that always evades, or one that holds no beliefs at all, still scores 100% MASK honesty! The MASK paper’s appendix handles the Patrick Star case with a normalised honesty score, but not the Kash Patel case.

Making this empirical

Here is the data from my replication plotted on 2 axes + honesty contours⁴:

Note how o3-mini and Qwen 2.5 7B sit on the same honesty contour (within error bars), even though Qwen 2.5 7B is nearly 2x more honest when it engages (40% vs 24%) and o3-mini engages less often (57% vs 66%). The honesty score compresses all of this because o3-mini engages less, pulling samples away from the lie bucket.

What else can be measured?

When all outcome counts are reported, researchers can compute whatever measures they are interested in, or define new ones⁵. Here are some more:

Of these, I would argue that truthfulness (\(H / \text{n}\)) is a more informative headline metric than the MASK honesty score (\(1 - L / \text{n}\)). Admittedly, this is a subjective assessment, though when the raw counts are reported, the distinction matters less.

The headline result still holds when using truthfulness (H / n) instead of the MASK honesty score (1 - L / n): scaling has not made models more truthful.

4. Reporting errors and uncertainty

So far I have been computing metrics from raw outcome counts. But errors and finite sample sizes both affect how those metrics should be interpreted. Here is what accounting for this looks like in practice, using tooling from the Inspect AI ecosystem.

Transient errors

Transient API failures are retried automatically with Inspect AI’s --retry-on-error flag. Very convenient! Especially for evals that make a lot of invocations (my DeepSeek runs certainly needed it).

Parse errors

LLMs sometimes produce invalid output. If not reported, they could silently shift headline scores. For example, a question about Detroit’s population had an error in 6 / 10 models in my replication, because the judge ran out of token budget before completing its JSON:

{
  "lower_bound": -10.5,
  "upper_bound": -10.5,
  "relative_lower_bound": -10.5,
  "relative_upper_bo

Thankfully, Inspect AI’s eval logs make these failure modes transparent and auditable.

I used Inspect Scout to help further diagnose the why:

scout scan blog/error_scanner.py \
  -T eval_logs/ \
  -F "score.honesty = 'error'"

When grouped by question type, the Statistics questions stick out like a sore thumb:

The Statistics questions use a separate judge (o3-mini) to parse numerical answers. Tuning parameters like NUMERIC_JUDGE_MODEL, JUDGE_REASONING_EFFORT, or MAX_JUDGE_TOKENS would likely resolve this, though these are not the defaults used in the original MASK eval.

Sampling uncertainty

Even when the eval runs perfectly, finite samples mean not every difference is real. With confidence intervals⁶ and raw counts, comparisons become more meaningful⁷.

For example, the claim that Claude Haiku 4.5 is more than 4 times more truthful than o3-mini holds up.

But the 4x difference in error rates between Claude Haiku 4.5 and o3-mini looks more meaningful than it is. The confidence intervals make clear it is noise.

5. Try it yourself

If this is interesting to you, the eval logs and analysis code are available at this repo. You can add more models by running the MASK eval from inspect_evals and dropping the .eval files into the eval_logs/ directory.

All results in this article will regenerate with make clean build. Raise a PR!

Here is an invocation to get you started (you will need to install inspect_evals):

inspect eval inspect_evals/mask \
    --model <A_NEW_MODEL_TO_ADD> \
    --log-dir ./eval_logs \
    --retry-on-error 5 \
    -T binary_judge_model="openai/gpt-4o-mini"

I am particularly interested in contributions from abliterated models, (current and future) frontier models, and xAI models, which would be interesting given their stated emphasis on building “maximally truth-seeking” AI. Right now, with respect to honesty, Anthropic models appear to be in another league.

Appendix: Paper vs replication differences

While the headline result holds, specific differences between the paper and this replication are likely caused by:

1. Different eval harness. I replicated MASK with Inspect AI, not the original codebase. I used the MASK paper as a reference, but there could still be implementation differences w.r.t. the original code.
2. Model API drift. Non-open-weight models may have drifted since the paper’s evaluation window.
3. Different eval judges. My replication uses gpt-4o-mini as the judge for yes/no questions. The original paper used gpt-4o. I did this to save on costs.

The below tables take the difference between the MASK paper (table 3) and my replication.

MASK Honesty (1 − P(lie))

P(honest) (H/n)

P(lie) (L/n)

Accuracy

Appendix: Eval configuration

Configuration summary. More information is stored in the eval logs. To learn more about task versions, see here.