detector-scope-peer-review-gpt4o

Detects GPT-4o-written reviews of machine-learning papers (trained on the AI Peer Review Detection Benchmark, human reviews vs. GPT-4o reviews). Gemma-2-2b fine-tuned as a review classifier, sparse autoencoders fine-tuned on it, and a sparse logistic-regression head over their features. Every prediction is a sum of contributions from sparse-autoencoder features, each with a validated natural-language description, so the detector can say which properties of the text drove its decision and where they occur. Paper: Building Auditable AI Text Detectors with Sparse Autoencoders; code: google-deepmind/detectorscope.

Use

pip install detectorscope

from detectorscope import AuditableDetector

det = AuditableDetector.from_pretrained("liamdugan/detector-scope-peer-review-gpt4o")
out = det("The authors propose a novel framework for ...")

out.p_ai       # probability that the text is AI-written
out.label      # "ai" or "human" (logit threshold +0.29)
out.features   # [Feature(id, importance, description, highlights), ...]

out.features lists every feature active in the text with a non-zero effect, largest effect first. importance is how much the logit falls if that feature is silenced: positive pushes toward AI, negative toward human, and out.logit == out.bias + sum(f.importance for f in out.features). A list of texts returns a list of detections.

To see where a feature fires:

out = det(text, return_highlights=True)
out.tokens                  # the text as a list of tokens
out.features[0].highlights  # [(token index, activation), ...] of the top feature

The default head is logistic regression over the features whose auto-interpretation score is at least 0.85. head="xgboost" or another tau (0.00 to 0.95 in steps of 0.05), on from_pretrained or on the call, selects one of the other 39 heads.

Performance

Default head; the decision threshold is fixed on the development data.

split	AUROC	F1	accuracy	documents
PRDB test, GPT-4o reviews	1.000	0.998	0.998	57,368

What it looks at

The features with the largest average contribution on the training data (weight × mean activation when active; density = share of the documents of each class in which the feature fires).

Toward AI

feature	description	mean contribution	AI density	human density
13_15950	academic paper review questions asking for more information details or clarification	+1.002	75%	1%
17_8459	academic review feedback or peer review decisions	+0.814	91%	0%
22_4800	double colons following bolded markdown section headers	+0.672	51%	0%
17_12216	activates on the word potential	+0.515	87%	14%
7_10900	the word particularly when used as an adverb to specify or emphasize a context	+0.438	73%	8%
17_11188	academic research terminology related to study design and methods	+0.388	62%	7%
7_4872	activates on the word potentially often used to hedge academic critique or discuss future implications	+0.363	41%	1%
13_15079	societal impact in research paper sections	+0.357	77%	0%
22_16295	the word beyond in academic or technical contexts	+0.320	47%	7%
7_5712	forms of the word enhance such as enhances enhancing and enhancements	+0.307	70%	9%

Toward human

feature	description	mean contribution	AI density	human density
13_5785	first person singular pronoun i at start of sentences or after punctuation	-0.919	10%	73%
17_13291	forms of the verb seem	-0.639	9%	49%
22_8699	academic review section headers like clarity and clarify	-0.574	1%	35%
7_6962	word for at the start of a sentence	-0.427	6%	35%
7_13654	math and variable notation indices and subscripts	-0.392	2%	38%
7_7454	the adverb very in various forms and cases	-0.341	11%	45%
13_5925	modal verb might	-0.334	4%	20%
7_23	academic document figure or fig references	-0.306	9%	32%
22_13623	polite imperative requests used in academic review feedback	-0.297	0%	20%
13_12569	words related to academic paper authors and roles like writer author or auth	-0.294	1%	12%

Variants

Branches of this repo hold the paper's other two settings on the same data: ft-sae (the unmodified Gemma-2-2b with the fine-tuned SAEs) and base-sae (the unmodified Gemma-2-2b with the Gemma Scope SAEs). Load one with revision="ft-sae" or revision="base-sae". The general-domain detector is detector-scope.

Licence

The language model under extractor/ is Gemma-2-2b fine-tuned on these reviews, provided under and subject to the Gemma Terms of Use (see NOTICE). The sparse autoencoders derive from Gemma Scope (CC-BY-4.0); the heads and feature descriptions are CC-BY-4.0.

Citation

@misc{dugan2026detectorscope,
  title={Building Auditable AI Text Detectors with Sparse Autoencoders},
  author={Liam Dugan and Callum McDougall and Matthew Denton and Phillip Pham and Christine Kaeser-Chen and Chris Callison-Burch},
  year={2026},
  eprint={2610.30006},
  archivePrefix={arXiv},
  url={https://arxiv.org/abs/2610.30006},
}