Abstract
Standard coding-agent benchmarks primarily evaluate binary task success, treating execution as a black box and obscuring the behaviors that produce those outcomes. We analyze a large corpus of 143,209 execution trajectories from 125 agent configurations spanning 117 unique models. To enable comparisons across heterogeneous agent interfaces, we develop a rule-based parser that maps raw execution traces to high-level actions such as repository search, file editing, and test execution, and audit its predictions against a six-model consensus reference on 500 events. We find that agents with similar solve rates can nevertheless differ substantially in how they explore, modify and verify code. Moreover, without using task outcomes, token counts, or tool names as features, aggregate behavioral profiles retrieve the corresponding agent stack across disjoint repository sets and programming languages, although task composition is not fully controlled in the cross-repository setting. Through analyses of frontier models and agent scaffolds, we further show that scaffold choice and reasoning effort systematically shift agents' search, editing, and testing behavior. We release the deterministic parser together with the full annotation and analysis pipeline alongside an annotated trajectory subset.
Community
Coding agents with the same solve rate still work in different ways, and the patterns are stable enough to identify the model and scaffold from behavior alone.
We parsed 143,209 SWE-rebench trajectories from 125 agent configurations (117 models) into action types such as search, read, edit, and test. The scaffold largely decides how agents edit files (native edit tools vs. shell or Python scripts), and higher reasoning effort spends the extra steps on exploration and workspace checks rather than repeated failures.
Accepted at the 1st Workshop on Interpreting Agent Behavior (IAB) at NeurIPS 2026.
Code: https://github.com/SWE-rebench/trajectory-analyzer
Trajectories: https://huggingface.co/datasets/nebius/SWE-rebench-leaderboard-trajectories
Tasks: https://huggingface.co/datasets/nebius/SWE-rebench-leaderboard
Leaderboard: https://swe-rebench.com
Two agents with the same solve rate behaving so differently is something I wish more leaderboards showed. I find it striking that higher reasoning effort goes into exploration and workspace checks rather than repeated failures. Does the scaffold or the model explain more of the editing style when you hold the other fixed?
Hi authors, thanks for the analysis. A question on Table 1: mutation type agrees on 338 of 491 resolved actions, which caps exact match on the 457 fully resolved actions at 74.0%, while the table gives 77.5%. Details in this GitHub issue: https://github.com/SWE-rebench/trajectory-analyzer/issues/1
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
- Before the First Edit? Measuring When Coding-Agent Guidance Actually Arrives (2026)
- ANCHOR: Assessing Naturalistic Constraint Hierarchies and Ordering in Repository-Grounded Agentic Coding (2026)
- Continue or Restart? The Limits of Early Failure Prediction in Coding Agents (2026)
- EvoX: Rethinking the Agent Harness as a Runtime Policy (2026)
- GUI-HARVEST: Self-Improving GUI Agents through Evidence-Driven Harness Evolution (2026)
- TokenEconomizer: Experience-Driven Black-Box Cost Control for Coding Agents (2026)
- When Agents Fail: Model or Harness--Who's to Blame? (2026)
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on Hugging Face checkout this Space
You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: @librarian-bot recommend
Get this paper in your agent:
hf papers read 2611.17440 Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash Models citing this paper 0
No model linking this paper
Datasets citing this paper 1
Spaces citing this paper 0
No Space linking this paper
Collections including this paper 0
No Collection including this paper