Production-assessed code-agent evaluation

AgentLens

AgentLens evaluates interactive coding agents on full trajectories: user requests, messages, tool calls, file edits, verification attempts, final answers, and repository state. It combines formal checks with written LLM-judge reviews so results explain both whether an agent solved the task and how safely, reliably, and pleasantly it worked.

Code Paper Blog post

Leaderboard

Java coding-agent results

Loading leaderboard...

All runs use GPT-5.4 as the LLM judge. Evaluated agents run with high reasoning effort where supported, except where noted. OpenAI and Anthropic models use their official providers, GLM-5.1 (self-hosted) is hosted locally, and the remaining models use OpenRouter.
* marks runs with provider-side instability; results are included for completeness.

Description

What AgentLens is designed to evaluate

AgentLens targets production-like Java coding workflows where a binary pass/fail signal is too coarse. The benchmark reviews the complete interaction, including instruction following, tool-use discipline, verification behavior, error recovery, and the quality of user-facing communication.

Methodology

How to read this leaderboard

Tasks

The current benchmark contains 16 coding-agent scenarios, each run with two user personas for 32 trajectories per evaluated agent. Scenarios cover realistic multi-step Java workflows and are designed to be extensible with additional task families.

Scoring

Score is the Quality Index (QI) - an unweighted average of all the other metics. Formal is the formal-verification pass rate. End result, Instruction compliance, Pitfalls, Pleasantness, and Tool calls are LLM-judge metric means. Formal checks capture objective task evidence where possible. LLM judges then review complete trajectories for End Result, Instruction Compliance, Pitfalls, Pleasantness, and Tool Calls.