arxiv:2609.15309
Published on Sep 14
· Submitted by
Wenhao Chai on Sep 15
Upvote
13
Authors:
,
,
,
,
,
,
,
,
Abstract
Elo-per-token analysis reveals that LLM agents initially scale faster than independent sampling but eventually slow, while parallel short sessions improve performance over single long runs.
Generated by thinkingmachines/Inkling-Small
Large language model (LLM) agents allocate test-time compute adaptively as they revise solutions, use tools, explore alternatives, and decide when to stop. This test-time strategy makes it difficult to measure how agent performance scales. We study open-ended tasks that provide continuous scores for intermediate submissions, making progress observable throughout long trajectories. We propose Elo-per-token analysis, which tracks the best solution found at each token budget and uses a Bradley-Terry model to aggregate within-task orderings into Elo ratings across tasks with different score scales. We apply it to four general-purpose agents on four open-ended benchmarks, with sessions of up to 100M tokens, and to three feedback-driven LLM optimization harnesses in controlled single-task interventions. Independent sampling provides a theoretically characterized reference, for which Elo grows linearly with log compute. Against this reference, agents can initially convert tokens into Elo faster than independent sampling, but their marginal gains diminish and eventually fall below the reference. In contrast, the strongest historical human contestants improve superlinearly over contest time on shared AtCoder Heuristic Contest tasks, providing evidence of continual learning and substantial headroom after agents slow down. We define the scaling inflection point as the per-session budget where marginal Elo gains match the independent-sampling reference. Using this point as the per-session budget, we split 100M tokens across parallel sessions on FrontierCS Polyomino Packing, gaining +264 Elo over one long session and +355 over ten short sessions.
View arXiv page View PDF Add to collection
Community
Paper submitter 8 days ago
hi!
8 days ago
Project Page: https://agent-tts.github.io/agent-tts/
7 days ago
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
-
Disagree to Explore, Agree to Commit: Routing-Guided Test-Time Scaling for Software Agents (2026)
-
AI4AI-Bench: Benchmarking LLM Agents in Algorithmic Design for Recursive Self-Improvement (2026)
-
$R^3$-Bench: LLMs Struggle with Resource-Rational Reasoning under Shared Budgets (2026)
-
Zero-Shot Self-Orchestration with Ledger-Based Control for Improved LLM Coding Performance (2026)
-
T1: Terminal Agent Reinforcement Learning for Long-Horizon Tasks (2026)
-
VRL-Bench: Benchmarking agents on computer control tasks under finite trial budgets (2026)
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on HF Mirror checkout this Space
You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: @librarian-bot recommend
Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.
Tap or paste here to upload images
· Sign up or log in to comment
Upvote
13
Models citing this paper 0
No model linking this paper
Cite arxiv.org/abs/2609.15309 in a model README.md to link it from this page.
Datasets citing this paper 0
No dataset linking this paper
Cite arxiv.org/abs/2609.15309 in a dataset README.md to link it from this page.
Spaces citing this paper 0
No Space linking this paper
Cite arxiv.org/abs/2609.15309 in a Space README.md to link it from this page.
Collections including this paper 0
No Collection including this paper
Add this paper to a collection to link it from this page.