arxiv:2609.20715
Published on Sep 17
· Submitted by
Juzheng Zhang on Sep 18
· Amazon
Upvote
40
Authors:
,
,
,
,
Abstract
Agent trajectories record what an agent does and what happens next. Yet standard supervised fine-tuning (SFT) applies loss only to agent-authored action tokens, using environment observations as context but not as prediction targets. We ask whether this convention provides the best initialization for subsequent reinforcement learning. We introduce ActObs, which also supervises the observation tokens already present in each trajectory. Although deployed agents never generate observations, learning to predict them encourages the policy to model action consequences without adding data, parameters, sequence tokens, or forward passes. The methods perform similarly after SFT but diverge after GRPO. On Qwen3-4B, GRPO from ActObs achieves higher pass@k at every evaluated sampling budget than its action-only counterpart on Terminal-Bench 2.0. On Qwen3-8B, it trades some pass@1 reliability for higher pass@k (+3.4 pp at pass@16) and solves more distinct tasks. The advantage extends to cross-domain code editing on aider-polyglot (+4.2 pp at pass@1 at 4B), whose tasks are unseen during SFT and RL. ActObs retains more entropy during RL while requiring less policy movement, leaving the final policy closer to its SFT initialization. Our analysis traces this difference to SFT: action and observation gradients rapidly become orthogonal, while action-only training leaves a large residual observation gradient and degrades environment prediction below the base model. Joint supervision prevents this one-sided specialization, preserving consequence prediction and preparing the policy for downstream exploration.
View arXiv page View PDF Add to collection
Community
Paper submitter 5 days ago
Supervising both actions and observations in agent trajectories produces a stronger initialization for downstream reinforcement learning, improving exploration and cross-domain generalization.
4 days ago
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
-
TailSFT: Filtered Fine-Tuning Improves Post-Training Performance (2026)
-
How do World Models and Policies Compose in LLM Agents? A Joint Spectral and Behavioral Account (2026)
-
State2State: Environment-Derived Mid-Training for LLM Agents (2026)
-
RMSWeb: Reflection, Failure-Mode Mining, and Salvage-DS for Web Agent Reinforcement Learning (2026)
-
Locked at the Entrance, Open Inside: Where RLVR Narrows the Solution Space (2026)
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on HF Mirror checkout this Space
You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: @librarian-bot recommend
2 days ago
Does the observation-supervision signal survive scale, or is this a small-model effect? The exploration gains you show are real, but I'd want to see the added loss on a 7B+ model — SFT objectives get diluted fast when you're stacking auxiliary losses. Also curious about training cost: does the extra supervision slow convergence enough to eat the exploration benefit? For anyone deploying this, the question is whether the exploration improvement is worth the extra training complexity, or if it's just a nice trick for small runs.
Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.
Tap or paste here to upload images
· Sign up or log in to comment
Upvote
40
Get this paper in your agent:
hf papers read 2609.20715
Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash
Models citing this paper 0
No model linking this paper
Cite arxiv.org/abs/2609.20715 in a model README.md to link it from this page.
Datasets citing this paper 0
No dataset linking this paper
Cite arxiv.org/abs/2609.20715 in a dataset README.md to link it from this page.
Spaces citing this paper 0
No Space linking this paper
Cite arxiv.org/abs/2609.20715 in a Space README.md to link it from this page.