arxiv:2609.20784
Published on Sep 17
· Submitted by
Zhengxi Lu on Sep 18
Upvote
52
Authors:
,
,
,
,
,
,
,
,
,
Abstract
Multi-turn agents trained with reinforcement learning (RL) receive a single scalar reward per trajectory, which motivates self on-policy distillation (OPD) to supply dense token-level supervision from a self-teacher with privileged task skills, letting a skill-free student internalize them. This recipe, however, is undermined by two findings in agentic tasks: privileged information alone does not always make a teacher reliable, and the benefit of teacher supervision is stage-dependent. We therefore propose RetireOPD (Self-Retiring On-Policy Distillation), which first optimizes a decoupled, skill-conditioned teacher with environment rewards and then trains a skill-free student jointly with RL and OPD. Rather than following a predefined distillation schedule, RetireOPD adopts Adaptive Retirement: the student drops the teacher on its own once their discrepancy stops shrinking and it reaches a target fraction of the teacher's success rate, after which training proceeds with RL alone. Across Qwen2.5 models from 1.5B to 7B, RetireOPD improves ALFWorld success rate over RL baseline by 14.1% to 18.8% and WebShop accuracy by 11.8% to 19.0%, and surpasses its own skill-conditioned teacher in every setting.
View arXiv page View PDF Add to collection
Community
Paper author Paper submitter 5 days ago
Rather than following a predefined distillation schedule, RetireOPD adopts Adaptive Retirement: the student drops the teacher on its own once their discrepancy stops shrinking and it reaches a target fraction of the teacher's success rate, after which training proceeds with RL alone.
Paper author Paper submitter 5 days ago
Code: https://github.com/ZJU-REAL/SDAR
4 days ago
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
-
AgentOPSD: Recursive Self-Distillation for Agentic Reinforcement Learning (2026)
-
From Scoring to Acting: Outcome-Verified Comparative Self-Distillation for LLM Agents (2026)
-
AHEAD: Adaptive Hindsight with Environment-Augmented Distillation for Agentic RL (2026)
-
Sequential Beats Joint: On the Interplay between On-Policy Distillation and RLVR (2026)
-
Agentic Reinforcement Learning with Observation-Calibrated Self-Distillation (2026)
-
EDGE: Experience-Distillation for Guided Exploration in Agentic Reinforcement Learning (2026)
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on HF Mirror checkout this Space
You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: @librarian-bot recommend
Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.
Tap or paste here to upload images
· Sign up or log in to comment
Upvote
52
Models citing this paper 0
No model linking this paper
Cite arxiv.org/abs/2609.20784 in a model README.md to link it from this page.
Datasets citing this paper 0
No dataset linking this paper
Cite arxiv.org/abs/2609.20784 in a dataset README.md to link it from this page.
Spaces citing this paper 0
No Space linking this paper
Cite arxiv.org/abs/2609.20784 in a Space README.md to link it from this page.