arxiv:2609.15051
Published on Sep 14
· Submitted by
Qianli Ma on Sep 15
Upvote
14
Authors:
,
,
,
,
Abstract
The framework dynamically adjusts training prompts via exploration potential scoring and scaffolded rewrites to improve reinforcement learning for multimodal language models.
Generated by thinkingmachines/Inkling-Small
Training prompts in online reinforcement learning (RL) differ substantially in how informative they are for the current policy: some are already saturated while others are too difficult to yield reliable learning signals, yet both receive equal rollout budget under standard training. We propose an exploration-guided prompt scaffolding framework that adapts the training prompt distribution dynamically throughout RL post-training of multimodal large language models (MLLMs). Central to our approach is the Exploration Potential Score (EPS), a lightweight rollout-based proxy for prompt utility derived from KL-regularized policy improvement theory, computable directly from on-policy rollout statistics without additional overhead. Rather than discarding low-utility prompts, we use a teacher model to generate scaffolded rewrites that preserve the original task intent while making subsequent training more informative, reframing teacher supervision as training-data refinement rather than output imitation. Integrated with GRPO on Geo3K and MMK12, our method consistently outperforms the baseline on both in-domain and out-of-distribution benchmarks, achieving up to 9.7% relative improvement in-domain and gains of 11.5% on MathVision and 11.1% on MMMU-Pro.
View arXiv page View PDF Project page Add to collection
Community
Paper author Paper submitter 8 days ago
In this paper, we introduce exploration-guided prompt scaffolding for multimodal RL post-training. Our Exploration Potential Score (EPS) identifies low-utility prompts from existing rollouts, guiding a teacher model to rewrite them into more informative training inputs rather than providing answers to imitate. Integrated with GRPO, our approach achieves up to 9.7% relative improvement in-domain, with gains of 11.5% on MathVision and 11.1% on MMMU-Pro.
🌐 Project page: https://mqleet.github.io/EPS-ProjectPage/
7 days ago
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
-
LEEPS: Latent-Guided Explore-Exploit Prompt Sampling for Efficient RLVR in Large Language Models (2026)
-
PAC: Progress-Augmented Advantage Curriculum for Multi-Task Reinforcement Learning of LLMs (2026)
-
Distilled Reinforcement Learning for LLM Post-training (2026)
-
From Rollouts to Recipes: Self-Contained Post-Training for LLMs (2026)
-
StructReward: Efficient Structured Process Rewards for Self-Correcting Multimodal Reasoning (2026)
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on HF Mirror checkout this Space
You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: @librarian-bot recommend
Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.
Tap or paste here to upload images
· Sign up or log in to comment
Upvote
14
Get this paper in your agent:
hf papers read 2609.15051
Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash
Models citing this paper 0
No model linking this paper
Cite arxiv.org/abs/2609.15051 in a model README.md to link it from this page.
Datasets citing this paper 0
No dataset linking this paper
Cite arxiv.org/abs/2609.15051 in a dataset README.md to link it from this page.
Spaces citing this paper 0
No Space linking this paper
Cite arxiv.org/abs/2609.15051 in a Space README.md to link it from this page.