arxiv:2609.13680
Published on Sep 12
· Submitted by
FeiYuan on Sep 16
Upvote
13
Authors:
,
,
Yilei Tu ,
,
,
Abstract
Fine-tuning instruct models often improves target performance while inducing behavioral drift from the reference model, which can degrade existing capabilities. Rather than treating this drift as an uncontrolled consequence of optimization, we specify a behavioral drift budget before optimization and ask how to boost the target-task performance within it. Locally, behavioral drift induces a shared geometry anchored at the reference model, with the drift budget defining a boundary within this space. In this space, drift determines distance from the reference, leaving update direction as the remaining degree of freedom. Fine-tuning updates can therefore be compared through their directional efficiency, naturally reformulating fine-tuning as a direction-selection problem. This reformulation makes a concrete prediction: changing the accessible directions can qualitatively alter the outcome of fine-tuning. We test this prediction in a stringent QA-only setting, where strong instruct models are fine-tuned only on final answers but must still generate multi-step reasoning at inference. Despite this mismatch, a coarse layer-selective probe reverses the failure of QA-only fine-tuning and reveals the existence of effective directions, with multiple neighboring configurations improving target performance while preserving reasoning and general capabilities. Across Qwen3-8B and Qwen3-14B, these directions substantially improve scientific reasoning and multilingual translation. Over more than 100 languages, the resulting models match or outperform dedicated translation systems and provide a stronger initialization for subsequent reinforcement learning. Our results suggest that fine-tuning is not just about how much a model changes, but how that change is spent. https://github.com/CONE-MT/DCO and https://huggingface.co/collections/LLaMAX/dco
View arXiv page View PDF Add to collection
Community
Paper submitter 7 days ago
We usually ask:
How do we fine-tune an instruct model without behavioral drift?
This work reverses the question:
Given a behavioral drift budget, how should we fine-tune the model?
This reversal predicts that where the model is allowed to move can qualitatively change the outcome. On Qwen3-8B and Qwen3-14B, under the exact same QA-only supervision, changing only the accessible update directions reverses fine-tuning failure preserving reasoning while improving the target task.
6 days ago
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
-
Source-Prior-Driven Selective Adaptation for Efficient Diffusion Model Finetuning (2026)
-
Weightless Fine-Tuning: Personalizing LLMs via Logit-Space Transport (2026)
-
Neuron-Guided Fine-Tuning: Unlocking Efficient Alignment Mechanisms for Large Language Models (2026)
-
MemSFT: Mitigating Alignment Tax with an External Parametric Memory (2026)
-
Behavior Quotient Learning for Low-Rank Adaptation of LLM Agents (2026)
-
Learning New Facts with QLoRA: An Acquisition-Retention Frontier (2026)
-
RRPO: Reference-Relative Policy Optimization with Stratified Conditional Rollouts (2026)
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on HF Mirror checkout this Space
You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: @librarian-bot recommend
Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.
Tap or paste here to upload images
· Sign up or log in to comment
Upvote
13
Get this paper in your agent:
hf papers read 2609.13680
Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash
Models citing this paper 0
No model linking this paper
Cite arxiv.org/abs/2609.13680 in a model README.md to link it from this page.
Datasets citing this paper 0
No dataset linking this paper
Cite arxiv.org/abs/2609.13680 in a dataset README.md to link it from this page.
Spaces citing this paper 0
No Space linking this paper
Cite arxiv.org/abs/2609.13680 in a Space README.md to link it from this page.