arxiv:2609.21346
Published on Sep 18
· Submitted by
LONGFEI XU on Sep 21
Upvote
92
Authors:
,
,
,
Abstract
Mixture-of-Experts (MoE) scales capacity, but existing designs cannot set three quantities independently. For a single token, participation is how many experts contribute knowledge to its output, execution is how many are actually computed (compute cost), and materialization is how many expert-sized parameter sets must be built and stored (memory cost). Sparse routing keeps execution and materialization low, but shrinks participation: for each token, only a few experts contribute. Dense output-mixing restores full participation, but its execution grows with the number of experts. Parameter-merging keeps execution at one expert, but its materialization grows with the number of routing decisions. We propose IntBMoE, a block-conditioned MoE that decouples all three by pairing dense expert composition with sparse block execution. Its blocks come from a small learned codebook, one per entry. At each internal layer, a lightweight hypernetwork merges all expert bases in that layer's pool into one composed expert. Participation is full, because every composed expert draws on the entire pool. Execution stays sparse, because a router sends each token to only a few blocks. Materialization is bounded, because the codebook, not the input, fixes how many blocks exist. Dual-Path Residual Gating (DPRG) further couples two independently composed paths through multiplicative gating. Experiments on image classification show consistent gains over representative sparse and dense MoE baselines. Additional experiments on language modeling and sequential recommendation validate its generalization beyond vision. IntBMoE is fully deployed in AMap's generative recommendation system, serving hundreds of millions of users under a 60ms latency budget, with a 2.4% relative UVCTR gain in online A/B testing. Our code is available at https://github.com/AMAP-ML/DreamX-Rec/.
View arXiv page View PDF Add to collection
Community
Paper author Paper submitter 1 day ago
IntBMoE learns reusable computational blocks by combining parameters from a shared expert pool, enabling full expert participation while routing each token to only a few blocks. These blocks can be precomputed and cached, making inference computation independent of the expert count when the block configuration is fixed. Experiments across computer vision, natural language processing, and recommendation demonstrate strong predictive performance and efficient inference, highlighting its effectiveness across diverse domains.
1 day ago
This is an automated message from the ResearchStudio team.
We created an interactive ResearchStudio Reel for this paper. It includes a visual poster, a video, and a blog, all available for download in editable formats.
Open the ResearchStudio Reel →
Download all files from HF Mirror
Please give this comment a thumbs up if you find the Reel helpful!
Want to explore or create Reels for more papers? Visit the ResearchStudio demo.
about 22 hours ago
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
-
Share First, Route What Remains: A Unified Framework for Token-Adaptive MoE Computation (2026)
-
PCoMoE: Shifting MoE Inference from Monolithic Expert Selection to Fine-Grained Path Composition (2026)
-
MoEGen: Mixture-of-Experts for Instance-Adaptive LoRA Generation (2026)
-
ExFold: Unified Expert Folding for Training-Free MoE Prefill-Decode Acceleration (2026)
-
UniMoMo: Expert Merging-Based MoE Acceleration for Large Recommendation Models (2026)
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on HF Mirror checkout this Space
You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: @librarian-bot recommend
Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.
Tap or paste here to upload images
· Sign up or log in to comment
Upvote
92
Get this paper in your agent:
hf papers read 2609.21346
Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash
Models citing this paper 0
No model linking this paper
Cite arxiv.org/abs/2609.21346 in a model README.md to link it from this page.
Datasets citing this paper 0
No dataset linking this paper
Cite arxiv.org/abs/2609.21346 in a dataset README.md to link it from this page.
Spaces citing this paper 0
No Space linking this paper
Cite arxiv.org/abs/2609.21346 in a Space README.md to link it from this page.