Designer-RSI: Evolving Procedural Memory from User Traffic for Agentic Graphic Design

Papers

arxiv:2609.22086

Published on Sep 18

· Submitted by

Hongyang Du on Sep 21

· Adobe

Upvote

Authors:

Abstract

Professional graphic design is a long-horizon agentic task in which structured, editable artifacts emerge from many interdependent actions, yet outcomes admit no reliable programmatic oracle. We introduce a continual adaptation framework in which a frozen frontier model operates professional design software through more than 230 tools, while an external procedural memory of natural-language skills accumulates and refines reusable design procedures from experience. The memory widens by acquiring procedures for recurring uncovered subtasks and deepens by revising existing procedures against their own successful and failed executions, while a matched replay gate admits only changes that repair failures without regressing observed successes. Five rounds over 1,406 real user briefs and 1,869 automatically graded trajectories, with no weight updates and no human labels, grow the bank from 76 documentation-derived skills to 139 and raise GenEval2 execution success on Claude-Sonnet-4 from 72.7% to 99.3% (+11.99 points in generation quality), with 61.8% and 67.6% win rates against the no-skill agent across four specialized design benchmarks on Claude-Sonnet-4 and Claude-Opus-4.6. We further show the two mechanisms are effective in combination: on 200 held-out briefs from user-traffic benchmark, widening or deepening alone reaches a 49.4% / 48.6% win rate over the no-skill agent, while their combination reaches 58.5% (p = 0.025). Procedural memory offers a practical route to continual adaptation of agents under noisy, unverifiable feedback.

View arXiv page View PDF Add to collection

Community

Hongyang-Du

Paper submitter 1 day ago

Designer-RSI: a graphic-design agent that gets better from user traffic, without any weight updates 🎨

A frozen frontier model drives professional design software through 230+ tools. What changes over time is its procedural memory: a bank of natural-language skills that anyone can read and edit.

How the skill bank evolves

Widening: finds recurring subtasks in user requests that no existing skill covers, then clusters them and distills them into new skills.
Deepening: finds skills tied to repeated failures and rewrites them by contrasting failed runs with successful runs of the same skill.
Replay gate: a candidate change is kept only if it fixes at least one replayed failure and causes no detected regression. Everything else, including the models, tools and grader, stays frozen.

Results (5 rounds, 1,406 briefs, 1,869 auto-graded trajectories, no human labels)

The skill bank grows from 76 documentation-derived skills to 139.
GenEval2 execution success on Claude-Sonnet-4 goes from 72.7% → 99.3%, with +11.99 in generation quality.
Win rates against the no-skill agent across 4 specialized design benchmarks are 61.8% (Sonnet-4) and 67.6% (Opus-4.6).
On 200 held-out user-traffic briefs, widening alone reaches a 49.4% win rate and deepening alone 48.6%. Both together reach 58.5% (p = 0.025).

Same brief, same pool of retrieved assets. Top: no-skill agent. Bottom: with the evolved skill bank. These are uncurated one-shot rollouts.

Takeaway: when outcomes can't be checked automatically, an external procedural memory is a practical way for frozen agents to keep adapting. Every improvement is a skill edit that a person can read and audit.

📄 Paper:

librarian-bot

about 22 hours ago

This is an automated message from the Librarian Bot. I found the following papers similar to this paper.

The following papers were recommended by the Semantic Scholar API

Please give a thumbs up to this comment if you found it helpful!

If you want recommendations for any Paper on HF Mirror checkout this Space

You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: @librarian-bot recommend

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment

Upvote

Get this paper in your agent:

hf papers read 2609.22086

Don't have the latest CLI?

curl -LsSf https://hf.co/cli/install.sh | bash

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2609.22086 in a model README.md to link it from this page.

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2609.22086 in a dataset README.md to link it from this page.

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2609.22086 in a Space README.md to link it from this page.

Abstract

Community

Models citing this paper 0

Datasets citing this paper 0

Spaces citing this paper 0

Collections including this paper 2

相关阅读