SEED 7/17 (arXiv 2607.14777): self-evolving on-policy distillation lifts agentic RL across ALFWorld / Search-QA / WebShop — Qwen3-1.7B jumps 46.1 → 92.0

2 sources

arXiv 2607.14777 (indexed 7/17) introduces SEED, a self-evolving on-policy distillation framework for long-horizon agentic RL that converts completed trajectories into hindsight skills and distills them back into the policy, eliminating the sparse-reward credit assignment problem. Two stages: (1) Hindsight Skill SFT fine-tunes the base policy on 1,440 trajectories (180 tasks × K0=8 rollouts) annotated by an external analyzer (GLM-5.2) to extract reusable natural-language skills; (2) Self-Evolving OPD has the current policy snapshot simultaneously act as rollout actor and trajectory analyzer, refreshed each iteration so actor and analyzer co-evolve. The same model plays both roles (actor + analyzer) sharing parameters; confidence-gated token-level distillation via sigmoid(β_opd × Δlog-prob); joint loss L_SEED = L_RL (GRPO with KL regularization) + λ_opd · L_OPD; gradients flow only through the ordinary student branch; at inference, the deployed policy acts from ordinary history alone — no skills, memory, or retrieval required. Reported benchmark numbers (ALFWorld / Search-QA / WebShop score / WebShop succ): Qwen2.5-3B-Instruct — GRPO 75.0 / 36.4 / 79.8 / 63.3 vs Seed 91.8 / 45.7 / 88.5 / 78.9; Qwen2.5-7B-Instruct — GRPO 81.2 / 42.0 / 80.9 / 72.6 vs Seed 96.1 / 48.6 / 89.7 / 78.1; Qwen3-1.7B-Instruct — GRPO 46.1 / 40.8 / 67.3 / 38.3 vs Seed 92.0 / 42.2 / 87.1 / 77.3; sample efficiency: Seed with 60% of training data (ALFWorld 80.7) exceeds full-data GRPO (75.0); cross-domain generalization: +15.3 points on ALFWorld Unseen split (86.2 vs 70.9); multimodal (Qwen2.5-VL-3B) Sokoban 82.0% / EZPoints 100.0%, averaging 91.0% vs GRPO 77.0%; ablations (ALFWorld avg) — removing Hindsight-Skill SFT → 86.0 (-5.8); removing Self-Evolving OPD → 87.0 (-4.8); replacing on-policy skills with static offline skills → 84.4 (-7.4).

Why now

SEED is the strongest agentic RL research of July — across 3 Qwen models × 3 agent benchmarks (ALFWorld / Search-QA / WebShop) consistent large jumps, cross-domain generalization +15.3 points, small model Qwen3-1.7B jumps from 46.1 to 92.0. Creators can frame this as 'self-evolving on-policy distillation is the general solution for sparse-reward agentic RL.'

Why it is worth publishing

Big demo surface: run Qwen2.5-3B-Instruct on ALFWorld comparing SEED vs GRPO to show the concrete effect of self-evolving hindsight-skill + on-policy distillation; walk through the three ablation pieces (-5.8 / -4.8 / -7.4) and their independent contributions.

Evidence basis

arXiv 7/17 indexing + consistent jumps across 3 models × 3 benchmarks + ALFWorld Unseen split +15.3 points + multimodal Sokoban 82.0% / EZPoints 100.0% — heat is medium-to-high as a single research event.

“SEED 7/17 hit arXiv — self-evolving on-policy distillation lifts agentic RL across ALFWorld / Search-QA / WebShop, with Qwen3-1.7B jumping 46.1 → 92.0 and ALFWorld Unseen split gaining +15.3.”

Angle

Frame SEED (arXiv 2607.14777) as 'the strongest agentic RL research of July — self-evolving on-policy distillation is the general solution for sparse-reward agentic RL' — bundle 'consistent jumps across 3 models × 3 benchmarks + cross-domain +15.3 + small-model +46 + three ablation pieces each contributing roughly half' into a multi-dimensional comparison card.

Format

Long-form explainer

Demo idea

Record a 12-minute three-segment demo: 4 minutes on 'why agentic RL suffers under sparse rewards' (without hindsight skills the only option is full-trajectory credit assignment); 4 minutes on 'SEED's two stages' (Hindsight Skill SFT uses GLM-5.2 as external analyzer / Self-Evolving OPD has current policy snapshot simultaneously act as actor + analyzer, refreshed each iteration so they co-evolve); 4 minutes on 'cross-3-model × cross-3-benchmark jumps + three ablation pieces (-5.8 / -4.8 / -7.4) each contribute roughly half + 60% data beats full-data GRPO showing sample efficiency'.

Platform notes

Benchmark numbers are SEED author self-reported (medium risk) — don't paraphrase as third-party validation; per-iteration rollout counts, exact reward-model architecture, and full author list beyond the captured summary are not stated (medium risk) — don't fill in from memory.

Usable claims

arXiv 2607.14777 (indexed as a Hugging Face daily paper on 2026-07-17) introduces SEED, a self-evolving on-policy distillation framework for long-horizon agentic RL that converts completed trajectories into hindsight skills and distills them back into the policy; SEED has two stages — (1) Hindsight Skill SFT fine-tunes the base policy on 1,440 trajectories (180 tasks × K0=8 rollouts) annotated by an external analyzer (GLM-5.2) to extract reusable natural-language skills; (2) Self-Evolving OPD has the current policy snapshot simultaneously act as rollout actor and trajectory analyzer, refreshed each iteration so actor and analyzer co-evolve; the same model plays two roles (actor + analyzer) sharing parameters; confidence-gated token-level distillation via sigmoid(β_opd × Δlog-prob); joint loss L_SEED = L_RL (GRPO with KL regularization) + λ_opd · L_OPD; gradients flow only through the ordinary student branch; at inference, the deployed policy acts from ordinary history alone — no skills, memory, or retrieval required; reported benchmark numbers (ALFWorld / Search-QA / WebShop score / WebShop succ) — Qwen2.5-3B-Instruct: GRPO 75.0 / 36.4 / 79.8 / 63.3 vs Seed 91.8 / 45.7 / 88.5 / 78.9; Qwen2.5-7B-Instruct: GRPO 81.2 / 42.0 / 80.9 / 72.6 vs Seed 96.1 / 48.6 / 89.7 / 78.1; Qwen3-1.7B-Instruct: GRPO 46.1 / 40.8 / 67.3 / 38.3 vs Seed 92.0 / 42.2 / 87.1 / 77.3; sample efficiency: Seed with 60% of training data (80.7 ALFWorld) exceeds full-data GRPO (75.0); cross-domain generalization: +15.3 points on ALFWorld Unseen split (86.2 vs 70.9); multimodal (Qwen2.5-VL-3B): Sokoban 82.0%, EZPoints 100.0%, averaging 91.0% vs GRPO 77.0%; ablations (ALFWorld avg): removing Hindsight-Skill SFT → 86.0 (-5.8); removing Self-Evolving OPD → 87.0 (-4.8); replacing on-policy skills with static offline skills → 84.4 (-7.4).

Evidence pipeline

From the news

Breakdown

Reading any single SEED fact point (self-evolving on-policy distillation / hindsight skills / actor + analyzer sharing parameters / GRPO + KL + λ_opd · L_OPD / Qwen3-1.7B 46.1 → 92.0 / ALFWorld Unseen +15.3 / 60% data beats full GRPO / three ablations -5.8 / -4.8 / -7.4) in isolation turns into 'cross-benchmark jump reading.' This piece explains how to use the 'self-evolving on-policy distillation is the general solution for sparse-reward agentic RL' frame — bundle 'consistent jumps across 3 models × 3 benchmarks + cross-domain +15.3 + small model +46 + three ablation pieces each contribute roughly half' into a multi-dimensional comparison card so creators can produce 'self-evolving + hindsight-skill + on-policy' three-piece independent-contribution content rather than cross-benchmark jump reading.

Sources

Risks

Pin links to each source; quote only what the captured summary states; do not paraphrase specific benchmark numbers, performance metrics, paper claims, integration milestones, or architectural details beyond what is stated.
Pin the link to the arXiv paper page; quote only what the paper summary states; do not paraphrase the headline scores as third-party validation; flag that all reported numbers are paper self-reported by the SEED authors.

Demo ideas

Run SEED vs GRPO on ALFWorld (Qwen2.5-3B-Instruct 75.0 → 91.8 / Qwen2.5-7B-Instruct 81.2 → 96.1 / Qwen3-1.7B-Instruct 46.1 → 92.0) to show the concrete effect of self-evolving hindsight-skill.
Build an 'ablation three-piece comparison card' — remove Hindsight-Skill SFT -5.8 / remove Self-Evolving OPD -4.8 / static offline skills -7.4 → ~18 points of total contribution, on-policy self-evolving and hindsight-skill each roughly half.
Record a 'why SEED lets small models catch up to large ones' demo — Qwen3-1.7B 46.1 → 92.0 vs Qwen2.5-7B 81.2 → 96.1, showing small models with SEED catch up to large-model baseline RL.