Abstract
Video understanding agents acquire evidence through an executable harness that controls what they observe and how they use those observations. However, execution traces contain only the evidence acquired by the current harness, leaving competing explanations for failure unresolved and limiting the basis for self-improvement. We introduce Video-RSI, a framework for recursive self-improvement in which a video understanding agent uses its own language model to revise its harness. Through active video investigation, the model revisits the original training videos to test competing failure explanations with additional observations, grounding proposed changes in evidence beyond the existing trace. Cost-aware harness evolution turns these diagnoses into reusable revisions and determines which revisions to retain by considering both answer accuracy and visual cost. Across our evaluation settings on video understanding benchmarks, the evolved agent improves accuracy while processing fewer frames and achieves competitive accuracy-efficiency trade-offs against existing video understanding agents. These results demonstrate the potential for video understanding agents to improve their own evidence acquisition and use through harness evolution.
Method
The Editor gathers additional visual, transcript, and OCR evidence, then revises the agent’s prompts, tools, memory, and execution logic. Useful revisions accumulate in the executable harness and seed subsequent attempts.
Evolution runs offline. The selected harness is frozen before evaluation on new questions.
Quantitative Results
Accuracy and visual frame usage on four video understanding benchmarks.
| Method | Accuracy ↑% | Frames ↓per question |
|---|---|---|
| Initial harness | 62.4 | 68.5 |
| VideoSeek | 68.1 | 83.9 |
| VideoHarness-RSI | 64.1 | 83.2 |
| Video-RSI (Ours) | 72.9 | 61.1 |
| Method | Accuracy ↑% | Frames ↓per question |
|---|---|---|
| Initial harness | 67.7 | 51.6 |
| VideoSeek | 70.9 | 71.9 |
| VideoHarness-RSI | 68.4 | 92.1 |
| Video-RSI (Ours) | 71.1 | 41.1 |
| Method | Accuracy ↑% | Frames ↓per question |
|---|---|---|
| Initial harness | 73.4 | 38.6 |
| VideoSeek | 78.7 | 17.8 |
| VideoHarness-RSI | 69.7 | 92.8 |
| Video-RSI (Ours) | 80.0 | 21.6 |
| Method | Accuracy ↑% | Frames ↓per question |
|---|---|---|
| Initial harness | 71.4 | 66.0 |
| VideoSeek | 73.2 | 72.3 |
| VideoHarness-RSI | 78.0 | 66.8 |
| Video-RSI (Ours) | 78.2 | 42.4 |
Selected comparisons from Tables 1–2. VideoSeek and VideoHarness-RSI are reproduced with the same DeepSeek-V4-Pro backbone and Qwen3.6-Plus visual model.
Frames count frame inputs to successful visual-model calls per question. Offline Editor investigation is excluded; this measure does not represent total compute cost.
Ablation Study
Active investigation. Revisiting videos improves accuracy over revision based only on existing trajectories across all four benchmarks.
Cost-aware selection. Compared with an accuracy-only gate, the retained harness uses fewer frames and achieves higher accuracy across these settings.
Ablation results
Accuracy (%) and frames per question.
| Variant | MLVU | LongVideoBench | Video-MME | EgoSchema | ||||
|---|---|---|---|---|---|---|---|---|
| Acc. ↑ | Frames ↓ | Acc. ↑ | Frames ↓ | Acc. ↑ | Frames ↓ | Acc. ↑ | Frames ↓ | |
| Initial harness | 62.4 | 68.5 | 67.7 | 51.6 | 73.4 | 38.6 | 71.4 | 66.0 |
| Trajectories only | 64.7 | 60.3 | 67.2 | 43.1 | 73.7 | 28.5 | 76.6 | 51.8 |
| Accuracy-only gate | 67.1 | 81.9 | 69.5 | 66.4 | 74.9 | 43.1 | 77.4 | 60.5 |
| Video-RSI (Ours) | 72.9 | 61.1 | 71.1 | 41.1 | 80.0 | 21.6 | 78.2 | 42.4 |
Source: Table 2. All variants share the initial harness and frozen models. Investigation and trajectory-only runs share training data and per-attempt limits.
Evolution Dynamics
The first accepted revision accounts for most of the development-set frame reduction. Transfer is not monotonic: MLVU accuracy initially falls, then recovers through later revisions. See Section 4.4.
BibTeX
@article{luo2026videorsi,
title={Video-RSI: Recursive Self-Improvement of Video Understanding Agents via Harness Evolution},
author={Luo, Bingjun and Guo, Jialin and Li, Siqi},
journal={arXiv preprint arXiv:2609.37950},
year={2026}
}