FastBench finds streaming VLMs miss fast real-world video events

What problem does FastBench isolate?

Streaming video-language models answer questions while frames arrive under a bounded context budget. That budget forces a three-way trade-off among history length, spatial resolution, and temporal granularity. Many public streaming benchmarks, the authors argue, are dominated by slow scenes where a 1–2 FPS sample still contains the evidence. FastBench is built to reject that comfort zone: if a question remains answerable after downsampling to 2 FPS, it is filtered out as pseudo-dynamic.

The construction pipeline is deliberate. A VLM proposes candidate QA pairs from native high-frame-rate clips; a low-FPS filter discards easy questions; a Vision Expert Verification stage grounds answers on object trajectories from SAM3 and CoTracker3 to suppress hallucinated evidence; three human inspection rounds remain. The retained set is small by LLM-benchmark standards—300 clips and 306 QA pairs per the GitHub README—but each item carries a human-annotated evidence interval and a temporal scope label (forward, instant, or backward).

Domains span sports, games, performing arts, animals, lifestyle, transportation, science and technology, and food. Capability tags cover action, predictive/causal reasoning, motion tracking, entity perception, temporal state dynamics, and online detection. That taxonomy matters when you compare against broader multimodal agents such as OneSearch-VL’s research agent: FastBench is not a tool-use contest, it is a perception-timing contest.

What do the author tables actually show?

Under the paper’s open-ended LLM-as-judge protocol, the strongest reported model, Gemini-3.5-Flash, reaches 50.7% overall. Denser uniform sampling helps open models: Qwen3-VL-8B improves from 32.9% at 2 FPS to 44.6% at 24 FPS, then saturates as history must be compressed to stay inside the context window. Those percentages are the authors’ evaluation rows; we did not re-judge answers.

Oracle-guided focusing—raising FPS when a privileged signal knows evidence is imminent—sets a higher bar than any autonomous policy in the paper. The gap is the point: current VLMs struggle to decide, from the visual stream alone, when finer temporal perception is worth the token cost. Domain and scope breakdowns in the appendix tables vary widely; a single headline number hides where forward prediction fails versus where instant detection fails.

Readers comparing robotics or world-model papers from the same arXiv day—such as DreamTrue’s action-faithful world model—should keep the task distinct. FastBench scores perception of recorded streams, not closed-loop control. Mixing those leaderboards would invent a comparison the paper does not make.

How does ProactiveFrame try to help?

ProactiveFrame is a training-free baseline packaged with the release. Instead of only sampling sparsely, the model may emit text tokens that request a higher future frame rate (Focus_Start) or restore sparse sampling. A dual-tier sliding window keeps recent high-FPS chunks dense while older high-FPS history degrades into sparse tokens, preserving some long context without paying full density forever.

Author results say ProactiveFrame improves over sparse uniform sampling by about 5.4 and 1.5 percentage points on the configurations they highlight, yet remains well below oracle-guided focusing. In other words, giving the model a dial helps a little; teaching it when to turn the dial from pixels alone remains unsolved. The README positions the code as evaluation plus this baseline, not as a new pretrained streaming checkpoint.

Operationally, streaming input is described as incremental chunks of up to one second. That chunking interacts with how judges see partial evidence and with how prefix caches behave in real serving stacks. If you reproduce, freeze the judge prompt, the chunk size, and the FPS schedule before comparing two models.

How do you run the open code?

The GitHub repository Ashone3/FastBench is MIT-licensed. It hosts evaluation code for the benchmark and ProactiveFrame. Stars were low (single digits) on our 11 October check; that is not a quality signal either way, only a popularity snapshot. Follow the README for dataset pointers and environment pins rather than inventing paths from the abstract alone.

Before you publish a “we beat FastBench” claim, match the paper’s filtering rule: if your eval set still answers correctly at 2 FPS, you are not measuring the same thing. Likewise, if you change the judge model, report that change. LLM-as-judge variance can move small benchmarks by more than ProactiveFrame’s reported lift.

For product teams shipping live camera features—adjacent to accessibility or sports overlays, or to video products such as Kandinsky 6.0 Video—FastBench is a diagnostic, not a shipping model. Use it to decide whether your default FPS and context packing are silently dropping the frames that matter.

What are the limits of this evidence?

Three hundred and six items is enough to expose a systematic failure mode and too small to rank every commercial model forever. Domains are curated; real deployments may see different motion statistics. Trajectory verification reduces some hallucinations but inherits errors from SAM3/CoTracker3. Human inspection rounds improve label quality without making the set exhaustive.

The paper’s strongest models are closed or large hosted VLMs evaluated by the authors’ harness. Independent labs should expect score drift when prompts, tools, or judge versions change. We did not download clips, run ProactiveFrame, or confirm every appendix cell beyond the abstract, HTML paper, and README.