Tsinghua, NVIDIA and Others Launch StreamEval, a Benchmark for Real-Time Interactive Video Generation

Tsinghua University, NVIDIA, CMU, UC Berkeley, Princeton and NUS have unveiled StreamEval, described as the first streaming interactive video generation benchmark. It spans five task categories — digital human dialogue, virtual human performance, world generation, instruction editing and reference editing — with 300 test cases and 38 models evaluated. The code, dataset and scoring rules have been fully open-sourced for developers to reproduce and submit new models.

Unlike conventional video benchmarks that score static outputs, StreamEval is designed around interactive streaming. Inputs such as speech, text instructions, reference images and source video frames are released progressively along a timeline, so models cannot see future requests or the expected answers used for scoring. Each case is run three times to reduce randomness, and sessions span 15 to 300 seconds.

The evaluation framework decomposes performance into four dimensions and ten metrics, including task execution, state consistency, visual fidelity and audiovisual coordination. Each metric is scored from 0 to 100, and the five tasks are ranked separately.

Results reveal a gap between generation quality and instruction following. For example, Vidu S2-Avatar scored highest in the virtual human performance task, yet its instruction-following score was only 36.90. In world generation, HappyOyster 1.0 Adventure led in quality and stability but scored just 5.56 on instruction following. Reference editing proved more challenging, with top scores well below other tasks.

The project is positioned as a public evaluation framework for a field that has yet to settle on common definitions. The StreamEval leaderboard is expected to be updated monthly with new models and results.