Verified · Aug 5, 2026
Independently verifiedVideoChat3 7/17 (arXiv 2607.14935): 4B fully open video MLLM with 3 response-state tokens controlling pixel budget, 2x faster inference than Qwen3-VL on H200
2 sourcesarXiv 2607.14935 (indexed 7/17, MCG-Nanjing University) introduces VideoChat3, a 4B-parameter fully open, efficient, and generalist video-centric MLLM; addresses three limitations of existing open-source video models: limited cross-domain generalization, high computational overhead, incomplete openness of training assets. Architecture: (1) Inflated 3D Vision Transformer (I3D-ViT) inflates a pretrained image tokenizer by extending 2D spatial self-attention into 3D spatiotemporal self-attention, groups consecutive frames into chunks of T=4, applies spatiotemporal self-attention, performs temporal pooling, combined with 2x2 spatial downsampling yields a 16x spatiotemporal compression ratio; (2) Adaptive Frame Resolution for Streaming Video Perception uses three response-state tokens (</Silence>, </Standby>, </Response>) that also control the next window's pixel quota: Silence/Response → 224² pixels (low), Standby → 448² pixels (high). Training datasets: 3M samples total — VideoChat3-Academic2M (2.27M), VideoChat3-LV116K (116.2K), VideoChat3-OL617K (617K); four training stages (tokenizer pre-training → video-language alignment → video instruction tuning → long & streaming instruction tuning). Full open-source — model weights, training code, training strategy, complete training datasets. Benchmark numbers (VideoChat3-4B vs open-weight Qwen3-VL-4B): MotionBench 61.7 vs 58.6, TempCompass 75.6 vs 70.8, Video-MME 70.1 vs 69.3, LVBench 56.7 vs 56.2, MMVU 56.4 vs 50.5, Charades TL mIoU 56.1 vs 46.4, VUE-TR V1 47.9 vs 32.9, VUE-TR V2 40.2 vs 19.6, MomentSeeker 25.9 vs 13.8; streaming: ODVBench 72.3 vs StreamForest 59.9 (+12.4), OVOBench task avg 62.5, StreamingBench Real-Time 83.0, River Avg 42.8, OVO-Timing Avg F1 35.5 vs Qwen3-VL-4B 8.1; efficiency on NVIDIA H200 at 2048 frames: total latency 20.412s vs Qwen3-VL 44.449s, total FLOPs 80.775 × 10^15 vs 106.913 × 10^15, GPU memory 80.775 GB vs 106.913 GB (saving 26.14 GB), visual tokens 100,352 vs 200,704 (half); best fully open results on MotionBench (61.7) and TempCompass (75.6); paper claims VideoChat3 'surpasses GPT-5 and Gemini 2.5 Flash on all three TimeLens splits.'
Why now
VideoChat3 is the strongest fully open video MLLM research of July — 4B parameters + fully open (weights + code + training strategy + training data) + 3 response-state tokens controlling pixel budget + 3M training data + 2x faster inference than Qwen3-VL on H200. Creators can frame this as 'open video MLLM surpasses GPT-5 and Gemini 2.5 Flash on TimeLens for the first time.'
Why it is worth publishing
Big demo surface: run VideoChat3-4B on the same video (dialogue vs teaching vs monitoring) vs Qwen3-VL-4B to show the 3 response-state tokens in action.
Evidence basis
arXiv 7/17 indexing + fully open (weights + code + training strategy + data) + 9 video benchmarks consistently beating Qwen3-VL-4B + 3 TimeLens splits beating GPT-5 and Gemini 2.5 Flash + 2x faster inference on H200 — heat is medium-to-high as a single research event.
“VideoChat3 7/17 hit arXiv — 4B fully open video MLLM (weights + code + training strategy + training data), 3 response-state tokens controlling pixel budget, 2x faster inference on H200 than Qwen3-VL, beating GPT-5 and Gemini 2.5 Flash on all three TimeLens splits.”
Angle
Frame VideoChat3 (arXiv 2607.14935) as 'the strongest fully open video MLLM research of July' — bundle '4B parameters + fully open (weights + code + training strategy + training data) + 9 video benchmarks beating Qwen3-VL-4B + 3 TimeLens splits beating GPT-5 and Gemini 2.5 Flash + 2x faster inference on H200' into one piece.
Format
Long-form explainer
Demo idea
Record a 10-minute three-segment demo: 3 minutes on 'why video MLLM is hard' (weak cross-domain / high compute / partial openness); 4 minutes on 'VideoChat3's three architectural moves' (Inflated 3D ViT's 16x spatiotemporal compression / 3 response-state tokens controlling pixel budget / 3M training data with four-stage training); 3 minutes on 'VideoChat3 vs Qwen3-VL-4B vs GPT-5 vs Gemini 2.5 Flash' (9 benchmarks consistently beat Qwen3-VL-4B + 2x faster inference on H200 + 26.14 GB GPU memory saved).
Platform notes
Benchmark numbers and the 'surpasses GPT-5 / Gemini 2.5 Flash' claim are MCG-Nanjing University author self-reported (medium risk) — don't paraphrase as third-party validation; specific author list beyond 'MCG-Nanjing University' and exact license (MIT / Apache / etc.) are not stated in the captured summary (medium risk) — don't fill in from memory.
Usable claims
- arXiv 2607.14935 (indexed as a Hugging Face daily paper on 2026-07-17) introduces VideoChat3, a 4B-parameter fully open, efficient, and generalist video-centric MLLM; addresses three limitations of existing open-source video models: limited cross-domain generalization, high computational overhead, incomplete openness of training assets; architecture: (1) Inflated 3D Vision Transformer (I3D-ViT) inflates a pretrained image tokenizer by extending 2D spatial self-attention into 3D spatiotemporal self-attention, groups consecutive frames into chunks of T=4, applies spatiotemporal self-attention, performs temporal pooling, combined with 2x2 spatial downsampling yields a 16x spatiotemporal compression ratio; (2) Adaptive Frame Resolution for Streaming Video Perception uses three response-state tokens (</Silence>, </Standby>, </Response>) that also control the next window's pixel quota: Silence/Response → 224² pixels (low), Standby → 448² pixels (high); training datasets: 3M samples total — VideoChat3-Academic2M (2.27M), VideoChat3-LV116K (116.2K), VideoChat3-OL617K (617K); four training stages (tokenizer pre-training → video-language alignment → video instruction tuning → long & streaming instruction tuning); full open-source — model weights, training code, training strategy, complete training datasets; benchmark numbers (VideoChat3-4B vs open-weight Qwen3-VL-4B): MotionBench 61.7 vs 58.6, TempCompass 75.6 vs 70.8, Video-MME 70.1 vs 69.3, LVBench 56.7 vs 56.2, MMVU 56.4 vs 50.5, Charades TL mIoU 56.1 vs 46.4, VUE-TR V1 47.9 vs 32.9, VUE-TR V2 40.2 vs 19.6, MomentSeeker 25.9 vs 13.8; streaming: ODVBench 72.3 vs StreamForest 59.9 (+12.4), OVOBench task avg 62.5, StreamingBench Real-Time 83.0, River Avg 42.8, OVO-Timing Avg F1 35.5 vs Qwen3-VL-4B 8.1; efficiency on NVIDIA H200 at 2048 frames: total latency 20.412s vs Qwen3-VL 44.449s, total FLOPs 80.775 × 10^15 vs 106.913 × 10^15, GPU memory 80.775 GB vs 106.913 GB (saving 26.14 GB), visual tokens 100,352 vs 200,704 (half); best fully open results on MotionBench (61.7) and TempCompass (75.6); paper claims VideoChat3 'surpasses GPT-5 and Gemini 2.5 Flash on all three TimeLens splits.'
Evidence pipeline
From the news
Breakdown
Reading the 9 VideoChat3 benchmark numbers (MotionBench 61.7 / TempCompass 75.6 / VUE-TR V1 47.9 / V2 40.2 / ODVBench 72.3 / etc.) in isolation turns into 'benchmark jump reading.' This piece explains how to use the '4B parameters + fully open + 3 response-state tokens controlling pixel budget + 2x faster inference on H200' frame — bundle '4B parameter fully open (weights + code + training strategy + training data) + 3 response-state tokens controlling pixel budget (Silence/Response 224², Standby 448²) + 2x faster inference on H200 at 2048 frames + 3 TimeLens splits beating GPT-5 and Gemini 2.5 Flash' into a multi-dimensional comparison card.
Sources
Risks
- Pin links to each source; quote only what the captured summary states; do not paraphrase specific benchmark numbers, performance metrics, paper claims, integration milestones, or architectural details beyond what is stated.
- Pin the link to the arXiv paper page; quote only what the paper summary states; explicitly flag that the benchmark numbers and the 'surpasses GPT-5 / Gemini 2.5 Flash' claim are paper self-reported by MCG-Nanjing University authors.
Demo ideas
- Run VideoChat3-4B vs Qwen3-VL-4B on the same video (dialogue / teaching / monitoring) on H200 to show 9 benchmarks consistently beat Qwen3-VL-4B + 2x faster inference on 2048 frames + 26.14 GB GPU memory saved.
- Build a '3 response-state tokens controlling pixel budget' information card: Silence/Response 224², Standby 448², mapping each to specific video monitoring / video dialogue / long-video teaching scenarios.
- Record a 'fully open vs partially open' demo — VideoChat3 simultaneously releases VideoChat3-Academic2M (2.27M), VideoChat3-LV116K (116.2K), and VideoChat3-OL617K (617K) training datasets + training code + weights + training strategy, explain 'why fully open matters more than partial open.'