Aug 5, 2026
paper
arXiv 2607.14935: VideoChat3 — Fully Open Video MLLM for Efficient and Generalist Video Understanding
Hugging Face 2026-07-17 indexed arXiv 2607.14935 (MCG-Nanjing University) introduces VideoChat3, a 4B-parameter fully open, efficient, and generalist video-centric MLLM; addresses three limitations of existing open-source video models: limited cross-domain generalization, high computational overhead, incomplete openness of training assets. Architecture: (1) Inflated 3D Vision Transformer (I3D-ViT) inflates a pretrained image tokenizer by extending 2D spatial self-attention into 3D spatiotemporal self-attention, groups consecutive frames into chunks of T=4, applies spatiotemporal self-attention, performs temporal pooling, combined with 2x2 spatial downsampling yields a 16x spatiotemporal compression ratio; (2) Adaptive Frame Resolution for Streaming Video Perception uses three response-state tokens (</Silence>, </Standby>, </Response>) that also control the next window's pixel quota: Silence/Response → 224² pixels (low), Standby → 448² pixels (high). Training datasets: 3M samples total — VideoChat3-Academic2M (2.27M), VideoChat3-LV116K (116.2K), VideoChat3-OL617K (617K); four training stages (tokenizer pre-training → video-language alignment → video instruction tuning → long & streaming instruction tuning). Full open-source — model weights, training code, training strategy, complete training datasets. Benchmark numbers (VideoChat3-4B vs open-weight Qwen3-VL-4B): MotionBench 61.7 vs 58.6, TempCompass 75.6 vs 70.8, Video-MME 70.1 vs 69.3, LVBench 56.7 vs 56.2, MMVU 56.4 vs 50.5, Charades TL mIoU 56.1 vs 46.4, VUE-TR V1 47.9 vs 32.9, VUE-TR V2 40.2 vs 19.6, MomentSeeker 25.9 vs 13.8; streaming: ODVBench 72.3 vs StreamForest 59.9 (+12.4), OVOBench task avg 62.5, StreamingBench Real-Time 83.0, River Avg 42.8, OVO-Timing Avg F1 35.5 vs Qwen3-VL-4B 8.1; efficiency on NVIDIA H200 at 2048 frames: total latency 20.412s vs Qwen3-VL 44.449s, total FLOPs 80.775 × 10^15 vs 106.913 × 10^15, GPU memory 80.775 GB vs 106.913 GB (saving 26.14 GB), visual tokens 100,352 vs 200,704 (half); best fully open results on MotionBench (61.7) and TempCompass (75.6); paper claims VideoChat3 'surpasses GPT-5 and Gemini 2.5 Flash on all three TimeLens splits.'
1 daily topicOriginally published Jul 17, 2026View source