Temporal continuity
Sparse frames miss brief events and invert causality. Dense frames preserve motion but repeat huge amounts of nearly identical evidence.
FIELD GUIDE / UPDATED AUG 2026
A video becomes a sequence before it becomes an answer.
Multimodal language models do not “watch” pixels. They sample time, carve frames into patches, encode motion and appearance as vectors, then spend a tightly managed token budget to let language attend to the result.
- Input
- T × H × W × 3
- Latent
- N × d
- Bottleneck
- context window
ENCODER TRACE
08 frames
196 patches/frame
032 pooled tokens
THE COMPRESSION PROBLEM
Raw video is an impossible prompt.
A 10-second 1080p clip at 30 fps contains more than 1.8 billion RGB values. The practical pipeline discards most of them on purpose, trying to preserve the evidence a question might need: objects, text, motion, ordering, causality, and sound.
V ∈ ℝT×H×W×3
fvideo
Z ∈ ℝN×d
where N must fit beside the text prompt
A FIVE-STAGE DATA PATH
From photons to language context
There is no single canonical stack. Some models fuse time inside the vision backbone; others encode frames independently and model time later. But most production systems can be read through these five operations.
-
SAMPLE / DECODE
Choose which moments survive.
Uniform sampling is cheap and predictable. Shot-aware or motion-aware sampling spends frames near cuts and action. Query-conditioned samplers retrieve moments that match the user’s question. Long-video systems often combine a sparse global pass with denser local clips.
video → {framet}t∈S -
PATCH / ENCODE
Turn pixels into spatial tokens.
A ViT slices each frame into fixed-size patches (often 14×14 or 16×16 pixels), flattens each patch, and linearly projects it to a vector. Position embeddings retain where it came from. CLIP or SigLIP-style encoders add semantic pretraining from paired text.
Nspatial = (H/P) × (W/P) -
MODEL / TIME
Let tokens exchange evidence across time.
3D convolutions mix local space-time neighborhoods. Factorized transformers alternate spatial and temporal attention. Tubelets embed short 3D blocks directly. The objective is not merely to recognize “a glass” and “a hand,” but to preserve that the hand picked the glass up.
zt,p ← Attn(z1:T,p) -
COMPRESS / PROJECT
Pay the connector tax.
A linear layer or MLP can project every visual token into the LLM width. Q‑Former and Perceiver Resampler modules instead use a fixed bank of learned queries to extract a smaller set. Pooling is where thousands of patch tokens can become dozens, but missing evidence cannot be recovered later.
Q′ = CrossAttn(Qlearned, Zvideo) -
FUSE / GENERATE
Make vision addressable by language.
Projected video tokens are inserted into the prompt sequence or exposed through cross-attention. Positional encodings carry space and time; instruction tuning teaches the language model how to ask those latents for answers, captions, timestamps, or actions.
[video tokens] + [text tokens] → decoder
WHY VIDEO IS HARD
Every gain spends something else.
Image understanding asks what is present. Video adds when, for how long, in which order, and what changed. Those extra axes multiply compute before the language model sees a single token.
Attention compute
Full self-attention grows quadratically with tokens. Factorization, local windows, pruning, and latent resampling keep sequences tractable.
spatial detail
temporal coverage
language context
budget
Resolution vs. length
OCR and small objects need high resolution. Long activities need more timestamps. A fixed budget rarely affords both uniformly.
Weak alignment
Captions describe only fragments of a clip. Contrastive training must learn from noisy pairs without collapsing background, speech, and action into one signal.
ARCHITECTURE LINEAGE
Eight models, eight bottlenecks
These systems are not direct substitutes. Some are vision backbones, some are training objectives, and some are end-to-end multimodal models. The useful comparison is where each places the compression and alignment work.
architecture published
architecture partial / undisclosed
| Model | Date | Core move | What it contributes to video LLMs | Disclosure |
|---|---|---|---|---|
| ViT ↗Vision Transformer | Patchify images; apply a Transformer to spatial tokens. | The frame encoder template. Video variants extend or factor its attention across time. | published | |
| VideoCLIP ↗Meta AI | Contrast temporally overlapping video–text pairs with hard negatives. | Shows that aligned video and text embeddings can transfer zero-shot to retrieval, QA, and localization. | published | |
| VideoMAE ↗NJU / Tencent | Reconstruct videos after masking 90–95% of spatiotemporal tubelets. | Data-efficient self-supervised features that capture motion and appearance without paired text. | published | |
| SigLIP ↗Google Research | Replace global softmax contrastive normalization with pairwise sigmoid loss. | A strong image–text encoder often reused per frame; SigLIP itself is not a temporal video model. | published | |
| V‑JEPA ↗Meta AI | Predict masked latent features rather than pixels; no text or negative pairs. | Learns action-sensitive video representations. V‑JEPA 2 (Jun 2025) extends toward prediction and planning. | published | |
| Qwen2‑VL ↗Alibaba / Qwen | Dynamic visual resolution plus multimodal rotary position embedding (M‑RoPE). | Adapts token count to input size and encodes time alongside 2D position. Qwen2.5‑VL (Feb 2025) adds absolute-time encoding. | published | |
| GPT‑4o ↗OpenAI | One end-to-end “omni” network across text, vision, and audio. | Official material states video input capability, but does not publish the video tokenizer, sampling policy, or connector design. | undisclosed | |
| Gemini 2.5 ↗Google DeepMind | Native multimodality with long context; internal video encoder details remain private. | The API guide documents 1 FPS sampling and about 300 tokens/s at default media resolution. | partial |
* Dates are first paper submissions or public announcements. Public capability does not imply a public architecture.
TOKEN BUDGET LAB
Compression has a bill.
Explore a simple accounting model. This is not a universal tokenizer: real systems crop, tile, merge, prune, or resample tokens differently. It makes the order of magnitude visible.
visual ≈ duration × fps × tokens/frame
17,400estimated input tokens
Visual15,480
Audio1,920
Frames60
Comfortable inside a 128K context, before the text prompt and response reserve.
Preset values mirror Google’s documented default Gemini video accounting: 1 frame/s, 258 visual tokens/frame, plus 32 audio tokens/s. Official guide ↗
DESIGN SPACE
Four places to change the answer
Architecture is a sequence of lossy decisions. Select a layer to see what it preserves, what it costs, and where it tends to fail.
FRAME SELECTION
Spend frames where information changes.
Uniform sampling is the robust baseline. Scene-cut sampling avoids redundant shots; optical-flow or entropy heuristics favor motion; learned selectors can use the question itself. Hierarchical designs first index a long video sparsely, then re-encode promising windows densely.
- Preserves
- coverage, salient events
- Costs
- decode I/O, missed transients
- Failure
- the answer happens between frames
SPATIAL FEATURES
Use patches, tubelets, or both.
2D ViTs reuse powerful image encoders and batch frames efficiently. 3D tubelet embeddings join space and time earlier, while 3D convolutions impose local motion bias. Dynamic tiling keeps small text legible but makes the token count input-dependent.
- Preserves
- objects, layout, fine text
- Costs
- tokens scale with resolution
- Failure
- appearance without event structure
TEMPORAL MIXING
Choose a receptive field for change.
3D convolutions are local and efficient. Temporal attention reaches across long intervals but can be expensive. Factorized space-then-time attention reduces cost; memory tokens or recurrent state extend effective duration without reprocessing every past patch.
- Preserves
- order, motion, long dependencies
- Costs
- attention and memory bandwidth
- Failure
- correct objects, wrong chronology
VISUAL CONNECTOR
Compress before the language model pays.
Average pooling is cheap but indiscriminate. Token merging and pruning remove redundancy. Q‑Former and Perceiver-style resamplers use learned queries to extract a fixed latent set. Text-conditioned resamplers keep evidence relevant to the prompt.
- Preserves
- task-relevant summaries
- Costs
- another learned bottleneck
- Failure
- irreversible information loss
ALIGNMENT OBJECTIVES
How vectors learn to mean the same thing
ContrastivePull matched video–text pairs together; push mismatches apart.
GenerativePredict captions or answers conditioned on visual latents.
Masked / predictiveRecover pixels, tubelets, or latent features from context.
Instruction tuningTeach the LLM when and how to consult video evidence.
WHAT THE LATENTS ENABLE
One embedding stack, different reads
The same video representation can be pooled into one vector for search, decoded into prose, grounded to timestamps, or kept as a rolling memory for an agent.
Video question answering
Cross-attend to visual tokens to answer about objects, events, order, intent, or causality. Strong QA needs temporal grounding, not just scene recognition.
Captioning & summarization
Decode dense captions for short windows or hierarchical summaries for long footage. Coverage and faithfulness compete with narrative compression.
Semantic retrieval
Map clips and queries into a shared space. Global vectors make search cheap; token-level late interaction improves fine-grained matching.
Perception for agents
Maintain a compact visual memory for robots, desktop agents, and assistants. Streaming encoders must update state without repeatedly ingesting the full past.
THE ENGINEERING HEURISTIC
Preserve the evidence the next module cannot reconstruct.
Sampling can erase an event. Patching can erase small text. Pooling can erase identity and order. The best embedding is not the largest one: it is the smallest sequence that still contains the evidence your task will ask for.
Replay from the first framePRIMARY SOURCES
Read the architectures
Dates below are initial arXiv submissions or official public announcements. Links open the original papers or first-party documentation.
- Dosovitskiy et al.An Image is Worth 16×16 Words
- Radford et al.Learning Transferable Visual Models From Natural Language Supervision (CLIP)
- Xu et al.VideoCLIP: Contrastive Pre-training for Zero-shot Video–Text Understanding
- Tong et al.VideoMAE: Masked Autoencoders are Data-Efficient Learners
- Alayrac et al.Flamingo: a Visual Language Model for Few-Shot Learning
- Li et al.BLIP‑2: Bootstrapping Language–Image Pre-training with Frozen Encoders
- Zhai et al.Sigmoid Loss for Language–Image Pre-training (SigLIP)
- Bardes et al.Revisiting Feature Prediction for Learning Visual Representations from Video (V‑JEPA)
- Wang et al.Qwen2‑VL: Perception of the World at Any Resolution
- OpenAIGPT‑4o System Card
- Bai et al.Qwen2.5‑VL Technical Report
- Assran et al.V‑JEPA 2: Understanding, Prediction, and Planning
- Google DeepMindGemini 2.5 Technical Report
- Google AI for DevelopersVideo understanding: sampling and token calculation