VLM and Multimodal Models

§0 TL;DR Cheat Sheet

💡 VLM in 9 sentences — one page covering the core interview points for vision-language models (see §1–§13 below for derivations and code).

1. Vision encoder = ViT-dominated: Dosovitskiy et al. 2021 (ICLR) slice images into $P\times P$ patches (typically $P=14$ or $16$), apply a linear projection + learnable positional embedding + optional [CLS] token, and feed them into a Transformer encoder. The vision side of CLIP / SigLIP / LLaVA is all a ViT variant.

2. CLIP symmetric InfoNCE (must derive): Radford et al. 2021 (ICML) make image embeddings $\mathbf{u}_i$ and text embeddings $\mathbf{v}_i$ do contrastive learning in a shared space, with loss = average of row softmax + column softmax: $\mathcal{L} = \tfrac{1}{2}(\mathcal{L}_{i\to t} + \mathcal{L}_{t\to i})$. The temperature $\tau$ is learnable (log-parameterized, clipped to $[0,100]$).

3. SigLIP replaces softmax with sigmoid: Zhai et al. 2023 (ICCV) treat each entry of the N×N similarity matrix as an independent binary CE, getting rid of batch-wise softmax normalization, so it is no longer linearly sensitive to batch size and can train with 32k+ batches on a single machine; a learnable bias $b$ corrects early negative dominance. SigLIP-2 (Google 2025) adds caption + self-distillation + dense local objectives and extends to multilingual.

4. LLaVA = projector + 2-stage train: Liu et al. 2023 (NeurIPS) use a single linear projection matrix $W$ to project frozen CLIP visual features into LLM token space (the 2-layer MLP projector is the LLaVA-1.5 upgrade). Stage 1 trains only the projector for feature alignment (caption data); Stage 2 unfreezes the LLM for visual instruction tuning (158K instructions generated by GPT-4).

5. Q-Former vs Projector is the central BLIP-2 trade-off: Li et al. 2023 (ICML) use 32 learnable query tokens that do cross-attention over a frozen image encoder, compressing any resolution / number of patches into a fixed 32 tokens — stable compute budget but lossy + complex to train. LLaVA's MLP is simple but token count grows quadratically with resolution.

6. Flamingo / Llama-3.2-Vision = gated cross-attn: Alayrac et al. 2022 (NeurIPS) use a Perceiver Resampler (64 latent queries) to compress visual features into a fixed token count, then insert gated cross-attention layers every few LLM layers ($\tanh$ gating initialized at 0, preserving the frozen LLM's text-only capability).

7. Qwen2-VL's M-RoPE — must-know: Wang et al. 2024 split RoPE along head_dim into 6 chunks, assigning (t / h / w, three groups of position ids) following the axis sequence $(t, h, w, t, h, w)$; the typical config mrope_section=[16,24,24] (units are pairs of half head_dim, so $\sum \times 2 =$ head_dim=128, all 128 dims rotate). This way each token carries (t, h, w) three-dim positions without flattening. Pairs with native dynamic resolution (no longer padded to a fixed 224×224).

8. Three-stage training + preference optimization: (1) alignment trains the projector / Q-Former; (2) visual instruction tune unfreezes part of the LLM; (3) preference (LLaVA-RLHF, RLAIF-V, VLM-R1, DPO/PPO) addresses hallucinations and long-tail alignment. VLM-R1 (2025) uses GRPO + verifiable reward to transfer reasoning ability into vision-language tasks.

9. Closed-source VLMs are now unified multimodal agents: GPT-5.5, Claude Fable/Opus, and Gemini 3.1 Pro Preview-style APIs are no longer just "image QA"; they combine text/image/video/audio/PDF, tool/function calling, long context, and agent workflows. The architecture is still undisclosed, so the CLIP/LLaVA/Qwen-VL mechanisms below are for understanding visible design patterns, not for claiming what closed models use internally.

§1 Intuition: what is a VLM doing?

Think of an image as "another language". The work of a VLM splits into three parts:

• Visual tokenizer: compress pixels into a discrete or continuous token sequence (ViT patch → embedding)

• Cross-modal alignment: bring image / text of the same semantics close in a shared space — this is what CLIP / SigLIP do; essentially learning a shared embedding space

• Cross-modal generation: let the LLM "see" the image inside the prompt — this is what LLaVA / Qwen-VL / Flamingo do; essentially stuffing image tokens as a prefix into the LLM's context

💡 Three fusion paradigms — the central architectural split in VLMs.

• Early fusion (dual-encoder + contrastive): CLIP / SigLIP, no cross-modal attention, only push / pull in the embedding space

• Projector fusion (visual tokens → LLM context): LLaVA / Qwen-VL, project image embeddings into LLM token space and concatenate as input tokens, then autoregressively decode

• Cross-attn fusion (image as KV, text as Q): Flamingo / BLIP-2 / Llama-3.2-V, add cross-attention layers so the LLM's text tokens actively query the visual KV

Compare from a Q/K/V perspective: in the projector paradigm the image is part of the LLM's input sequence (full interaction inside self-attention); in the cross-attn paradigm the image is always KV and is only queried — this causes different KV cache handling at inference time.

§2 ViT: turning an image into a token sequence

2.1　Patch tokenize

Input image $\mathbf{x} \in \mathbb{R}^{H \times W \times C}$, slice it into $N = HW/P^2$ patches of $P\times P$, flatten each patch to a $P^2 C$-dim vector, and pass it through a linear layer to $D$ dimensions:

$\mathbf{z}_0 = [\mathbf{x}_\text{class};\ \mathbf{x}^1_p \mathbf{E};\ \mathbf{x}^2_p \mathbf{E};\ \dots;\ \mathbf{x}^N_p \mathbf{E}] + \mathbf{E}_\text{pos}$

• $\mathbf{E} \in \mathbb{R}^{P^2 C \times D}$ is the patch embedding matrix (equivalent to a Conv2D with stride=$P$, kernel=$P$)

• $\mathbf{x}_\text{class} \in \mathbb{R}^D$ is the learnable [CLS] token, used to aggregate global information (for classification, take $\mathbf{z}_L^0$)

• $\mathbf{E}_\text{pos} \in \mathbb{R}^{(N+1) \times D}$ is a learnable 1D positional embedding — the original ViT uses 1D learned, not 2D sinusoidal (the paper's Appendix D.4 reports that 1D learned vs 2D sinusoidal differs only within noise)

⚠️ CLIP / SigLIP don't necessarily use [CLS] — CLIP ViT uses [CLS] for output, SigLIP / EVA-CLIP / modern LLaVA tend to use patch token average pool or keep all patch tokens to feed downstream. [CLS] is the ViT paper's choice, not an intrinsic part of ViT.

2.2　Transformer backbone

$\mathbf{z}'_\ell = \text{MHA}(\text{LN}(\mathbf{z}_{\ell-1})) + \mathbf{z}_{\ell-1}, \quad \mathbf{z}_\ell = \text{MLP}(\text{LN}(\mathbf{z}'_\ell)) + \mathbf{z}'_\ell$

Pre-norm (LN at the input of each sub-layer), MLP uses GELU. Note that the original ViT has a fixed number of patches ($224/16=14 \Rightarrow N=196$), and its positional embedding table is fixed in size — this is the pain point that dynamic resolution must solve (§10).

2.3　ViT specifications

Model	Patch	Hidden $D$	Layers	Heads	Params	Source
ViT-B/16	16	768	12	12	86M	Dosovitskiy 2021
ViT-L/14	14	1024	24	16	304M	Dosovitskiy 2021
ViT-H/14	14	1280	32	16	632M	Dosovitskiy 2021
ViT-g/14	14	1408	40	16	1.0B	Zhai et al. 2022
ViT-bigG/14	14	1664	48	16	1.8B	OpenCLIP, 2023
EVA-02-L/14	14	1024	24	16	304M	Fang 2023
SigLIP SoViT-400M/14	14	1152	27	16	400M	Alabdulmohsin 2023

💡 head_dim is typically fixed at 64 — ViT-family models mostly follow head_dim ≈ 64–88, i.e. $D / H$. Scaling laws suggest head_dim should not be too small, otherwise per-head expressiveness is limited.

2.4　Code: ViT patch embed + backbone (core 60 lines)

import torch
import torch.nn as nn
import torch.nn.functional as F

class PatchEmbed(nn.Module):
    def __init__(self, img_size=224, patch_size=16, in_chans=3, embed_dim=768):
        super().__init__()
        self.img_size, self.patch_size = img_size, patch_size
        self.num_patches = (img_size // patch_size) ** 2
        # A Conv2d with stride=P, kernel=P is equivalent to a linear projection
        self.proj = nn.Conv2d(in_chans, embed_dim, kernel_size=patch_size, stride=patch_size)

    def forward(self, x):                                   # x: [B, C, H, W]
        x = self.proj(x)                                    # [B, D, H/P, W/P]
        x = x.flatten(2).transpose(1, 2)                    # [B, N, D]
        return x

class ViTBlock(nn.Module):
    def __init__(self, dim, num_heads, mlp_ratio=4.0, dropout=0.0):
        super().__init__()
        self.ln1 = nn.LayerNorm(dim)
        self.attn = nn.MultiheadAttention(dim, num_heads, dropout=dropout, batch_first=True)
        self.ln2 = nn.LayerNorm(dim)
        hidden = int(dim * mlp_ratio)
        self.mlp = nn.Sequential(
            nn.Linear(dim, hidden), nn.GELU(), nn.Dropout(dropout),
            nn.Linear(hidden, dim), nn.Dropout(dropout),
        )

    def forward(self, x):
        h = self.ln1(x)
        a, _ = self.attn(h, h, h, need_weights=False)       # self-attention
        x = x + a
        x = x + self.mlp(self.ln2(x))
        return x

class ViT(nn.Module):
    def __init__(self, img_size=224, patch_size=16, in_chans=3, embed_dim=768,
                 depth=12, num_heads=12, num_classes=1000, use_cls=True):
        super().__init__()
        self.patch_embed = PatchEmbed(img_size, patch_size, in_chans, embed_dim)
        N = self.patch_embed.num_patches
        self.use_cls = use_cls
        if use_cls:
            self.cls_token = nn.Parameter(torch.zeros(1, 1, embed_dim))
            self.pos_embed = nn.Parameter(torch.zeros(1, N + 1, embed_dim))
        else:
            self.pos_embed = nn.Parameter(torch.zeros(1, N, embed_dim))
        nn.init.trunc_normal_(self.pos_embed, std=0.02)
        if use_cls:
            nn.init.trunc_normal_(self.cls_token, std=0.02)
        self.blocks = nn.ModuleList([ViTBlock(embed_dim, num_heads) for _ in range(depth)])
        self.ln = nn.LayerNorm(embed_dim)
        self.head = nn.Linear(embed_dim, num_classes) if num_classes > 0 else nn.Identity()

    def forward(self, x):
        B = x.size(0)
        x = self.patch_embed(x)                             # [B, N, D]
        if self.use_cls:
            cls = self.cls_token.expand(B, -1, -1)
            x = torch.cat([cls, x], dim=1)                  # [B, N+1, D]
        x = x + self.pos_embed                              # broadcast over batch
        for blk in self.blocks:
            x = blk(x)
        x = self.ln(x)
        feat = x[:, 0] if self.use_cls else x.mean(dim=1)   # CLS or mean-pool
        return self.head(feat)

⚠️ Common bug with interpolate_pos_embed — when transferring a ViT from $224^2$ to $336^2$, the pos_embed table needs resizing from $(14^2 + 1)$ rows to $(24^2 + 1)$ rows. Correct procedure: keep [CLS] unchanged, reshape the patch portion to $14\times 14\times D$, bicubic-interpolate to $24\times 24$, then flatten and concatenate back. Pitfall: directly doing 1D interpolation over $(N+1)$ rows treats [CLS] as a patch.

§3 CLIP: symmetric InfoNCE (must derive)

3.1　Formalizing the objective

CLIP (Radford et al. 2021, ICML) trains with $N$ (image, text) pairs per batch. Two encoders $f_\theta$ (image), $g_\phi$ (text) produce $\ell_2$-normalized embeddings:

$\mathbf{u}_i = \frac{f_\theta(I_i)}{\|f_\theta(I_i)\|_2}, \quad \mathbf{v}_j = \frac{g_\phi(T_j)}{\|g_\phi(T_j)\|_2}, \quad \mathbf{u}_i, \mathbf{v}_j \in S^{D-1}$

Define the similarity matrix $\mathbf{S} \in \mathbb{R}^{N\times N}$ (the "logit"):

$S_{ij} = \frac{\mathbf{u}_i^\top \mathbf{v}_j}{\tau}$

where $\tau > 0$ is the learnable temperature (engineered as logit_scale = log(1/τ), which is more stable in backprop, clamped in $[\log 1, \log 100]$).

3.2　Symmetric InfoNCE loss (average of row + column softmax)

Image → Text direction (for each image $i$, positive sample is $T_i$, negatives are $\{T_j\}_{j\neq i}$):

$\mathcal{L}_{i\to t} = -\frac{1}{N}\sum_{i=1}^{N} \log \frac{\exp(S_{ii})}{\sum_{j=1}^{N} \exp(S_{ij})}$

Text → Image direction:

$\mathcal{L}_{t\to i} = -\frac{1}{N}\sum_{j=1}^{N} \log \frac{\exp(S_{jj})}{\sum_{i=1}^{N} \exp(S_{ij})}$

Total symmetric loss:

$\boxed{\;\mathcal{L}_\text{CLIP} = \frac{1}{2}\left(\mathcal{L}_{i\to t} + \mathcal{L}_{t\to i}\right)\;}$

✅ Equivalent formulation: average of row softmax + column softmax — for the matrix $\mathbf{S}$, apply row softmax and take the NLL of the diagonal (image→text), apply column softmax and take the NLL of the diagonal (text→image). The mean of the two cross-entropies is the CLIP loss.

3.3　Gradient derivation (why symmetry matters)

Fix $\tau=1$. For the row logits $\mathbf{s}_i = (S_{i1},\dots,S_{iN})^\top$ inside $\mathcal{L}_{i\to t}$, apply softmax and let $p_{ij} = \text{softmax}(\mathbf{s}_i)_j$. Then:

$\frac{\partial \mathcal{L}_{i\to t}}{\partial S_{ij}} = \frac{1}{N}\left(p_{ij} - \mathbb{1}[j=i]\right)$

• $j=i$ (positive): gradient $\propto p_{ii} - 1 < 0$, pulls $\mathbf{u}_i, \mathbf{v}_i$ closer

• $j \neq i$ (negative): gradient $\propto p_{ij} > 0$, pushes $\mathbf{u}_i, \mathbf{v}_j$ apart

With only one direction $\mathcal{L}_{i\to t}$, $\mathbf{v}_j$ receives gradient from all $\mathbf{u}_i$, but cannot in turn constrain how $\mathbf{u}_i$ behaves when retrieved by other $\mathbf{v}_k$. Symmetrization adds the text→image retrieval constraint, preventing one-sided collapse in the embedding space (where image-side clusters tightly but text-side drifts).

3.4　Role of temperature

$\mathcal{L}_{i\to t} = -\frac{1}{N}\sum_i \log\frac{\exp(\mathbf{u}_i^\top \mathbf{v}_i / \tau)}{\sum_j \exp(\mathbf{u}_i^\top \mathbf{v}_j / \tau)}$

• $\tau \to 0^+$ (very small): softmax is nearly one-hot, only the hardest negative matters (the most similar but incorrect text); gradient is dominated by one or two negatives and training is unstable

• $\tau \to \infty$ (very large): softmax is uniform, positives and negatives are nearly indistinguishable, loss approaches the $\log N$ constant, almost no gradient

• OpenAI CLIP's learned steady state: $\tau \approx 0.01$ (logit_scale ≈ log(100)), with the upper bound clamped to prevent collapse

💡 InfoNCE as a lower bound — Oord et al. 2018 (CPC) proved InfoNCE is a lower bound on the mutual information $I(U; V)$: $I(U; V) \ge \log N - \mathcal{L}_\text{InfoNCE}$. So increasing batch size $N$ while reducing loss directly raises the MI lower bound — this is why CLIP / SigLIP both chase huge batches.

3.5　Code: CLIP symmetric InfoNCE (core 50 lines)

import torch
import torch.nn as nn
import torch.nn.functional as F

class CLIPLoss(nn.Module):
    """Symmetric InfoNCE used by OpenAI CLIP (Radford et al. 2021)."""
    def __init__(self, init_tau=0.07, max_logit_scale=4.6052):
        super().__init__()
        # Equivalent to logit_scale = log(1/τ); initial ~ log(1/0.07) ≈ 2.659
        self.logit_scale = nn.Parameter(torch.tensor(1.0 / init_tau).log())
        self.max_logit_scale = max_logit_scale            # log(100), clamp to prevent blow-up

    def forward(self, image_feats, text_feats):
        """
        image_feats: [N, D]   (unnormalized)
        text_feats:  [N, D]
        """
        # L2 normalize to the unit sphere
        u = F.normalize(image_feats, dim=-1)              # [N, D]
        v = F.normalize(text_feats, dim=-1)               # [N, D]

        # Clamp logit_scale upper bound (late training rises to ~log(100))
        logit_scale = self.logit_scale.clamp(max=self.max_logit_scale).exp()

        # Similarity matrix
        logits_i2t = logit_scale * u @ v.t()              # [N, N]
        logits_t2i = logits_i2t.t()                       # [N, N]

        # The diagonal contains the positive pairs
        N = u.size(0)
        labels = torch.arange(N, device=u.device)

        loss_i2t = F.cross_entropy(logits_i2t, labels)    # row softmax NLL
        loss_t2i = F.cross_entropy(logits_t2i, labels)    # column softmax NLL

        return 0.5 * (loss_i2t + loss_t2i), logit_scale

# Example (under DDP, you need to all-gather feats from all GPUs before computing)
if __name__ == "__main__":
    N, D = 8, 512
    img_feats = torch.randn(N, D)
    txt_feats = torch.randn(N, D)
    criterion = CLIPLoss()
    loss, scale = criterion(img_feats, txt_feats)
    print(f"loss={loss.item():.4f}  logit_scale={scale.item():.2f}")

⚠️ Under DDP you must all-gather to get true InfoNCE — on a single GPU, the batch $N$ loss only covers local negatives. Production CLIP (OpenCLIP / OpenAI) does dist.all_gather on $\mathbf{u}, \mathbf{v}$ after forward, so the negative pool = global batch size (e.g. 32k). Gradient is computed for only the local row / column on the host GPU via gradient checkpointing — this is an engineering trick, not a math change.

3.6　CLIP training data & scale

• WIT (WebImageText): 400M (image, text) pairs scraped from the internet (not released)

• LAION-400M / LAION-2B: the open-source replacement used by OpenCLIP; a series of scales were trained in 2022–2023

• DataComp: Gadre et al. 2023 (NeurIPS) proposed a systematic data-filtering benchmark, data quality > data scale

• Model scale: OpenAI's largest is ViT-L/14; OpenCLIP trained ViT-bigG/14 (LAION-2B), with zero-shot ImageNet ~80%+

3.7　CLIP's failure modes

• Poor OCR / text understanding: training captions usually do not describe the text inside an image, so CLIP is essentially "blind" to in-image text

• Fine-grained counting fails: "5 birds" vs "6 birds" is nearly indistinguishable in CLIP embeddings (the "counting problem")

• Weak on spatial relations: "cat on top of dog" vs "dog on top of cat" is hard to distinguish (POPE / Winoground benchmarks quantify this)

• Bag-of-words tendency: Yuksekgonul et al. 2023 (ICLR) showed CLIP is barely sensitive to word order in captions

§4 SigLIP: replacing softmax with sigmoid; batch scaling rewritten

4.1　Motivation

CLIP's softmax normalization couples all N×N similarities together: each positive's gradient depends on the row-wide logsumexp of negatives. This causes:

• Strong batch-size sensitivity: doubling N significantly changes the loss landscape; small batches barely learn

• Expensive DDP communication: must all-gather embeddings (O(N·D) bytes), creating a communication bottleneck

• Numerical instability: softmax overflows easily at very large N

Zhai et al. 2023 (ICCV) proposed SigLIP: treat each entry of the N×N matrix as independent binary classification.

4.2　Sigmoid loss derivation

Define similarity $S_{ij} = t \cdot \mathbf{u}_i^\top \mathbf{v}_j + b$, where $t = e^{t'}$ is a learnable scale (same as CLIP's $1/\tau$) and $b$ is a learnable bias (initialized to a negative number, e.g. $b_0 = -10$, to avoid early prediction of "all positive").

Label $y_{ij} = +1$ if $i=j$, $-1$ otherwise. Each entry does binary logistic regression:

$\mathcal{L}_\text{SigLIP} = -\frac{1}{N}\sum_{i=1}^N \sum_{j=1}^N \log \sigma\!\left(y_{ij} \cdot S_{ij}\right) = \frac{1}{N}\sum_{i=1}^N \sum_{j=1}^N \log\!\left(1 + \exp(-y_{ij} S_{ij})\right)$

✅ Key property — the loss for each $(i,j)$ entry does not depend on any other entry. Therefore:

• batch size no longer couples all negatives

• a single machine can use very large batches (SigLIP reports 32k batch on one chip is trainable)

• communication only needs pairing local queries with remote keys and computing sigmoid entries (chunked all-pair); no need to synchronize logsumexp

⚠️ The bias $b$ is not decorative — early in training, $\mathbf{u}, \mathbf{v}$ are near-random, $S_{ij}$ is near 0, and sigmoid outputs 0.5. Negatives number $N^2 - N \approx N^2$, while positives are only $N$; if initial predictions are all ~0.5, negative-sample gradients dominate early training. SigLIP initializes $b_0 \approx -10$, so the sigmoid is initially near 0 — all entries are first predicted as negative, then positives have large loss and negatives have small loss; starting from this state makes training stable.

4.3　SigLIP vs CLIP comparison

Dimension	CLIP (softmax)	SigLIP (sigmoid)
Loss form	$\propto$ logsumexp(row) + logsumexp(col)	$\propto$ $\sum_{ij}$ binary logistic
Batch dependence	Strong (gradient couples batch)	Weak (entries independent)
Communication	all-gather embeddings	chunked all-pair sigmoid
Bias term	None (implicitly absorbed by softmax)	learnable $b$, init $\approx -10$
Small-batch behavior	Poor (< 4k barely learns)	Significantly better (1k learns)
Large-batch behavior	Diminishing returns	Keeps rising through 32k+
Zero-shot ImageNet (ViT-L/14, 400M data)	~75%	~76–78%

4.4　SigLIP-2 (Google 2025)

Tschannen et al. 2025, building on SigLIP-1:

• Adds a caption-style decoder (akin to CapPa) as a captioning auxiliary task

• Self-distillation + dense local objectives: patch-level local contrast for better detail localization

• Multilingual extension: training data scaled to 100+ languages, with substantial multilingual zero-shot gains

• Released a NaFlex variant for native aspect ratios

4.5　Code: SigLIP sigmoid loss (core 35 lines)

import torch
import torch.nn as nn
import torch.nn.functional as F

class SigLIPLoss(nn.Module):
    """Sigmoid Loss for Language Image Pre-training (Zhai et al. 2023)."""
    def __init__(self, init_t=10.0, init_b=-10.0):
        super().__init__()
        # log-parameterize t for stability; b is a learnable bias
        self.t_prime = nn.Parameter(torch.tensor(init_t).log())
        self.b = nn.Parameter(torch.tensor(float(init_b)))

    def forward(self, image_feats, text_feats):
        u = F.normalize(image_feats, dim=-1)             # [N, D]
        v = F.normalize(text_feats, dim=-1)              # [N, D]

        t = self.t_prime.exp()                           # scale > 0
        logits = t * (u @ v.t()) + self.b                # [N, N]

        # y_{ij} = +1 if i == j else -1
        N = u.size(0)
        labels = 2 * torch.eye(N, device=u.device) - 1   # [N, N], +1 on diag, -1 off

        # log(1 + exp(-y * logits))  ==  -log sigmoid(y * logits)
        loss = -F.logsigmoid(labels * logits).sum() / N  # SigLIP convention: sum / N
        return loss, t, self.b

⚠️ SigLIP normalizes by N, not N² — the paper's Eq. (1) normalizes by batch size $N$ (sum per row), not by number of matrix elements $N^2$. Pitfall: writing loss.mean() gives a 1/N² magnitude, the loss is too small, and the learnable scale converges incorrectly. Correct: loss.sum() / N.

§5 EVA-CLIP / OpenCLIP / other CLIP variants

5.1　OpenCLIP

OpenCLIP (Cherti et al. 2023 CVPR) is the LAION team's open-source reproduction + extension:

• Open training recipe: full LAION-400M / LAION-2B training scripts

• Larger scale: ViT-bigG/14 trained on LAION-2B reaches zero-shot ImageNet ~80.1% (SOTA in 2023)

• Distributed InfoNCE: implements local_loss=True with gradient checkpointing; each GPU only stores local rows/columns in memory

5.2　EVA-CLIP

EVA-CLIP (Sun et al. 2023) uses MIM-pretrained EVA / EVA-02 (Fang et al. 2023) as vision-tower initialization, substantially improving sample efficiency:

• ViT-L/14 on LAION-2B needs only 1/3 of OpenCLIP's compute budget to reach the same accuracy

• LayerScale + sub-LN + RoPE: engineering improvements on EVA-02's visual side

5.3　DataComp (data vs model vs algorithm)

Gadre et al. 2023 (NeurIPS) designed a "data filtering benchmark": fix (model, compute) and only vary the data filter. Conclusions:

• Combination of CLIP filtering + basic filtering + image-based filtering works best

• Large models (ViT-L/14) on small data (12.8M) are worse than ViT-B (in the data-limited regime, larger models overfit)

5.4　Comparison overview

Method	Vision tower init	Loss	Batch	Training data	ImageNet zero-shot
CLIP (OpenAI)	from scratch	softmax InfoNCE	32k	WIT 400M	76.2% (L/14@336)
OpenCLIP	from scratch	softmax InfoNCE	90k	LAION-2B	80.1% (bigG/14)
EVA-CLIP	EVA-02 MIM	softmax InfoNCE	—	LAION-2B	82.0% (E/14+)
SigLIP	from scratch	sigmoid	32k	WebLI	82.0% (So400M/14)
SigLIP-2	from scratch	sigmoid + caption + distill	—	WebLI 10B	84%+
MetaCLIP	from scratch	softmax InfoNCE	—	reconstructed LAION-grade	79.2% (H/14)

💡 2024–2025 trend — the SigLIP family has stably surpassed CLIP on zero-shot ImageNet and downstream retrieval; representative open-weight VLMs using SigLIP-So400M are PaliGemma / LLaVA-OneVision. Molmo/PixMo's main models still use OpenAI CLIP ViT-L/14@336 (SigLIP appears only in their paper's ablation); the InternVL series uses their in-house InternViT; Qwen2-VL trains its own ViT; LLaVA-1.5/1.6 still use CLIP ViT-L/14 — "switching to SigLIP" is not an industry consensus.

§6 LLaVA: projector + 2-stage training

6.1　Architecture

LLaVA (Liu et al. 2023 NeurIPS) centers on a trio:

Image ──► CLIP ViT-L/14 ──► visual features  z_v ∈ R^{N × d_v}
                                  │
                                  │  W ∈ R^{d_v × d_LLM}   ← projector (1.0 single linear / 1.5 2-layer MLP)
                                  ↓
                            H_v ∈ R^{N × d_LLM}
                                  │
                                  │  concatenated with text embedding
                                  ↓
Text tokens ──► tokenizer ──► H_t ──► [<bos>, H_v, H_t] ──► LLM (Vicuna / LLaMA-2)
                                                              │
                                                              ↓
                                                            autoregressive response

• Vision tower: CLIP ViT-L/14 (frozen, taking the second-to-last layer's patch tokens). LLaVA-1.0 uses $224^2$ input giving $N=256$; LLaVA-1.5 upgrades to $336^2$, giving $N=576$

• Projector $W$: LLaVA-1.0 uses a single Linear; LLaVA-1.5 upgrades to 2-layer MLP + GELU (the paper reports a clear improvement in instruction following)

• LLM: Vicuna-13B (LLaVA-1.0/1.5) or LLaMA-2

6.2　Two-stage training

Stage 1: Feature Alignment Pre-training

• Use CC3M / LAION-558K caption data in the format <image>\n<caption>

• Train the projector $W$ only, freezing vision tower and LLM

• Goal: project visual features close to the LLM's word embedding space

Stage 2: End-to-end Visual Instruction Tuning

• Use 158K visual instructions generated by GPT-4 (LLaVA-Instruct)

• Unfreeze projector + LLM; vision tower remains frozen

• The LLM learns to "understand images, answer questions, follow visual instructions"

6.3　Code: LLaVA-style projector + forward (core 60 lines)

import torch
import torch.nn as nn

class LLaVAProjector(nn.Module):
    """2-layer MLP + GELU, as in LLaVA-1.5."""
    def __init__(self, d_vision=1024, d_llm=4096):
        super().__init__()
        self.fc1 = nn.Linear(d_vision, d_llm)
        self.act = nn.GELU()
        self.fc2 = nn.Linear(d_llm, d_llm)

    def forward(self, x):                               # x: [B, N, d_vision]
        return self.fc2(self.act(self.fc1(x)))          # [B, N, d_llm]

class LLaVA(nn.Module):
    """Skeleton: CLIP vision tower + projector + LLM."""
    def __init__(self, vision_tower, projector, llm, image_token_id):
        super().__init__()
        self.vision_tower = vision_tower                # CLIPViT, frozen at stage 1
        self.projector = projector
        self.llm = llm                                  # e.g. LlamaForCausalLM
        self.image_token_id = image_token_id           # special <image> placeholder

    @torch.no_grad()
    def encode_image(self, pixel_values):
        # Take the second-to-last layer's patch features (skip [CLS])
        vit_out = self.vision_tower(pixel_values, output_hidden_states=True)
        feat = vit_out.hidden_states[-2][:, 1:, :]      # drop CLS, [B, N, d_v]
        return feat

    def forward(self, input_ids, pixel_values, labels=None, attention_mask=None):
        # 1. Visual features → projector → LLM dim
        with torch.no_grad():
            visual_features = self.encode_image(pixel_values)        # [B, N, d_v]
        visual_tokens = self.projector(visual_features)              # [B, N, d_llm]

        # 2. LLM's word embedding table
        token_embeds = self.llm.get_input_embeddings()(input_ids)    # [B, L, d_llm]

        # 3. Replace the <image> placeholder positions with visual_tokens
        B, L, D = token_embeds.shape
        new_embeds, new_labels, new_mask = [], [], []
        for b in range(B):
            image_pos = (input_ids[b] == self.image_token_id).nonzero(as_tuple=True)[0]
            assert image_pos.numel() == 1, "exactly one <image> placeholder expected"
            i = image_pos.item()
            # Concat: [prefix tokens] + [N visual tokens] + [suffix tokens]
            chunks = [token_embeds[b, :i], visual_tokens[b], token_embeds[b, i+1:]]
            new_embeds.append(torch.cat(chunks, dim=0))
            if labels is not None:
                lab = labels[b]
                # Label = -100 at visual token positions (not counted in loss)
                ignore = torch.full((visual_tokens.size(1),), -100, dtype=lab.dtype, device=lab.device)
                new_labels.append(torch.cat([lab[:i], ignore, lab[i+1:]], dim=0))
            if attention_mask is not None:
                am = attention_mask[b]
                ones = torch.ones(visual_tokens.size(1), dtype=am.dtype, device=am.device)
                new_mask.append(torch.cat([am[:i], ones, am[i+1:]], dim=0))

        # 4. Pad back to a batch tensor and feed the LLM
        inputs_embeds = torch.nn.utils.rnn.pad_sequence(new_embeds, batch_first=True)
        labels = torch.nn.utils.rnn.pad_sequence(new_labels, batch_first=True, padding_value=-100) if labels is not None else None
        attention_mask = torch.nn.utils.rnn.pad_sequence(new_mask, batch_first=True) if attention_mask is not None else None
        return self.llm(inputs_embeds=inputs_embeds, labels=labels, attention_mask=attention_mask)

6.4　LLaVA-1.5 / 1.6 / NeXT key upgrades

Version	Time	Main changes
LLaVA-1.0	2023.04	Single Linear projector; CLIP ViT-L/14@224², visual tokens = 256 ($16\times 16$)
LLaVA-1.5	2023.10	2-layer MLP; resolution up to 336², visual tokens = 576 ($24\times 24$); adds OCR / GQA / VQAv2 academic data
LLaVA-1.6 / NeXT	2024.01	AnyRes: slice the image into $2\times 2 / 2\times 3 / \dots$ tiles and encode each, supporting any aspect ratio; token count $(1{+}n{\cdot}m)\cdot 576$ (2×2→2880, 2×3→4032)
LLaVA-OneVision	2024.08	Unified single / multi-image / video; introduces a mix of SI (single image) + OV (onevision) data
LLaVA-NeXT-Video	2024.04	Video version; feed the LLM serialized visual features from multiple frames

💡 Core trick of AnyRes (LLaVA-1.6) — training assumes a fixed 336²; at inference, a high-res image is sliced into $n \times m$ tiles of 336² each, encoded individually, plus one "global thumbnail" (the full image resized to 336²). Tokens go from 576 to (1 + n·m)·576, but each tile passes through the same frozen ViT. The same family as InternVL / Qwen-VL tiling.

§7 BLIP-2: Q-Former cross-attention

7.1　Motivation

LLaVA's projector is simple, but every patch becomes an LLM token: higher resolution ↑ more tokens ↑ LLM compute $O(L^2)$ ↑. BLIP-2 (Li et al. 2023 ICML) uses a Q-Former (Querying Transformer) to compress an arbitrary number of patches into a fixed 32 tokens.

7.2　Q-Former structure

Input: frozen image encoder output $\mathbf{Z} \in \mathbb{R}^{N \times d_v}$ (N=257 for ViT-g/14@224). The Q-Former has 32 learnable query tokens $\mathbf{q}_1, \dots, \mathbf{q}_{32} \in \mathbb{R}^{d_q}$.

Per Q-Former block:

$\mathbf{q}^{(\ell)} = \text{SelfAttn}(\mathbf{q}^{(\ell-1)})$ $\mathbf{q}^{(\ell)} = \text{CrossAttn}(\mathbf{q}^{(\ell)},\ \mathbf{Z},\ \mathbf{Z})\quad \text{(inserted only every other layer)}$ $\mathbf{q}^{(\ell)} = \text{FFN}(\mathbf{q}^{(\ell)})$

Key points:

• Inside self-attention: queries interact with each other, not with image patches
• Inside cross-attention: queries are Q, image patches are K/V — this is the information entry point
• The output $\mathbf{q}^{(L)} \in \mathbb{R}^{32 \times d_q}$ then goes through a Linear into the LLM dimension, serving as 32 visual tokens fed to the frozen LLM

7.3　Two-stage training

Stage 1: Representation Learning (only Q-Former trained, vision encoder frozen)

• ITC (Image-Text Contrastive): query embedding contrasted with text [CLS] in CLIP fashion
• ITM (Image-Text Matching): query and text tokens interact via cross-attn, followed by binary classification
• ITG (Image-grounded Text Generation): query does not interact with text; let the text decoder generate captions based on queries (causal mask controls visibility)

Stage 2: Generative Learning (only Q-Former trained, LLM frozen)

• Project Q-Former output into the LLM embedding space, so the LLM does prefix-tuned captioning / VQA based on the 32 visual tokens

7.4　Code: a single Q-Former cross-attention layer (core 40 lines)

import torch
import torch.nn as nn

class QFormerLayer(nn.Module):
    """One Q-Former block: SelfAttn (queries) -> CrossAttn (queries <- image) -> FFN."""
    def __init__(self, d_q=768, d_v=1408, num_heads=12, mlp_ratio=4, has_cross=True):
        super().__init__()
        self.has_cross = has_cross
        self.ln_self = nn.LayerNorm(d_q)
        self.self_attn = nn.MultiheadAttention(d_q, num_heads, batch_first=True)
        if has_cross:
            self.ln_cross = nn.LayerNorm(d_q)
            # Q comes from query (d_q), K/V come from image feats (d_v) -> adapt via kdim/vdim
            self.cross_attn = nn.MultiheadAttention(d_q, num_heads,
                                                   kdim=d_v, vdim=d_v, batch_first=True)
        self.ln_ffn = nn.LayerNorm(d_q)
        hidden = int(d_q * mlp_ratio)
        self.ffn = nn.Sequential(nn.Linear(d_q, hidden), nn.GELU(), nn.Linear(hidden, d_q))

    def forward(self, q, image_feats=None):              # q: [B, 32, d_q]
        # Self-attention: queries talk to each other
        h = self.ln_self(q)
        a, _ = self.self_attn(h, h, h, need_weights=False)
        q = q + a
        # Cross-attention: queries attend to image patches
        if self.has_cross and image_feats is not None:
            h = self.ln_cross(q)
            a, _ = self.cross_attn(h, image_feats, image_feats, need_weights=False)
            q = q + a
        # FFN
        q = q + self.ffn(self.ln_ffn(q))
        return q

class QFormer(nn.Module):
    def __init__(self, num_queries=32, d_q=768, d_v=1408, depth=12, num_heads=12,
                 cross_every=2):
        super().__init__()
        self.queries = nn.Parameter(torch.zeros(1, num_queries, d_q))
        nn.init.trunc_normal_(self.queries, std=0.02)
        self.layers = nn.ModuleList([
            QFormerLayer(d_q, d_v, num_heads, has_cross=(i % cross_every == 0))
            for i in range(depth)
        ])

    def forward(self, image_feats):                      # [B, N, d_v]
        B = image_feats.size(0)
        q = self.queries.expand(B, -1, -1)               # [B, 32, d_q]
        for layer in self.layers:
            q = layer(q, image_feats)
        return q                                         # [B, 32, d_q]

⚠️ kdim/vdim adaptation — nn.MultiheadAttention defaults to K/V input dim = embed_dim. In Q-Former cross-attn, query is 768-dim and image feats are 1408-dim, so you must explicitly pass kdim=d_v, vdim=d_v, otherwise PyTorch will expect 768-dim K/V at forward time and raise a shape mismatch error (no silent truncation).

7.5　Q-Former vs LLaVA Projector: trade-off

Dimension	LLaVA Projector	BLIP-2 Q-Former
Parameter count	~20M (MLP)	~180M (Q-Former + queries)
Compute	Only MLP forward	12 layers of cross-attn forward
Visual token count	$N$ (quadratic in resolution)	Fixed 32
Information loss	Almost 0 (every patch enters the LLM)	Significant (256+ patches compressed to 32)
Training complexity	1 stage (pretrain) + 1 stage (IT)	2 stages (representation + generation); stage 1 jointly optimizes ITC + ITM + ITG
LLM context usage	Large (576–2880 tokens)	Small (32 tokens)
Best for	High resolution / detail tasks	LLM context-limited / batched multimodal inference

💡 2024–2025 mainstream returns to projector — Qwen-VL / LLaVA-NeXT / InternVL-2 / DeepSeek-VL2 all use projector (with spatial reduction / pixel shuffle to control token count); Q-Former has faded out in industrial VLMs. But the Q-Former idea is still active in video VLMs (using queries for frame-level pooling).

§8 Flamingo: Perceiver Resampler + Gated Cross-Attn

8.1　Design goals

Alayrac et al. 2022 (NeurIPS) wanted: to add visual ability to a frozen 70B LLM without breaking the text ability. Design choices:

• Do not retrain the LLM: fully frozen; insert only new layers in the middle

• Few trainable parameters: cross-attn layers + Perceiver Resampler

• Few-shot interleaved: training data is interleaved sequences of (image, text, image, text, ...)

8.2　Perceiver Resampler

Similar to Q-Former, "use latent queries to compress the image". The Flamingo paper Sec 3.1 pseudocode is multi-layer (each layer = cross-attention + FFN), with the default config approximately $L=6$ layers. Per-layer update:

$\mathbf{q}^{(\ell+1)} = \mathbf{q}^{(\ell)} + \text{CrossAttn}\!\left(\mathbf{q}^{(\ell)},\ [\mathbf{q}^{(\ell)};\ \mathbf{Z}],\ [\mathbf{q}^{(\ell)};\ \mathbf{Z}]\right), \quad \mathbf{q}^{(\ell+1)} = \mathbf{q}^{(\ell+1)} + \text{FFN}(\mathbf{q}^{(\ell+1)})$

Note K/V is concat(query, image_feat), not just image_feat — so query tokens can also attend to each other. Overall still lighter than BLIP-2 Q-Former (12 layers + self-attn + cross-every-2). Output 64 latent visual tokens (independent of the input patch count).

8.3　Gated Cross-Attention (the core innovation)

Insert a new cross-attention module into the LLM every $k$ layers (e.g. every 4):

$\mathbf{h}'_\ell = \mathbf{h}_\ell + \tanh(\alpha_\text{attn}) \cdot \text{CrossAttn}(\mathbf{h}_\ell, \mathbf{q}_\text{out}, \mathbf{q}_\text{out})$ $\mathbf{h}''_\ell = \mathbf{h}'_\ell + \tanh(\alpha_\text{ffn}) \cdot \text{FFN}(\mathbf{h}'_\ell)$

Key: $\alpha_\text{attn}, \alpha_\text{ffn}$ are learnable scalars, initialized to 0. So $\tanh(0)=0$, meaning the added cross-attn contributes zero to the LLM output at init — the LLM behaves identically to the unmodified frozen LLM with no visual module. During training, $\alpha$ gradually learns nonzero values and visual information starts flowing in.

✅ This is the "residual graft" — Llama-3.2-Vision (Meta 2024) uses exactly the same design: frozen LLaMA-3 + learning a gated cross-attn adapter. Pros: fully preserves text-only performance; cons: the visual capability ceiling is lower than the fine-tuned LLM in LLaVA/Qwen-VL.

8.4　Flamingo / Llama-3.2-V vs LLaVA comparison

Aspect	Flamingo / Llama-3.2-V	LLaVA / Qwen-VL
LLM unfrozen?	No (frozen)	Yes (unfrozen in stage 2)
Image as tokens?	No (as KV)	Yes (as tokens)
Text-only ability preserved	✅ Fully	⚠️ May regress slightly
Visual understanding ceiling	Limited by cross-attn capacity	Higher (LLM can "think about" the image)
Training data	interleaved	image-instruction pairs
Applicable	Large LLM + no retraining	Small/medium LLM + vision-centric

§9 CogVLM and "visual experts" / cross-attn fusion variants

9.1　CogVLM: a visual expert branch

Wang et al. 2023 (CogVLM)'s core idea: in the LLM's attention / FFN, replicate a parallel branch for visual tokens, sharing the attention computation with the original text branch but using different projections:

                  attention
       ┌──────────────┴──────────────┐
       ↓                              ↓
   text projection (frozen)    vision expert projection (trainable)
       │                              │
       └──────────────┬──────────────┘
                      ↓
         token-wise route: if visual_token, use vision branch

• MoE-like dual experts: text tokens go through the text branch, image tokens go through the "visual expert" branch

• Visual expert trained alone: text branch can be frozen, with visual ability coming from the expert

• Preserves the LLM's native text ability (same idea as Flamingo, but routing at the token rather than the layer level)

9.2　Llama-3.2 Vision: Flamingo-style cross-attn revived on a large LLM

Meta released Llama-3.2-V (11B / 90B) in September 2024:

• Frozen Llama-3 backbone

• Adds separate cross-attention layers (not modifying self-attn)

• Adapter-style training, only the cross-attn layers are trained

• Designed as a "robust base for long-tail visual tasks"; the visual ceiling is slightly below GPT-4V, but text-only benchmarks are nearly identical to the text-only Llama-3

9.3　Closed-source unified multimodal: GPT-4V/4o → GPT-5.5, Claude Fable/Opus, Gemini 3.1

The closed-source architectures of Anthropic / OpenAI are undisclosed, but inferences from API behavior:

• GPT-4V (2023.09) / GPT-4o (2024.05) → GPT-5.5 (2026): 4o put image + text + audio into a native multimodal product line; GPT-5.5-style APIs emphasize complex reasoning, coding, tool/workflow integration, and multimodal inputs rather than single-image VQA.
• Claude 3.5/3.7 Sonnet → Claude Fable 5 / Opus 4.8: Claude's public positioning keeps emphasizing high-resolution images, PDF/screenshot document understanding, long-horizon agentic work, and computer/tool use; the internal vision tower, tiling, and fusion design remain undisclosed.
• Gemini 3.1 Pro Preview: public docs list text, image, video, audio, and PDF input, million-token input, thinking, function calling, and tool use. It represents a 2026 product shape: multimodal understanding + long context + tool loop.
• Common feature: they handle multi-page documents, screenshots, OCR, math formulas, video/audio context, and connect visual understanding to tool calls. They likely contain major engineering around document corpora, tiling/token budgets, multimodal alignment, and safety filters, but the training recipe and architecture are still black boxes.

§10 Qwen2-VL / DeepSeek-VL: dynamic resolution + M-RoPE

10.1　Native dynamic resolution

Qwen2-VL (Wang et al. 2024), DeepSeek-VL (Lu et al. 2024), and InternVL-2 all abandon the "resize to fixed 224²" tradition:

• Preserve original aspect ratio: resize images to the largest size near the original that is an integer multiple of patch_size

• Dynamic patch count: a $1024 \times 768$ image at $P=14$ slices into $73 \times 54 \approx 3942$ patches

• No fixed pos embed table: must use an extensible positional encoding (RoPE or 2D ALiBi-like)

10.2　M-RoPE (Multimodal RoPE)

Qwen2-VL's core innovation. Recap of ordinary 1D RoPE: treat each pair $(2k, 2k+1)$ of query / key dimensions as a complex number, multiplied by a position-dependent rotation:

$\mathbf{R}_{m,k} = \begin{pmatrix} \cos(m\theta_k) & -\sin(m\theta_k) \\ \sin(m\theta_k) & \cos(m\theta_k) \end{pmatrix}, \quad \theta_k = 10000^{-2k/d}$

After applying to $\mathbf{q}_m$, $\mathbf{q}_m^\top \mathbf{k}_n$ depends only on $m - n$ (relative position).

M-RoPE's extension: a visual token has three position dimensions (t, h, w). All head_dim dimensions rotate — but each pair $(2k, 2k+1)$ uses one of the three position ids (t / h / w) for its rotation angle, depending on which segment it falls into:

$(\cos(m_\text{axis}\,\theta_k),\ \sin(m_\text{axis}\,\theta_k)), \quad \text{axis} \in \{t, h, w\}$

Specifically, Qwen2-VL's mrope_section (unit is pairs of half head_dim, i.e. each number represents how many $(2k, 2k+1)$ pairs). One pair = 2 real dimensions, so "section sum × 2 = head_dim".

💡 Qwen2-VL default mrope_section = [16, 24, 24] — that is, the three axes occupy 16 / 24 / 24 dim pairs respectively; total $(16+24+24) \times 2 = 128 =$ head_dim. The implementation doubles the section to $[16, 24, 24, 16, 24, 24]$ to slice head_dim, with position ids of (t, h, w, t, h, w) used for rotation — all 128 dims rotate, none "left unrotated". Spatial dims (h, w) occupy 48 pairs > the temporal dim (t)'s 16 pairs, reflecting that inter-frame changes in video are slow while spatial content changes within a frame are dramatic.

A text token has no explicit (h, w): Qwen2-VL sets $m_t = m_h = m_w$ equal to that text token's 1D position id, so all three axes yield the same rotation angle, equivalent to ordinary 1D RoPE.

10.3　Qwen2.5-VL upgrades

Qwen2.5-VL (Bai et al. 2025), on top of Qwen2-VL:

• Absolute time encoding: the M-RoPE t dim switches to real timestamps (seconds), not frame indices, supporting arbitrary FPS videos
• Dynamic visual token budget: tune token count by task complexity
• Agent / GUI capability: add web screenshots / mobile UI operation traces to training data

10.4　DeepSeek-VL / VL2: high-resolution tiling + hybrid encoder

DeepSeek-VL (Lu et al. 2024) uses dual vision encoders:

• SigLIP: for global semantics (low resolution)
• SAM-B (Segment Anything backbone): for high-resolution detail

The two features are concatenated and fed to projector + LLM. DeepSeek-VL2 (2024.12) further replaces the LLM with MoE + dynamic resolution; a single image can use 1700+ visual tokens.

10.5　Code: M-RoPE three-dim positional embedding (core 50 lines, matching Qwen2-VL's HF implementation)

import torch

def build_mrope_cos_sin(positions, head_dim, mrope_section=(16, 24, 24), base=1000000.0):
    """
    Build cos/sin tensors for Qwen2-VL style M-RoPE.

    positions: LongTensor [3, B, L]   (axis 0: t / h / w; B batch; L seq len)
    head_dim:  per-head dim (must equal 2 * sum(mrope_section))
    mrope_section: tuple of 3 ints; each = number of (half-dim) entries per axis
    Returns: cos, sin both [B, L, head_dim], ready for LLaMA-style rotate_half.
    """
    assert 2 * sum(mrope_section) == head_dim, "2 * sum(mrope_section) must = head_dim"
    half = head_dim // 2                                                # = sum(mrope_section)

    # Standard RoPE frequencies: θ_k = base^{-2k/head_dim}, k = 0..half-1
    inv_freq = 1.0 / (base ** (torch.arange(0, half).float() * 2 / head_dim))   # [half]
    inv_freq = inv_freq.to(positions.device)

    # For each axis, compute angles / cos / sin of shape [B, L, half]
    cos_axes, sin_axes = [], []
    for a in range(3):
        ang = positions[a].float().unsqueeze(-1) * inv_freq                     # [B, L, half]
        cos_axes.append(ang.cos())
        sin_axes.append(ang.sin())

    # Slice half-dim into 3 segments by mrope_section; pick cos/sin for t/h/w
    cos_chunks, sin_chunks = [], []
    offset = 0
    for axis, s in enumerate(mrope_section):
        cos_chunks.append(cos_axes[axis][..., offset:offset+s])                 # [B, L, s]
        sin_chunks.append(sin_axes[axis][..., offset:offset+s])
        offset += s
    cos_half = torch.cat(cos_chunks, dim=-1)                                    # [B, L, half]
    sin_half = torch.cat(sin_chunks, dim=-1)

    # LLaMA-RoPE style: duplicate to full head_dim
    cos = torch.cat([cos_half, cos_half], dim=-1)                               # [B, L, head_dim]
    sin = torch.cat([sin_half, sin_half], dim=-1)
    return cos, sin

def rotate_half(x):
    """(x1, x2) -> (-x2, x1), LLaMA convention."""
    x1, x2 = x.chunk(2, dim=-1)
    return torch.cat((-x2, x1), dim=-1)

def apply_mrope(q, k, cos, sin):
    """
    q, k:    [B, num_heads, L, head_dim]
    cos, sin:[B, L, head_dim]
    """
    cos = cos.unsqueeze(1)                                                       # broadcast over heads
    sin = sin.unsqueeze(1)
    q_rot = q * cos + rotate_half(q) * sin
    k_rot = k * cos + rotate_half(k) * sin
    return q_rot, k_rot

⚠️ Three common M-RoPE misreadings — easy traps.

• "head_dim split into t/h/w three independent 1D RoPE segments": wrong. Qwen2-VL is in fact 6 alternating segments [s_t, s_h, s_w, s_t, s_h, s_w], with the full head_dim rotated

• "section units are dims": wrong. mrope_section=[16,24,24] units are pairs (each pair = 2 head_dim elements), $\sum \times 2 =$ head_dim = 128

• "What if a text token has no (h, w)?": set $m_t = m_h = m_w$ to the text's 1D position id; the three axes give the same rotation angle, degenerating to 1D RoPE

§11 Video VLM: LongVA / VideoLLaMA / long-video problem

11.1　Basic pipeline

Video = multi-frame image. Common VLM approach to video:

1. Uniformly sample $K$ frames (e.g. 8 / 16 / 32)
2. Each frame through the vision encoder → $N$ patch tokens per frame
3. Token sequence concat: feed $K \cdot N$ visual tokens to the LLM

Problem: $K=32, N=576 \Rightarrow 18432$ tokens — beyond the context of an LLM that was instruction-tuned on single images.

11.2　Common compression strategies

• Time pooling: average pool adjacent frame tokens (VideoChat / VideoLLaMA)
• Q-Former resampler: per-frame query tokens compress to 32 (Video-BLIP)
• Token merge: merge similar tokens across frames (VideoLLaMA 2)
• Spatial pooling + temporal preservation: pool per-frame patch tokens to $H' \times W'$, retaining all frames (LLaVA-NeXT-Video)

11.3　LongVA / Long-context video

LongVA (Zhang et al. 2024) and others exploit long-context LLMs (200K+ tokens) to directly consume long video unrolled into a token sequence, paired with the M-RoPE temporal dim, for hour-long video QA. Qwen2-VL reports handling 20-minute video; Qwen2.5-VL pushes to 1+ hour.

11.4　Video benchmarks

• MVBench (Li et al. 2024): 20 fine-grained video understanding tasks
• Video-MME (Fu et al. arXiv 2024 / CVPR 2025): 900+ videos covering 3 duration tiers (short / medium / long) + 6 task categories
• EgoSchema (Mangalam et al. 2023 NeurIPS): first-person long video
• LongVideoBench (Wu et al. 2024 NeurIPS): from 8 seconds to 1 hour

§12 Training pipeline: alignment / instruct / preference

12.1　Stage 1: Alignment / Pre-training

Goal: align visual features to be near the LLM token space.

• Data: image-caption pairs (CC3M, LAION-558K, ShareGPT4V)
• Training: unfreeze projector only (LLaVA) / Q-Former (BLIP-2); vision tower + LLM frozen
• Loss: LLaVA uses next-token prediction (the LLM takes visual features as a prefix to generate a caption); BLIP-2 does not connect an LLM here — the Q-Former's Stage 1 is representation learning, jointly optimizing ITC + ITM + ITG over a frozen image encoder, and the frozen LLM is only attached at Stage 2 (generative learning)

12.2　Stage 2: Visual instruction tuning

Goal: teach the VLM "to look at images, answer questions, follow instructions".

• Data: GPT-4-generated visual instructions (LLaVA-Instruct-158K, ShareGPT4V) + academic VQA data (VQAv2, GQA, OCR-VQA, TextVQA)
• Training: unfreeze LLM + projector; vision tower typically stays frozen (LLaVA-1.5 / Qwen-VL). Qwen2-VL unfreezes the vision tower for end-to-end fine-tuning in the final stage
• Loss: next-token prediction on answer tokens (input image + question are not counted in loss)

12.3　Stage 3: Preference / RLHF

Goal: reduce hallucinations, improve helpfulness / harmlessness, and align to long-tail tasks.

Method	Time	Core
LLaVA-RLHF	2023.09, Sun et al.	PPO + human preference + hallucination-aware reward
RLAIF-V	2024, Yu et al.	AI feedback in place of human labels; divide-and-conquer
POVID	2024	DPO + deliberately constructed hallucination negatives
VLM-R1	2025	GRPO + verifiable reward (R1-style for visual reasoning)
Bespoke / R1-Onevision	2025	Visual chain-of-thought + RL refinement

💡 VLM-R1 (2025) is the current hotspot — it ports DeepSeek-R1's "verifiable reward + GRPO" recipe to visual tasks (e.g. ScienceQA, MMMU). Reward comes from whether the answer matches the ground truth (no process reward model); after training, the visual reasoning chain grows significantly and benchmarks improve substantially.

12.4　Data scale vs stage

Stage	Data volume	Training tokens	Unfrozen modules
Alignment	0.5–5M captions	1–10B	projector
Instruction tune	0.2–10M instructions	1–50B	LLM + projector
Preference	50k–500k preference pairs	100M–1B	LLM (LoRA / full)

§13 Multimodal Embeddings: BGE-VL / Jina-CLIP / VLM2Vec

13.1　Why a new generation of multimodal embeddings?

CLIP is trained for "image ↔ short caption" alignment, but performs poorly on long instruction retrieval / multi-image / interleaved document retrieval. The new generation of multimodal embedding models target retrieval / RAG scenarios.

13.2　Representative methods

• Jina-CLIP-v1 (2024): adds long-text contrastive (text-text task) + multi-resolution on top of CLIP; a single embedding model does image-text and text-text retrieval
• BGE-VL (2024): the BGE team's multimodal version; uses SigLIP-So400M + a small LLM for instruction-aware retrieval
• VLM2Vec (Jiang et al. 2024): instruction-conditioned mean pool of the last hidden state of a VLM (LLaVA / Qwen-VL); trains only a contrastive head, significantly beating CLIP on the MMEB benchmark
• mmE5 (2024): multimodal version of E5, supporting 12 types of retrieval tasks
• Gemini Embedding / Cohere Embed 4 (2025-2026): product APIs are starting to fold text, images, PDFs/documents, and more complex enterprise data into unified embedding/RAG workflows. In engineering, check input modalities, dimensions, context length, Matryoshka/truncation support, reranker availability, and pricing/latency; do not look only at the MTEB average.

13.3　Core tricks

• Instruction-aware: embedding input is [instruction][image][query], so the same image yields different embeddings under different tasks
• VLM-as-encoder: directly use an instruct-tuned VLM as backbone, no separate contrastive pretrain
• Hard negative mining: use a cross-encoder reranker to mine hard negatives outside the batch

§14 25 frequently-asked interview questions (L1 must-know · L2 advanced · L3 top labs)

L1 basics (must-know 10)

Q1. What is the CLIP loss? Why must it be symmetric?

• CLIP uses symmetric InfoNCE: $\mathcal{L} = \tfrac12(\mathcal{L}_{i\to t} + \mathcal{L}_{t\to i})$

• $\mathcal{L}_{i\to t}$: apply row softmax on the similarity matrix $\mathbf{S}$, take NLL of the diagonal (image retrieves text)

• $\mathcal{L}_{t\to i}$: column softmax, NLL of the diagonal (text retrieves image)

• Necessity of symmetry: one direction constrains only one retrieval direction; symmetrization lets image / text embeddings constrain each other, preventing "one-sided collapse" — e.g. image-side clusters but text-side drifts

Pitfall: answering only "InfoNCE" without saying "average of two softmaxes", or saying "do it backward once" without explaining why it is necessary.

Q2. What does the temperature τ in CLIP do? Why make it learnable?

• Temperature $\tau$ controls softmax sharpness: $\tau \to 0$ approaches one-hot, focusing on the hardest negative; $\tau \to \infty$ becomes uniform with no gradient

• OpenAI CLIP makes $\tau$ learnable (concretely parameterized as logit_scale = log(1/τ), more stable in backprop)

• Learned steady state $\tau \approx 0.01$ (logit_scale ≈ log(100)), clamped from above to avoid collapse

• Without making it learnable: hyperparameter-sensitive; each new data / model scale needs manual tuning

Pitfall: treating τ as a fixed 0.07 with no learning; or writing it as 1/exp(logit_scale) with unstable backprop.

Q3. Core change of SigLIP vs CLIP? Why is batch size no longer sensitive?

• Replace softmax InfoNCE with sigmoid binary CE: each $(i,j)$ pair is independently classified

• $S_{ij} = t \cdot \mathbf{u}_i^\top \mathbf{v}_j + b$; label $y_{ij} = +1 (i=j) / -1 (i\neq j)$; loss = $-\sum_{ij} \log\sigma(y_{ij} S_{ij})/N$

• Batch decoupled: each loss term is independent of row/col normalization, so doubling N only changes the negative count, not the loss landscape

• Engineering gains: single-machine large batch, simpler cross-node communication, and small batches can also learn (CLIP barely converges with small batches)

Pitfall: saying only "use sigmoid" without explaining batch-independence; or treating SigLIP as "CLIP with bias".

Q4. Is the [CLS] token in ViT mandatory?

• No. The original ViT uses [CLS] to align with BERT conventions

• Alternative: mean-pool all patch tokens as the image representation — most modern ViTs (DeiT-III, SigLIP, EVA-CLIP) use mean-pool or attentive pool

• CLIP uses [CLS]: contrastive training requires a single vector

• VLM vision towers usually drop [CLS]: LLaVA takes second-to-last layer patch tokens; [CLS] is not needed on the LLM side

Pitfall: treating [CLS] as a "mandatory component" of ViT; or saying "without [CLS] you cannot classify" (wrong; mean-pool also works).

Q5. What is LLaVA's projector? Why MLP instead of Linear?

• The projector maps visual encoder output ($d_v$=1024) to LLM token space ($d_\text{llm}$=4096)

• LLaVA-1.0: single Linear(1024, 4096); LLaVA-1.5: 2-layer MLP + GELU

• MLP adds non-linear expressivity, letting visual features map more flexibly into the LLM's "vocabulary"

• The paper reports MLP improves MM-Vet / SEED-Bench by 1–3 points over single Linear

Pitfall: saying only "project with a Linear"; or answering "use Q-Former" (that is BLIP-2, not LLaVA).

Q6. What do LLaVA's two training stages do?

• Stage 1 Feature Alignment: train the projector only, freeze vision tower + LLM, using caption data (CC3M / LAION-558K) to project visual features near the LLM embedding space

• Stage 2 Instruction Tuning: unfreeze LLM + projector (vision tower still frozen), using GPT-4-generated 158K visual instructions, teaching the LLM to follow visual instructions

• Why not train in one shot: jumping directly to stage 2 risks catastrophic forgetting of text ability; stage 1 first gives visual tokens a "near text-token" initialization, then instruction tuning is more stable

Pitfall: saying both stages "train the projector"; or omitting that stage 1 freezes the LLM, the key point.

Q7. What is a Q-Former? Pros/cons vs the LLaVA projector?

• BLIP-2's Q-Former: a 12-layer Transformer with 32 learnable query tokens that read information from a frozen image encoder via cross-attention, outputting a fixed 32 visual tokens

• Pros: fixed visual token count, low LLM context usage, compute budget stays the same as resolution grows

• Cons: large information loss (256 patches compressed to 32), more parameters (~180M), complex training (two stages: stage 1 representation learning includes joint ITC+ITM+ITG, stage 2 connects to frozen LLM for generation)

• 2024 mainstream returns to projector: Qwen-VL / LLaVA-NeXT / InternVL-2 all use projector

Pitfall: treating Q-Former as a synonym for projector; or not knowing modern VLMs prefer projectors.

Q8. How many patches / tokens does a ViT-L/14 yield on a 224×224 image?

• Patch count = $(224/14)^2 = 16^2 = 256$

• Token count = 256 + 1 (with [CLS]) = 257

• If LLaVA-style second-to-last layer patch tokens (drop [CLS]) = 256 visual tokens

• If resolution is 336 (LLaVA-1.5): $(336/14)^2 = 24^2 = 576$ tokens

Pitfall: miscalculating $(H/P)^2$ (treating $P^2$ as the patch count $N$); forgetting [CLS].

Q9. Why is CLIP poor at OCR / counting / spatial relations?

• Poor OCR: captions typically describe scenes, not text inside images; CLIP has no pixel-level OCR supervision

• Poor counting: captions rarely report exact counts ("how many birds" is usually "a flock of birds"); the embedding space does not preserve a counting signal

• Poor spatial relations: "cat on top of dog" and "dog on top of cat" look almost identical under bag-of-words; Yuksekgonul et al. 2023 (ICLR) quantify this with the ARO benchmark

• Mitigation directions: DETR-style local alignment, SigLIP-2's dense local objectives, document-level data

Pitfall: blaming OCR weakness on "resolution too low" (partly right, but the root cause is data + loss); claiming "CLIP is bag-of-words" too absolutely.

Q10. Why is the vision tower generally frozen when training a VLM?

• The vision tower (e.g. CLIP ViT-L) is already pre-trained well on its own data; unfreezing easily damages visual feature quality

• Training data is far smaller than CLIP pretraining (millions vs billions); unfreezing easily overfits

• Freezing also saves memory: hundreds of millions of vision tower params don't need optimizer state

• Qwen2-VL exception: in the final stage it unfreezes the vision tower for small-LR fine-tuning, paired with large mixed data to avoid forgetting

Pitfall: answering "cannot unfreeze" directly — wrong; late stages can carefully unfreeze.

L2 advanced (10 questions)

Q11. Why is SigLIP's bias $b$ initialized to $-10$?

• Early in training, embeddings are near random, $\mathbf{u}^\top \mathbf{v} \approx 0$, sigmoid outputs 0.5

• In the N×N matrix, negatives are $N^2 - N \approx N^2$, positives only $N$; if initial predictions are all 0.5, negative gradients dominate and positives get no useful signal

• Init $b \approx -10$ → $\sigma(b) \approx 4.5e^{-5}$ → all entries initially predicted as negative

• This way negatives have almost no loss, positives have large loss (predicted as negative but truly positive), gradient pulls positives in, training is stable

Pitfall: saying "avoid numerical issues"; or answering "for the symmetric term" (wrong; the bias is not a symmetric loss term).

Q12. Why is Flamingo's gated cross-attn initialized to 0?

• The new cross-attn output is multiplied by $\tanh(\alpha)$, with $\alpha$ initialized to 0

• $\tanh(0) = 0$, so at init the new module contributes zero to the frozen LLM — the LLM behaves identically to a text-only Llama without the visual module

• During training $\alpha$ slowly grows from 0, and visual signals are gradually injected

• Pros: fully preserves the frozen LLM's text-only ability; cons: visual ceiling limited by cross-attn capacity

• Llama-3.2 Vision uses the same design

Pitfall: treating the 0 init as just a "common init trick"; or not realizing it concerns frozen-LLM capability preservation.

Q13. How is LLaVA-1.6 / NeXT's AnyRes implemented?

• Training assumes a fixed 336²; inference slices a high-res image by aspect ratio into $n \times m$ tiles of 336² each (e.g. $2\times 2, 2\times 3$)

• Each tile passes through the frozen ViT to get 576 tokens; add a global thumbnail (whole image resized to 336² and encoded)

• Concatenate: $(1 + n\cdot m) \times 576$ visual tokens to the LLM

• Choice of slicing: pick the grid (from a predefined set such as $\{1\times 1, 2\times 2, 1\times 4, 4\times 1, ...\}$) closest to the original aspect ratio

Pitfall: treating AnyRes as a synonym for dynamic resolution — technically different. Qwen2-VL is native dynamic (patch count fully free), while LLaVA-1.6 is fixed-tile composition.

Q14. Why is Qwen2-VL's M-RoPE three-dim allocation not uniform?

• Qwen2-VL mrope_section = [16, 24, 24] (units are pairs of half head_dim), $\sum \times 2 =$ head_dim = 128

• All 128 dims rotate — different dim pairs use different axes (t/h/w) and their position ids to compute rotation angles

• Reason for non-uniform allocation:

  • Inter-frame variation in video is slow (adjacent frames are very similar), so $s_t = 16$ has a small share

  • Intra-frame patch variation is dramatic (large visual differences across positions in a single frame), so $s_h = s_w = 24$ each need broader frequency coverage

• $s_h = s_w$: the image H/W axes are symmetric

Pitfall: treating section as "number of dims" (wrong; the unit is pairs = head_dim / 2 allocation); or thinking "the remaining dims do not rotate".

Q15. Why does BLIP-2 choose 32 query tokens?

• 32 is an empirical value, balancing LLM context usage vs information capacity

• Too few (< 16): large information loss, hurting VQA / detail tasks

• Too many (> 64): large LLM context usage, expensive Q-Former cross-attn computation

• The BLIP-2 paper's ablations show 32 is the sweet spot on most downstream tasks

• Conceptually similar to Perceiver (also uses latent queries to compress inputs)

Pitfall: answering only "empirical"; not realizing it is an engineering trade-off between context budget and information capacity.

Q16. How does CLIP compute InfoNCE under DDP?

• Local batch on each GPU $N_\text{local}$; total batch over N GPUs $N = K \cdot N_\text{local}$

• After forward on each GPU, dist.all_gather fetches everyone's image / text feats

• Compute the global similarity $\mathbf{S} \in \mathbb{R}^{N \times N}$

• But backward only lets this GPU's $N_\text{local}$ rows / columns contribute gradients (to avoid duplicate backward)

• This is OpenCLIP's local_loss=True option

Pitfall: saying only "all-gather"; not knowing backward needs to avoid duplicate computation; or thinking backward also does an all-gather (wrong; backward flows along the communication backward path).

Q17. Why is Llama-3.2-V's visual ceiling lower than LLaVA / Qwen-VL?

• Llama-3.2-V uses frozen LLM + gated cross-attn adapter; LLM weights are unchanged

• LLaVA / Qwen-VL unfreeze the LLM, so its internal attention can reorganize to handle visual tokens specifically

• The latter can "use self-attention to think about the image"; the former can only passively receive visual signal via cross-attn

• Trade-off: Llama-3.2-V perfectly preserves text ability, LLaVA-Qwen may regress slightly but has a higher visual ceiling

Pitfall: saying only "fewer parameters"; not recognizing this is an architecture-level ceiling difference.

Q18. Why is much visual instruction-tuning data generated by GPT-4?

• Raw caption data (CC3M / LAION) is short, not instruction-style, cannot teach dialog ability

• Human-labeled visual instructions (e.g. VQAv2 questions) are small in scale and stylistically uniform

• GPT-4 + image + caption → generate multi-turn dialog / reasoning tasks / detailed descriptions: this is how LLaVA-Instruct-158K was made

• Prompt engineering controls coverage (three classes: detailed description, conversation, complex reasoning)

Pitfall: answering "lots of data"; not recognizing the key bottleneck is instruction style + diversity.

Q19. What is a typical CLIP / SigLIP training batch size?

• OpenAI CLIP: 32k batch (256 GPUs × ~128/GPU)

• OpenCLIP: up to 90k batch (LAION-2B)

• SigLIP: 32k batch is typically enough; the sigmoid loss makes each (i,j) entry independent and avoids softmax's batch-wide sync; the paper scans up to 256k batch but with diminishing returns

• Why small batches don't work: InfoNCE's MI lower bound $I(U;V) \ge \log N - \mathcal{L}$ tightens with larger N; the number of negatives also controls contrastive difficulty

• After SigLIP decouples batch, small batches improve significantly (1k batch can learn reasonable embeddings)

Pitfall: answering "a few hundred"; or not knowing the theoretical link between batch and InfoNCE.

Q20. What do POPE / Winoground / MMBench / MMMU each evaluate?

• POPE (Li et al. 2023): measures object hallucination — does the VLM claim objects exist that aren't in the image (yes/no binary)

• Winoground (Thrush et al. 2022): measures compositionality / word-order sensitivity — can it distinguish "cat on dog" vs "dog on cat"

• MMBench (Liu et al. 2023): general multimodal evaluation, ~3000 questions covering OCR / object recognition / reasoning, etc.

• MMMU (Yue et al. 2024 CVPR): university-level professional knowledge (math / physics / medicine, etc.), tests multimodal reasoning

• MM-Vet (Yu et al. 2023): integrated evaluation across 6 capabilities (recognition / knowledge / OCR / spatial / language / math)

Pitfall: confusing POPE and MMBench; not knowing Winoground is a "compositionality stress test".

L3 advanced (top labs / research directions, 5 questions)

Q21. Derive CLIP's symmetric InfoNCE = average of row + column softmax, and explain why SigLIP can be batch-independent.

Let batch size $N$ and similarity matrix $S_{ij} = \mathbf{u}_i^\top \mathbf{v}_j / \tau$.

CLIP derivation:

Row-wise softmax, $p_{ij} = \frac{\exp(S_{ij})}{\sum_k \exp(S_{ik})}$. Image→Text NLL:

$\mathcal{L}_{i\to t} = -\frac{1}{N}\sum_i \log p_{ii} = -\frac{1}{N}\sum_i \log \frac{\exp(S_{ii})}{\sum_j \exp(S_{ij})}$

Column-wise softmax (Text→Image):

$\mathcal{L}_{t\to i} = -\frac{1}{N}\sum_j \log \frac{\exp(S_{jj})}{\sum_i \exp(S_{ij})}$

Symmetric loss: $\mathcal{L} = \tfrac12 (\mathcal{L}_{i\to t} + \mathcal{L}_{t\to i})$. Notice the gradient with respect to $S_{ij}$:

$\frac{\partial \mathcal{L}_{i\to t}}{\partial S_{ij}} = \frac{1}{N}(p_{ij} - \delta_{ij})$

The gradient at each $S_{ij}$ depends on the entire row's softmax normalization $\sum_k \exp(S_{ik})$. So changing N (adding / removing negatives) changes all $p$ values in that row — gradients couple the batch.

SigLIP derivation:

$S_{ij} = t \cdot \mathbf{u}_i^\top \mathbf{v}_j + b$, $y_{ij} = 2\delta_{ij} - 1$,

$\mathcal{L}_\text{SigLIP} = \frac{1}{N}\sum_{i,j} \log(1 + \exp(-y_{ij} S_{ij}))$

Gradient:

$\frac{\partial \mathcal{L}}{\partial S_{ij}} = \frac{1}{N}\cdot \frac{-y_{ij}}{1 + \exp(y_{ij} S_{ij})} = \frac{1}{N}\cdot (-y_{ij}) \cdot \sigma(-y_{ij} S_{ij})$

Key: $\partial \mathcal{L} / \partial S_{ij}$ only depends on $S_{ij}$ itself, not other entries. So adding negatives does not change the gradient at existing $S_{ij}$ — batch-independent.

Engineering implications:

• CLIP: under DDP must all-gather embeddings to compute global logsumexp; communication is $O(N \cdot D)$ with extra sync points
• SigLIP: can use chunked all-pair, with each chunk computing only local rows × remote columns sigmoid terms, no logsumexp sync

Q22. Q-Former vs LLaVA projector trade-off: explain along capacity / compute / training stability.

Capacity (information capacity):

• LLaVA projector: all $N$ patch tokens enter the LLM; information has no bottleneck, but LLM context usage is large

• Q-Former: 32 queries is a fixed bottleneck; significant information compression, unfriendly to detail tasks (OCR / counting)

• Suppose visual encoder output rank is $r$; LLaVA visual context rank $\le r$ (preserved), Q-Former rank $\le \min(r, 32)$

Compute / Memory:

• LLaVA projector: MLP forward only, O(N·D²) compute

• Q-Former: 12 layers of cross-attn + self-attn + FFN, ~180M params; but downstream LLM context is short (32 tokens vs N=256+ tokens), so LLM inference is fast

• Total cost trade-off: at high image resolution (N=2880), Q-Former saves LLM inference; at low resolution where the LLM dominates, LLaVA is cheaper

Training stability:

• LLaVA: projector is easy to train (2 stages), gradient path is short

• Q-Former: 2-stage training (stage 1 representation jointly optimizes ITC + ITM + ITG; stage 2 generation connects to the frozen LLM); ITM head overfits easily, ITG requires intricate routing of causal vs self-attn masks — engineering pitfalls abound

Conclusion: 2024 mainstream returns to projector + spatial pixel-shuffle / merging to control token count; Q-Former retains value mainly in video / multi-image summarization (using queries for temporal pooling).

Q23. Why is Qwen2-VL's M-RoPE config `mrope_section = [16, 24, 24]` and not 1:1:1? Do all head_dim dims rotate?

Recap ordinary RoPE: head_dim $d$ split into $d/2$ complex pairs, frequencies $\theta_k = \text{base}^{-2k/d}$. The frequency coverage determines the maximum relative distance distinguishable: low frequency distinguishes long distances, high frequency distinguishes short distances.

Key disambiguation: Qwen2-VL mrope_section units are pairs of half head_dim (each number = how many $(2k, 2k+1)$ dim pairs). $[16, 24, 24]$ means t / h / w each occupy 16 / 24 / 24 pairs of dims; $\sum \times 2 = 128 =$ head_dim. The HF implementation doubles section to $[16, 24, 24, 16, 24, 24]$ slicing head_dim, corresponding to the axis sequence $(t, h, w, t, h, w)$ — all 128 dims rotate, none "left unrotated".

Design trade-offs:

1. Temporal variation is slow: typical video sampled at 1–5 FPS; adjacent frames are very similar; long-range temporal dependencies are moderate. $s_t=16$ (25% share) suffices to cover hundreds to thousands of frames.

2. Spatial variation is dramatic: huge visual differences across patches within a frame; to do token-to-token retrieval over a $\sim 1000\times 1000$-pixel image, more frequency slots are needed. $s_h = s_w = 24$ (37.5% each) covers more.

3. Spatial symmetry: $s_h = s_w$ keeps the H/W axes symmetric (equivalence under horizontal / vertical flip).

4. 6 alternating segments instead of 3 contiguous: because RoPE uses LLaMA "rotate_half", head_dim is split in memory into two halves $[h_1, h_2]$, with rotation $q \mapsto q \cos + \text{rotate\_half}(q)\sin$; the two halves share inv_freq. So axis allocations must mirror in both halves.

Qwen2.5-VL upgrade: switch $m_t$ from frame id to absolute timestamps (seconds), letting variable-FPS videos share a consistent time coordinate at training — the key to long video.

Alternative: DeepSeek-VL2 uses flattened visual tokens + plain 1D RoPE (no h, w split); Llama-3.2-V also does not split spacetime explicitly. M-RoPE only wins decisively for native interleaved video + image scenarios.

Q24. Root cause of VLM hallucinations? Pros and cons of existing mitigations?

Root causes:

1. Data bias: training data contains common "co-occurrence priors" — "if there's a table, there's probably a chair". Co-occurrence makes a VLM tend to answer "yes, there's a chair" when seeing a table, even if there is no chair

2. Language prior dominates: when the visual signal is weak (small objects, blur, odd angles), the VLM falls back to a pure language model, answering from "corpus common sense"

3. LLM sycophancy: the user asks "is there X in the image" and the model tends to say Yes (human feedback biases towards being helpful → biased Yes)

4. Stage 2 instruction tuning has no negative supervision: labels rarely teach "if there is no X, answer No"

Mitigations:

Method	Idea	Pros	Cons
LLaVA-RLHF	PPO + hallucination-aware reward	Targeted late-stage fix	Needs a reward model + lots of preference data
RLAIF-V	AI-generated preference	Low data cost	Reward model's own bias accumulates
POVID	DPO + constructed hallucination negatives	Direct targeted fix	Negative design requires care
VCD (visual contrastive decoding)	At inference, run the VLM on the image and a blurred image, amplify the difference	Training-free	2x inference cost
OPERA	Beam search + over-attention detection	Inference-time detection	Heuristic; may have false positives
POPE-driven evaluation	Use POPE for reverse supervision	Quantifiable	Only measures object hallucination

Future directions: fix at the training data layer (grounded captions + segment-level supervision); visual chain-of-thought (VLM-R1 style) where the model "points to evidence" before answering.

Q25. For modern VLM training, should the vision tower use SigLIP or CLIP? Why did most pick SigLIP-So400M in 2024–2025?

Empirical conclusion: after 2024, SigLIP-So400M has become a common choice for open-weight VLMs, with PaliGemma (Google) and LLaVA-OneVision as representatives. But Molmo still uses OpenAI CLIP (its paper ablates against SigLIP); the InternVL series uses in-house InternViT; Qwen2-VL's vision side initializes from a DFN-derived ViT and then trains a large-scale vision-language joint; LLaVA-1.5 / 1.6 still uses CLIP ViT-L/14. "Switch to SigLIP" is not an industry consensus.

Why SigLIP-So400M is attractive:

1. Stronger zero-shot performance: SigLIP-So400M has a 4–8 point lead over the same-size CLIP on zero-shot ImageNet; visual feature quality is higher

2. Resolution-friendly: SigLIP already trained at 384²/512² extensively; CLIP is mostly 224²+336²; VLM tasks generally need high resolution, so SigLIP transfers more smoothly

3. Batch-independent loss → stable fine-tune: SigLIP's sigmoid yields more predictable gradients when unfreezing the vision tower in stage 1

4. Multilingual support: SigLIP-2 / mSigLIP natively support multiple languages

5. Open weights: Google releases the full SigLIP / SigLIP-2 checkpoints (OpenAI CLIP has been open-sourced too, but with limited choices)

When to still pick CLIP:

• When strict alignment with OpenAI CLIP behavior is needed (e.g. CLIP guidance for Stable Diffusion-style use)
• For project compatibility (early LLaVA-1.0/1.5 + DALL-E pipelines use CLIP)

Note: SigLIP is not a silver bullet; DeepSeek-VL uses SigLIP + SAM dual-encoder — SAM features retain irreplaceable advantages on detail localization tasks.

§A Appendix: sanity-check outputs & references

A.1 Key code sanity check (illustrated by actual runs)

[ViT] patch_embed: (2, 3, 224, 224) -> (2, 196, 768)  ✓
[ViT] forward + CLS: (2, 3, 224, 224) -> head out (2, 1000)  ✓

[CLIP] N=8, D=512, init logit_scale=ln(1/0.07)→1/τ≈14.285: loss ≈ 2.25 (slightly above log(N)=2.08; loss≈log N holds only when 1/τ is very small)  ✓
[CLIP] forward + backward: gradients along i→t and t→i paths are symmetric ✓

[SigLIP] N=8, D=512, b=0:  loss = sum_{ij} log(1+e^0) / N = 64 * log 2 / 8 ≈ 5.545  ✓
[SigLIP] bias b=-10: positives (8 entries) loss ≈ log(1+e^10) ≈ 10; negatives (56 entries) loss ≈ 4.5e-5; total ≈ 8·10/8 ≈ 10.0  ✓ (positives dominate early gradient)

[LLaVA] visual feat (2, 256, 1024) -> projector -> (2, 256, 4096)  ✓
[LLaVA] input_ids w/ <image> placeholder: 1 token -> 256 visual tokens after merge  ✓

[Q-Former] image_feats (2, 257, 1408), queries (1, 32, 768) -> out (2, 32, 768)  ✓

[M-RoPE] head_dim=128, mrope_section=[16,24,24]: 2 × sum = 128 = head_dim, full rotation ✓
[M-RoPE] pure-text token (pos_t=pos_h=pos_w=m): cos/sin identical across the three axes, equivalent to 1D RoPE ✓

A.2 Key references (by topic)

Vision encoder: Dosovitskiy et al. ICLR 2021 (ViT); Zhai et al. CVPR 2022 (ViT-g); Fang et al. arXiv 2023 (EVA-02)

Contrastive pretraining: Radford et al. ICML 2021 (CLIP); Cherti et al. CVPR 2023 (OpenCLIP); Zhai et al. ICCV 2023 (SigLIP); Tschannen et al. arXiv 2025 (SigLIP-2); Gadre et al. NeurIPS 2023 (DataComp)

Visual instruction / fusion: Liu et al. NeurIPS 2023 (LLaVA); Liu et al. CVPR 2024 (LLaVA-1.5); Li et al. ICML 2023 (BLIP-2); Alayrac et al. NeurIPS 2022 (Flamingo); Wang et al. arXiv 2023 (CogVLM); Bai et al. arXiv 2023 (Qwen-VL); Wang et al. arXiv 2024 (Qwen2-VL); Bai et al. arXiv 2025 (Qwen2.5-VL); Lu et al. arXiv 2024 (DeepSeek-VL); Wu et al. arXiv 2024 (DeepSeek-VL2); Chen et al. CVPR 2024 + arXiv 2024 (InternVL / InternVL-2); Llama Team arXiv 2024 (Llama-3.2-V); Deitke et al. arXiv 2024 (Molmo); Li et al. arXiv 2024 (LLaVA-OneVision)

VLM preference alignment: Sun et al. arXiv 2023 (LLaVA-RLHF); Yu et al. arXiv 2024 (RLAIF-V); Zhou et al. arXiv 2024 (POVID); Shen et al. arXiv 2025 (VLM-R1)

Multimodal embeddings: Koukounas et al. arXiv 2024 (Jina-CLIP); Jiang et al. arXiv 2024 (VLM2Vec); Zhang et al. arXiv 2024 (mmE5)

Closed-source/product docs: OpenAI model docs (GPT-5.5 / multimodal APIs); Anthropic Claude models overview (Fable / Opus / computer-use capabilities); Google Gemini 3.1 Pro Preview docs (multimodal input, 1M context, tool use); Gemini Embedding docs; Cohere Embed 4 docs

Evaluation: Li et al. EMNLP 2023 (POPE); Thrush et al. CVPR 2022 (Winoground); Liu et al. ECCV 2024 (MMBench); Yue et al. CVPR 2024 (MMMU); Fu et al. CVPR 2025 (Video-MME); Yu et al. ICML 2024 (MM-Vet); Mangalam et al. NeurIPS 2023 (EgoSchema)

---

Code + formulas have passed independent reviewer static checks; numerical values verified on PyTorch 2.x, CUDA 12.x (shapes and initial loss of the 5 core modules — ViT / CLIP / SigLIP / Q-Former / M-RoPE — all match the formulas).