Zero-shot prompt classification

Fix image and text encoder maps $f_{\theta }:\mathsf{X}\to {\mathbb{R}}^d$ and $g_{\varphi }:\mathsf{T}\to {\mathbb{R}}^d$ into one shared embedding space, and a real log-scale $\tau$ with $s:=\exp (\tau )>0$. The inputs are an image $x\in \mathsf{X}$, an ordered nonempty list of distinct class descriptions $C=(c_1,\dots ,c_K)$, and a finite nonempty list of prompt templates $T=(t_1,\dots ,t_M)$, where each $t_m$ maps a class description to a text input. The admissible input conditions are $f_{\theta }(x)\mathrm{\ne }0$, $g_{\varphi }(t_m(c_k))\mathrm{\ne }0$ for every $k,m$, and a nonzero average text embedding for every class.

1. Normalize the image embedding: $u:=f_{\theta }(x)\mathrm{/}\mathrm{\parallel }{f}_{\theta }(x){\mathrm{\parallel }}_2$.

2. For each class $c_k$ and template $t_m$, form the text embedding $v_{km}:=g_{\varphi }(t_m(c_k))\mathrm{/}\mathrm{\parallel }{g}_{\varphi }(t_m(c_k)){\mathrm{\parallel }}_2$.

3. Average the normalized prompt embeddings in the shared embedding space and normalize the average to obtain the class prototype $q_k:=\left(\frac{1}{M}{\sum }_{m=1}^M{v}_{km}\right)/{\parallel \frac{1}{M}{\sum }_{m=1}^M{v}_{km}\parallel }_2$. This is prompt-template averaging; it averages embeddings before classification, not class probabilities.

4. For each class compute the scaled similarity score $z_k:=s⟨u,q_k⟩$ and softmax probability $p_k:=\exp (z_k)\mathrm{/}{\sum }_{r=1}^K\exp (z_r)$.

5. Return the vector $(p_1,\dots ,p_K)$ and class $c_{k^{\ast }}$, where $k^{\ast }$ is the least index maximizing $z_k$.

The returned classifier is zero-shot in the precise sense that no labeled examples from the target classification dataset are used to fit its class prototypes or classifier. The encoder maps, log-scale, class descriptions, and templates are fixed inputs to this inference procedure; it performs no training or parameter update. The procedure makes no accuracy or distribution-generalization guarantee.

Radford et al., §3.1.2: text-class candidates, scaled cosine scores, and softmax classification; §3.1.4: prompt engineering and embedding-space ensembling

Machine Learning II lecture notes, “Zero-shot transfer” and “Prompt engineering”/“Prompt ensembling”

Symmetric image-text contrastive loss: supplies the shared image/text embedding space, normalized inner-product score convention, and positive learned exponential scale.

The real exponential function and the number $e$ by a power series: defines the real exponential used in the logit scale and softmax.

The exponential is a continuous bijection from $\mathbb{R}$ onto $(0,\mathrm{\infty })$: gives positivity of exponential scores and softmax denominators.