Overview
Vision-language models are usually consumed as a black box: text in, similarity out. This pipeline opens the box in four directions at once, because the interesting failures live between the parts rather than inside any one of them.
The four pipelines
- Image-text grounding: zero-shot matching of images to prompts with OpenCLIP ViT-B/32.
- Attention visualization: custom hooks extract multi-head attention from the Vision Transformer, overlaid as heatmaps so you can see which patches carried a match.
- Image generation: Stable Diffusion v1.5 through
diffusers, with configurable conditioning, to probe the same semantics from the generative side. - Embedding analysis: cosine similarity matrices plus t-SNE and UMAP projections to inspect how well the two modalities actually align.
Why the combination matters
Grounding scores tell you that a match happened; attention maps tell you where; the embedding analysis tells you whether the space is separable at all or whether a good score was luck. Running all four over the same inputs is what makes a VLM result interpretable rather than merely reported.