What is the CLIP model?

Radford and colleagues describe it in Learning Transferable Visual Models From Natural Language Supervision. Usual vision models, they write, predict a fixed set of object categories, so a new concept needs new labels.

CLIP’s pre-training task is to predict which caption goes with which image. They collected 400 million pairs from the internet.

Natural language then refers to visual concepts, including ones described at test time, which they call zero-shot transfer.

They benchmark more than 30 existing vision datasets, including OCR, action recognition, geo-localization, and fine-grained object classification.

The paper says the model transfers non-trivially to most of those tasks and is often competitive with a fully supervised baseline without dataset-specific training.

One figure they highlight: matching the original ResNet-50’s ImageNet accuracy zero-shot, without using that dataset’s 1.28 million training examples. They released code and weights.

Those comparisons are the paper’s, on the datasets it names.

Why the candidate sentence changes the result

CLIP scores an image against the text you supply. A different wording is a different candidate.

Include a description that names the background as well as one that names the object, and see which one wins. Multimodal AI is the broader pattern of a model that takes more than text.

CLIP is one training recipe inside that pattern, not a general description of every image model.

What a similarity score does not prove

A high score does not date the photo, name the camera, or show that the scene happened. Provenance still comes from metadata and from sources outside the model.

The paper’s zero-shot results are also not a claim about images the model never saw in the form you care about, such as a private product photo.

Common questions

What is a CLIP model used for?

In the paper, it is used to score images against text and to transfer to vision tasks without a new labeled training set for each dataset. Later products reuse the idea.

Check which checkpoint you are running.

Does CLIP check whether an image is authentic?

No. It compares an image with text. Authenticity, date, and origin need evidence outside that score.

Sources & further reading

  1. Learning Transferable Visual Models From Natural Language Supervision ↗
How this story was made

Written by Kristian Kostov with AI assistance and checked against the linked sources. Company performance claims are attributed to the company. Analysis reflects AiLookout’s interpretation; we have not independently tested the products discussed. Cover photography is illustrative and does not depict the specific announcement or product.

Our editorial standards
Back to all stories