Sources checked
Test extraction against difficult original images
An original receipt assistant could inspect an image and extract merchant, date, and total. Check small text, rotated pages, and ambiguous images against the original rather than trusting a neatly formatted answer.
A visual answer can include invisible details
A model can misread a picture or infer details that are not visible. Match the task to the supported inputs and verify important extracted information before using it.
What “multimodal” specifies
The word only means the model was built to take more than text. One system embeds an image into the same space as tokens and then predicts text.
Another transcribes audio first and only then calls a text model. Those are different pipelines.
In the second case, an error in the transcript never reaches the reasoning step as audio at all.
Resolution and length are limits. An image may be resized until a label is unreadable.
A video may be sampled as a few frames. A long recording may be clipped.
If you need a serial number in the corner of a photo, test that photo at the size the API will see. The model card’s phrase “accepts images” does not state that size.
What to test besides the demo
Demos use clean, centered pictures. Your users send glare, handwriting, and screenshots of screenshots.
Build a set that includes the bad cases. Score extraction separately from description.
A caption can sound right while the number you needed is wrong.
Images and audio are data-handling problems as well as modeling problems. A photo of a passport is not “just an image” in a retention policy.
Read whether the provider stores the bytes, and for how long. A model that can describe a face is not a reason to upload a face you do not have a reason to share.
| Input | A limit to ask about | A failure that still looks fluent |
|---|---|---|
| Image | Resize and crop before the model | A confident description that skips the small text |
| Audio | Whether you get a transcript first | A summary of words the person did not say |
| Video | Which frames are kept | An action that happened between the frames |
What to record before you rely on it
Record the pipeline: whether the API sees the original file or a transcript, a resized image, or a sample of frames. Note the maximum size you tested and the smallest text that still came back correctly.
Keep the ugly examples, not only the clean demo. Add the retention rule for the bytes you upload.
A model card that says “accepts images” is not a record of any of those facts.
Keep the model id, the date, and the inputs you used beside that note, so a later change can be compared with the same test.
Common questions
Does a multimodal model see the original file?
It sees the representation the pipeline builds. That can be a downscaled image or a transcript.
Ask what transform runs before the model, and test the detail you care about after that transform.
Can I assume text and image agree?
No. The image and the surrounding text can contradict each other. A test should include that case, because a model may follow whichever channel it weights more strongly.
Sources & further reading
How this story was made
Written by Kristian Kostov with AI assistance and checked against the linked sources. Company performance claims are attributed to the company. Analysis reflects AiLookout’s interpretation; we have not independently tested the products discussed. Cover photography is illustrative and does not depict the specific announcement or product.
Our editorial standards





