Test extraction against difficult original images

An original receipt assistant could inspect an image and extract merchant, date, and total. Check small text, rotated pages, and ambiguous images against the original rather than trusting a neatly formatted answer.

A visual answer can include invisible details

A model can misread a picture or infer details that are not visible. Match the task to the supported inputs and verify important extracted information before using it.

What “multimodal” specifies

The word only means the model was built to take more than text. One system embeds an image into the same space as tokens and then predicts text.

Another transcribes audio first and only then calls a text model. Those are different pipelines.

In the second case, an error in the transcript never reaches the reasoning step as audio at all.

Resolution and length are limits. An image may be resized until a label is unreadable.

A video may be sampled as a few frames. A long recording may be clipped.

If you need a serial number in the corner of a photo, test that photo at the size the API will see. The model card’s phrase “accepts images” does not state that size.

What to test besides the demo

Demos use clean, centered pictures. Your users send glare, handwriting, and screenshots of screenshots.

Build a set that includes the bad cases. Score extraction separately from description.

A caption can sound right while the number you needed is wrong.

Images and audio are data-handling problems as well as modeling problems. A photo of a passport is not “just an image” in a retention policy.

Read whether the provider stores the bytes, and for how long. A model that can describe a face is not a reason to upload a face you do not have a reason to share.

More inputs, more places to lose detail
InputA limit to ask aboutA failure that still looks fluent
ImageResize and crop before the modelA confident description that skips the small text
AudioWhether you get a transcript firstA summary of words the person did not say
VideoWhich frames are keptAn action that happened between the frames

What to record before you rely on it

Record the pipeline: whether the API sees the original file or a transcript, a resized image, or a sample of frames. Note the maximum size you tested and the smallest text that still came back correctly.

Keep the ugly examples, not only the clean demo. Add the retention rule for the bytes you upload.

A model card that says “accepts images” is not a record of any of those facts.

Keep the model id, the date, and the inputs you used beside that note, so a later change can be compared with the same test.

Common questions

Does a multimodal model see the original file?

It sees the representation the pipeline builds. That can be a downscaled image or a transcript.

Ask what transform runs before the model, and test the detail you care about after that transform.

Can I assume text and image agree?

No. The image and the surrounding text can contradict each other. A test should include that case, because a model may follow whichever channel it weights more strongly.

Sources & further reading

  1. Hugging Face: Image-text-to-text ↗
How this story was made

Written by Kristian Kostov with AI assistance and checked against the linked sources. Company performance claims are attributed to the company. Analysis reflects AiLookout’s interpretation; we have not independently tested the products discussed. Cover photography is illustrative and does not depict the specific announcement or product.

Our editorial standards
Back to all stories