Hidden in Plain Sight: Evaluating Abstract Shape Recognition in Vision-Language Models

We built a test to see whether modern vision models can recognise shapes hidden inside natural scenes, and found that the most advanced systems almost completely fail. Humans spot these forms easily; the models get entirely distracted by the background.

If these systems are going to be reliable in the real world, they need to see overall shape rather than only texture.

Why this was hard

Earlier attempts to measure shape understanding relied on artificially modifying images, like pasting a cat's texture onto an elephant's body. Datasets built that way, such as Stylized-ImageNet, use legacy style transfer that destroys natural contrast and never produces a coherent scene. That makes it impossible to tell whether a model genuinely understands shape or is simply confused by an unnaturally uniform image.

Testing with naturalistic scenes has its own problem: manually collecting thousands of good optical illusions is slow and subjective. We needed a way to embed clear, unambiguous shapes into complex, high-quality backgrounds without degrading image quality, forcing a model to separate an object's form from the material it is made of, while keeping the illusion obvious to a human eye.

How it works

IllusionBench is a dataset of over 32,000 images generated with ControlNet, an architecture that injects spatial conditioning into text-to-image diffusion models. We feed ControlNet a binary mask of a target shape, a bicycle or a corporate logo, and pair it with a text prompt describing a complex scene such as a medieval village or a coral reef. Stable Diffusion then synthesises a photorealistic image in which the target shape emerges from the arrangement of scene elements rather than being drawn explicitly.

We evaluate vision-language models including GPT-4o and LLaVA-Next by prompting them to identify the hidden shape, the background scene, or both. Rather than a classification head, we measure shape recall: how often the correct shape class appears in the model's generated text. Three splits probe different things. IllusionBench-IN uses standard physical objects, IllusionBench-LOGO uses brand logos, and IllusionBench-ICON uses abstract semantic concepts across hundreds of emojis.

The gap is large. Human annotators identify the shapes over 95% of the time; zero-shot shape recall for current models falls between 10% and 30%. The models overwhelmingly default to describing background texture. In-context learning barely helps: passing a few labelled examples in the prompt does almost nothing against the texture bias, and in some cases shape examples make performance worse.

We also formulate a domain generalisation task, training a contrastive vision-language encoder on shapes hidden in five environments and testing on six unseen ones. Linear probing fails entirely, but full-parameter fine-tuning lets the model learn domain-invariant shape features and recovers accuracy to 73%.

What it does not do

The generation pipeline hides a single shape per image, so it cannot test whether a model can recognise multiple abstract shapes interacting in one scene. The experiments are closed-set: the model is given a list of possible classes. And because we parse free-text responses, our metric may slightly underestimate a model that recognises the shape but describes it with vocabulary outside our target list.

Accepted to the datasets and benchmarks track at NeurIPS 2024. Dataset, code and examples are on the project page.