>>Exposing the Hidden Biases of AI Image Generators
Manos Plitsis & Giorgos Bouritsas
If one asks an AI image generator like Stable Diffusion or DALL·E to picture "a doctor", they'll probably get a picture of a man. If they ask for "a nurse", they'll probably get a woman. AI models, among other technological artefacts, are known to inherit common stereotypes from their human creators (designers, developers, data collectors, annotators, etc.).
Researchers and investigative journalists have been documenting these stereotypes for years, putting pressure on the academic community and AI companies to "debias" their models, either via retraining or by internal adjustments, to give more balanced results. But here is the catch: these fixes mostly work on typical prompts investigators test with, e.g. "a photo of a doctor" or "an engineer at work" — generic sentences that reflect societal biases we are mostly aware of.
What happens when real users type something more descriptive instead: "a scientist with intense focus" or "a doctor with compassionate eyes"?
In our new research paper (ICML 2026) we set out to answer that question. We built an automated tool, coined Bias-Guided Prompt Search (BGPS), that hunts down natural-sounding prompts that trigger biased outputs, to conduct rigorous bias auditing[1] for image generators.
Using our method of bias discovery, we found prompts that trigger gender, race and age-related biases in numerous open-weight image generators (Stable Diffusion 1.5, 2.1, 3.0, SDXL, Flux). Our findings suggest that even supposedly "fair" image generators are still stereotyping in subtle ways that went unnoticed by only using standard evaluations. We further investigate how contextual wording triggers biased generation.
| 
|
The problem with current bias tests
Most bias evaluations rely on benchmarks consisting of hand-curated (or LLM-generated) lists of prompts, usually variations of "A photo of a <job title>". When an image generator produces a roughly 50/50 mix of men and women on those prompts, it is considered debiased with respect to gender.
The trouble is that real prompts in the wild are far more colourful and varied: people add adjectives, scenes, moods, clothing, contexts. The model has been trained on billions of those richer descriptions from the internet, where stereotypes are deeply baked in. In other words, we cannot be sure a priori if the typical benchmarks used for evaluation cover all the possible sources of bias that every specific model contains.
Dealing with this requires designing benchmarks that can stress-test and identify edge cases, beyond the average usage of the model — a process frequently referred to as adversarial auditing[2]. This is precisely what we aim to do with BGPS.
A smarter prompt-hunter
Our bias discovery tool works as follows: it pairs (1) a regular language model (the kind that writes natural sentences, e.g. the well-known GPT) that produces prompts, with (2) a "detector" network that peeks inside the image generator and predicts, before any image is even drawn, the resulting image's likely gender, race, age, or any other potentially biased attribute.
The language model is instructed to generate candidate prompts that sound like something a human would type, but do not contain any reference to the attribute of interest (e.g. it excludes gender-revealing words such as pronouns, names, etc.). The detector network steers it toward word choices that increase bias. As a result, we obtain completely natural-sounding and attribute-neutral sentences that nonetheless cause the image generator to produce heavily skewed results.
What we found
Biases are spread out across all image generator models we tested
We conducted experiments for gender bias across various image generation models, including Stable Diffusion (SD) 1.5, SD 2.1, SD XL, Flux and SD 3.0. Testing with typical profession-related prompts, such as "a photo of a person working as a <profession>", produced imbalanced results with respect to gender for all models tested, echoing and extending the Bloomberg investigation. Moving one step forward, testing with BGPS, we found sets of prompts that further increased male/female representation (up to 95% and up to 83%, respectively, for the SDXL model), illustrating severe vulnerabilities under stress testing.
Racial bias is even more pronounced on the SD 1.5 model, with white people appearing significantly more frequently when querying the model with arbitrary prompts. Similarly here, BGPS discovered tailored prompts that increase white/black representation (up to 82% and up to 27%, respectively - observe here the overall underrepresentation of the latter), exposing inherited stereotypes.
"Debiased" models can still produce biased images
Interestingly, this phenomenon is present in models that have undergone debiasing. A state-of-the-art debiased version of SD 1.5 that produced 49% male images on standard test prompts shot up to 76% male when given prompts discovered by BGPS, a stark reminder that passing a fairness benchmark is not the same as being fair.
Subtle words have huge effects
We qualitatively analysed the prompts that can be automatically discovered by holding the profession constant and asking BGPS to generate only the descriptive words. Here is the same model (SDXL) generating a "teacher":
| 
|
Same effect with a doctor:
| 
|
In both cases, images are shifted toward male characters when adding thinking-focused descriptors and female ones when adding an empathetic-focused tone. These additions are not semantically associated with a particular gender, but undoubtedly perpetuate conservative societal associations.
Studying the model's biased associations
When we mapped out which individual words most reliably pushed outputs toward male or female depictions, two distinct vocabularies emerged:
| 
|
The resulting groupings reveal how the image generator "views" each gender. Technology, music, and thought-related terms reliably nudged images toward men, while arts, crafts, literature, and emotion-related descriptors nudged them toward women. The model has, in effect, learned that thinking is masculine and feeling is feminine, and it expresses that through the images it generates, even when no one explicitly mentions gender.
Racial stereotypes appeared too, in unsettling places
We also inspected our methodology on prompts triggering racial biases. Each of the prompts below was written to be racially neutral, yet each one reliably produced Black subjects:
| 
|
| 
|
| 
|
| 
|
| 
|
| 
|
| 
|
| 
|
A farm labourer and a pastor on one side; a security guard and a "poor" street artist on the other. The model has learned to map race onto a specific narrow set of contexts: security, manual labour, certain styles of art, poverty, without any explicit racial words in the prompt.
What do our discoveries entail?
Our motivation for the project stems from the extent to which text-to-image models are now used. Their popularity exceeds academic and high-tech environments, with AI-generated images being used in stock photography, advertising, design tools, and commercial products that millions of people interact with. Importantly, this type of content is currently flooding the internet (see the term AI slop)[3].
The visual information circulated in the world shapes our perspectives about reality. Text-to-image models' stereotypes did not emerge in a vacuum. They are deeply embedded in their training data, which in turn reflect the beliefs of a large part of society. But when stereotypes, especially subtle ones, are repeatedly depicted at an enormous scale, they are reinforced.
Our paper's main point is that fairness cannot be achieved by balancing the results of a fixed set of test prompts. Biases exist in the far richer space of how we describe people, in moods, postures, clothing, settings, and adjectives. Truly fixing them requires confronting the fact that the model has learned millions of small, statistical associations from human language, most of which the model's evaluators never explicitly catalogued.
Our project aims to highlight the need to build model-specific bias-triggering catalogues, instead of relying on standard fairness benchmarks. This opens the door to more targeted debiasing[4], but also to a deeper understanding of the harmful associations that the training data carry.
Read our paper
Try SDXL on HuggingFace
[1] Algorithmic auditing is an inspection process that an algorithm undergoes in order to ensure that it complies with certain standards, constraints, regulations, best practices, etc., or to identify violations thereof. It may be performed internally by the algorithm designers (e.g. red teaming) or externally by regulatory bodies, independent actors, etc., as a tool for accountability.
[2] Eticas, a non-profit organisation, frequently advocates in favour of adversarial testing by independent actors (e.g. community-led AI audits).
[3] This often disturbing phenomenon ignites additional severe concerns beyond biases, over aspects such as privacy, copyright, originality, creativity, aesthetics, misinformation, manipulation, and more.
[4] We tested this in an initial experiment: fine-tuning a model with BGPS-produced prompts balances gender proportions, making the model less vulnerable to further BGPS attempts to find adversarial prompts.
Images and prompts in this post are reproduced from the source paper, "Exposing Hidden Biases in Text-to-Image Models via Automated Prompt Search," and were generated by Stable Diffusion XL using prompts discovered by the BGPS method described in that work.