MIRAGE: Multi-model Interface for Reviewing and Auditing Generative Text-to-Image AI

What is MIRAGE?

MIRAGE is a web-based tool that lets anyone compare outputs from four text-to-image AI models on a single screen and report harmful biases through a structured workflow. It appeared as a first-author works-in-progress paper at HCOMP 2024, built during a research summer at Carnegie Mellon University's Human-Computer Interaction Institute.

MIRAGE stands for Multi-model Interface for Reviewing and Auditing Generative Text-to-Image AI. It is a web tool where anyone can type a prompt, watch four different text-to-image models respond on one screen, and file a structured report on what they noticed. The premise is simple. One model's output can look normal in isolation and clearly skewed the moment three alternatives sit next to it.

I built MIRAGE in the summer of 2024 during a research summer at Carnegie Mellon University's Human-Computer Interaction Institute, working with Wesley Hanwen Deng, Dr. Jason Hong, Dr. Ken Holstein, and Dr. Motahhare Eslami. It was my first project at CMU and my first first-author paper, presented in the works-in-progress track at HCOMP 2024. The demo still runs at mirage.matheus.wiki.

Why everyday users should audit generative AI

Text-to-image models reproduce harmful societal stereotypes, and the people best positioned to catch those harms are often the everyday users the outputs affect. MIRAGE builds on end-user auditing research, adding the structured feedback channel that existing model comparison playgrounds still lack.

Text-to-image systems have a documented bias problem. Prior audits found Stable Diffusion rarely depicting women in professional roles and showing racial bias around crime-related prompts. Developers run internal evaluations, and those still miss harms that only become obvious to someone with the right lived experience. A growing line of research argues that diverse end users can spot harmful AI behaviors that development teams overlook.

Existing tools each solved part of the problem. IndieLabel supports end-user audits of sentiment analysis models. LLM Comparator helps developers compare language model outputs. Replicate's Zoo, a text-to-image playground, lets anyone generate from several models, though it offers no way to report what you find, and its vertical layout makes comparison tedious. MIRAGE puts the pieces together: multiple image models, one screen, and a reporting path built in. That gap is what it fills.

How MIRAGE structures bias reports

MIRAGE walks auditors through five steps. Enter a prompt, answer reflection questions while models generate, review a single model's output, compare all four side by side, then submit a structured report. The questions between steps turn a casual reaction into a usable audit record.

The workflow is deliberate about order. You start with a prompt and answer a few questions while the images generate: why you chose it, what you expect the outputs to look like, whether they match what your experience anticipated. Then you see a single model's output first, and only after reviewing it do the other three appear. Your answers accumulate into the final audit report, so the reflection is the documentation.

That sequencing captures a before-and-after. If the side-by-side view changed what you noticed, the report records exactly when the change happened. We also connected an optional discussion forum so auditors could share findings, since auditing works better as a collective activity than a solitary one.

What five participants found

In a preliminary IRB-approved study, five participants audited one shared prompt and two prompts of their own choosing. Comparing four models at once surfaced biases that single-model review missed, and participants drew directly on their identities and lived experiences to spot them.

Sessions ran roughly 30 to 45 minutes. Everyone started with the same predefined prompt, a fancy dinner party, then explored two prompts of their own, chosen with their backgrounds and identities in mind. All five found the tool easy to use. The interesting part was what comparison did to their attention.

One participant prompted for an intern working in the city of Pittsburgh. With one model, they only noted the workers were outdoors. With four, they saw darker-skinned workers in construction clothes while lighter-skinned workers wore office attire and carried keyboards and notebooks. Cultural context surfaced the same way: participants who expected the dinner party prompt to default to white men also noticed every table was long and rectangular, a Western convention, where round tables are common in Asian dining.

Three of the five described the same strategy shift. With one model they inspected individual images. With four, they read the overall distribution of outputs, looking for patterns in a model's behavior instead of flaws in a single picture. That shift is the finding I care most about.

From MIRAGE to Seeing Twice

MIRAGE was a works-in-progress paper with five participants, so the same team scaled the idea into Seeing Twice, a fifteen-participant study published at ACM Collective Intelligence 2025 that tested a contrast-first workflow and surfaced a language-fidelity gap across English, Portuguese, and Chinese prompts.

A five-person study can suggest, and it cannot conclude. So the same team carried the idea forward. In Seeing Twice, published at ACM CI 2025, fifteen participants used a refined interface with four predefined models, and we watched the same shift happen at scale. Once the side-by-side step appeared, most participants moved from analyzing individual images to reading general output patterns.

Two findings went beyond the original paper. Participants coined persistent model personalities, shorthand like cartoonish or saturated, that helped them predict how each model would behave on future prompts. And bilingual participants surfaced a language-fidelity gap: English prompts produced more accurate images than Portuguese or Chinese ones, an issue that single-model review tends to miss entirely. Comparison changed what people could see. That held up twice.

Where multi-model auditing goes next

The paper sketches three directions. Anonymous auditing would let proprietary model developers collect user feedback without exposing their models, a text-to-image model supermarket would help people choose models by use case, and a community leaderboard would rank models the way Chatbot Arena ranks chatbots.

The works-in-progress format exists for ideas that are still moving, and we used the discussion section to sketch three of them. Anonymous auditing would let companies submit proprietary models and receive structured feedback while protecting their intellectual property. A model supermarket would let anyone scan many models at once before picking one, the way a parent might want cartoonish outputs where a designer wants photorealism. And a community leaderboard, similar to Chatbot Arena, would rank models by how often users choose their outputs, giving developers a reason to fix the biases users find.

MIRAGE taught me that evaluation interfaces shape what people notice, a thread that runs from Seeing Twice to my agent benchmarking work at Microsoft Research. Auditing tools decide which harms get seen. I want to keep building the ones that widen the view.