Seeing Twice: How Side-by-Side T2I Comparison Changes Auditing Strategies
ACM CI, 2025 · Carnegie Mellon University
Matheus Kunzler Maldaner
Wesley Hanwen Deng
Jason I. Hong
Ken Holstein
Motahhare Eslami
What is Seeing Twice?
Seeing Twice is a first-author paper at ACM Collective Intelligence 2025 studying how side-by-side comparison of text-to-image models changes human auditing behavior. Fifteen participants used the MIRAGE tool to view up to four models on one screen, and most shifted from judging single images to reading model-level output patterns.
I wrote Seeing Twice during my research at Carnegie Mellon University's Human-Computer Interaction Institute, working with Wesley Hanwen Deng, Dr. Jason Hong, Dr. Ken Holstein, and Dr. Motahhare Eslami. The paper appeared at ACM Collective Intelligence 2025 and extends MIRAGE, the auditing tool for text-to-image (T2I) models, systems that turn a written prompt into pictures, that I first presented as a demo at AAAI HCOMP 2024.
The question behind it is simple. Platforms like Hugging Face already place model outputs side by side, yet we had little scientific understanding of how that layout affects auditing and red-teaming. So we ran fifteen user sessions and watched what happened when a second, third, and fourth model entered the screen. Quite a lot changed.
MIRAGE and the contrast-first auditing workflow
MIRAGE is a web-based auditing tool built on a contrast-first workflow. Users select up to four text-to-image models, submit one prompt, and view eight images per model on a single screen. Reflection questions occupy users while images render in the background, hiding generation latency inside a moment of thinking.
MIRAGE is the tool at the center of the study. You pick up to four text-to-image models, type one prompt, and see every model's answer on a single screen. Each model returns eight images, so a full grid holds 32 outputs for the same sentence. We call this a contrast-first workflow because comparison is the default state of auditing.
One design detail I like: while images render, the interface asks reflection questions such as why you chose that prompt and what you expect to see. Generation happens in the background while you answer, and several participants assumed the results were pre-cached. The waiting disappears into thinking.
How side-by-side comparison changes AI auditing
Side-by-side model comparison changed how participants audited text-to-image systems. With one model visible, people inspected individual images. Once four models appeared, they shifted to scanning output distributions for patterns, a strategy that surfaced skin-tone and occupation biases they had missed during the single-model phase of the very same session.
Participants first saw output from a single model and rated it. Then the other three models appeared. That one change moved most people from inspecting individual images to reading patterns across a model's output distribution. One participant said they became less likely to look at individual images and more at surface-level outputs and trends.
The shift matters because pattern-level reading catches different harms. A participant prompted for an intern working in Pittsburgh and initially noted only that the scene was outdoors. With four models on screen, they noticed darker-skinned figures wearing construction clothes while lighter-skinned figures carried keyboards and notebooks in office attire. The single-model view had hidden that split.
Interactive demo: spot the bias
an illustrative recreation of the study's core moment. all "generated images" are stylized drawings made in code, not real model outputs
One model returns eight images. Notice anything?
illustrative recreation
Now four models share the screen
Scan across each model's full row before judging any single image, then click the images you think participate in a pattern. Watch what the skin-tone swatch in each corner pairs with.
pattern images found: 0 of 12
illustrative recreation
Model personalities: how users describe T2I systems
Model personalities emerged when participants viewed four text-to-image systems side by side. Users assigned persistent descriptors such as cartoonish, saturated, colorful, and exaggerated to specific models, then used those characterizations to predict how each system would behave on future prompts. Comparison gave people durable mental models of generative AI.
Once all four models were visible, several participants started assigning each one a persistent character. Kandinsky 2.2 was colorful and vibrant. Latent Consistency read as cartoonish and exaggerated. Stable Diffusion kept producing figures with proportions participants described as off. ByteDance's SDXL-Lightning drew praise for quality and variation, though one participant found it the least culturally diverse.
These personalities did real work. Participants used them to predict how each model would behave on prompts they had never tried, exactly the mental model an auditor needs. A personality is a hypothesis about a model's failure modes. Comparison gave people enough evidence to form one.
Interactive demo: match the model personalities
an illustrative recreation. all "output strips" are stylized drawings made in code, not real model outputs
Participants who saw four models side by side started assigning each one a persistent character. Try it yourself: click a descriptor, then click the anonymous output strip it fits best.
illustrative recreation
The language fidelity gap in image generation
The language fidelity gap describes how text-to-image models produce more accurate images from English prompts than from Portuguese or Chinese equivalents. Bilingual participants in the Seeing Twice study surfaced this disparity by comparing outputs across languages and across four models at once, a bias single-model interfaces routinely let slip past.
Several participants were bilingual, and some sessions ran in Portuguese or Chinese. They tried the same idea in English and in their first language. English prompts produced more accurate images than their Portuguese or Chinese equivalents, and seeing the gap repeat across four models made it read as systemic rather than as one model's quirk.
This is the finding I keep coming back to. A single-model interface makes language bias easy to miss because there is no baseline. The side-by-side grid supplied one. Growing up in Brazil, I have felt what it means when technology serves one language better than another. This time the interface made it visible.
Interactive demo: the same prompt in three languages
an illustrative recreation. meter levels are qualitative, calibrated to the paper's finding, not measured scores
how faithfully each model's images tracked the prompt, shown as a qualitative meter
Study design with fifteen participants
The Seeing Twice user study ran fifteen sessions with volunteers of varied ages, education levels, and AI experience. Every session used the same four text-to-image models, starting from a predefined prompt about a fancy dinner party before participants wrote two prompts of their own grounded in lived experience.
We ran fifteen sessions with volunteers recruited through convenience sampling. The group was varied: eight women and seven men, education from high school to master's degrees, and self-rated AI experience spanning the full scale. Nine participants were primarily English speakers, with the rest bringing Portuguese or Chinese.
Each session used the same four models for consistency. Everyone started with a predefined prompt, a fancy dinner party, then wrote two prompts of their own drawn from lived experience, since prompts about your own life surface harms a stock benchmark never would. One participant noticed that every generated dinner table was rectangular, a Western default that quietly erased the round tables common in Asian households.
Why comparative interfaces accelerate bias discovery
Comparative interfaces accelerate bias discovery by giving auditors a baseline for every judgment. Seeing Twice found that placing four text-to-image models on one screen, with no new detection algorithm, led everyday users to form expectations, test them, and catch harmful patterns they had overlooked minutes earlier in single-model viewing.
The headline implication is that a simple interface change can do the work of an elaborate auditing pipeline. We added no new detection algorithm. We placed outputs next to each other, and participants began forming expectations, testing them, and catching biases they had walked past minutes earlier. Comparison turned casual viewers into auditors.
We are honest in the paper that this was a pilot study, so the next step is a controlled experiment quantifying how much side-by-side workflows improve bias detection, and for whom, since factors like identity and AI literacy likely mediate the effect. Small interfaces can move a field. That is the bet.
Abstract
While generative AI systems have gained popularity in diverse applications, their potential to produce harmful outputs limits their trustworthiness and utility. A small but growing line of research has explored tools and processes to better engage non-AI expert users in auditing generative AI systems. In this work, we present the design and evaluation of MIRAGE, a web-based tool exploring a 'contrast-first' workflow that allows users to pick up to four different text-to-image (T2I) models, view their images side-by-side, and provide feedback on model performance on a single screen. In our user study with fifteen participants, we used four predefined models for consistency, with only a single model initially being shown. We found that most participants shifted from analyzing individual images to general model output patterns once the side-by-side step appeared with all four models; several participants coined persistent 'model personalities' (e.g., cartoonish, saturated) that helped them form expectations about how each model would behave on future prompts. Bilingual participants also surfaced a language-fidelity gap, as English prompts produced more accurate images than Portuguese or Chinese, an issue often overlooked when dealing with a single model. These findings suggest that simple comparative interfaces can accelerate bias discovery and reshape how people think about generative models.
Frequently Asked Questions
How do you audit a text-to-image model?
Send the same prompt to several models and compare the grids instead of judging one image at a time. Write prompts from your own life, culture, and language, since those surface harms generic benchmarks miss. Then check specific dimensions: skin tone and occupation pairings, cultural defaults like furniture and clothing, and accuracy outside English.
Do AI image generators work worse in other languages?
In our study, yes. Bilingual participants worked in their preferred language and found Portuguese and Chinese prompts produced less accurate images than English ones across all four models. If you use a generator in one language only, this degradation is nearly invisible, which is why bilingual auditors are unusually well positioned to catch it.
What is the MIRAGE tool?
MIRAGE is the web application behind Seeing Twice. Its modular codebase supports roughly 30 text-to-image models and includes a Discourse forum integration designed for community discussion of findings, though we did not use that feature in the study. A live demo runs at mirage.matheus.wiki.
Can non-experts audit AI systems?
Yes, and that is the point. Our participants ranged from 18 to 76 years old, needed no AI expertise, and were compensated with 15 dollar gift cards for 45-minute IRB-approved sessions. Diverse end users repeatedly caught harms that small, homogeneous developer teams overlook, because they audit against their own lived experience.
How is Seeing Twice different from the original MIRAGE paper?
The HCOMP 2024 paper was a demo introducing the tool concept. Seeing Twice, at ACM CI 2025, contributes the full multi-model web application, findings from fifteen user sessions, and future directions including an anonymous auditing mode for proprietary models and a community leaderboard ranking T2I systems by user preference.
Where can I read the Seeing Twice paper?
The paper is on arXiv as 2511.21547, and the ACM Collective Intelligence 2025 proceedings host the official PDF. The full author list is Matheus Kunzler Maldaner, Wesley Hanwen Deng, Jason I. Hong, Ken Holstein, and Motahhare Eslami, with the work done at Carnegie Mellon University.
BibTeX
@inproceedings{maldaner2025seeingtwice,
title={Seeing Twice: How Side-by-Side T2I Comparison Changes Auditing Strategies},
author={Maldaner, Matheus Kunzler and Deng, Wesley Hanwen and Hong, Jason I. and Holstein, Ken and Eslami, Motahhare},
booktitle={Proceedings of the ACM Conference on Collective Intelligence (CI)},
year={2025}
}