For behavioral scientists struggling to recruit enough subjects for their studies, artificial intelligence (AI) offers a tantalizing solution: artificial "participants" that can stand in for real people. Researchers have already reported that these so-called silicon samples produce humanlike responses in some surveys and experiments – and some even hope they could simulate the responses of minorities or other groups who are often underrepresented in studies.

But a new preprint calls for caution before researchers make the leap. Those who turn to AI samples face a dizzying array of choices – from which large language model (LLM) to use, to its settings, to precisely what information it is fed. According to the paper, posted to arXiv in September, 2025, these decisions can result in wide-ranging and even contradictory results – and no specific combination of choices produces data that best matches human responses.

By studying the combined effect of researchers’ decisions, the finding bolsters previous work that has studied the impact of individual choices, says Indira Sen, a computational social scientist at the University of Mannheim who was not involved with the study. "It definitely seems that this risk is very real."

Source https://www.science.org/content/article/ai-generated-participants-can-lead-social-science-experiments-astray-study-finds

Another high-profile case has interested scientists. Read now