Abstract

Factorial Survey Experiments (FSEs) are widely used to investigate judgment principles and subgroup differences by presenting respondents with systematically varied descriptions of scenarios, individuals, or objects. Numerous studies have explored how to optimize design features of FSEs, such as presentation format and vignette complexity. Most of them rely on non-probability or convenience samples. However, the underlying assumption that the cognitive response processes of non-probability samples are equivalent to those of probability-based samples remains largely untested in an FSE setting. Non-probability samples often comprise individuals with higher survey familiarity and extrinsic motivation (e.g., monetary incentives or academic credit), which may systematically influence their engagement and response behavior in FSEs. Here we want to answer whether non-probability samples yield systematically different responses in methodological FSE experiments compared to probability samples? As a basis for our study we replicate and adapt (due to time constraints) the experiment on presentation style (table vs. text), and how to sort vignettes (random vs. extreme-cases-first) by Sauer et al. (2020) on fairness of wage in Germany. The vignette experiment will comprise of a set of the original eight vignette dimension used by Sauer et al. excluding some specific dimension levels to improve d-efficiency, resulting in a vignette universe of 10496. We present 240 vignettes divided into 20 decks with 12 vignettes each. This design will be employed later this year in a probability sample drawn from the Federal Employment Agency consisting of 1,500 respondents and a non-probability sample from various online access panels resulting in a sample of 1,500 respondents. The poster will present preliminary results examining differences in response patterns, attention, response times, and judgment consistency across presentation style, vignette order, and sample types, providing empirical evidence on the robustness of methodological FSE experiment findings when generalized on non-probability samples. The study offers practical guidance for the design of future methodological research on FSEs and contributes to the broader understanding of how sample types may influence experimental outcomes in survey research.