ABSTRACT

 Best practices in quantitative user research in security and privacy, such as using a-priori power analysis to determine necessary sample sizes before conducting studies or reporting and interpreting effect sizes with statistical hypothesis tests, are still not regularly applied in our field. Our meta-scientific work explores the status quo of power, effect size, and reporting and aims to support researchers in following best practices. We first focused on the field of developer-centered usable security, where recruiting participants is especially challenging, and thus, data from such studies should be used with care. We found many studies in this area do not conduct power analysis, leading to underpowered studies that may fail to detect even large effects. We extended this manual analysis to a larger sample of studies, using an LLM-supported process and conducted a meta-analysis to aid future research in conducting power analysis and interpreting effect sizes. To also understand the perspective of researchers on these topics, we additionally conducted interviews and surveys with researchers, from which we learned more about how they interpret and understand effect sizes. We hope this work sparks discussion and development around making statistical results more accessible and useful for readers and meta-analysts.