Interactive IR User Study Design, Evaluation, and Reporting (Synthesis Lectures on Information Concepts, Retrieval, and Services)

Since user study design has been widely applied in search interactions and information retrieval (IR) systems evaluation studies, a deep reflection and meta-evaluation of interactive IR (IIR) user studies is critical for sharpening the instruments of IIR research and improving the reliability and validity of the conclusions drawn from IIR user studies.

To this end, we developed a faceted framework for supporting user study design, reporting, and evaluation based on a systematic review of the state-of-the-art IIR research papers recently published in several top IR venues (n=462). Within the framework, we identify three major types of research focuses, extract and summarize facet values from specific cases, and highlight the under-reported user study components which may significantly affect the results of research. Then, we employ the faceted framework in evaluating a series of IIR user studies against their respective research questions and explain the roles and impacts of the underlying connections and "collaborations" among different facet values. Through bridging diverse combinations of facet values with the study design decisions made for addressing research problems, the faceted framework can shed light on IIR user study design, reporting, and evaluation practices and help students and young researchers design and assess their own studies.

Jiqun Liu is a Ph.D. candidate in the Department of Library and Information Science at School of Communication and Information, Rutgers University. His main research interests lie in understanding how people's problematic situations, information seeking intentions, and search strategies are connected in search interactions and what this means for the design and evaluation of user-centered interactive search systems. Jiqun Liu received his M.S. in Information Science from the Department of Information Management, Peking University and a double B.S. in Library Science and Finance from the Business School, Nankai University.

Chirag Shah is an Associate Professor in both the School of Communication and Information (SC&I) and the Department of Computer Science at Rutgers University. His research interests include studies of interactive information retrieval/seeking, trying to understand the task a person is doing, and providing proactive recommendations. Dr. Shah received his M.S. in Computer Science from University of Massachusetts (UMass) at Amherst, and Ph.D. in Information Science from University of North Carolina (UNC) at Chapel Hill. He directs the InfoSeeking Lab at Rutgers University where he investigates issues related to information seeking, human-computer interaction (HCI), and social media, supported by grants from National Science Foundation (NSF), National Institute of Health (NIH), Institute of Museum and Library Services (IMLS), Amazon, Google, and Yahoo.

Gary Marchionini is the Cary C. Boshamer Professor of Information Science in the School of Information and Library Science at the University of North Carolina at Chapel Hill. His Ph.D. is from Wayne State University in mathematics education with an emphasis on educational computing. His research interests are in information seeking in electronic environments, digital libraries, human-computer interaction, digital government and information technology policy. He has had grants or contracts from the National Science Foundation, the U.S. Department of Education, the Council on Library Resources, the National Library of Medicine, the Library of Congress, the Kellogg Foundation, and NASA, among others. He was the Conference Chair for the 1996 ACM Digital Library Conference and program chair for the 2002 ACM-IEEE Joint Conference on Digital Libraries. He is editor-in-chief for ACM Transactions on Information Systems and serves on the editorial boards of a dozen scholarly journals. He has published more than 150 articles, chapters, and conferencepapers in the information science, computer science, and education literatures. He founded the Interaction Design Laboratory at UNC-CH.