News

Two papers accepted for CSCW 2019

CSCW 2019 logo

The Crowd Lab had two papers accepted for the upcoming ACM Computer Supported Cooperative Work and Social Computing (CSCW 2019) conference in Austin, TX, USA, November 9-13, 2019. The conference had a 31% acceptance rate.

Ph.D. student Sukrit Venkatagiri will be presenting “GroundTruth: Augmenting expert image geolocation with crowdsourcing and shared representations,” co-authored with Jacob Thebault-Spieker, Rachel Kohler, John Purviance, Rifat Sabbir Mansur, and Kurt Luther, all from Virginia Tech. Here’s the paper’s abstract:

Expert investigators bring advanced skills and deep experience to analyze visual evidence, but they face limits on their time and attention. In contrast, crowds of novices can be highly scalable and parallelizable, but lack expertise. In this paper, we introduce the concept of shared representations for crowd–augmented expert work, focusing on the complex sensemaking task of image geolocation performed by professional journalists and human rights investigators. We built GroundTruth, an online system that uses three shared representations—a diagram, grid, and heatmap—to allow experts to work with crowds in real time to geolocate images. Our mixed-methods evaluation with 11 experts and 567 crowd workers found that GroundTruth helped experts geolocate images, and revealed challenges and success strategies for expert–crowd interaction. We also discuss designing shared representations for visual search, sensemaking, and beyond.

Ph.D. student Tianyi Li will be presenting “Dropping the baton? Understanding errors and bottlenecks in a crowdsourced sensemaking pipeline,” co-authored with Chandler J. Manns, Chris North, and Kurt Luther, also from VT. Here’s the abstract:

Crowdsourced sensemaking has shown great potential for enabling scalable analysis of complex data sets, from planning trips, to designing products, to solving crimes. Yet, most crowd sensemaking approaches still require expert intervention because of worker errors and bottlenecks that would otherwise harm the output quality. Mitigating these errors and bottlenecks would significantly reduce the burden on experts, yet little is known about the types of mistakes crowds make with sensemaking micro-tasks and how they propagate in the sensemaking loop. In this paper, we conduct a series of studies with 325 crowd workers using a crowd sensemaking pipeline to solve a fictional terrorist plot, focusing on understanding why errors and bottlenecks happen and how they propagate. We classify types of crowd errors and show how the amount and quality of input data influence worker performance. We conclude by suggesting design recommendations for integrated crowdsourcing systems and speculating how a complementary top-down path of the pipeline could refine crowd analyses.

Congratulations to Sukrit, Tianyi, and their collaborators!

Welcome, Summer 2019 interns!

We are excited to welcome three undergraduate interns to the Crowd Lab’s Arlington location for Summer 2019. Their research is funded by the Research Experience for Undergraduates (REU) program of the National Science Foundation.

Efua Akonor is a Computer Science major at Wellesley College. She will be working on the GroundTruth project.

David Mitchell is a Computer Science major at the University of Illinois at Urbana-Champaign. He will be working on the PairWise project.

Sarwat Kazmi is an Information Science and Government & Politics double-major at the University of Maryland, College Park. She will be working on the GroundTruth project.

Presented at ACWM grand opening

Credit Penelope M. Carrington / The American Civil War Museum

Dr. Luther gave an invited presentation on Civil War Photo Sleuth at the grand opening celebrations of the American Civil War Museum in Richmond, VA, on May 4. He was one of eight Emerging Scholars invited to speak. The museum described the event and program as follows:

On Saturday, May 4, 2019, the American Civil War Museum celebrates the grand opening of its new museum building and exhibits. As part of that program, the ACWM will highlight some of the most interesting work of the next generation of writers, communicators, and thinkers of Civil War era history/public history with a series lightning talks by emerging professionals in their field. Over the winter, ACWM staff reviewed many applications and selected eight individuals in the early phases of their careers who represented a blend of compelling scholarship and communication skills.

You can read more about the grand opening of the museum here.

Received Purdue Outstanding Technology Alumni Award

Dr. Luther (left) with his undergraduate mentor, Prof. Ronald Glotzbach of Purdue University

Dr. Luther was honored with the Outstanding Technology Alumni Award from his alma mater, Purdue Polytechnic Institute (formerly Purdue University’s College of Technology), at an awards event at Purdue University on April 26.

According to the press release, recipients of the Outstanding Technology Alumni Award “have made significant and recognized impacts in their fields, and they remain involved with the Purdue University community.” The university also profiled Dr. Luther’s accomplishments on the college website.

Dr. Luther received his B.S. with highest distinction and departmental honors in computer graphics technology, with a minor in art and design, from Purdue in 2006.

Participated in Intelligence Studies Consortium panel

Dr. Luther joined Prof. Aaron Brantly (VT Political Science), Prof. Chad Levinson (VT Government and International Affairs), and moderator Ms. Christine Callsen (VT Hume Center) on a panel titled, “Social Computing and Its Impact on Intelligence,” at the Emerging Trends: New Tools, Threats and Thinking symposium. The event was sponsored by the National Capital Region Intelligence Studies Consortium (ISC) and held at Marymount University on April 25.

Presented at IUPUI Computer Science Seminar

Dr. Luther gave an invited presentation, titled “Solving Photo Mysteries with Expert-Led Crowdsourcing,” at the Indiana University-Purdue University Indianapolis (IUPUI) Department of Computer & Information Science Seminar on April 19. Here is the abstract for the presentation:

Investigators in domains such as journalism, intelligence analysis, and human rights advocacy frequently analyze photographs of questionable or unknown provenance. These photos can provide invaluable leads and evidence, but even experts must invest significant time in each analysis, with no guarantee of success. Crowdsourcing, with its affordances for scalability and parallelization, has great potential to augment expert performance, but little is known about how crowds might fit into photo analysts’ complex workflows. In this talk, I present my group’s research with two communities: open-source investigators who geolocate and verify social media photos, and antiquarians who identify unknown persons in 19th-century portrait photography. Informed by qualitative studies of current practice, we developed a novel approach, expert-led crowdsourcing, that combines the complementary strengths of experts and crowds to solve photo mysteries. We built two software tools based on this approach, GroundTruth and Photo Sleuth, and evaluated them with real experts. I conclude by discussing some broader takeaways for crowdsourced investigations, sensemaking, and image analysis.

Participated in Return on Creativity panel

Dr. Luther joined Prof. Ben Knapp (VT Institute for Creativity, Arts, and Technology) and Ms. Natalie Hart (VT Advancement) for a panel titled, “Return on Creativity: From Creativity to Innovation.” The event was sponsored by Arlington Economic Development (AED) and held at the Virginia Tech Research Center-Arlington on April 4. The abstract for the panel was as follows:

For the past seven years, Virginia Tech’s Institute for Creativity, Arts, and Technology (ICAT) has been pushing the envelope of creative exploration. Through partnerships with all the colleges at Virginia Tech, ICAT has assembled teams of scientists, engineers, artists, and designers to tackle some of the most complex innovation challenges that drive economic development. Join us to hear about the Creativity and Innovation District at Virginia Tech, ICAT’s role within it and the critical importance of human-centered design.