Presented at University of Maryland CASCI Seminar

Dr. Luther gave an invited presentation at the Center for the Advanced Study of Communities and Information (CASCI) at the University of Maryland on November 19, 2019. The title of his presentation was, “Crowd Sleuths: Solving Mysteries with Crowdsourcing, Experts and AI.” The abstract was as follows:

Professional investigators in fields such as journalism, law enforcement, and academia have long sought the public’s help in solving mysteries, typically by providing tips. However, as social technologies capture more digital traces of daily life and enable new forms of collaboration, members of the public are increasingly leading their own investigations. These efforts are perhaps best known for high-profile failures characterized by sloppy research and vigilantism, such as the 2013 Boston Marathon Bombing manhunt on Reddit and 4chan. However, other crowdsourced investigations have led to the successful recovery of missing persons and apprehension of violent criminals, suggesting real potential. I will present three projects from my research group, the Crowd Intelligence Lab, where we helped to enable novice crowds to discover a hidden terrorist plot within large quantities of textual evidence documents; collaborate with expert investigators to geolocate and verify (or debunk) photos and videos shared on social media; and use AI-based face recognition to identify unknown soldiers in historical portraits from the American Civil War era.

Presented at AAAI Fall Symposium on Artificial Intelligence and Work

Photo courtesy of Kenneth Huang via Twitter.

Dr. Luther gave an invited presentation at the AAAI Fall Symposium on Artificial Intelligence and Work on November 8, 2019. The title of his presentation was, “Solving AI’s last-mile problem with crowds and experts.”

Dr. Luther’s position paper accompanying the presentation is available online. The abstract for the paper is as follows:

Visual search tasks, such as identifying an unknown person or location in a photo, are a crucial element of many forms of investigative work, from academic research, to journalism, to law enforcement. While AI techniques like computer vision can often quickly and accurately narrow down a large search space of thousands of possibilities to a shortlist of promising candidates, they usually cannot select the correct match(es) among those, a challenge known as the last-mile problem. We have developed an approach called crowd-augmented expert work to leverage the complementary strengths of human intelligence to solve the last-mile problem. We report on case studies developing and deploying two visual search tools, GroundTruth and Photo Sleuth, to illustrate this approach.

Presented on photo sleuthing at The Washington Post

Dr. Luther gave an invited presentation to an audience of engineers and journalists at The Washington Post on October 23, 2019. The title of his talk was, “Photo sleuthing: Helping investigators solve photo mysteries using crowdsourcing and AI.” The abstract for the talk was:

Journalists, intelligence analysts, and human rights investigators frequently analyze photographs of questionable or unknown provenance, trying to identify the people and places depicted. These photos can provide invaluable leads and evidence, but even experts must invest significant time in each analysis, with no guarantee of success. Collective human intelligence (via crowdsourcing) and artificial intelligence (via computer vision) offer great potential to support expert photo analysis. However, we must first understand how to leverage the complementary strengths of these techniques to support investigators’ real-world needs and work practices. 

In this talk, I present my lab’s research with two “photo sleuthing” communities: (1) open-source intelligence (OSINT) analysts who geolocate and verify photos and videos shared on social media, and (2) researchers and collectors who identify unknown soldiers in historical portraits from the 19th century. Informed by qualitative studies of current practice, we developed a novel approach that combines the complementary strengths of expert investigators, novice crowdsourcing, and computer vision to solve photo mysteries. We built two software tools based on this approach, GroundTruth and Photo Sleuth, and evaluated them with real expert investigators.

Two posters/demos accepted for HCOMP 2019

The Crowd Lab had two posters/demos accepted for AAAI HCOMP 2019! Both of these papers involved substantial contributions from our summer REU interns, who will be attending the conference at Skamania Lodge, Washington, to present their work.

Crowd Lab interns Sarwat Kazmi (left) and Efua Akonor (right) presenting their poster at HCOMP 2019.

It’s QuizTime: A study of online verification practices on Twitter was led by Crowd Lab Ph.D. student Sukrit Venkatagiri, with co-authors Jacob Thebault-Spieker, Sarwat Kazmi, and Efua Akonor. Sarwat and Efua were summer REU interns in the Crowd Lab from the University of Maryland and Wellesley College, respectively. The abstract for the poster is:

Misinformation poses a threat to public health, safety, and democracy. Training novices to debunk visual misinformation with image verification techniques has shown promise, yet little is known about how novices do so in the wild, and what methods prove effective. Thus, we studied 225 verification challenges posted by experts on Twitter over one year with the aim of improving novices’ skills. We collected, annotated, and analyzed these challenges and over 3,100 replies by 304 unique participants. We find that novices employ multiple tools and approaches, and techniques like collaboration and reverse image search significantly improve performance.

Crowd Lab intern David Mitchell presenting his demo and poster at HCOMP 2019.

PairWise: Mitigating political bias in crowdsourced content moderation was led by Crowd Lab postdoc Jacob Thebault-Spieker, with co-authors Sukrit Venkatagiri, David Mitchell, and Chris Hurt. David was a summer REU intern from the University of Illinois, and Chris was a Virginia Tech undergraduate. The abstract for the demo is:

Crowdsourced labeling of political social media content is an area of increasing interest, due to the contextual nature of political content. However, there are substantial risks of human biases causing data to be labelled incorrectly, possibly advantaging certain political groups over others. Inspired by the social computing theory of social translucence and findings from social psychology, we built PairWise, a system designed to facilitate interpersonal accountability and help mitigate biases in political content labelling.

Two papers accepted for CSCW 2019

CSCW 2019 logo

The Crowd Lab had two papers accepted for the upcoming ACM Computer Supported Cooperative Work and Social Computing (CSCW 2019) conference in Austin, TX, USA, November 9-13, 2019. The conference had a 31% acceptance rate.

Ph.D. student Sukrit Venkatagiri will be presenting “GroundTruth: Augmenting expert image geolocation with crowdsourcing and shared representations,” co-authored with Jacob Thebault-Spieker, Rachel Kohler, John Purviance, Rifat Sabbir Mansur, and Kurt Luther, all from Virginia Tech. Here’s the paper’s abstract:

Expert investigators bring advanced skills and deep experience to analyze visual evidence, but they face limits on their time and attention. In contrast, crowds of novices can be highly scalable and parallelizable, but lack expertise. In this paper, we introduce the concept of shared representations for crowd–augmented expert work, focusing on the complex sensemaking task of image geolocation performed by professional journalists and human rights investigators. We built GroundTruth, an online system that uses three shared representations—a diagram, grid, and heatmap—to allow experts to work with crowds in real time to geolocate images. Our mixed-methods evaluation with 11 experts and 567 crowd workers found that GroundTruth helped experts geolocate images, and revealed challenges and success strategies for expert–crowd interaction. We also discuss designing shared representations for visual search, sensemaking, and beyond.

Ph.D. student Tianyi Li will be presenting “Dropping the baton? Understanding errors and bottlenecks in a crowdsourced sensemaking pipeline,” co-authored with Chandler J. Manns, Chris North, and Kurt Luther, also from VT. Here’s the abstract:

Crowdsourced sensemaking has shown great potential for enabling scalable analysis of complex data sets, from planning trips, to designing products, to solving crimes. Yet, most crowd sensemaking approaches still require expert intervention because of worker errors and bottlenecks that would otherwise harm the output quality. Mitigating these errors and bottlenecks would significantly reduce the burden on experts, yet little is known about the types of mistakes crowds make with sensemaking micro-tasks and how they propagate in the sensemaking loop. In this paper, we conduct a series of studies with 325 crowd workers using a crowd sensemaking pipeline to solve a fictional terrorist plot, focusing on understanding why errors and bottlenecks happen and how they propagate. We classify types of crowd errors and show how the amount and quality of input data influence worker performance. We conclude by suggesting design recommendations for integrated crowdsourcing systems and speculating how a complementary top-down path of the pipeline could refine crowd analyses.

Congratulations to Sukrit, Tianyi, and their collaborators!

Poster presented at HCIC 2018

Crowd Lab Ph.D. student Sukrit Venkatagiri and postdoc Jacob Thebault-Spieker presented a poster on their research using crowdsourcing to analyze satellite imagery for geolocation purposes at the Human-Computer Interaction Consortium (HCIC 2018) conference in Watsonville, CA. The poster, seen below, was titled, “Verifying Truth from the Ground: Leveraging Human Strengths in the Image Geolocation Process“.

HCIC poster