Our Civil War Photo Sleuth project was featured in a TIME article by Olivia Waxman, “How Artificial Intelligence Is Helping Identify Thousands of Unknown Civil War Soldiers.” The article includes interviews with several prominent CWPS users showing how the software has benefited them by identifying unknown soldiers in Civil War-era photos.
Congrats to Vikram Mohanty, lead developer of CWPS and Crowd Lab Ph.D. student, for this publicity!
Dr. Luther gave an invited presentation at the Center for the Advanced Study of Communities and Information (CASCI) at the University of Maryland on November 19, 2019. The title of his presentation was, “Crowd Sleuths: Solving Mysteries with Crowdsourcing, Experts and AI.” The abstract was as follows:
Professional investigators in fields such as journalism, law enforcement, and academia have long sought the public’s help in solving mysteries, typically by providing tips. However, as social technologies capture more digital traces of daily life and enable new forms of collaboration, members of the public are increasingly leading their own investigations. These efforts are perhaps best known for high-profile failures characterized by sloppy research and vigilantism, such as the 2013 Boston Marathon Bombing manhunt on Reddit and 4chan. However, other crowdsourced investigations have led to the successful recovery of missing persons and apprehension of violent criminals, suggesting real potential. I will present three projects from my research group, the Crowd Intelligence Lab, where we helped to enable novice crowds to discover a hidden terrorist plot within large quantities of textual evidence documents; collaborate with expert investigators to geolocate and verify (or debunk) photos and videos shared on social media; and use AI-based face recognition to identify unknown soldiers in historical portraits from the American Civil War era.
Dr. Luther gave an invited presentation at the AAAI Fall Symposium on Artificial Intelligence and Work on November 8, 2019. The title of his presentation was, “Solving AI’s last-mile problem with crowds and experts.”
Dr. Luther’s position paper accompanying the presentation is available online. The abstract for the paper is as follows:
Visual search tasks, such as identifying an unknown person or location in a photo, are a crucial element of many forms of investigative work, from academic research, to journalism, to law enforcement. While AI techniques like computer vision can often quickly and accurately narrow down a large search space of thousands of possibilities to a shortlist of promising candidates, they usually cannot select the correct match(es) among those, a challenge known as the last-mile problem. We have developed an approach called crowd-augmented expert work to leverage the complementary strengths of human intelligence to solve the last-mile problem. We report on case studies developing and deploying two visual search tools, GroundTruth and Photo Sleuth, to illustrate this approach.
The Crowd Lab won the Best Poster/Demo Award at HCOMP 2019! Congratulations to Crowd Lab postdoc and team leader Jacob Thebault-Spieker, Ph.D. student Sukrit Venkatagiri, and undergraduate researchers David Mitchell and Chris Hurt. Their poster/demo was titled, “PairWise: Mitigating political bias in crowdsourced content moderation.”
The Crowd Lab also won the Best Poster/Demo Award at HCOMP 2018.
Dr. Luther and Dr. Sylvester Johnson, director of the Center for Humanities at Virginia Tech, co-presented on “The future of AI and what it means for humans” to local journalists at a media event called “On the Record with Virginia Tech” on October 17, 2019. The press release for the event described it as follows:
Technology is the changing the way we live and work. For centuries, being human has been described by emphasizing the ability to think and reason. But now technology innovation using Artificial Intelligence (AI) can help us mimic human-like behavior to make complicated decisions and solve world problems.
Virginia Tech’s Innovation Campus in Alexandria will focus on the intersection between technology and the human experience, leading the way not just in technical domains but also looking at the policy and ethical implications to ensure that technology doesn’t drive inequity.
What will it mean to be human as intelligent machines continue to advance? How is AI improving our lives? What are the dangers that more powerful AI might bring?
In this talk, Virginia Tech humanities scholar Sylvester Johnson and computer scientist Kurt Luther will share recent discoveries and explore how the latest technological advances in AI are changing our lives.
A video clip of the event was broadcast on a local TV news channel, WDVM.
Dr. Luther gave an invited presentation to an audience of engineers and journalists at The Washington Post on October 23, 2019. The title of his talk was, “Photo sleuthing: Helping investigators solve photo mysteries using crowdsourcing and AI.” The abstract for the talk was:
Journalists, intelligence analysts, and human rights investigators frequently analyze photographs of questionable or unknown provenance, trying to identify the people and places depicted. These photos can provide invaluable leads and evidence, but even experts must invest significant time in each analysis, with no guarantee of success. Collective human intelligence (via crowdsourcing) and artificial intelligence (via computer vision) offer great potential to support expert photo analysis. However, we must first understand how to leverage the complementary strengths of these techniques to support investigators’ real-world needs and work practices.
In this talk, I present my lab’s research with two “photo sleuthing” communities: (1) open-source intelligence (OSINT) analysts who geolocate and verify photos and videos shared on social media, and (2) researchers and collectors who identify unknown soldiers in historical portraits from the 19th century. Informed by qualitative studies of current practice, we developed a novel approach that combines the complementary strengths of expert investigators, novice crowdsourcing, and computer vision to solve photo mysteries. We built two software tools based on this approach, GroundTruth and Photo Sleuth, and evaluated them with real expert investigators.
Our article, Read-Agree-Predict: A crowdsourced approach to discovering relevant primary sources for historians, was published in Human Computation Journal. Congratulations to Crowd Lab Ph.D. alumnus Dr. Nai-Ching Wang, who led the paper, along with co-authors Dr. David Hicks (Education) and Dr. Paul Quigley (History). The abstract for the article is:
Historians spend significant time looking for relevant, high-quality primary sources in digitized archives and through web searches. One reason this task is time-consuming is that historians’ research interests are often highly abstract and specialized. These topics are unlikely to be manually indexed and are difficult to identify with automated text analysis techniques. In this article, we investigate the potential of a new crowdsourcing model in which the historian delegates to a novice crowd the task of labeling the relevance of primary sources with respect to her unique research interests. The model employs a novel crowd workflow, Read-Agree-Predict (RAP), that allows novice crowd workers to label relevance as well as expert historians. As a useful byproduct, RAP also reveals and prioritizes crowd confusions as targeted learning opportunities. We demonstrate the value of our model with two experiments with paid crowd workers (n=170), with the future goal of extending our work to classroom students and public history interventions. We also discuss broader implications for historical research and education.
Dr. Luther was invited to present at the Machine Learning + Libraries Summit held on September 20, 2019 at the Library of Congress in Washington, DC. The title of his presentation — which was allocated extra time as a featured project — was, “Civil War Photo Sleuth: Combining Crowdsourcing and Face Recognition to Identify Historical Portraits.” According to the event organizers:
This one-day conference convened 75 cultural heritage professionals (roughly 50 from outside the Library of Congress and 25 staff from within) to discuss the on-the-ground applications of machine learning technologies in libraries, museums, and universities. Hosting this conference was part of a larger effort to learn about machine learning and the role it could play in helping the Library of Congress reach its strategic goals, such as enhancing discoverability of the Library’s collections, building connections between users and the Library’s digital holdings, and leveraging technology to serve creative communities and the general public.
A detailed report on the outcomes of the event is available online.
The Crowd Lab had two posters/demos accepted for AAAI HCOMP 2019! Both of these papers involved substantial contributions from our summer REU interns, who will be attending the conference at Skamania Lodge, Washington, to present their work.
It’s QuizTime: A study of online verification practices on Twitter was led by Crowd Lab Ph.D. student Sukrit Venkatagiri, with co-authors Jacob Thebault-Spieker, Sarwat Kazmi, and Efua Akonor. Sarwat and Efua were summer REU interns in the Crowd Lab from the University of Maryland and Wellesley College, respectively. The abstract for the poster is:
Misinformation poses a threat to public health, safety, and democracy. Training novices to debunk visual misinformation with image verification techniques has shown promise, yet little is known about how novices do so in the wild, and what methods prove effective. Thus, we studied 225 verification challenges posted by experts on Twitter over one year with the aim of improving novices’ skills. We collected, annotated, and analyzed these challenges and over 3,100 replies by 304 unique participants. We find that novices employ multiple tools and approaches, and techniques like collaboration and reverse image search significantly improve performance.
PairWise: Mitigating political bias in crowdsourced content moderation was led by Crowd Lab postdoc Jacob Thebault-Spieker, with co-authors Sukrit Venkatagiri, David Mitchell, and Chris Hurt. David was a summer REU intern from the University of Illinois, and Chris was a Virginia Tech undergraduate. The abstract for the demo is:
Crowdsourced labeling of political social media content is an area of increasing interest, due to the contextual nature of political content. However, there are substantial risks of human biases causing data to be labelled incorrectly, possibly advantaging certain political groups over others. Inspired by the social computing theory of social translucence and findings from social psychology, we built PairWise, a system designed to facilitate interpersonal accountability and help mitigate biases in political content labelling.
The Crowd Lab had a paper, titled, “Second Opinion: Supporting last-mile person identification with crowdsourcing and face recognition,” accepted for the upcoming AAAI Human Computation and Crowdsourcing (HCOMP 2019) conference at the Skamania Lodge in Stevenson, WA, USA, October 28-30, 2019. The conference had a 25% acceptance rate.
Ph.D. student and lead author Vikram Mohanty will present the paper, co-authored with Dr. Luther and Crowd Lab undergraduate researchers Kareem Abdol-Hamid and Courtney Ebersohl. Here’s the paper’s abstract:
As AI-based face recognition technologies are increasingly adopted for high-stakes applications like locating suspected criminals, public concerns about the accuracy of these technologies have grown as well. These technologies often present a human expert with a shortlist of high-confidence candidate faces from which the expert must select correct match(es) while avoiding false positives, which we term the “last-mile problem.” We propose Second Opinion, a web-based software tool that employs a novel crowdsourcing workflow inspired by cognitive psychology, seed-gather-analyze, to assist experts in solving the last-mile problem. We evaluated Second Opinion with a mixed-methods lab study involving 10 experts and 300 crowd workers who collaborate to identify people in historical photos. We found that crowds can eliminate 75% of false positives from the highest-confidence candidates suggested by face recognition, and that experts were enthusiastic about using Second Opinion in their work. We also discuss broader implications for crowd–AI interaction and crowdsourced person identification.
