Our Civil War Photo Sleuth project was featured in a TIME article by Olivia Waxman, “How Artificial Intelligence Is Helping Identify Thousands of Unknown Civil War Soldiers.” The article includes interviews with several prominent CWPS users showing how the software has benefited them by identifying unknown soldiers in Civil War-era photos.
Congrats to Vikram Mohanty, lead developer of CWPS and Crowd Lab Ph.D. student, for this publicity!
Dr. Luther gave an invited presentation at the Center for the Advanced Study of Communities and Information (CASCI) at the University of Maryland on November 19, 2019. The title of his presentation was, “Crowd Sleuths: Solving Mysteries with Crowdsourcing, Experts and AI.” The abstract was as follows:
Professional investigators in fields such as journalism, law enforcement, and academia have long sought the public’s help in solving mysteries, typically by providing tips. However, as social technologies capture more digital traces of daily life and enable new forms of collaboration, members of the public are increasingly leading their own investigations. These efforts are perhaps best known for high-profile failures characterized by sloppy research and vigilantism, such as the 2013 Boston Marathon Bombing manhunt on Reddit and 4chan. However, other crowdsourced investigations have led to the successful recovery of missing persons and apprehension of violent criminals, suggesting real potential. I will present three projects from my research group, the Crowd Intelligence Lab, where we helped to enable novice crowds to discover a hidden terrorist plot within large quantities of textual evidence documents; collaborate with expert investigators to geolocate and verify (or debunk) photos and videos shared on social media; and use AI-based face recognition to identify unknown soldiers in historical portraits from the American Civil War era.
Dr. Luther gave an invited presentation at the AAAI Fall Symposium on Artificial Intelligence and Work on November 8, 2019. The title of his presentation was, “Solving AI’s last-mile problem with crowds and experts.”
Dr. Luther’s position paper accompanying the presentation is available online. The abstract for the paper is as follows:
Visual search tasks, such as identifying an unknown person or location in a photo, are a crucial element of many forms of investigative work, from academic research, to journalism, to law enforcement. While AI techniques like computer vision can often quickly and accurately narrow down a large search space of thousands of possibilities to a shortlist of promising candidates, they usually cannot select the correct match(es) among those, a challenge known as the last-mile problem. We have developed an approach called crowd-augmented expert work to leverage the complementary strengths of human intelligence to solve the last-mile problem. We report on case studies developing and deploying two visual search tools, GroundTruth and Photo Sleuth, to illustrate this approach.
Dr. Luther gave an invited presentation to an audience of engineers and journalists at The Washington Post on October 23, 2019. The title of his talk was, “Photo sleuthing: Helping investigators solve photo mysteries using crowdsourcing and AI.” The abstract for the talk was:
Journalists, intelligence analysts, and human rights investigators frequently analyze photographs of questionable or unknown provenance, trying to identify the people and places depicted. These photos can provide invaluable leads and evidence, but even experts must invest significant time in each analysis, with no guarantee of success. Collective human intelligence (via crowdsourcing) and artificial intelligence (via computer vision) offer great potential to support expert photo analysis. However, we must first understand how to leverage the complementary strengths of these techniques to support investigators’ real-world needs and work practices.
In this talk, I present my lab’s research with two “photo sleuthing” communities: (1) open-source intelligence (OSINT) analysts who geolocate and verify photos and videos shared on social media, and (2) researchers and collectors who identify unknown soldiers in historical portraits from the 19th century. Informed by qualitative studies of current practice, we developed a novel approach that combines the complementary strengths of expert investigators, novice crowdsourcing, and computer vision to solve photo mysteries. We built two software tools based on this approach, GroundTruth and Photo Sleuth, and evaluated them with real expert investigators.
Dr. Luther was invited to present at the Machine Learning + Libraries Summit held on September 20, 2019 at the Library of Congress in Washington, DC. The title of his presentation — which was allocated extra time as a featured project — was, “Civil War Photo Sleuth: Combining Crowdsourcing and Face Recognition to Identify Historical Portraits.” According to the event organizers:
This one-day conference convened 75 cultural heritage professionals (roughly 50 from outside the Library of Congress and 25 staff from within) to discuss the on-the-ground applications of machine learning technologies in libraries, museums, and universities. Hosting this conference was part of a larger effort to learn about machine learning and the role it could play in helping the Library of Congress reach its strategic goals, such as enhancing discoverability of the Library’s collections, building connections between users and the Library’s digital holdings, and leveraging technology to serve creative communities and the general public.
A detailed report on the outcomes of the event is available online.
The Crowd Lab had a paper, titled, “Second Opinion: Supporting last-mile person identification with crowdsourcing and face recognition,” accepted for the upcoming AAAI Human Computation and Crowdsourcing (HCOMP 2019) conference at the Skamania Lodge in Stevenson, WA, USA, October 28-30, 2019. The conference had a 25% acceptance rate.
Ph.D. student and lead author Vikram Mohanty will present the paper, co-authored with Dr. Luther and Crowd Lab undergraduate researchers Kareem Abdol-Hamid and Courtney Ebersohl. Here’s the paper’s abstract:
As AI-based face recognition technologies are increasingly adopted for high-stakes applications like locating suspected criminals, public concerns about the accuracy of these technologies have grown as well. These technologies often present a human expert with a shortlist of high-confidence candidate faces from which the expert must select correct match(es) while avoiding false positives, which we term the “last-mile problem.” We propose Second Opinion, a web-based software tool that employs a novel crowdsourcing workflow inspired by cognitive psychology, seed-gather-analyze, to assist experts in solving the last-mile problem. We evaluated Second Opinion with a mixed-methods lab study involving 10 experts and 300 crowd workers who collaborate to identify people in historical photos. We found that crowds can eliminate 75% of false positives from the highest-confidence candidates suggested by face recognition, and that experts were enthusiastic about using Second Opinion in their work. We also discuss broader implications for crowd–AI interaction and crowdsourced person identification.
Dr. Luther gave an invited presentation on Civil War Photo Sleuth at the grand opening celebrations of the American Civil War Museum in Richmond, VA, on May 4. He was one of eight Emerging Scholars invited to speak. The museum described the event and program as follows:
On Saturday, May 4, 2019, the American Civil War Museum celebrates the grand opening of its new museum building and exhibits. As part of that program, the ACWM will highlight some of the most interesting work of the next generation of writers, communicators, and thinkers of Civil War era history/public history with a series lightning talks by emerging professionals in their field. Over the winter, ACWM staff reviewed many applications and selected eight individuals in the early phases of their careers who represented a blend of compelling scholarship and communication skills.
You can read more about the grand opening of the museum here.
Our Civil War Photo Sleuth project continues to attract press and media attention. Here is a roundup of additional articles since the last post:
Our paper, “Photo Sleuth: Combining Human Expertise and Face Recognition to Identify Historical Portraits,” received the Best Paper Award at IUI 2019 in Los Angeles, CA. This award recognized the best paper among 282 submissions. Congratulations to lead author Vikram Mohanty (CS Ph.D. student), David Thames (CS undergraduate), and Sneha Mehta (CS Ph.D. student). A video of the talk (presented by Dr. Luther is embedded below.
Dr. Luther gave an invited presentation, titled “Civil War Photo Sleuthing: Past, Present, and Future” at Civil War Photo Talks in Arlington, VA, co-sponsored by Military Images Magazine and Civil War Faces. Other invited speakers included Ann Shumard, National Portrait Gallery; Micah Messenheimer, Library of Congress; Bryan Cheeseboro, National Archives; and Rick Brown, Military Images. The abstract for Dr. Luther’s talk was as follows:
People have struggled to identify unknown soldiers and sailors in Civil War photos since even before the war ended. In this talk, I trace the 150-year history of photo sleuthing, showing how the passage of time has magnified some challenges, but also unlocked exciting new possibilities. I show how technologies like social media, face recognition, and digital archives allow us to solve photo mysteries that have eluded families and researchers for a century and a half.
