Applying Facial Recognition to Civil War Photos

Facial recognition technology for genealogy: Civil War photos and more

A new website applies facial recognition technology to unidentified Civil War photos, aiming to “rediscover the lost names and stories of every photo of American Civil War soldiers and sailors.” To do this, Civil War Photo Sleuth  plans to curate the world’s biggest Civil War-era photo portrait archive, apply facial recognition technologies to the images, and enlist the interest and knowledge of a dedicated community of historians and enthusiasts to compile their stories.

Users of the site are encouraged to upload their own Civil War-era portrait photos, along with any known information about the person’s identity military service. Users can then run a search for other photos of the same person. The site’s advanced facial recognition technology looks at “27 different facial landmarks on your unknown photo, and then compares the ratio of those landmarks to the ratios of all the other photos in our database,” explains site overseer Kurt Luther, Assistant Professor of Computer Science at Virginia Tech. “This allows us to compare two different views of the same person, because even if her head is turned at a different angle, the ratios stay the same.”

Trying out the technology

I tried Civil War Photo Sleuth to see how it works. First, I attempted to upload an image from my own family archive, but it didn’t have good enough quality for the site to recognize a face. So I uploaded a Civil War photo I found online of Samuel Watson Black (it’s in the public domain). A link to the University of Pittsburgh’s Digital Collections, which posted the image, says it was taken by famed photographer Mathew Brady. The Civil War Photo Sleuth database did, in fact, identify another image of Black:

The site is free to use. It’s currently funded by multiple grant sources and individual donations, with primary support from the National Science Foundation and Virginia Tech.

A genealogist’s questions for Civil War Sleuth

Thinking of all those unidentified photos in genealogists’ hands, I asked Luther whether he plans to expand the site’s scope. “For now we’re focused on the Civil War era, and that’s keeping us busy,” he responded. “But [the site itself] is also a research project, and we’re committed to sharing what we’ve learned to help researchers working in other areas.”

Next, I asked whether the technology can currently predict a degree of relatedness, to help identify relatives in photos. Luther explains, “Our software does generate a similarity percentage for each face in the search results, but we choose not to display that information, because it can be misleading. The research on face recognition suggests that for close comparisons, humans still outperform computers, because humans can consider more information in making their decision, including hair and facial hairstyles, clothing, and biographical details.”

“We want to discourage the idea that if two faces have a certain percentage similarity, it’s definitely a match,” Luther continues. “Instead, we emphasize the face recognition is a way to narrow down possibilities or reveal some candidates you might have missed. It’s the start of a research process, not the end of one.”

More photo recognition technology for family history

Out of curiosity, I reached out to Joe Bott, founder of the online photo archive DeadFred.com. Would or could he use technology like this on the site to help users identify photos of the same people? He responded immediately. “Yes, I would very much like to offer Face Recognition….A tool that makes finding a subject in a photo more accurate and faster…will become part of the genealogical toolbox, sooner than later.” He doesn’t currently have the ability to add face recognition technology to DeadFred.com but would love to make it happen.

One genealogy website that already uses photo recognition technology is MyHeritage.com. Chief Genealogist Daniel Horowitz explains: “MyHeritage (on the web and in the free Family Tree Builder software) uses face recognition to recognize faces in the images. When you upload a picture, MyHeritage will detect the face and select it with a square around it. This facilitates the tagging of people on it.” Tagging images adds them to those relatives’ individual profiles in your family tree.

Learn more about Civil War photos from The Photo Detective and Family Tree Magazine columnist Maureen A. Taylor, or read the fascinating ebook Life in Civil War America, available instantly as a digital download.