Born and raised in Philadelphia, Henry Lazarus was one of two children born to Esther Lyons and Isaac Lazarus. In 1835, Henry married his first cousin, Merriem Moss, daughter of John Moss and Rebecca Lyons. They had one child, George Gaines Lazarus, born in 1836.

Depicted beside Lazarus is a figure recorded as Isaac P. Jacobs, about whom nothing is known.