Benjamin Dores Lazarus was the fifteenth of seventeen children born to Richa Benjamin and Marks Lazarus of South Carolina. Born in Charleston, Benjamin counted among his older siblings Emma and Joshua. His middle name testified to the Sephardic heritage he bore from his mother’s side—aas it condensed the Spanish “De Torres.”

Benjamin was raised in a large and happy Charleston family. By the time of his birth in 1800, his three oldest sisters were in their teens and likely doted on this baby brother. His father, a middle class shopkeeper, had served during the Revolutionary War in Lushington’s Militia Company, sometimes called the Jews’ Company for the prevalence of Jewish men within its ranks.

In 1840, at age forty, Benjamin married his niece, Cornelia Cohen, the daughter of his eldest sister Leah and the wealthy Polish-born peddler-turned-planter Mordecai Cohen. Benjamin and Cornelia had five sons and a daughter (named Cornelia after her mother), whom they raised in Charleston.