Little is known about the early life of Abraham Levy. He was born in Amsterdam and by the late 1790s—the exact date remains uncertain—he had married Sarah Rachel Cornelia Bernard. The first of their five children was born in 1799, the last in 1813. It is not clear exactly when they came to the United States, but by 1820, when Levy was already past 50 years of age, the family was listed as living in Richmond, Virginia. There, according to an obituary, he amassed a small fortune in dry goods.