Abraham Wormser was born in Michelstadt, Hesse, the son of Seckel Löb Mattes Wormser, a celebrated rabbi, kabbalist, and a Baal Shem (miracle worker). He lived an ascetic life, abstaining from meat, and ran a yeshiva in Michelstadt. He enjoyed a reputation for treating illness and lunacy throughout southern Germany. Abraham was the youngest of the rabbi’s thirteen children, born to his second wife, Johanna Wormser, when his father was already sixty years old.

Abraham moved to New York as young man, and there, in 1831, he married Fannie Wormser, herself an immigrant from Prussia, and with whom he had two children. Like his father, Abraham served as a communal leader, associated with a congregation on Avenue C.