Abraham was the fourth of nine children born to Joshua and Sarah Rodriguez Brandon Moses, a group well represented by a series of daguerreotypes in the collections of the American Jewish Historical Society. His siblings included Isaac Jr., Israel, Lionel, Benjamin and Joseph. His father was a merchant and the son of German immigrant Isaac Moses, and his mother the daughter of Sephardic Merchant Abraham Rodriguez Brandon.