Abraham was the fourth of nine children born to Joshua and Sarah Rodrigues Brandon Moses, a group well represented by a series of daguerreotypes in the collections of the American Jewish Historical Society. Abraham’s siblings included Isaac Jr., Israel, Lionel, Benjamin, and Joseph. His father was a merchant and the son of German immigrant Isaac Moses, and his mother the daughter of Sephardic Merchant Abraham Rodrigues Brandon.