Moses was, together with his twin brother Isaac, the second of of nine children born to Joshua and Sarah Rodriguez Brandon Moses, a group well represented by a series of daguerreotypes in the collections of the American Jewish Historical Society. His siblings included Israel, Lionel, Benjamin and Joseph. His father was a merchant and the son of German immigrant Isaac Moses, and his mother the daughter of Sephardic Merchant Abraham Rodriguez Brandon.