Born in New York in 1823, Joseph was one of Joshua Moses and Sarah Rodriguez Brandon Moses’ nine children. His siblings included Isaac Jr., Israel, Lionel, Abraham, and Benjamin. His father was a merchant and the son of German immigrant Isaac Moses, and his mother the daughter of Sephardic merchant Abraham Rodriguez Brandon.