The daughter of Abraham Rodrigues Brandon and Sarah Esther Lopez Brandon, Sarah Rodrigues Brandon was born on the island of Barbados in 1799. Her family had grown wealthy from the sugar trade that dominated the Barbadian economy and they maintained ties—both religious and mercantile—throughout the Atlantic world, especially with New York. Most of the Brandon children would leave Barbados, and both Sarah and her brother Isaac would marry children of Isaac Moses of New York, she to Joshua in 1817, and her brother to Lavinia seven years later. Together Sarah and Joshua had ten children, including a set of twins. One of the twins, Isaac Moses Jr., would serve as a high-ranking Union officer in the Civil War.