Born in Barbados, Isaac Rodriguez Brandon was the son of Abraham Rodriguez Brandon and Sarah Esther Lopez, both natives of the island. The Rodriguez Brandon family, as were nearly all Jewish families in the thriving British colony, was involved in the manufacture and export of sugar and its derivatives, the principle products of the island’s slave-based agricultural economy.

The Rodriguez Brandon family maintained strong ties throughout the Caribbean and to New York, where they often traveled and would eventually reside for a time.

In 1817, during the family’s sojourn in New York, Isaac’s sister Sarah married Joshua Moses, the son of distinguished New York merchant Isaac Moses and Reyna Levy. Seven years later, Isaac married the Moses’ daughter, Lavinia, in an apparent love match. However, theirs was to be a tragically short union. Lavinia died in 1828, while giving birth to their second child, a daughter who would in turn bear her mother’s name. The younger Lavinia barely lived to see age two. Heartbroken at the loss of his beloved wife and child, Isaac Rodriguez Brandon never remarried.