Born in Philadelphia, Albert Cardozo was raised in New York, one of the seven children of Sephardim Michael Hart and Ellen Cardozo.

Cardozo was a dedicated student and passed the New York bar and began practicing law in 1849. In 1864, he became a justice on the Supreme Court of New York, however his tenure was marked by controversy. He became part of Boss William Tweed’s Tammany Hall ring and, in 1868, played a role in a scandal involving Jay Gould of the Erie Railroad, whom Cardozo had helped evade vast personal debts. The incident led directly to the creation of the Association of the Bar of the City of New York and to Cardozo’s resignation from the bench in 1872. He continued to practice law until his death.

In 1854, Cardozo married Rebecca Washington Nathan, daughter of Sarah Mendes Seixas and Isaac Mendes Seixas Nathan. They had seven children, including Albert Cardozo, Jr. and future Supreme Court justice Benjamin Cardozo. The family belonged to Congregation Shearith Israel, where Albert served as vice president and trustee.