He died in 1845, his wife, the baroness, predeceasing him in 1836.
Presently he elected to retire to Germany, and thence to England, where he married Margaret, daughter of Admiral George Keith Elphinstone, Lord Keith, and after the latter's death Baroness Keith in her own right.
At one end of the table, the old chamberlain was heard assuring an old baroness that he loved her passionately, at which she laughed; at the other could be heard the story of the misfortunes of some Mary Viktorovna or other.
By his marriage with Mary, daughter of Edward Wortley Montagu of Wortley, Yorkshire, who in 1761 was created Baroness Mount Stuart of Wortley, and through whom he became possessed of the enormous Wortley property, he had, besides six daughters, five sons, the eldest of whom, John, Lord Cardiff (1744-1814), succeeded him as 4th earl and was created a marquess in 1796.
On his death in 1891 his widow was created Baroness Macdonald of Earnscliffe.