The estate was bought from the seventh Earl of Cork for £94,400 in 1776 by James Alexander (later first Earl of Caledon), an East India Company Nabob. The Earls of Cork and Orrery had only acquired the estate by marriage from the Hamilton family in 1738, but during the forty years of ownership, they had made it into a by-word for fashionable landscape design, complete with a lodge decorated with statues and Latin epigrams, a hermitage and a bone house. Despite this earlier work and the recency of their own connection with Caledon the Alexanders nevertheless invested heavily in the improvement of their houses and demesnes there.
Between 1806 and 1811 the first Earl of Caledon spent over £17,000 when he employed the celebrated English architect, John Nash, to extend and repair
His proposals were ambitious, and his first contract, dated February 1806, included "two domed pavilions, one at each end of the present house, advancing forward in the point of the same and enclosing between the two wings with columns forming a colonnade connecting the said wings...." as well as a new library and alterations to the earls private apartments. Three years later, in July 1809, Nash gave an estimate for the repairs which were required in the Earl of Cork's original part of the house, and these included new ceilings, windows and doors in the old library, drawing room, parlour and dining room.
All of this work was carried out while the Earl of Caledon was in South Africa, where he was Governor of the Cape Colony, and this was possibly one reason why the relationship between Nash and the earl's agent, John Pringle, who oversaw the work, became strained. The difficulties culminated in a disagreement between Nash and the earl, after the latters return from the Cape in 1811, over the architect's final account, and were exacerbated when the earl held Nash to account for the leaks which began to appear around the new domes. Nash's response was rather plaintive. In March 1812, he wrote:
"I could wish to take off the stucco from the domes in order to ascertain the points of leakage but it is a much greater object with me to relieve your mind of all anxiety on the subject. I am sure you are too liberal and candid to attribute blame to me, it was a misfortune I could not foresee and had no reason to expect but I wish to impress on your lordship a belief that I wish to remedy the evil"
The problems obviously continued, and four years later, in 1816, Nash was still involved in trying to remedy them. In January of that year he wrote:
"My remarks on the damage buildings receive from damp by being unoccupied, of course, cannot apply to those times they are actually occupied, when they are not, I will only remind your lordship that Caledon with all its charms, has not the climate of Italy, but that Providence considering fruitful soil a greater blessing than a dry house, has given you a horrid atmosphere."
Nash concluded by assuring the Earl that he would send his assistant, Bevan, to inspect the leaking roof as soon as possible.