In the 1871 census Thomas Potten (24, gardener) was living in the Gardener's cottage of Caldecot House, with his wife, Ellen, and young family. Henry (3) was born in Bushey and Ann (2) was born at Aldenham. Walter Drummond (40, Independent Income) of Caldecot House was there with a larger and older family. Harry (3) had been born in Putney, while  Mary (1) had been born in Aldenham. The children's births suggest that Walter moved to Caldecott House in about 1868 and immediately took on Thomas as his gardener.

A news item about a damaged fence in the Watford Observer of 4th January 1868, shows that Walter Drummond owned property in Aldenham at the end of 1867, while the Bucks Herald of 11th September 1869, announced the birth of a daughter to the wife of Walter Drummond, Esq., of Caldecote House, Bushey Heath.

On the 9th November 1872 the Hertford Mercury (see cutting)  reported that Anstole Rowbottom had been found guilty of searching for rabbits on Walter Drummond's land and Thomas Potten, Drummond's employee, was one of the witnesses.

Walter Drummond was still at the house in 1874 as the Caldecott House team won a cricket match .

See CRICKET: Caldecott House v. Stanmore

In November 1876 Acting Sub-Lieutenant Eric Morton Drummond, second son of Walter Drummond of Caldecott House, Bushey Heath, died aged 21.

By the time of the 1881 census Thomas Potten was a gardener living in Watford, and the ages and place of birth of the younger children suggest that they moved to Bushey in 1876/7. It would seem that Thomas changed jobs when the Drummond family left Caldecott House early in 1877.

The 1878 Post Office directory lists Charles Edward Jones living at Caldecote House and was still there in 1886, but not 1890.

Caldecote House is listed as unoccupied at the time of the 1891 census (not even a resident caretaker).

On 25th May, 1895, Noel George Keble, the infant son of George Clarke & Ann Catherine Keble, died at Caldecote House, Bushey Heath.. (Gloucester Journal, 1st June, 1895)

In 1901 Theodor A Vasmer (50, Export merchant, born Germany), his wife Mary A, and son George C Vasmer, plus servants, were at Caldecote House. Theodor died in 1909.

In 1911 Mary Agnes Vasmer, a 57 year old widow, was living in Caldecote with her son George Charles Vasmer (29, single). In 1912, 1922 & 1926 a Mrs, Vasmer was living at Caldecote House - she died in 1929.

George Vasmer was there in 1929, 1933 & 1937. He is also listed, with family, in the 1939 register. George died in 1940.