James Lucas was the son of a very rich Liverpool businessman and land-owner, who owned large sugar plantations in the West Indies and - until emancipation - many slaves, James was born in 1811 and when he was about ten years old the family came to settle on the family estate at Great Wymondley, near Stevenage, Elmwood House, the family house, was a moderate sized country house at the hamlet of Redcoat's Green between Stevenage .and Hitchin. James Lucas was always a moody and eccentric boy and grew more so as he grew up: he alarmed the countryside by wildly riding about, tied by a cord to an old fashioned saddle, his long hair flowing behind. In 1849 James' mother, to whom he was devoted, died and James became rather more than eccentric. For 13 weeks he refused to have her buried but sat constantly by the coffin until the magistrates ordered burial.

James Lucas had inherited Elmwood House, as well as a large fortune: his brother left at once, all the servants were dismissed and  James became a complete recluse. All the rooms of the house were shut up and James lived in the kitchen, which contained nothing but one table and one chair. He wore only a blanket but never allowed the fire to go out: as the ashes accumulated, he scraped them back with his hand and they gradually filled the whole room. In this room James lived, black with soot and filth, for he slept on the ashes and never washed, for 25 years.

James' terror of his relatives and others made him bar all the windows and doors with logs and iron bars. He lived mainly on bread and milk, the milk can padlocked while it was brought from the farm, the bread chosen from a number of loaves.

Towards the latter part of his life he employed two watchmen.As his fame spread, Mad James Lucas received an increasing number of visitors to whom he talked for hours through the barred window. He was a highly intelligent man and had an excellent memory. His favourite visitors, however, were tramps and children. Tramps he would closely question and if they could give a convincing account of themselves he would give Protestants a penny and Catholics two pence, but if he caught them lying he swore at them and sent them away with no money or gin. He was happiest talking to children and used to distribute pennies, buns and sweets to dozens of children on Good Friday and Christmas Day.

In 1861 Charles Dickens, who frequently stayed with Bulwer Lytton at Knebworth House a few miles to the south, paid a visit to James Lucas (although the hermit later denied he ever came). Dickens wrote a semi-fictional account of this visit which was published under the title 'Tom Tiddler's Ground': the description of 'Mr, Mopes' the hermit, a morbid and misanthropical man, secluding himself for the sake of notoriety, probably helped further to embitter James Lucas.