Trying to track down gypsies can be difficult - but a clue is that they normally lived in caravans - and all people living in "mobile" accommodation would have been recorded as such on the census, as they were living somewhere which was not on the official census schedule. I therefore checked the censuses on Ancestry to see where Joseph White was living and there is nothing to suggest a "traveller" connection and I ahev not (at this stage) followed up this aspect of your question,

Between 1851 and 1901 Joseph never moved more than a few miles within Hertfordshire. In 1851 and 1861 he was living at Hooks Cross with his parents - his father being described as an agricultural labourer.  In 1871, 1881 and 1891 Joseph was living with his wife and family and was described as an agricultural labourer. In 1901 Joseph was in Benington and is described as "Horse Keeper on Farm" and the word "Ag" was added to indicate that this was counted for census purposes as an agricultural occupation. (If you have another look at the page of the census returns you will see that a large number of occupations have been qualified by such a later "clarification". Many people - when looking up things in the census - simply grab the lines with the name they were looking for and fail to look at what the neighbours were doing - as this can often provide important additional context and clues.)

It is important to realise that in the 19th century and earlier horses were everywhere. On an average farm the farmer would have a horse to ride round the farm (today it would be a quad bike) and possibly to go hunting. There would be a different horse to pull the trap that took the family to town or to church on Sundays. In addition there would be heavy horses to plough and pull the farm carts. Someone would need to look after the horses. On a large farm with a well-to-do farmer every member of the family might have their personal riding horse (or pony for the younger children), there would be a number of working horses, and possibly some brood mares with foals. In such a situation some of the more experienced agricultural labourers would be employed as  "horse keepers" working full time with the horses.

If you look at the adjacent pages on the 1901 census for Benington you will find quite a few people employed as "horse keepers" and even a "domestic coachman" who presumably worked at a nearby large house.