An article is posted on the website of Historic England (formerly known as English Heritage) about the Lygon family’s association with Walmer Castle in Kent. The article is part of the website’s promotion of sites in the care of Historic England that have LGBTQ associations. Evelyn Waugh knew the Lygon family from their Worcestershire home at Madresfield Court. Walmer Castle was available to them from 1914 when Lord Beauchamp was made Lord Warden of the Cinque Ports which entitled him to reside in the castle. The Historic England report does not explain very well that the Lygons made only occasional useage of Walmer Castle. They travelled there as a family on an annual basis for a seaside holiday:
Lygon enjoyed the pomp and ceremony that came with his role. Dover was still the main entry point for visiting foreign dignitaries, and it was among his duties to dress in his finery and welcome them on behalf of the King. However, when the family were away he was rumoured to have thrown parties, to which he invited his high-class friends, along with local fishermen and youths.
According to Jane Mulvagh, the family had three other homes of their own to visit or live in as they saw fit:
Walmer was uncomfortable and “so cold in our quarters that the wind blew the carpets off the floors,” recalled Sibell [Lygon]. As the family’s private section of the house was small, the  children had to share just two rooms, which provoked squabbles and discontent. (Madresfield, 2008, pp. 254, 266)
But Lord Beauchamp also made private visits to Walmer Castle without the family and used it sometimes as a venue for homosexual gatherings, as noted in the Historic England article. The article is also accompanied by several interesting photographs taken of the family on their visits to Walmer Castle. It mentions the connection of the Lygons with the Flyte family as depicted by Evelyn Waugh in Brideshead Revisited. There is nothing to suggest, however, that Waugh ever accompanied them to Walmer Castle. It seems unlikely that he would have done so since his friendship dated from the period after Lord Beauchamp had been forced into foreign exile following his outing as an active homosexual (then a criminal offense) by his jealous and vindictive brother-in-law. Paula Byrne mentions that Waugh’s Oxford friend Robert Byron had accompanied the Lygons to Walmer, but his relationship with the family predated Waugh’s (Mad World, 2010, p. 88).
A related article in Huffington Post UK notes that February is LGBT history month:
The focus for this year’s LGBT History Month is the fiftieth anniversary of the decriminalisation of gay men in England and Wales. The Sexual Offences Act 1967 was a turning point, allowing the development of an organised gay rights movement. In the first half of the twentieth century there was a vibrant gay community in Britain. Evelyn Waugh’s Brideshead Revisited is full of homosexual overtones. Authorities turned a blind eye from time to time. Decriminalisation meant gay men no longer had to behave so furtively. It was not until 1980 that being gay was decriminalised in Scotland, then 1982 in Northern Ireland. Stonewall was founded in 1989.