Tilbury Fort / Coalhouse Fort

Anyone tried to fly near or around Tilbury/Coalhouse Fort? On Sunday took a drive to this location and upon reaching the car park at Coalhouse Fort, an on their sign at the entrance it does say no drones. So we left the area and went over to Tilbury Fort. They don’t have a sign tat entwined anything about drone restrictions, however just to be sure we enquired at the ticket office at the entrance and we were told that flying drones is not permitted on their property, showed me some printed document which although not an official headed document from CAA, suggests that we have to apply for permission etc.
I’d be interested to hear if anyone managed to fly their drones there and whether they took off and landed outside of the fort premises. Would it be wrong for us to fly the drone outside of their premises and without flying directly over it, but circling around it above 50m, would we break any rules or regulations? I want to fly responsibly, keep within the CAA regulations.