I have just added this to the map of places to fly your drone at Drone Scene:
Land owner permission requirements unknown.
Close to an area of High-Intensity Radio Transmissions to the north and the Joint Services Diving School (active military establishment) to the south.
I’ve flown here too @JackGolding 
Parking is easy, I used the parking right outside and TOAL was from the park immediately to the north which according to the Fareham council website is public land.
Good views of the castle from that location, although had to walk around to the North Side to get much of the water views in order to remain VLOS since the park by the car parking area is pretty limited by the high walls of the castle and the trees.
It was a busy day with several other drone operators present - but I had time to kill while my other half was inside the castle looking around.
Do be mindful of the Fleet lands FRZ immediately to the West however.
Video to follow in due course! (still editing my Gilkicker video).
