I have just added this to the map of places to fly your drone at Drone Scene:
Land owner permission not required.
The originator declared that this location was not inside a Flight Restriction Zone at the time of being flown on 02/04/2021. It remains the responsibility of any pilot to check for any changes before flying at the same location.
I so want to do this flight but Hadleigh Castle is National Trust who prohibit drones so I think you are maybe flying illegally 
I once slept there on Halloween many years ago…
I was there a little while ago with another dronie flying a Mavic Air Pro. It’s actually owned by English Heritage who have a rather more relaxed policy on drone flying on their land. Got asked a few perfectly normal and polite questions by a number of people. Just do it mate, you’ll be fine but I would suggest picking a time when it’s not going to be busy. Early morning perhaps.
