I have just added this to the map of places to fly your drone at Drone Scene:
https://www.youtube.com/watch?v=7GjgtGODGCo
Land owner permission not required.
The originator declared that this location was not inside a Flight Restriction Zone at the time of being flown on 02/04/2021. It remains the responsibility of any pilot to check for any changes before flying at the same location.
I so want to do this flight but Hadleigh Castle is National Trust who prohibit drones so I think you are maybe flying illegally 
I once slept there on Halloween many years ago…
I was there a little while ago with another dronie flying a Mavic Air Pro. It’s actually owned by English Heritage who have a rather more relaxed policy on drone flying on their land. Got asked a few perfectly normal and polite questions by a number of people. Just do it mate, you’ll be fine but I would suggest picking a time when it’s not going to be busy. Early morning perhaps.
Please be aware: English Heritage does not permit drone flying from or over sites in our care, except by contractors or partners undertaking flights for a specific purpose, who satisfy stringent CAA criteria, have the correct insurances and permissions, and are operating under controlled conditions. Please see our drone filming guidelines for more details, or email our Filming team. 
Who do they think they are, National Trust? 
Just starting out and it seems to be an absolute minefield as to what you can and can’t do. It’s actually putting me off before i even have my first flight.
So can i fly at Hadleigh Castle legally or not?
Best option is to take off outside of the castle grounds.
You can overfly with no issues, as long as you follow the rules of flying.
However may be try your first flight in an area where you have good visibility - perhaps open countryside or a park etc.
Don’t be put off by land owners saying that you can’t fly over their land.
They can stop you from taking off on their land however.
Enjoy your flying 