The Wildlife act makes it a criminal offence to intentionally or recklessly damage/disturb whatever makes an SSSI special. Whether you know its an SSSI or not.
But following from what @DroneGeek said, if the special feature is a rock, or a flower, your drone is unlikely to be an issue. Birds are the most problematic and the things to watch out for.