@Boydey I have flown at al times of the day but it generally depends on what / where I am. So when I flew in a local public park recently I deliberately went at a time when schools were still in session and most folks are in work.
Because of my passion for landscapes (and Long exposures in particular) I am almost always found near the seaside or generally in places where the general public need to put in some effort to get too - not always but it helps. Then as you suggest there is ‘the time of day’. I do fly in the morning but predominantly it’s evening and around sunset time.
For flying, just follow the drone code and I’ll add ensure you take off from public property. If folks approach explain this and explain what / why you’re doing what you do. My experience is that nobody has yet “had a go” though I do expect it.
Being 6ft, 15 stone and with a #1 haircut it can put folks off approaching me though Just remember - none of these organisations own the airspace so they can only ask you not to fly over and not to fly from off of their land. But they can’t stop you if you take off and land from public land. The beach, a promontory, footpath, end of a cliff walk, etc… all good places IMO.