“POIs” are a Litchi Mission construct, and this information isn’t uploaded to the drone as a result of the limitations of the DJI API that has to be used.
The DJI API doesn’t know what a POI is - it only knows WPs and the camera orientation at each and can extrapolate angles between WPs uniformly (on a distance/time basis) between each.
This post explains the extreme situation when flying in a straight line past a POI - and the same thing happens on a curved or polygonal track to a greater/lesser extent: Some notes on using Litchi - #2 by OzoneVibe
You are probably getting the issues that this causes.
The “curved” waypoints used to orbit have to create an accurate circular track for the POI to be viewed as accurately in the centre at all times by the footage.
If the track is 50p-piece shaped, for example, then the POI won’t remain central to the line of the camera - in pan or in pitch.