I assume you are add/removing the filter with the Mini 2 powered off?
Gimbals are very sensitive to weight, the slightest imbalance can cause this behaviour. Some users have even resorted to adding a bit of sticky tape behind the camera as a counterweight.
More issues regarding Mini 2 gimbal and 3rd party filters here: