As @isazi said even if you doesn’t notice that it’s quite likely that this happens because of the crown contact with something. It happens for me all the time during winter if crown touches the cuffs of my jacket during cross-country skiing.
What I would suggest:
Select zoom level you like on map screen. Switch to non-map screen - open control panel by long click on bottom button - enable lock screen. Then switch back to map screen. Now zoom is locked and won’t change. Personally I solved the problem by using long strap and wearing watch over-jacket