Yup, that was what I thought. If you mainly use the viewfinder, then leave that setting turned off. It is intended (when turned on) for those that use the touch screen for viewing and choosing the focus point.
What happened to me was the touch screen was on and if my nose touched the screen whilst using the viewfinder, I would move the focus point. It was so annoying I have the touch screen turned off completely
X100f stuck in A and P mode!
I just got my x100f back from servicing and even when I move the aperture ring and shutter dial away from A - it is still in A mode- if I move the aperture ring and shutter dial to A- I am in P mode.
I want M mode- I am pulling my hair out. Am I missing some step in settings or is my camera damaged from servicing?
By chance do you have the settings for touch screen focus turned on? If so, then every time you look through the view finder, the focus point will bounce around like you are describing. Make certain it is turned off in the menus. That setting is more intended to be used with the big lcd screen.