I seriously doubt that what I am going to say will solve your problem (sorry), but its related so I thought I'd throw this in.
Overall I love the X100v (and also have a Sony (a/7/r/4). I also usually use single shot autofocus also smallest or, more often, next smallest box. My frustration with the focus is keeping the point of focus in the middle of the frame. I want to start each shot with the focus point being directly in the middle and then moving it when/if I deem necessary. Whether I use the touch screen to center it, or if I use the button (lever? I don't know what to call it) - give it a double click to center it - it moves but... it doesnt stay there. Its. just too easy to touch either the lever or the screen as one is shooting and then the next shot(s) are crap.
There is no way I've found to actually lock it in a way that stays locked in the center (on the other hand, my Sony does this very well). Its too easy to touch the screen or the lever accidentally. So, if I'm shooting quickly sometimes i miss the fact that it moved and the shot is out of focus because its focussing on one corner or another.
I assume you would have detected this so, apologies that this does not address your issue. But, if others reading this have had the same issue or same frustration, please weigh in
