Having Fuji, Olympus and Sony, I would advise you to try an Olympus body: its IBIS technology is fantastic and would allow you to put that "extra-shake" behind your concerns. Or, if you really want to go Fuji, I guess you would be very happy with a 2nd-hand X-H1 (I had one and replaced it with an X-T3): it is heavier and the IBIS is also great.
But I'd check the Olympus before anything...