I came from Nikon D3 to an X-T4. Yes it was a bit small (almost anything is after a D3) so I added a SmallRig grip. Dramatic improvement in how it felt in the hand. It also added slightly to the height making it all a bit more manageable. I then decided I wanted something very small and compact for street or discrete photography and bought an X-E2. Ergonomically its a bit of a disaster but the images are lovely and I'm happy to forgive it because of that. I can drop it in a pocket and snap off a couple of shots in a few seconds and put it back in a pocket nice and quickly.
Since then I've moved on to a GFX100s II and that is simply in a different league. Ergonomics are very good (especially if you are used to PASM dials and finger and thumbwheels for shutter and aperture. Buttons are all programmable and you can customise to your hearts content. The only thing is the joystick control is a bit random in how it operates - especially when pixel peeping and the touchscreen operation is inconsistent - it works with some things but not others. Fort example scrolling an image about or selecting focus points but doesn't work when in the menu system.
If you're having issues with the X-T20 try a grip and give it a little more time to get used to.
As for whats best for you it depends entirely on what you shoot. For example I'd never use my GFX for street photography or sports photography but conversely I wouldn't use the X-E" for landscapes or wildlife. The X-T4 is a jack of all trades and well suited to sports photography (very high frame rate and crop sensor) where the other two aren't. Horses for courses.
I recently went from Nikon Dx series -> Fuji X-T4 -> GFX100s II.
I've now had the GFX for almost 3 weeks and find it generally to be pretty intuitive and the overall design to be very good. It fits neatly in the hand and although there's a reasonable amount of heft its handling minimises it well.
All the buttons are programmable to put the functions you need close at hand and the customised menus are easy to use. The only thing I'm not sure about is the joystick operation and the inconsistent use of the touchscreen - it works for some things but not others.
All that being said the image quality is simply astonishing SOOC.
As a flexible high resolution system it works very well. You can get adapters to use non-Fuji lenses as if they are native (with some caveats about sensor coverage). For example smart adapters are available to use the GFX with Nikon AF-S or Canon EF-S lenses. Some will give full sensor coverage and some won't but even as a full frame 35mm camera its still 60MP.