I myself had the X-E2 and then, X-E2S. II have a X-E3 for about 6 months now. It is a wonderful deal.
What can I say : The joystick is not perfect when you dive too much in in the menus. Sometimes I hit and miss. I use the Jpg to Raw in camera conversion menu, I cannot go as fast as with the d-pad. Nevertheless, the joystick works very well for choosing focus. I have no big issues with swipes, it works well for me.
May i add that the X-E3 is a totally different camera tha