Hi Jim,
I myself had the X-E2 and then, X-E2S. II have a X-E3 for about 6 months now. It is a wonderful deal.
What can I say : The joystick is not perfect when you dive too much in in the menus. Sometimes I hit and miss. I use the Jpg to Raw in camera conversion menu, I cannot go as fast as with the d-pad. Nevertheless, the joystick works very well for choosing focus. I have no big issues with swipes, it works well for me.
May i add that the X-E3 is a totally different camera that its predecessors. Its more advanced in features and sensor. While the image quality, on the specs sheet, is better, the rendering of the image is different, i still find the X-E2s image quality wonderful.
The two negatives for the X-E2s, the skin smoothing on images at higher ISO and its video quality made want to upgrade, not the image quality or the resolution. The video quality on the X-E3 is okay but not as nice as the X-T2 slightly cropped version.
I much prefer the X-E1, X-E2, X-E2s's design to the X-E3.
I hope this help you ...