e3_user

I upgraded, and love it.  I do with the viewfinder was larger, however it is still quite good.  Even when wearing my glasses I can see the full frame.  I turned off the touch screen, because I'm a left eye shooter... nose... screen... no good.  I still had enough customizable buttons for all the features I use most.  The ISO doesn't bother me because I used a button for that.
Auto focus speed is up, and photo quality is pretty amazing with the newish sensor.  I couldn't go back.  I love the small size and shape.  It's a beautiful camera, yet doesn't attract huge attention (and as an introvert that is a great asset).
Are the SLR Styled camera's more flexible?  probably.  Do I need that flexibility?  No.  The real question for you is, what do you use.  What do you need?

Love using the XE-3, however I disabled the touchscreen as it was truly annoying. In general I find touchscreens useless. I never use the little flashes built in flashes, so that isn't a big deal to me. In fact I taped down the flash on the XE-2 as I would hit the button from time to time and it would pop up, I was worried about snapping it off. The XE-3 focus is better and the buffer is larger (raw plus jpg)  than the XE-2. While I still like the 2 I find it has limitations that modern cameras should not have. Not sure what the complaints with EVF is for people like me with glasses, also a non issue for me.

I had the X-E2 and traded up to the X-E3 . The X-E3 comes with an external flash( included with the camera )which attaches to the flash shoe when needed and is MUCH better than the built in flash of the X-E2 , which was pretty much useless.
I also use glasses and I can't say that I noticed any difference when using the viewfinder . And I use it almost exclusively .
As far as other differences :  the new larger sensor and processor produce noticeably better pictures and the X-E3 has HDMI live view output ( which I really like because now I can plug in an external monitor for focusing when using my old manual lenses )
well, all the other improvements/differences are easy enough to get from the spec sheets ,so there is no point to list them here.
All in all, I liked the X-E2 but am very happy  that I made the upgrade .