If you don't have any specific requirements (i.e. headshots or sweeping landscapes), I'd start with the 23mm. If I had to choose one lens, it would be 23mm. The aperture will be really handy in low light, and eventually you can pair it with the upcoming 50mm f/2.
I have the XF18-55 which I bought recently. The camera (XT10) came with the XC16-50. I bought the XF because it was more expensive (I know, easily led!) and it had a metal mount and it was an XF rather than an XC ( I know, easily led). I wish I hadn't. The XC is just as good as the XF, I can't tell the difference in the images with each. I shouldn't have wasted my money on the XF. I would recommend the XC16-50.
Personally, I'd recommend the 18-55 over the 16-50, especially as an only lens.
I got the 16-50 for my wife to use with my X-E2, and the build is like a Canon Rebel kit lens. The autofocus gets really slow in low light, and the aperture is pretty limiting.
Having said that, I'd prefer the 16-50 + 23 over the 18-55 alone.