As I started shooting street with a Canon 7D equiped with a 35mm f1,4 Sigma EX, size didnt matter for me...than I switched to Fuji X and started to shoot the equivalent 35mm f1.4 and loved it.
With the x-e1 body I was much more stealthy. Than I switched to the 27mm pencake to take the stealthyness to the next level and I loved it much more.
For me, the 23mm never was an option for street photography in terms of focal length and that was the only problem I had with the X100S, which is a great camera, but in my oppinion it´s too wide.