I know exactly what you mean...
The 35mm is so nice it has to be there.
23mm? mmmm tough call: I use that focal all the time on my X100s, so maybe yes?
On the wide side, 16 or 14mm? I love the wideness of the 14mm, but the latest test shots I saw from the 16mm are just great: very interesting to see the capacity for smooth bokeh with such a wide focal.
Then what to do about the 56mm? I didn't properly try it yet, but it seems a wonderful lens...
Final call, if only 3 primes can stay?
Jeez, you're killing me... 16mm, 35mm, 56mm