I have the 14mm, 23mm, and the 10-24mm........and I'm getting the new 23mm in January. Every lens works best at what it's designed for. Primes and zooms both have advantages and diadvantages, all of which I'm sure have been discussed above. The answer to your dilemma is not a life or death issue, it's what I'd call a 'first world' problem.
Depends on what you shoot, how often you shoot it, how important fine, exacting detail is fir you. And of course how much you're willing to invest monetarily to get that detail. Only you can answer those questions. But if it were up to me, I say to hell with the mortgage and college for the kids!