I think you should try both - usability/handling/personal preferences will matter more than IQ, IMHO.
I am a long time Nikon user (30y), with the D750 being my most recent camera, and have also developed a little Leica M system on the side - a luxury position I admit.
For the last few years I tended to use my M9 with 2-3 lenses for most of my pictures, and only use the Nikon for "action" photography. When the Fuji X system came out it looked like a good compromise and could potentially replace both my Nikon and Leica systems: nice selection of quality lenses, small size and weight, traditional controls I'm found of, convenience of AF...
After 2 years of testing the Fuji X system, I decided to resell it and keep my Nikon and Leica equipment. Image quality was not the issue and I find Fuji's lenses truly excellent (with perhaps the exception of the 18mm f/2 which, while good, is clearly not a match for the Nikon 28mm 1.8, not to mention the Summicron 28mm).
The main issues, to me, were:
my inability to totally "bind" with the EVF, in particular outside with strong sunlight where I often couldn't see much at all. I am a rangefinder lover so this may have biased my judgement;
the sometimes unreliable AF behaviour, both in single AF (as with many contrast-based & hybrid AF based cameras, there is a tendency to sometimes focus on background items while I find PD-only cameras are better at identifying foreground elements) and continuous AF (I find Nikon's "group" AF tracking combined with Lock-On particularly efficient in fast-paced sports).
But I've rarely been disappointed by the image quality coming from the Fujis