I find that the camera's ISO3200 is acceptable, it doesn't bother me. Perhaps if you posted a sample of your ISO3200 we could tell you if it's similar to what we're experiencing?
I shoot street photography almost exclusively with my X100V. I shoot in raw format and process with Lightroom. I just use the built-in Fuji presets or I'll use Adobe's default presets.