I highly recommend the Viltrox 56mm based on your list. I just got it and very pleased with people portraits. I think it's worth more than I paid for. You need to control for correct exposure and scene as it can be more prone to fringing more than fujifilm lens. I have the Fujifilm 50mm f2 which is very sharp with fast AF, but use it mostly for pet portraits. The 35 1.4 is wonderful too for wider portraits and general, shop around for used if $ is a factor. Good luck!
What kind of portraits do you take? Is WR necessary in your environments? Do you shoot a lot in low light- why do you need a fast lens? Back in the day my first DSLR was a Nikon d70 with the 18-70 lens for weddings and studio portraits, it served the clients very well, and took some of my best wedding shots with it. I still use that same lens when I *have* to shoot dslr. If you're happy with the look of 16-80, which is a very nice versatile lens, stick with it for awhile and gather data on what focal length you most use for portraits. All the lenses mentioned here are good it's really down to the ideal selection for your purpose.