The 35 f/2 should be reasonably quiet.
All Fujifilm lenses faster than f/2, and that includes your 23mm and 56mm, have to move a lot of glass to change focus. That requires powerful AF motors which are slow and noisy.
The quietest lenses are those with linear motors, which you can easily recognize by the LM in their official designation. At the moment, most of those are zooms. The XF90mmF2 R LM WR is the only LM prime I know of, although it does require four linear motors to shove its focusing glass around.
