Hello, @Sjonsnaps hopefully You've figured out the topic on Your own by now but here's how I always did it on Fujifilm and on X-T5 currently:
Enable AF+MF
Disengage shutter AF in AF-S
Set DOF scale for MF to film (rather than pixel)
From now on You operate in AF-S only in a Back-Button-Focusing manner. When You want to set focus distance, half press the shutter and turn the ring - the distance will stay the same, no matter what and without accidental change, unless You half press the shutter and turn the ring simultaniously again . This works in A more no problem.
Hope this helps!