Right, so I found a weird but arguably very useful workaround: when you're in video mode, put your camera into AF-C (Continuous) mode, and engage the clutch whenever necessary to pull focus - disengage to leave focus on a certain area. Maybe my lens is already very used but even just a slight push upwards on the clutch (i.e. without fully engaging it) the AF mechanism kicks in, pulls focus, and as soon as I release the slight push, it disengages the AF
It's kinda fun to use now, not gonna lie... However, the AF on Fuji isn't the best to begin with, so having an option to still manually focus would be very much appreciated; but at least there is a workaround! Semi-manual AF lol