Everyone calls it normal, but I consider it to be a major design flaw. It either needs a mechanism to dampen the movement or some sort of mechanical lock.
I also carry a Canon MPE 65mm with Fringer adapter for hiking and backpacking, and may replace the 80mm with a Canon 100mm f/2.8 macro. I normally do not use AF for macrophotography to gain more precise focusing or for focus bracketing. The AF is useful when shooting bees, other flying insects and birds in flight. With the 80mm in my pack and a X-H1 with 100-400mm around my neck, I have had numerous occaisions where I was slowly approaching birds or other wildlife only to see them panic or get wary following the sound of the thud. Aside from this, the thud sound is just plain annoying and makes you wonder about the lifespan of the lens.
Fujifilm makes great glass with excellent mechanical build quality and ergonomics, but some of their design decisions (a dozen or so filter sizes across the lineup, inconsistent use of OIS across the lineup, numerous missing primes in their lineup, etc.) are frustrating and suggest a lack of engineering design guidelines to restrict poor decision making.