Thanks everyone for your input, and sorry it took me a while to respond!
At least for now, I am tempted to try a few lenses every time I'm out and shooting, since I'm new to the interchangeable lens camera field.
For now, I have decided to use the following method, which is probably not the most efficient way to do it but I think it is rather safe:
1) Take and hold the camera's body cap and the lens's rear cap in one hand.
2) Press the release button and turn the lens. Bring the camera's cap close, and when you remove the lens, cover the sensor with the camera's cap simultaneously.
3) Hold the removed lens's back towards the ground while removing it, and cover it with its rear cap. Place it inside your bag, etc.
4) Take the new lens with its rear part facing downwards, remove its rear cap and bring it close to the camera body. Then remove the camera's cap and immediately place the lens onto the camera.
5) Store your caps properly.