GOT TO IT!!! It't actually the lens, I don't know why actually but it doesn't seem to work when attached properly... (with analog everything works well)
First, I just mounted it with adapter, no converter and (duh) the same thing happened so I again started messing with screw and aperture ring (it is possible to set it to "A" without converter) and such.
Apparently to make it work, you have to click the removing button and twist it a bit. So... its probably not the safest solution and clearly that's not the way lens should be mounted if its not "clicked" in place but for macro at home... well, it will work for me
edit: the f stop needs to be chacked if doing it - depending on level of unscrewing I can't go below f3.5 if went too far or f22 is not closed enough if loosened too little.
With just the adapter that shouldn't be much of a problem - it has lock, so you can unscrew the lens a bit (it's still in place) and have this second way lock on the adapter, so fairly/enough safe for me - I would (/will) give it a shot outside.