Software to gain pixelshift-like superresolution with Fuji X cameras

[wilecoyote]

Hi all,

 

I am working on a program, called VerySharp, which is inspired by the pixel-shifting feature used by some modern cameras.

It aims to provide an algorithm that enhances photo resolution using multiple exposures. This is realized by extracting sub-pixel information from the little image shifts in a series of hand-held images, yielding a relolution gain of approximately factor two under good circumstances.

 

I am a Fuji X user and therefore development an testing of the program was mainly done using my X-T10. It seems like the X-Trans sensor works well with VerySharp.
Verysharp is open source and therefore completely free. Please keep in mind that I am developing this in my free time and it is still in a quite early development stage. Especially, as I am a Linux user, I am still experimenting regarding packagin VerySharp for Windows users. So far, the provided exe file seems to run well on Windows 8 and above if recent updates are installed. So have fun while giving it a try please feel free to share feedback and any bugs you encounter

 

https://wilecoyote2015.github.io/VerySharp/

[kimcarsons]

Is this any different than using align_image_stack on an image series and taking the mean?

[George_P]

Interesting... a while ago I read about the automated focus rail slides that some people build to take stacked photos (to achieve greater depth of field in macrophotography). They advance the camera by the means of an Arduino-controlled stepper motor turning a wormdrive.

 

Would it be an idea to use something like this to move the camera along the x and y axis (as opposed to the z axis used in macrophotography, assuming z is the movement towards/from the subject) to achieve manageable pixel shifts ?

[kimcarsons]

Interesting... a while ago I read about the automated focus rail slides that some people build to take stacked photos (to achieve greater depth of field in macrophotography). They advance the camera by the means of an Arduino-controlled stepper motor turning a wormdrive.

 

Would it be an idea to use something like this to move the camera along the x and y axis (as opposed to the z axis used in macrophotography, assuming z is the movement towards/from the subject) to achieve manageable pixel shifts ?

 

That would cost more and be harder to use than a Pentax or Olympus with pixel-shift built-in, so what would be the point?

[George_P]

That would cost more and be harder to use than a Pentax or Olympus with pixel-shift built-in, so what would be the point?

 

The entertainment factor would be much bigger and who knows what kind of ground-breaking discovery might be made along the way  ​ And one can teach one's kids a trick or two while at it. Btw, the cost of components for this sort of thing is next to nothing.

 

But you are right, of course. It was just an idea.

Edited November 15, 2016 by George_P

[wilecoyote]

Is this any different than using align_image_stack on an image series and taking the mean?

 

Yes. The point is that VerySharp upscales the images prior to alignment so that, under the assumption that the shifts between the images are represented by an even distribution on a subpixel level, the resulting image stack on the upscaled subpixel grid can be interpreted as a high resolution image convolved with the superposition of a box kernel with the size of the upscaling factor and the upscaling operator. Thus, proper deconvolution recovers that high resolution image to some extent, which is also performed by VerySharp.

 

Also, as far as I know, panotools use a feature-based approach for image alignment, which is the best solution for the purpose of panotools, whereas most probably it does not provide the accuracy needed here, which is the reason for using the slower iterative ECC algorithm provided by OpenCV.

 

 

Interesting... a while ago I read about the automated focus rail slides that some people build to take stacked photos (to achieve greater depth of field in macrophotography). They advance the camera by the means of an Arduino-controlled stepper motor turning a wormdrive.

 

Would it be an idea to use something like this to move the camera along the x and y axis (as opposed to the z axis used in macrophotography, assuming z is the movement towards/from the subject) to achieve manageable pixel shifts ?

 

That is an interesting idea

However, rotating the camera (shifting would not work due to perspectivic distortions) could not be done with the required precision using normal tools, and the projection of the lenses prevents a homogeneous shift of the image over the whole sensor area during rotation...

Doing sub-pixel alignments when shooting the images is not even necessary because doing random sampling (shooting handheld    ) and reconstructing the shifts with sub-pixel alignment does the job quite well. Of cause there is always room for improvement, but I doubt that the extensive effort would pay off in terms of resolution gain.

Edited November 15, 2016 by wilecoyote

[George_P]

That is an interesting idea

However, rotating the camera (shifting would not work due to perspectivic distortions) ...

 

May I ask, maybe you know, does the pixel-shift mechanism in Olympus rotate the sensor ? I thought it would be shifting the sensor. I am asking just out of curiosity, I know nothing about it. Another question that comes to my mind is if we are theoretically considering moving the camera to the order of the size of a pixel, if distortion plays a role at normal focal lengths. But you must have reason for mentioning that and you know the subject, as opposed to me, so I don't argue, I am just wondering.

 

To come back (just for fun) to my idiotic idea, to rotate the camera would possibly be easier than to move it along two axis and it could be built to make very small increments in the angle of rotation. But I say this just for amusement or maybe for some special purpose that is totally remote from normal photography, don't take this seriously.

 

On the other hand, your effort to use a set of random hand-held images to calculate an increase in resolution could be very practical, it is a stroke of genius if you ask me and I find it very interesting. Not because I think I need more pixels or because I think it would improve my photographic or artistic skills, but because as a technical idea and for people who do need more resolution for their work, I think it is a fascinating idea and effort on your part.

 

Compliments & Cheers.

 

Edit: For macro, rotate the subject ?    

Edited November 16, 2016 by George_P

[pipetman2]

First of all, thanks for making this available. I'm a big fan of free/open source software and this one will come in very handy.

Just downloaded it and tried to run it, but encountered a few snags. Initially, I got the entry point error, which went away after multiple OS updates and installing the latest C++ redistributable. But running the exe after that just leads to a cmd window quickly flashing by and disappearing. After many tries, I managed to get a screenshot:

Welcome, dear visitor! As registered member you'd see an image here…

Simply register for free here – We are always happy to welcome new members!

[wilecoyote]

Thank you both for your kind words!

 

 

 

May I ask, maybe you know, does the pixel-shift mechanism in Olympus rotate the sensor ? I thought it would be shifting the sensor. I am asking just out of curiosity, I know nothing about it. Another question that comes to my mind is if we are theoretically considering moving the camera to the order of the size of a pixel, if distortion plays a role at normal focal lengths. But you must have reason for mentioning that and you know the subject, as opposed to me, so I don't argue, I am just wondering.

 

I have used the wrong term, sorry In fact, you have to pan and tilt the camera, which corresponds to the shifting of the sensor in Olympus cameras.

 

 

First of all, thanks for making this available. I'm a big fan of free/open source software and this one will come in very handy.

Just downloaded it and tried to run it, but encountered a few snags. Initially, I got the entry point error, which went away after multiple OS updates and installing the latest C++ redistributable. But running the exe after that just leads to a cmd window quickly flashing by and disappearing. After many tries, I managed to get a screenshot:

Clipboard01.jpg

 

Thank you for that useful report! I will look into it as soon as I have access to a Windows machine.

[MirrorMirror]

Sorry mate. I'm sure your intentions are honourable but personally I wouldn't dream of downloading and installing software from someone's forum post. It's just too dodgy in my opinion. Each to their own though, I'm sure you'll get some good feedback from others.

[wilecoyote]

Hi,

As you can easily see on the link, this is not a "download from a forum post", but a project hosted on github, which one of the most popular hosting services for Open Source software and considerably more reliable than many portals providing bloatware-infected "freeware" . Also, as the source code is provided, you are free to check the software for malware by yourself if you feel inconfident with downloading binaries.

[mdm]

as the source code is provided, you are free to check the software for malware by yourself...

 

nice advice but wrong forum...

[MirrorMirror]

A host site called github doesn't exactly inspire me with confidence!

http://www.urbandictionary.com/define.php?term=Git

 

Seriously though, not for me.  I wouldn't know what to do or like to take the risk; software fully developed for my PC is for me

 

Good luck though....

Edited November 27, 2016 by MirrorMirror

[wilecoyote]

 

Seriously though, not for me.  I wouldn't know what to do or like to take the risk; software fully developed for my PC is for me

 

 

That is a very understandable view, of course

 

 

Good luck though....

 

 

Thank you!

[johant]

 

 

A host site called github doesn't exactly inspire me with confidence!

You are clearly not a software developer [emoji14]

[MirrorMirror]

It was my take on the word git that was the issue, and it was meant humorously.

 

I'm a user of software, certainly not a developer.

[johant]

It was my take on the word git that was the issue, and it was meant humorously.

 

I'm a user of software, certainly not a developer.

Ahh yes, but Github is an essential tool nowadays for software development teams. "Everyone" uses it.

 

@OP: thanks! I will certainly take a look!

[MirrorMirror]

As I said I'm not decrying the use of a hub, forum, testing bed, call it what you will to enable software developers to develop.

 

Even though personally I've never heard of it.

 

Its more the name that doesn't do it justice and gives me a sense of wariness. If cola was called shitcola I probably wouldn't buy a can. A hub called github makes me move on.

 

Think they should market themselves more professionally. Maybe its a cultural thing with the word git. It no doubt stands for something technical or maybe they just don't care. They haven't got my custom, and that's the point I was making.

[lauriebrett]

This sounds like a really exciting piece of software. I'm not at all programming literate and the install for Mac sounds daunting for me. If you make a easy to install version for Mac I would be happy to pay for it.

Good luck though, I hope this takes off.

[dlabrosse]

This sounds very interesting. I will wait however until a more friendly Mac installation exists.