“Lensless” imaging with advanced machine learning for next-generation image sensing solutions

“Lensless” imaging with advanced machine learning for next-generation image sensing solutions

A diagram of how the lensless imaging process works, from light collection to signal coding to post-processing with computer algorithms. Credit: Xiuxi Pan from Tokyo Tech

A camera generally requires a lens system to capture a focused image, and the lens camera has been the dominant imaging solution for centuries. A lens camera requires a complex lens system to achieve high quality, bright, aberration-free imaging. The last few decades have seen an increase in demand for smaller, lighter and cheaper cameras. There is a clear need for next-generation cameras with high functionality that are compact enough to be installed anywhere. However, miniaturization of the lens camera is limited by the lens system and the focusing distance required by refractive lenses.

Recent advances in computer technology can simplify the lens system by substituting computing for certain parts of the optical system. The entire lens can be dropped through the use of image reconstruction computing, allowing for a lensless, ultra-thin, lightweight, and inexpensive camera. The lensless camera has recently gained popularity. But so far, the image reconstruction technique has not been established, resulting in inadequate image quality and tedious computation time for the lensless camera.

Recently, researchers have developed a new image reconstruction method that improves computation time and provides high quality images. Describing the initial motivation behind the research, a core member of the research team, Professor Masahiro Yamaguchi of Tokyo Tech, says, “Without the limitations of a lens, the lensless camera could be ultra-miniature, which which could enable new applications that are beyond our imagination.” Their work has been published in Optical letters.

“Lensless” imaging with advanced machine learning for next-generation image sensing solutions

Vision Transformer (ViT) is a state-of-the-art machine learning technique, which is better for reasoning global characteristics due to its novel structure of multi-stage transformer blocks with overlapping “patchify” modules. This allows it to efficiently learn the features of the image in a hierarchical representation, which enables it to deal with the multiplexing property and avoid the limitations of conventional CNN-based deep learning, thus enabling better reconstruction. of picture. Credit: Xiuxi Pan from Tokyo Tech

The typical optical hardware of the lensless camera simply consists of a thin mask and an image sensor. The image is then reconstructed using a mathematical algorithm. The mask and sensor can be fabricated together in established semiconductor fabrication processes for future production. The mask optically encodes incident light and projects patterns onto the sensor. Although the cast patterns are completely uninterpretable to the human eye, they can be decoded with explicit knowledge of the optical system.

However, the decoding process, based on image reconstruction technology, remains difficult. Traditional model-based decoding methods approximate the physical process of lensless optics and reconstruct the image by solving a “convex” optimization problem. This means that the result of the reconstruction is sensitive to imperfect approximations of the physical model. Moreover, the calculation necessary to solve the optimization problem is time-consuming because it requires an iterative calculation. Deep learning could help avoid the limitations of model-based decoding, as it can learn the model and decode the image through a non-iterative direct process instead. However, existing deep learning methods for lensless imaging, which use convolutional neural network (CNN), cannot produce high quality images. They are inefficient because CNN processes the image based on the relationships of neighboring “local” pixels, whereas lensless optics transform local scene information into “global” information superimposed on all image sensor pixels, via a property called “multiplexing”. “

“Lensless” imaging with advanced machine learning for next-generation image sensing solutions

The lensless camera consists of a mask and an image sensor with a separation distance of 2.5 mm. The mask is made by deposition of chromium in a synthetic silica plate with an opening size of 40×40 µm. Credit: Xiuxi Pan from Tokyo Tech

The Tokyo Tech research team is studying this property of multiplexing and has now proposed a new machine learning algorithm dedicated to image reconstruction. The proposed algorithm is based on an advanced machine learning technique called Vision Transformer (ViT), which is better for global feature reasoning. The novelty of the algorithm lies in the structure of multi-stage transformer blocks with overlapping “patchify” modules. This allows it to efficiently learn the features of the image in a hierarchical representation. Therefore, the proposed method can deal well with the multiplexing property and avoid the limitations of conventional CNN-based deep learning, allowing better image reconstruction.

While classical model-based methods require long computation times for iterative processing, the proposed method is faster because direct reconstruction is possible with an iteration-free processing algorithm designed by machine learning. The influence of model approximation errors is also greatly reduced as the machine learning system learns the physical model. Moreover, the proposed ViT-based method uses global features in the image and is suitable for processing molded models over a large area on the image sensorwhile conventional decoding methods based on machine learning mainly learn local relationships by CNN.

“Lensless” imaging with advanced machine learning for next-generation image sensing solutions

The targets are the images displayed on an LCD screen (two columns from the left) and the objects in nature (two columns from the right; doll waving cat and teddy bear), respectively. The first row shows the ground truth images displayed on the screen and the shooting scenes for objects in the wild. The second line shows the patterns captured on the sensor. The last three rows illustrate the images reconstructed by the proposed model-based and CNN-based methods, respectively. The proposed method produces the highest quality and visually appealing images. Credit: Xiuxi Pan from Tokyo Tech

In summary, the proposed method solves the limitations of conventional methods such as iterative processing based on image reconstruction and CNN-based machine learning with ViT architecture, enabling the acquisition of high quality images in a short computation time. The research team further performed optical experiments – as reported in their latest publication in – which suggest that the lensless camera with the proposed reconstruction method can produce high quality and visually appealing images while the speed post-processing calculation is high enough for real-time capture.

“We realize that miniaturization should not be the only advantage of the lensless camera. The lensless camera can be applied to imaging in invisible light, in which the use of a lens is impractical or even impossible. Additionally, the underlying dimensionality of the optical information captured by the lensless camera is greater than two, making single-shot 3D imaging and post-capture refocusing possible. the camera without a lens. The ultimate lens of a camera without a lens camera is miniature but powerful. We are excited to be at the forefront of this new direction for next-generation imaging and sensing solutions,” says the study’s lead author, Mr. Xiuxi Pan of Tokyo Tech, while talking about their future work.


Extension of infrared microspectroscopy with the Lucy-Richardson-Rosen computational reconstruction method


More information:
Xiuxi Pan et al, Transformer Image Reconstruction for Mask-Based Lensless Imaging, Optical letters (2022). DOI: 10.1364/OL.455378

Quote: “Lensless” Imaging Using Advanced Machine Learning for Next-Generation Image Sensing Solutions (2022, April 28) Retrieved April 28, 2022 from https://phys.org/news/2022-04- lensless-imaging-advanced-machine-image. html

This document is subject to copyright. Except for fair use for purposes of private study or research, no part may be reproduced without written permission. The content is provided for information only.

Leave a Reply

Your email address will not be published.