Bringing it all into focus

When designing microscopes, optical engineers have long faced a trade-off between speed, field of view and resolution. But improving one of these properties typically comes at the expense of another. Now, a UC Berkeley-led team of researchers has found a way to address this challenge, opening the door to new possibilities in the field of microscopy.

In a study published today in Nature Photonics, researchers designed a computational microscope capable of capturing micron-scale resolution across multi-centimeter areas at faster-than-video rates. Using an array of 48 camera sensors and computational imaging “tricks,” their new microscope was able to capture videos at a rate of 25.2 billion pixels/second — an unprecedented combination of highly detailed magnification, wide field of view and high-speed imaging.

“This is really a breakthrough in the field of computational microscopy,” said Laura Waller, professor of electrical engineering and computer sciences and the study’s principal investigator. “With our microscope, we were able to achieve 3-micron resolution across 5 square centimeters at 120 frames per second, which isn’t possible with traditional ways. It’s the largest space-bandwidth time product of any practical microscope we know of.”

She added, “Our approach could be used to image many live organisms simultaneously and to monitor samples over time.”

For many years, optical engineers have struggled with spatiotemporal throughput limits. This means that the microscope cannot simultaneously capture highly detailed spatial information, such as high resolution or a large field of view, and rapid temporal information, like a fast frame rate. These constraints make it difficult to capture fine details of highly dynamic samples, such as freely moving organisms.

To overcome this issue and achieve snapshot gigapixel-scale imaging, the researchers built a microscope that combines a camera sensor array, an engineered diffractive optical element and an optimization algorithm for processing.

Their novel microscope uses a prefabricated array of 48 disjoint sensors, which can fit into a single circuit board about the size of a credit card. When the microscope is in use, the array acts like a single, giant sensor, increasing the number of pixels it can capture and enabling greater image fidelity. But this design also creates a problem: there are gaps between the sensors in the array where light —and, therefore, data — isn’t captured, making it difficult to completely reconstruct the full image.

To fill these gaps, explained Waller, the researchers used compressed sensing “tricks.”

“Specifically, we fabricated a custom-designed phase mask — a glass plate that diffracts light in the microscope — so that light that would normally fall between the sensors is redirected onto them instead,” she said. “And using a computational algorithm, we were able to fill in the data gaps and reconstruct the image.”

The researchers then demonstrated the versatility of their microscope by imaging static and dynamic samples. For the static samples, they found that the global features of their reconstructions closely matched those of low-resolution traditional images, but their reconstructions achieved higher resolution. For the dynamic samples, they imaged dozens of freely moving C. elegans nematodes at high speed —120 frames per second — for 15 seconds with impressive results.

“The simultaneous high-resolution, wide-field-of-view, high-speed capabilities of our computational microscope allowed us to track the freely moving C. elegans, as well as some structures within them,” said lead author Kevin C. Zhou, formerly a postdoctoral researcher in Waller’s lab and now an assistant professor at the University of Michigan. “From the video reconstruction, for example, we were able to track individual worms and perform functional imaging of their rapid pharyngeal pumping, a part of their feeding behavior.”

Overview of the researchers’ large-scale compressive microscope. (a) The imaging system uses an engineered diffractive optical element to generate a distributed multi-spot point spread function (PSF) at the image plane. Images are captured by an array of 48 sensors arranged in a grid, with gaps between individual sensors. (b) The system forward model is a masked convolution with the PSF, which compressively encodes the parts of the image that would normally fall on the gap areas. A computational inverse problem then recovers the full-scale image. (Image courtesy of the researchers)

According to the researchers, their computational approach also eliminated the need for the laborious manual calibration that a microscope this large would normally require, saving valuable time and effort. “I think calibration-free capabilities like our method will prove to be one of the key ingredients for scaling up imaging systems in the future,” noted co-author Chaoying Gu, a Ph.D. student in the Department of Electrical Engineering and Computer Sciences.

Waller believes that this work could ultimately help advance the field of computational microscopy. “There’s a lot of progress being made, and still to be made, on very large-scale video microscopy,” she said. “Our study shows that computational imaging — the joint design of hardware and software — has a lot to offer in scaling up to large data acquisition.”

Researchers across multiple institutions and disciplines collaborated on this study. In addition to Waller, Zhou and Gu, co-authors include Muneki Ikeda and Saul Kato, UCSF; Tina M. Hayward and Rajesh Menon, University of Utah; Nicholas Antipa, UC San Diego; and Roarke Horstmeyer, Duke University and Ramona Optics.

This work was supported by the Office of Naval Research, the Air Force Office of Scientific Research, the National Institutes of Health and the Japan Society for Promotion of Science. Kevin C. Zhou was supported by the Schmidt Science Fellows, in partnership with the Rhodes Trust. Laura Waller is a Chan Zuckerberg Biohub SF investigator.