bit2bit: 1-bit quanta video reconstruction via self-supervised photon prediction

Yehe Liu 1, 2, Alexander Krull 3,*, Hector Basevi 3, Aleš Leonardis 3, Michael Jenkins 1, 2, *
1 Case Western Reserve University, 2 OpsiClear LLC, 3 University of Birmingham,
NeurIPS 2024
* Joint Supervision
Paper Data Code arXiv

bit2bit reconstructs quality videos from 1-bit binary raw data collected from high-speed single photon avalanche diode (SPAD) cameras, at the original spatial-temporal resolution, using only the data itself. Raw SPAD binary data | Reconstructed smooth video

Abstract

Quanta image sensors, such as SPAD arrays, are an emerging sensor technology, producing 1-bit arrays representing photon detection events over exposures as short as a few nanoseconds. In practice, raw data are post-processed using heavy spatiotemporal binning to create more useful and interpretable images at the cost of degrading spatiotemporal resolution. In this work, we propose bit2bit, a new method for reconstructing high-quality image stacks at the original spatiotemporal resolution from sparse binary quanta image data. Inspired by recent work on Poisson denoising, we developed an algorithm that creates a dense image sequence from sparse binary photon data by predicting the photon arrival location probability distribution. However, due to the binary nature of the data, we show that the assumption of a Poisson distribution is inadequate. Instead, we model the process with a Bernoulli lattice process from the truncated Poisson. This leads to the proposal of a novel self-supervised solution based on a masked loss function. We evaluate our method using both simulated and real data. On simulated data from a conventional video, we achieve 34.35 mean PSNR with extremely photon-sparse binary input (less than 0.06 photons per pixel per frame). We also present a novel dataset containing a wide range of real SPAD high-speed videos under various challenging imaging conditions. The scenes cover strong/weak ambient light, strong motion, ultra-fast events, etc., which will be made available to the community, on which we demonstrate the promise of our approach. Both reconstruction quality and throughput substantially surpass the state-of-the-art methods (e.g., Quanta Burst Photography (QBP)). Our approach significantly enhances the visualization and usability of the data, enabling the application of existing analysis techniques.

Denoised Mandrill Image
Noisy Mandrill Image
Single frame of input| Reconstructed output

Video summary

remember to change the display quality to 1080p

Introduction


Single-photon avalanche diode (SPAD)

single photon avalanche diode (SPAD) is a highly sensitive photodetector that operates above its breakdown voltage, allowing it to detect individual photons by triggering a self-sustaining avalanche current when a single photon hits the device. SPADs have evolved significantly since their early development, particularly with CMOS implementation in the early 2000s, and are now used in various applications including LIDAR, PET scanning, fluorescence lifetime microscopy, and quantum key distribution, with recent advances achieving higher fill factors, lower noise, and integration into arrays with over 1 million pixels.


Quanta image generation process

SPAD process diagram

The process of quanta image generation is shown above. A continuous signal in spacetime generates photons according to a Poisson process, where the rate parameter λ is proportional to the signal intensity. These photons are then detected by the SPAD array with a certain quantum efficiency, producing binary measurements that indicate whether at least one photon was detected during each exposure period. For a more detailed discussion of this process, please refer to the very nice review by Dr. Stanley Chan.


Poisson/Bernoulli denoising

Generative accumulation of photons has shown promise in reconstructing high-quality images from extremely noisy photon counting data by using binomial sampling to create training pairs. However, the binary nature of SPAD data makes the Poisson distribution assumption inadequate. The process is modeled using a Bernoulli lattice process derived from truncated Poisson distribution. The splitting process creates complementary pixel dependencies, leading to significant reconstruction artifacts as shown in the figure below.

Masking process

We show that by masking the complementary pixels in the loss computation, we can eliminate the reconstruction artifacts and generalize the method for extremely photon sparse 1-bit data.

Key Contributions

  1. Self-supervised framework: Developing a self-supervised method that denoises photon-sparse binary quanta images, which effectively handles the binary nature of the data through a novel masking strategy.
  2. Higher dimensionality integration: Enhancing the performance of reconstruction by leveraging the temporal information that is readily available in most quanta image data.
  3. New sampling strategy: Providing insights on stochastic sampling, training, and regularization techniques to manage overfitting and improve the reconstruction quality.
  4. New data: Presenting a novel dataset with real and simulated 1-bit SPAD data to support further research and quantitative evaluation of quanta image processing.

Simulated quanta video

Ground truth | Simulated data | Reconstruction

Real SPAD data

Simulated 3D microscopy data (XYZ)

Real SPAD data from QBP (SIGGRAPH 2020)

BibTeX

@inproceedings{
          liu2024bitbit,
          title={bit2bit: 1-bit quanta video reconstruction via self-supervised photon prediction},
          author={Yehe Liu and Alexander Krull and Hector Basevi and Ales Leonardis and Michael W. Jenkins},
          booktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},
          year={2024},
          url={https://openreview.net/forum?id=HtlfNbyfOn}
          }