AI-Enhanced Vision: Seeing the Invisible

Register here for the EE Seminar Series (via Zoom)

If you point your camera to a scene, and the camera registersnothing—does it mean that nothing was really there? Hardly! The camera pixelsmeasure “raw” light intensity where the encoded information often is muchricher than a human observer could tell just by looking at the pixels on ascreen. Which algorithms, then, should one apply to decode the raw intensityand reveal the hidden scene?  

In this seminar, I will describe how to use Deep NeuralNetworks (DNNs), a form of Machine Learning (ML) algorithm, to perform thisdecoding. During the training stage of the DNN, physically generated objectsare used to produce the encoded raw intensities. From these pairs of objectsand raw intensities the DNN learns the association between the scenes and theirencoded representations. After training, given a new scene, the DNN decodes itcorrectly to produce a final reconstructed image that is meaningful to a humanobserver.

With my research group, we applied this approach to threechallenging instances of invisibility: transparent objects, also known as“phase objects,” whose raw intensities are highly rippled diffraction patterns;phase objects that are also very dark, i.e. the diffraction patterns arealso highly attenuated; and objects hidden behind or surrounded by diffusers,e.g. frosted glass or multiple layers of glass patterned with sharplight-scattering features.

It is important to emphasize that in our work ML is not usedin the traditional way to interpret the scenes; rather, it is used to forminterpretable representations of scenes in situations where traditional MLwould be helpless due to physical limitations in the optics. The cooperation ofML with physical models proved to be very powerful in this work and, beyond, iscertain to impact many fundamental and applied aspects of physical and lifesciences and engineering.