Electrical Engineering Seminars

AI-Enhanced Vision: Seeing the Invisible

George Barbastathis, Professor of Mechanical Engineering, MIT

Sep 11, 2020


Register here for the EE Seminar Series (via Zoom)

If you point your camera to a scene, and the camera registersnothing—does it mean that nothing was really there? Hardly! The camera pixelsmeasure “raw” light intensity where the encoded information often is muchricher than a human observer could tell just by looking at the pixels on ascreen. Which algorithms, then, should one apply to decode the raw intensityand reveal the hidden scene?  

In this seminar, I will describe how to use Deep NeuralNetworks (DNNs), a form of Machine Learning (ML) algorithm, to perform thisdecoding. During the training stage of the DNN, physically generated objectsare used to produce the encoded raw intensities. From these pairs of objectsand raw intensities the DNN learns the association between the scenes and theirencoded representations. After training, given a new scene, the DNN decodes itcorrectly to produce a final reconstructed image that is meaningful to a humanobserver.

With my research group, we applied this approach to threechallenging instances of invisibility: transparent objects, also known as“phase objects,” whose raw intensities are highly rippled diffraction patterns;phase objects that are also very dark, i.e. the diffraction patterns arealso highly attenuated; and objects hidden behind or surrounded by diffusers,e.g. frosted glass or multiple layers of glass patterned with sharplight-scattering features.

It is important to emphasize that in our work ML is not usedin the traditional way to interpret the scenes; rather, it is used to forminterpretable representations of scenes in situations where traditional MLwould be helpless due to physical limitations in the optics. The cooperation ofML with physical models proved to be very powerful in this work and, beyond, iscertain to impact many fundamental and applied aspects of physical and lifesciences and engineering.

Speaker Bio

George Barbastathis received the Diploma in Electrical and Computer Engineering in 1993from the National Technical University of Athens (Πολυτεχνείο) and the MSc and PhD degrees in Electrical Engineering in 1994 and1997, respectively, from the California Institute of Technology (Caltech.)After post-doctoral work at the University of Illinois at Urbana-Champaign, hejoined the faculty at MIT in 1999, where he is now Professor of MechanicalEngineering. He has worked or held visiting appointments at Harvard University,the Singapore-MIT Alliance for Research and Technology (SMART) Centre, theNational University of Singapore, and the University of Michigan – ShanghaiJiao Tong University Joint Institute (密西根交大學院) in Shanghai, People’s Republic of China. His research interests are inmachine learning and optimization for computational imaging and inverseproblems; and optical system design, including artificial optical materials andinterfaces. He is member of the Society for Photo Instrumentation Engineering(SPIE), the Institute of Electrical and Electronics Engineering (IEEE), and theAmerican Society of Mechanical Engineers (ASME). In 2010 he was elected Fellowof the Optical Society of America (OSA) and in 2015 he was a recipient ofChina’s Top Foreign Scholar (“One Thousand Scholar”) Award. Website: 


Demba Ba


Jessica Brenn