Google researchers introduce Lightlab: A diffusion-based AI method for physically reasonable fine-grained light control in a single image

The lighting conditions in the image after capturing are challenging. Traditional methods rely on 3D graphics methods that reconstruct scene geometry and properties before using physical lighting models to simulate new lighting. Although these techniques provide clear control over the light source, restoring an accurate 3D model from a single image is still a problem, often resulting in unsatisfactory results. Modern diffusion-based image editing methods have become an alternative to using powerful statistical priors to bypass physical modeling requirements. However, these methods cannot accurately control parameters due to their inherent randomness and dependence on text conditions.

Generating image editing methods have been applied to various re-tasks and the results are mixed. Portrait-focused approaches often use light-stage data to supervise the generative model, while object reprocessing methods may fine-tune the diffusion model using synthetic datasets adjusted on the environment map. Some approaches assume that outdoor scenes (such as the sun) adopt a single primary light source, while indoor scenes present more complex multi-resilient challenges. Various methods solve these problems, including inverse rendering networks and methods that manipulate Stylegan’s latent space. Flash Photography Research shows progress in multi-refresh editing by using the technology of Flash/No-Flash pairs to unzip and manipulate scene lighting agents.

Researchers at Google, Tel Aviv University, Richmann University and Hebrew University in Jerusalem have proposed LightLab, a diffusion-based approach that allows explicit parameter control of light sources in images. It targets two basic characteristics of light source, intensity and color. LightLab provides control over ambient lighting and tone mapping effects, creating a comprehensive set of editing tools that allow users to manipulate the overall look and feel of an image through lighting adjustments. This method shows the effectiveness of indoor images containing visible light sources, although other results show hope for outdoor scenes and outdoor examples. Comparative analysis confirms that LightLAB is pioneering in providing high-quality, precise control of visible local light sources.

LightLab uses a pair of images to implicitly model the light changes controlled by implicitly in image space and then trains a dedicated diffusion model. Data collection combines real photos with synthetic renderings. The photography dataset consists of 600 original image pairs captured using mobile devices on a tripod, each showing the same scene, with only one visible light source turned on or off. Automatic exposure settings and post-capture calibration ensure proper contact. Augment this series using physics-based rendering in a blender from a comprehensive set of images rendered in 20 indoor 3D scenes created by the artist. The synthetic pipeline randomly samples camera browsing around the target object, and the program assigns light source parameters including intensity, color temperature, area, area, and tapered angle.

Comparative analysis showed that weighted mixtures using real capture and synthetic rendering achieved the best results in all settings. Quantitative improvements from adding synthetic data to real capture are relatively small, at only 2.2% in PSNR, probably because details in the low-frequency image range in these metrics mask obvious local illumination changes. Qualitative comparisons of the evaluation datasets show that LightLab is superior to competitive methods such as Omnigen, RGB↔X, Scribblelight, and IC-Light. These alternatives often introduce unnecessary lighting changes, color distortion, or geometric inconsistency. In contrast, LightLab provides faithful control of the target light source while producing physically reasonable lighting effects throughout the scene.

In short, the researchers introduced LightLab, an advancement in image manipulation based on diffusion. Using the principle of light linearity and synthesized 3D data, the researchers created high-quality paired images that implicitly simulate complex lighting changes. Despite its advantages, Lightlab faces limitations of dataset bias, especially with regard to light source types. This can be solved by integrating with an unpaired fine-tuning method. Furthermore, while a simple data capture process using consumer mobile devices for post-capture exposure calibration facilitates easier dataset collection, it prevents accurate re-acquisition of absolute physical units, suggesting that further refinement can be carried out in future iterations.

View paper and project pages. All credits for this study are to the researchers on the project. Also, please stay tuned for us twitter And don’t forget to join us 90K+ ml reddit.

Sajjad Ansari is a final year undergraduate student from IIT Kharagpur. As a technology enthusiast, he delves into the practical application of AI, focusing on understanding AI technology and its real-world impact. He aims to express complex AI concepts in a clear and easy way.