Develop a Novel Computational Photography Pipeline

Description

Why this is important?

For many consumers, the camera and its associated features are among the most important factors when choosing a new smartphone. Modern capabilities such as night-mode photography and extreme digital zoom increasingly rely on neural networks to enhance image quality, for example by improving the signal-to-noise ratio (SNR). A notable example is the Google Pixel 10 Pro’ pro-res zoom which uses on-device diffusion models and generative AI to reconstruct fine detail when zooming beyond the optical range.

With the introduction of Arm’s Scalable Matrix Extension (SME2) in commercial handsets, these advanced neural networks can now run efficiently on the CPU itself. This enables a wider range of devices to incorporate neural networks directly into their photography pipelines, while also promoting more portable software designs that are not tightly coupled to a specific Neural Processing Unit (NPU).

Project Summary

This challenge focuses on designing and implementing a novel computational photography pipeline on an Arm-based device. Your solution must implement one of the following example pipelines, or a closely related variant:

•	Night-mode imaging: Improve the signal-to-noise ratio (SNR) of photographs captured in low-light conditions.
•	Portrait-mode imaging: Apply realistic bokeh blur effects to background regions in portrait photographs.
•	Neural ray denoising: Use neural networks to denoise ray-based or rendering-related image data.
•	Generative AI–based pipelines: Leverage GenAI models, such as style-transfer or diffusion-based approaches, to enhance or transform images.

For example, your approach could involve fine-tuning an existing model, implementing a novel pipeline. The objective is to optimize your pipeline for both inference latency and perceived image quality and the technical criteria, of your project, on which you will be judged, are as follows:

Must use a quantized neural network, preferably less that 16-bits (i.e. quantized to less the FP16)
Preferably use the Executorch runtime, designed for running neural networks with the Arm Neural Technology with the ML Extensions for Vulkan backend
Algorithms should run on an Arm-based CPU with single-instruction multiple data (SIMD) such as SVE or SME or an Neural-Graphics enabled GPU (either simulation or physical handset once available)

Prerequisites

Experience with traditional image processing algorithms (e.g., convolution, filters).
Intermediate experience or willingness to learn about machine learning for graphics and photography (e.g., neural ray denoising).
Familiarity with build systems and associated tools such as CMake, docker.
Experience with object-orientated programming and scripting with langauges such as C++ and Python.
Some experience or willingness to learn about low-level programming and using APIs such as Vulkan.

Support Level

This project is designed to be self-serve but comes with opportunity of some community support from Arm Ambassadors, who are part of the Arm Developer program. If you are not already part of our program, click here to join. Additionally, if you would like to engage with this project and need assistance with access to hardware, please email us at Arm-Developer-Labs@arm.com.

Benefits

Standout project contributions to the community will earn digital badges. These badges can support CV or resumé building and demonstrate earned recognition.

To receive the benefits, you must show us your project through our online form. Please do not include any confidential information in your contribution. Additionally if you are affiliated with an academic institution, please ensure you have the right to share your material.

Attributions

By User:Omegatron - Created by User:Omegatron using the GIMP, CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?curid=1836856

Description

Prerequisites

Resources from Arm and our partners

Support Level

Benefits

Attributions