RoboMix²: Mixed Event-frame-based On-device Learning on Mixed Spiking-artificial Neural Networks for Robotics
About RoboMix²
RoboMix² builds on three advances. Event cameras — modeled after biological visual pathways — offer low motion blur, high dynamic range, and lower power than frame-based cameras, which complement them with dense, easier-to-process features. Spiking neural networks (SNNs), which mimic biological neurons, complement classical artificial neural networks (ANNs) with noise robustness and low-latency spike-driven computation — a natural fit for the asynchronous output of event cameras. Tiny machine learning (TinyML) algorithms address domain shift, the degradation that occurs when offline-trained models encounter real-world conditions, through continual on-device adaptation.
Deploying this combination on miniaturized platforms remains an open problem. Current state-of-the-art systems either exploit only a subset of these modalities or depend on devices consuming several Watts, exceeding the power budgets of miniaturized robots. RoboMix² targets the first generation of ultra-low power (ULP) neuromorphic devices for mixed event-frame sensing and mixed SNN-ANN inference, with µs-level resolution interfaces for simultaneous event and frame streams, energy-efficient hardware accelerators, and support for self-supervised continual fine-tuning on-device.
The project validates its approach on visual odometry across two complementary platforms: a kg-scale unmanned ground vehicle (UGV), to demonstrate general applicability, and a sub-50 g nano-unmanned aerial vehicle (nano-UAV), representing the tightest constraints in power envelope, payload, and form factor. The resulting Neuro Device architecture is designed to generalize beyond robotics, with potential applications in autonomous driving and edge-based scene analysis.
Project Partners
First in-person consortium meeting — IDSIA, Lugano, June 26, 2026
Latest News
Funded by the Swiss National Science Foundation (SNSF) under the Project Funding scheme · Grant Number 10004854