Vitis AI-ML on the VCS³ 3D Kit with Prophesee Event Camera

A LogicTronix tutorial enabling Vitis AI/DPU on the VCS³ 3D Kit, along with Prophesee IMX636 event-based vision camera.

Our partner LogicTronix, an FPGA Design and Machine Learning Company, has been busy working with the VCS³ 3D Kit. Read on to see how they’ve demonstrated object detection and tracking using the kit.

Original Article by LogicTronix HERE

Credits: LogicTronix FPGA, Embedded, and Machine Learning Team

Story

This article demonstrates how to create an FPGA design for the 3D Kit, integrating the Prophesee IMX636 event-based vision sensor (aka, Neuromorphic or DVS sensor) and deploying a machine learning model to process event-based data in real time.

They are using 2022.2 tool version, VIVADO and Petalinux for this application development.

VIVADO Pipeline – Overview

VIVADO IP pipeline for – MIPI CSI2 Rx, DPU and MIPI DSI2 Tx hierarchy:

The VIVADO pipeline consists of IMX636 MIPI Pipeline for Prophesee event sensor data acquisition, DPU hierarchy for ML inference and Display Pipeline for showing results of EventML at VCS³ 3D kit.

They are using Vitis AI 3.0 and DPU B512 variant for ML inference. With these IP blocks in the design, below is the resource usage by the complete pipelines:

Petalinux Development

They developed the Petalinux project and built artefacts for the VCS³ kit, based on the template provided. VCS³ has three major approaches to booting Linux, i.e. JTAG, QSPI and eMMC.

They are booting BOOT.BIN, boot.scr and image.ub at one partition of eMMC and Rootfs at 2nd partition.

To acquire the Prophesee IMX636 event-based sensor data, they used the metavision SDK (openEB core) on the Petalinux image. And the camera access, some of the data handling is happening at Metavision SDK.

The same app, including metavision SDK, runs the DPU-based ML inference and displays the result on the display monitor, which came along with VCS³ 3D kit.

The machine learning model demonstrated above processes event-based data captured by the sensor in real time. With fewer than 400k parameters, the model is highly optimised for edge AI inference, enabling low-latency and power-efficient execution on the compact Sundance VCS³ MPSoC platform.

We want to thank the LogicTronix FPGA, Embedded, and Machine Learning team for demonstrating that even a small form-factor MPSoC can effectively run real-time event-based vision applications without requiring high-end computing hardware!

Original Article by LogicTronix HERE

Looking to purchase the VCS³ 3D Kit? Have a look at the product on the Sundance Store by clicking the button below:

VCS³ 3D Kit Store Page