Screen Shot 2016-05-07 at 11.38.52

Chihuahua-VR: Advanced Media Processing

This is continuation of “Software Stack Introduction” and part of  Chihuahua-VR platform (see also VR tags).

Full Media Pipeline

I am focused on 3 groups of problems: capturing, processing and media/info distribution. The workflow is like follows:

  • the first group is mostly responsible on handling video frames from image sensor and placing the content into Host or CUDA memory
  • the second group is responsible for multiple media actions: transformation, converting, analytics, learning and producing new content
  • the third group is mostly responsible for media and info distribution over network, media interfaces and system services

The second group is a crucial and full Chihuahua-VR custom business logic of the pipeline. The basic actions are media transformation and converting. The advanced actions are learning, analytics and new content producing. The main goal is to place the incoming frames into CUDA memory and run custom processes on it.

Each group is fully configurable and extensible with custom plugins and libraries on every level. This custom implementation is based on CUDA with libraries: NPP, cuDNN and OpenGL, VisionWorks.

Chihuahua-VR media processing

Media Capturing and Processing

There are multiple ways to capture and process media. I am focused on USB and CSI media transport between CMOS Image Sensor and internal Host memory. In currently developed platform I do try to pipe the image via double ISP. First step of ISP is done in the camera module and the second is done on the TX1. I would consider to extend (or replace) the model with third ISP which would be the custom FPGA implementation, placed between camera module and TX1. FPGA/ISP should support MIPI/CSI-2 standard.

I consider 3 options of media capturing up to CUDA memory…

ISP support


Multiple parts of my blogpost have its source in remote articles, blogposts and wiki for which I have no rights. I am not able to link all external sources to my blogpost. I would like to say thank you to everyone who shares the knowledge publicly. If you think I have illegally used any of your thoughts, products, patents please let me know and I will remove related information asap.