Screen-Shot-2016-05-10-at-17.35.54-2

Art 3D/VR challenge – weeks 3, 4, 5 until 9

Previous week Next weeks coming soon…

Weekly diary

Week 3: FullHD processing on portable GPU vs CPU

Tegra X1 makes things very easy. For this project there is a requirement to upload a multiple RAW/4K videos at 30 fps and process them on multiple levels. For the version 1.0 the video must be stitched (stereoscopic mode), undistorted (lenses), computed for depth informations and processed by the keying algorithm.

I have verified that by applying OpenGL shader for each step it is fairly easy to accomplish a proper throughput of entire pipeline from RAW frames entrance up to encoders level…


CPU: Unfortunately, processing two streams of 4K@30 and applying all required steps was NOT possible with ARM-57/64bit processor.


Week 4: Hardware encoding on portable GPU

Tegra X1 chipset has support for multiple encoders on the hardware level. I haven’t verified all of them manually, but instead I have went through the source code of plugins gst-omx provided by Nvidia. It has support for video codecs: mpeg4, h263, h264, h265, vp8 and audio AAC. More information can be found along with Linux for Tegra releases. Please find more here:

I have verified VP8 encoder as the most interesting for this project. It works perfectly fine with 4K realtime encoding at 30fps.

Please refer here to find out what parameters are exposed by hardware encoders. The VP8 encoder omxvp8enc has following parameters:

  bitrate             : Target bitrate
                        flags: readable, writable, changeable in NULL, READY, PAUSED or PLAYING state
                        Unsigned Integer. Range: 0 - 4294967295 Default: 4000000 
  quant-i-frames      : Quantization parameter for I-frames (0xffffffff=component default)
                        flags: readable, writable, changeable only in NULL or READY state
                        Unsigned Integer. Range: 0 - 4294967295 Default: 4294967295 
  quant-p-frames      : Quantization parameter for P-frames (0xffffffff=component default)
                        flags: readable, writable, changeable only in NULL or READY state
                        Unsigned Integer. Range: 0 - 4294967295 Default: 4294967295 
  quant-b-frames      : Quantization parameter for B-frames (0xffffffff=component default)
                        flags: readable, writable, changeable only in NULL or READY state
                        Unsigned Integer. Range: 0 - 4294967295 Default: 4294967295 
  iframeinterval      : Encoding Intra Frame occurance frequency
                        flags: readable, writable, changeable only in NULL or READY state
                        Unsigned Integer. Range: 0 - 4294967295 Default: 0 
  quality-level       : Encoding quality-level
                        flags: readable, writable, changeable only in NULL or READY state
                        Unsigned Integer. Range: 0 - 2 Default: 0 
  low-latency         : Whether or not set low latency attribute of encoder
                        flags: readable, writable, changeable only in NULL or READY state
                        Boolean. Default: false
  SliceIntraRefreshEnable: Enable Slice Intra Refresh while encoding
                        flags: readable, writable, changeable only in NULL or READY state
                        Boolean. Default: false
  SliceIntraRefreshInterval: Set SliceIntraRefreshInterval
                        flags: readable, writable, changeable only in NULL or READY state
                        Unsigned Integer. Range: 0 - 4294967295 Default: 60 
  bit-packetization   : Whether or not Packet size is based upon Number Of bits
                        flags: readable, writable, changeable only in NULL or READY state
                        Boolean. Default: false
  vbv-size            : virtual buffer size = vbv-size * (bitrate/fps)
                        flags: readable, writable, changeable only in NULL or READY state
                        Unsigned Integer. Range: 0 - 30 Default: 10 
  temporal-tradeoff   : Temporal Tradeoff value for encoder
                        flags: readable, writable, changeable only in NULL or READY state
                        Enum "GstOmxVideoEncTemporalTradeoffType" Default: 0, "Do not drop frames"
                           (0): Do not drop frames - GST_OMX_VIDENC_DROP_NO_FRAMES
                           (1): Drop 1 in 5 frames - GST_OMX_VIDENC_DROP_1_IN_5_FRAMES
                           (2): Drop 1 in 3 frames - GST_OMX_VIDENC_DROP_1_IN_3_FRAMES
                           (3): Drop 1 in 2 frames - GST_OMX_VIDENC_DROP_1_IN_2_FRAMES
                           (4): Drop 2 in 3 frames - GST_OMX_VIDENC_DROP_2_IN_3_FRAMES

Week 5: Media framework on portable GPU

I have verified only the GStreamer to be properly working with Nvidia GPU. Of course the media framework has nothing to do with GPU support but rather provides API for plugins or extensions to accelerate video/audio processing with hardware APIs. If GPU is supported by OS and expose the API for the application then any media framework will work properly.

Week 6: OpenGL/OpenCV on portable GPU

Both are perfectly supported by Nvidia on Jetson TX1. Please refer to Nvidia documentation e.g. https://developer.nvidia.com/embedded/linux-tegra

Week 7: 3D/VR tricks using media frameworks and GPU

This topic has been started by me but not really evolved to the working state. I was mainly focus on capturing-playback tricks for reducing amount of data (bitrate) on the pipeline. This is important due to limited computing (GPU) power on the mobile/playback side. To reduce bitrate there are many tricks based on capturing, encoding and streaming only this parts of scene which would be presented in the VR headset.

I will share more knowledge and my experience in the following blogposts. You can see (below) a screenshot of chroma keying algorithm (current implementation) for extracting background/foreground layers of the input video stream by presenting colours in 3D space.

 

Week 8: Configurable cameras setup

I have made some progress on camera mounting solutions. You can read more in the blogpost I have shared a week ago. Also there is a progress on selecting the right cameras for the version 1.0 and 2.0. Please see dedicated blogpost.

Week 9: Audio input and output

There is also a little progress on the audio side. I have research and tested a few of small and cheap devices which might be working properly in this project. Please find more in the blogpost I have already written.

Attachment – parts & tools tests

Parts Check List (current)

Name I Arrived Tested Notes SW HW W
8×8 LED display i2c Y  Y tested with Jetson TK1 + Ubuntu + Bash + i2c cli
Touch Buttons i2c Y N
Relays GPIO Y Y tested with Jetson TK1 + Ubuntu + Bash scripts
Servo Controller UART Y Y tested with Jetson TK1 + Ubuntu + Bash + Python
IMU sensor i2c Y N
USB Hub 3.0 Y Y Tested with Jetson TK1 + Ubuntu + Gstreamer + Cameras
e-con FullHD camera, fxed focus USB 3.0 Y Y Tested with Jetson TK1 + Ubuntu + Gstreamer + FFmpeg + VLC + RAW data
Lenses  M12  Y  Y Tested with M12 camera holders. Tested FOV for 120H x 85V x 170D with 1/3″ CMOS Image Sensor
Auvidea J100 Carrier board Jetson TX1 USB3.0, MIPI/CSI-2, HDMI, Wifi, GPIO, UART, I2C Y  Y  Works properly
Audio Card Stereo Out + Mic  USB2.0  Y  Y Tested with Jetson TK1 and Jetson TX1 on Ubuntu 14.04 with kernel 3.10.*
Camera OV5693  MIPI/CSI-2  Y Y  Tested with Jetson TX1, ISP and Gstreamer

I – Interface
SW – software
HW – hardware
W – week

Frameworks Check List (current)

Name Tested Notes
RealTime Panorama Stitching for 30 fps with 4 cameras (2 x left, 2 x right)  N  (in development)
RealTime 3D/Depth calculation with stereo vision  N  (in development)
RealTime Chroma Keying  Y Gstreamer with FullHD stereo video (3840×1080) – Jetson TK1
RealTime Primatte Inspired Keying  N  (in development)
RealTime Hardware Encoding to VP8  Y Gstreamer + OMX plugins with FullHD stereo video (3840×1080/videotestsrc/camera) – Jetson TK1/TX1
Camera settings via e-con Systems QtCAM software for linux  Y Ubuntu 15.04 attempt failed – missing libs, does not compile
Compatibility of USB2.0/3.0 cameras with Gstreamer v4l2src plugin  Y  Tested Jetson TK1/TX1, works fine