ART-Challenge with UltraHD/4K + 3D/VR hardware accelerated, relatime HDMI/RTP/WebRTC live streaming and communication - portable device

Why?

For last months I have been working for new product based on real-time 3D/VR live streaming into the mobile browser. It was a huge challenge to deliver such a platform to the market because there are tons of missing, very tiny elements which must be done from the scratch. It affects mechanics, electronics, software and basically entire architecture on every level. This journey has shown me that using brute-force model we can solve a lot of problems if we drop the matter of price, size and common usability. However there are problem which are not trivial to solve or simply not to buy! One of them is FullHD 3D/VR ready camera which would fit some generic use cases.

In my opinion it would be worth to research more deeply into 3D camera topic, VR streaming topic and idea of bringing a "portable 3D/VR platform" to the regular homes.

As I said this is very complicated topic. It requires huge computing power, high precision of optics and tons of lines of software. Fortunately, there many open source hardware & software projects right now, which are like "dust in the wind" - if you do not have recipe for how to connect them together. If you know clearly your goal, which is very specific in this case, I claim that building this very advanced platform is possible right now at my house basement!

Why no one did it before?

Actually it has been done for many times in last years. The problem is about the size, the price and incredible complexity on every level (cameras, converters, cables, server towers etc). Most of this usages require a lot of power and are not portable! Most of them has been built using brute-force with the goal to deliver anything just WORKING!

Below a little draft overview of such platform which requires: video/audio inputs, GPU for processing video and many CPUs for encoding if not supported by hardware.

What are benefits of this challenge for business and home usage?

• REDUCE IN COST OF 75% - comparing to "obvious" brute-force solutions

• REDUCE IN SIZE OF 75% - comparing to "obvious" brute-force solutions

• INCREASED MOBILITY - must be small and portable

• INCREASED CONFIGURABILITY - must be easy to configure

• INCREASED EFFICIENCY

• FULL REAL-TIME OPERATIONS

In a few words I wanna use upcoming new open source hardware platform and open source software for media processing and communication to deliver small, powerful, customizable device for professional and home usage within incredible and fascinating VR world!

The GOALs

How could it look like?

The idea is to build very compact, portable platform which would combine CPUs, GPUs, CAMs, encoders, converters, bridges, storage and other required elements into small box.

Inputs:

• CAM-L + configurable lens and position
• CAM-R + configurable lens and position
• MIC-L
• MIC-R
• Audio-IN (optional)
• Power

Outputs:

• HDMI
• LAN (RTP, WebRTC)

What is the practical use case?

- 3D/VR-based conferencing

- Home Studio Streaming

- Professional Studio Streaming

- Medical services

- Entertainment services

- Anything you can imagine...

What's next?

I have already spent a lot of time on research of portable CPU/GPU platform which would be a heart of such device. I have also researched a multiple camera and lens options which would capture stereoscopic video and audio. Some of this components are on the way to my lab. The other ones are waiting to be officially released to the market by end of March 2016.

The most realistic date for having all required components is the beginning of April 2016. It is because incredible advanced components are gonna be released to the market in next 1-2 months. Some of them are designed and produced in US, some of them in Germany. Let's hope there will be no delay on any team:)

I am gonna share my first steps in the following blogposts. At this time I wanna say that I do not consider to use any of common used and expensive solutions like: GoPro Hero, Decklink converters, Decklink video capture cards, advanced CPU like Intel Skylake and high efficient GPU systems.

Plan (starting on 28 February 2016) - "A brief history of VR":

- Art 3D/VR challenge – week 1 – Abstract overview

- Art 3D/VR challenge – week 2 – Looking for performance

- Art 3D/VR challenge – week 3 – FullHD processing on portable GPU vs CPU

- Art 3D/VR challenge – week 4 – Hardware encoding on portable GPU

- Art 3D/VR challenge – week 5 – Media framework on portable GPU

- Art 3D/VR challenge – week 6 – OpenGL/OpenCV on portable GPU

- Art 3D/VR challenge – week 7 – 3D/VR tricks using media frameworks and GPU

- Art 3D/VR challenge – week 8 – Configurable cameras setup

- Art 3D/VR challenge – week 9 – Audio input and output

- Art 3D/VR challenge – week 10 – Controlling and management of media pipeline

- Art 3D/VR challenge – week 11 – REST API and CLI tooling

- Art 3D/VR challenge – week 12 – HDMI and RTP output

- Art 3D/VR challenge – week 13 – Advanced tuning and management

- Art 3D/VR challenge – week 14 – Introduction to embedded WebRTC client

- Art 3D/VR challenge – week 15 – Introduction to Intelligent Chroma Keying

- Art 3D/VR challenge – week 16 – Introduction to Intelligent Chroma Keying part 2

- Art 3D/VR challenge – week 17 – Embedded on-board 3D scene rendering

- Art 3D/VR challenge – week 18 – Embedding VR video into 3D scene

- Art 3D/VR challenge – week 19 – 3D into 2D scene dumping

- Art 3D/VR challenge – week 20 – Introduction to embedded WebRTC server

- Art 3D/VR challenge – week 21 – 3D/VR and 2D live streaming

Contribution

Feel free to contact me if you are interested in meeting the team and contribution to this project in any programming language (go, php, ruby, js, node.js, objective-c, java...). This project is parked on Github.

See my contact page if required.

© COPYRIGHT KRZYSZTOF STASIAK 2016. ALL RIGHTS RESERVED