Screen Shot 2016-05-10 at 11.15.14

Art 3D/VR challenge – Image Signal Processing (ISP) – Tegra X1

In the previous blogpost I have researched for the optimal stereo cameras setup. At the beginning of this project (see index) I was not really aware of the importance and complexity of image processing in area of picture tuning, but after detailed research I had to step back to gain more informations.

More information about ISP and signal processing itself you can find on the wiki.

Typical workflow

(sourced from Khronos presentation)

Hardware

There are multiple technologies which implement the ISPs. Most of the common are the ASIC, ASSP, SoC and of course FPGA which seems to be heavily used at the moment. The main advantage of FPGA is the flexibility for continues optimisation and customization of camera device features and quality. Solutions such ASIC, ASSP, SoC are less flexible and basically design for mass production and not customization.

Typical features (currently)

Most of hardware implementation provides following features:

  • Support RGB Bayer progressive image sensor
  • Support 8 ~ 14 bit input data Bayer
  • Support image sensor of 256*256 ~ 8192*8192 size
  • Defect Correction
  • Lens Shading Correction
  • High quality interpolation
  • 3D Motion Adaptive noise reduction and 2D noise reduction
  • Color correction by 3×3 matrix
  • Gamma correction
  • HDR processing for Multiple exposure images and HDR bayer image
  • WDR (Shadow/Highlight compensation, back light compensation)
  • 2D edge enhancement
  • support AE, AWB and AF
  • Saturation, contrast and brightness control
  • Support special effects (sepia, negative, solarization)
  • Support cpecial scene modes

Khronos standardisation

Khronos group implements the standards for handling the camera data processing. They introduce OpenKCam for advanced camera and sensor control. OpenKCam is based on FCAM (2010) Stanford/Nokia, open source.

(sourced from Khronos presentation)

Main needs, requirements and design philosophy:

  • Portable access to growing sensor diversity: e.g. depth sensors and sensor arrays
  • Cross sensor synch: e.g. synch of camera and MEMS sensors
  • Advanced, high-frequency per-frame burst control of camera/sensor: e.g. ROI
  • Multiple input, output re-circulating streams with RAW, Bayer or YUV Processing
  • Reduce extreme fragmentation for ISVs wanting more than point and shoot
  • Application control over ISP processing: Includes multiple, re-entrant ISPs
  • Control multiple sensors:  Stereo pairs, Plenoptic arrays, TOF or structured light depth cameras
  • Enhanced per frame detailed control: – Format flexibility, Region of Interest (ROI) selection
  • Global timing & synchronization
  • Flexible processing/streaming
  • C-language API starting from proven designs: FCAM, Android camera platform
  • Design alignment with wWidely used hardware standards: MIPI, CSI
  • Focus on mobile, power-limited devices
  • Provide support for vendor-specific extensions

How does Nvidia implement the ISP in Tegra X1?

Nvidia is working very hard to implement a very advances Image Signal Processing into SoC solutions. Each next (newer) generation of e.g. Tegra chipset contains improved and more efficient ISP or even multiple ISPs. I will focus on Tegra X1 (as this is the core of Jetson TX1) the part of current VR box project.


Tegra X1 implements Dual ISP with support for 4096 focus points. It improves throughout up to 1.3 GigaPixel/s and supports up to 100MP image matrix. It also support for 6 cameras!


Nvidia provides the open source SDK to access the ISP features. It is so called “NV GST” and can be found along with Linux for Tegra. The Nvcamerasrc (gst plugin) handles the imaging pipeline for still photos, video, and performs tasks like Bayer to RGB conversion (demosaicing), 3A (Autofocus, Auto Exposure, Auto white balance), noise reduction, lens correction, and many other actions:

wbmode              : White balance affects the color temperature of the photo
                        flags: readable, writable
                        Enum "GstNvCamWBMode" Default: 1, "auto"
                           (0): off              - GST_NVCAM_WB_MODE_OFF
                           (1): auto             - GST_NVCAM_WB_MODE_AUTO
                           (2): incandescent     - GST_NVCAM_WB_MODE_INCANDESCENT
                           (3): fluorescent      - GST_NVCAM_WB_MODE_FLUORESCENT
                           (4): warm-fluorescent - GST_NVCAM_WB_MODE_WARM_FLUORESCENT
                           (5): daylight         - GST_NVCAM_WB_MODE_DAYLIGHT
                           (6): cloudy-daylight  - GST_NVCAM_WB_MODE_CLOUDY_DAYLIGHT
                           (7): twilight         - GST_NVCAM_WB_MODE_TWILIGHT
                           (8): shade            - GST_NVCAM_WB_MODE_SHADE
                           (9): manual           - GST_NVCAM_WB_MODE_MANUAL
  scene-mode          : property to set scene mode
                        flags: readable, writable
                        Enum "GstNvCamSceneMode" Default: 0, "face-priority"
                           (0): face-priority    - GST_NVCAM_SCENE_MODE_FACE_PRIORITY
                           (1): action           - GST_NVCAM_SCENE_MODE_ACTION
                           (2): portrait         - GST_NVCAM_SCENE_MODE_PORTRAIT
                           (3): landscape        - GST_NVCAM_SCENE_MODE_LANDSCAPE
                           (4): night            - GST_NVCAM_SCENE_MODE_NIGHT
                           (5): night-portrait   - GST_NVCAM_SCENE_MODE_NIGHT_PORTRAIT
                           (6): theatre          - GST_NVCAM_SCENE_MODE_THEATRE
                           (7): beach            - GST_NVCAM_SCENE_MODE_BEACH
                           (8): snow             - GST_NVCAM_SCENE_MODE_SNOW
                           (9): sunset           - GST_NVCAM_SCENE_MODE_SUNSET
                           (10): steady-photo     - GST_NVCAM_SCENE_MODE_STEADYSHOT
                           (11): fireworks        - GST_NVCAM_SCENE_MODE_FIREWORKS
                           (12): sports           - GST_NVCAM_SCENE_MODE_SPORTS
                           (13): party            - GST_NVCAM_SCENE_MODE_PARTY
                           (14): candle-light     - GST_NVCAM_SCENE_MODE_CANDLE_LIGHT
                           (15): barcode          - GST_NVCAM_SCENE_MODE_BARCODE
  color-effect        : property to set color effect
                        flags: readable, writable
                        Enum "GstNvCamColorEffectMode" Default: 1, "off"
                           (1): off              - GST_NVCAM_COLOR_EFFECT_OFF
                           (2): mono             - GST_NVCAM_COLOR_EFFECT_MONO
                           (3): negative         - GST_NVCAM_COLOR_EFFECT_NEGATIVE
                           (4): solarize         - GST_NVCAM_COLOR_EFFECT_SOLARIZE
                           (5): sepia            - GST_NVCAM_COLOR_EFFECT_SEPIA
                           (6): posterize        - GST_NVCAM_COLOR_EFFECT_POSTERIZE
                           (7): aqua             - GST_NVCAM_COLOR_EFFECT_AQUA
  auto-exposure       : property to set auto exposure mode
                        flags: readable, writable
                        Enum "GstNvCamAutoExposureMode" Default: 2, "on"
                           (1): off              - GST_NVCAM_AUTO_EXPOSURE_OFF
                           (2): on               - GST_NVCAM_AUTO_EXPOSURE_ON
                           (3): OnAutoFlash      - GST_NVCAM_AUTO_EXPOSURE_AUTOFLASH
                           (4): OnAlwaysFlash    - GST_NVCAM_AUTO_EXPOSURE_ALWAYSFLASH
                           (5): OnFlashRedEye    - GST_NVCAM_AUTO_EXPOSURE_AUTO_RED_EYE
  flash               : property to set flash mode
                        flags: readable, writable
                        Enum "GstNvCamFlashMode" Default: 0, "off"
                           (0): off              - GST_NVCAM_FLASH_OFF
                           (1): on               - GST_NVCAM_FLASH_ON
                           (2): torch            - GST_NVCAM_FLASH_TORCH
                           (3): auto             - GST_NVCAM_FLASH_AUTO
  flicker             : property to select flicker detection and avoidance mode
                        flags: readable, writable
                        Enum "GstNvCamFlickerMode" Default: 3, "auto"
                           (0): off              - GST_NVCAM_FLICKER_OFF
                           (1): 50Hz             - GST_NVCAM_FLICKER_50HZ
                           (2): 60Hz             - GST_NVCAM_FLICKER_60HZ
                           (3): auto             - GST_NVCAM_FLICKER_AUTO
  contrast            : property to adjust contrast value
                        flags: readable, writable
                        Float. Range:               0 -               1 Default:               0 
  saturation          : property to adjust saturation value
                        flags: readable, writable
                        Float. Range:               0 -               2 Default:               1 
  tnr-strength        : property to adjust temporal noise reduction strength
                        flags: readable, writable
                        Float. Range:               0 -               1 Default:               0 
  tnr-mode            : property to select temporal noise reduction mode
                        flags: readable, writable
                        Enum "GstNvCamTNRMode" Default: 0, "NoiseReduction_Off"
                           (0): NoiseReduction_Off - GST_NVCAM_NR_OFF
                           (1): NoiseReduction_Fast - GST_NVCAM_NR_FAST
                           (2): NoiseReduction_HighQuality - GST_NVCAM_NR_HIGHQUALITY
  edge-enhancement    : property to adjust edge enhnacement value
                        flags: readable, writable
                        Float. Range:               0 -               1 Default:              -1 
  trigger             : property to fire pre capture trigger
                        flags: readable, writable
                        Enum "GstNvCamCaptureTrigger" Default: 0, ""
                           (1): idle             - GST_NVCAM_TRIGGER_IDLE
                           (2): start            - GST_NVCAM_TRIGGER_START
  intent              : property to select capture intent
                        flags: readable, writable
                        Enum "GstNvCamCaptureIntent" Default: 1, "preview"
                           (1): preview          - GST_NVCAM_INTENT_PREVIEW
                           (2): still            - GST_NVCAM_INTENT_STILL
                           (3): video            - GST_NVCAM_INTENT_VIDEO
                           (4): video-snapshot   - GST_NVCAM_INTENT_VIDEO_SNAPSHOT
                           (5): zsl              - GST_NVCAM_INTENT_ZSL
  sensor-id           : Set the id of camera sensor to use. Default 0.)
                        flags: readable, writable
                        Integer. Range: 0 - 255 Default: 0 
  enable-exif         : Return Exif data with captured frame
                        flags: readable, writable
                        Boolean. Default: false
  enable-meta         : Return Sensor meta data with captured frame
                        flags: readable, writable
                        Boolean. Default: false
  aeRegion            : Property to set region of interest for auto exposure.
			 Use GArray, with values of ROI coordinates (top,left,bottom,right)
			 and weight in that order, to set the property.
                        flags: readable, writable
                        Boxed pointer of type "GArray"
  wbRegion            : Property to set region of interest for white balance.
			 Use GArray, with values of ROI coordinates (top,left,bottom,right)
			 and weight in that order, to set the property.
                        flags: readable, writable
                        Boxed pointer of type "GArray"
  fpsRange            : Property to set FPS range.
			 Use string with values of FPS Range (low, high)
			 in that order, to set the property.
                        flags: readable, writable
                        String. Default: null
  exposure-time       : Property to adjust exposure time.
			 This property will have effect only when Exposure mode is Off.
                        flags: readable, writable
                        Float. Range:               0 -    3.402823e+38 Default:           0.033 
  wbManualMode        : Property to set manual mode of White balance.
			 This property will have effect only when wbMode is manual.
                        flags: readable, writable
                        Enum "GstNvCamWBManualMode" Default: 0, "off"
                           (0): off              - GST_NVCAM_WB_MANUAL_OFF
                           (1): reset            - GST_NVCAM_WB_MANUAL_RESET
                           (2): restore          - GST_NVCAM_WB_MANUAL_RESTORE
                           (3): external-gains   - GST_NVCAM_WB_MANUAL_EXTGAINS
  wbGains             : Property to set external white balance Gains.
			 Use GArray, with values of WB gains (R, GR, GB, B)
			 in that order, to set the property.
			 This will have effect only when wbMode is manual.
                        flags: readable, writable
                        Boxed pointer of type "GArray"
  stitchConfig        : Property to set stitching windows.
			 Use GArray, with values of sensor id and
			 window coordinates (sensor-id,top,left,bottom,right)

                        flags: readable, writable
                        Boxed pointer of type "GArray"
  callback            : Property to set user callback. It is required if user wants 
			 to change capture settings per frame. Provide address 
			 of GstNvCamSrcCallbacks type struct as value. 

                        flags: writable
                        Pointer. Write only
  dump-bayer          : Dump raw bayer data in still capture mode.
                        flags: readable, writable
                        Boolean. Default: false
  num-sensors         : Property to get number of sensors.
			 Returns GArray containing number of sensors {usb,csi}

                        flags: readable
                        Boxed pointer of type "GArray"
  orientation         : Sensor orientation value. This must be accessed in READY state.
                        flags: readable
                        Integer. Range: 0 - 360 Default: -1

The Dual ISP Tegra X1 technology is ported from Tegra K1 which retains compatibility with the Chimera 1.0+ features. The image below gives the overview of the processing/workflow, but it is not the Tegra X1 implementation.

(sourced from Nvidia presentation)

 

Nvidia along with VisionWorks frameworks gives the access to the power of image processing.

Comparing to FPGA solutions the X1 is SoC implementation of ISP and does not allow for any customization. It is very limiting for the new/future products, but works very well along with typical/average needs!

Contribution

Feel free to contact me if you are interested in meeting the team and contribution to this project in any programming language (puppet, go, php, ruby, js, node.js, objective-c, java… and hardware, PCB, 3D printing, UI/UX, PR… and financial support…)

See my contact page if required.

© COPYRIGHT KRZYSZTOF STASIAK 2016. ALL RIGHTS RESERVED