Videoinfrastruktur für den verteilten Congress des Chaos Computer Club

 

FKTG Seminar / Media Event Ilmenau, 23.02.2021

Alexander Votteler & Anton Schubert

Background image by Mitch Altman (Flickr, CC-BY-SA)

Alexander Votteler

  • active member of FeM e.V. since 2016
  • Chaos Communication Congress Streaming since 2015
  • doing freelance work in the video industry
  • employed at a local Internet service provider in Aachen
  • graduation master MT at TU Ilmenau in 2018
  • development engineer at Riedel Communications
  • active member of FeM e.V. since 2014
  • C3VOC streaming since 2014

Anton Schubert

  • student association at TU-Ilmenau
  • providing Internet access for 2000 students in dormitories
  • operation of iSTUFF - a student TV station
  • recording of Chaos Communication Congress since 2005

Forschungsgemeinschaft

elektronische Medien

  • CCC is Europe's largest association of hackers
  • "galactic community of life forms"
  • organized in local communities called "Erfas"

Chaos Computer Club

Background image by Yves Sorge (Flickr, CC-BY-SA)

  • congress with lectures and workshops surrounding technical, political and social issues
  • 2019: 17.000 visitors at Messe Leipzig (36C3)
  • organized by volunteers
  • 2020: first time as virtual event (rC3)

Chaos Communication Congress

Background image by Simon Waldherr (Flickr, CC-BY-SA)

  • working group of video enthusiasts in the CCC
  • streaming, recording and publishing of lectures
  • strong use of open source software and technologies

 

C3VOC - "CCC video operations center"

Background image by evilscientress (Flickr, CC-BY)

  • virtual event in place of the congress
  • 27.12.2020 - 30.12.2020
  • three main parts
    • rC3 World - 2D online multiplayer game
    • online workshops via video conferences
    • streaming of lectures

Remote Chaos Experience (rC3)

  • 25 studios
  • 19 live broadcast channels
  • over 350 talks and events

rC3 Streaming

  • over 9000 peak concurrent livestream viewers
  • >25 Gigabit/s peak outgoing stream bandwidth
  • about 95% of streams in adaptive formats

rC3 Streaming

Statistics

  • custom content and streaming schedules
  • content produced by local CCC chapters
  • streaming and releasing via C3VOCs own infrastructure
  • diverse ways of producing video:
    • OBS
    • hardware mixers

rC3 streaming

Community channels

  • curated content and continuous schedule
  • production handled by 12 live studios
  • mixing and recording by master control room in Ilmenau

rC3 streaming

Main channels

  • studio goes live
    • introduction by herald (announcer)
    • live/prerecorded presentation
    • Q/A session
  • handover to next studio

rC3 streaming

Programming Schedule

  • recording managers
  • stream operators (MCR)
  • studio crews
  • herald crew
  • signal angels
  • translators
  • VOC

rC3 streaming

Teams

  • voice communication via Mumble
  • shared Kanban boards (Wekan) and ticket system
  • file sharing for prerecorded talks

rC3 streaming

Communication channels

rC3 streaming

Communication channels

rC3 streaming

Challenges

  • missing live-video experience
  • coordinating the chaos
  • timing with varying video-delay
  • mixing of the signal for the mains streams
  • audio processing
  • embedding of live translation

Master Control Room

MCR Overview

  • vMix software video mixer
  • Riedel Mediornet
  • Lawo V__matrix
  • Allen & Heath dLive
  • Linux machines with SDI and MADI I/O

Master Control Room

Equipment

Master Control Room

vMix

Master Control Room

Translator Ingest

Master Control Room

Audio

Master Control Room

Technical Challenges

  • only a single day for building the setup
  • SDI interoperability is also bad
  • crappy hardware encoders and bad uplinks

C3VOC Infrastructure

Upgrades for rC3

C3VOC Infrastructure

Challenges

  • more viewers than before
  • greater expectations for the streams
  • more streams than before
  • production not in hands of the C3VOC

C3VOC Infrastructure

Streaming CDN

  • dynamic Conference schedule
  • custom web-player
  • Adaptive quality switching
  • Multi-Language support
  • MPEG-DASH (webM-based) and HLS

Streaming Website

C3VOC Infrastructure

Streaming CDN upgrades

  • more load-balancers and edge relays
  • 80 Gbit/s total CDN bandwidth in 7 locations
  • more transcoders
  • streaming-capacacity was sufficient
  • some ddos on day 2

C3VOC Infrastructure

Achieving better Stability

  • load-testing of CDN edge nodes
  • load-testing of software transcoders

github.com/voc/stream-tools

C3VOC Infrastructure

Encoding h.264 on GPUs

  • 2019 first tests with h.264 encoding on Intel GPUs
  • 2020 quality comparisons using Netflix VMAF
  • Intel h.264 GPU encoding beats x264 for realtime
  • using our full hardware: h.264 on GPU, VP9 on CPU

github.com/voc/voctoquality

  • RTMP is the de facto standard
  • used servers with nginx-rtmp plugin
  • added a custom authentication backend

C3VOC Infrastructure

Stream Ingest changes

https://github.com/voc/rtmp-auth

  • RTMP is less than ideal
    • only supports one audio-stream
    • limited codec support
    • TCP-based
  • so we developed a SRT ingest solution

C3VOC Infrastructure

Getting rid of RTMP

https://github.com/voc/srtrelay

  • improve speaker audio/video quality
  • make streaming easier for external parties Projects

Future plans

  • translation muxing in the cloud
  • plug&play capable RaspberryPi-based speaker video kit
  • custom WebRTC conferencing app (Open Source Vmix Call)
    (https://github.com/voc/kevin)
  • streaming automation and web-based streaming backend

Projects

  • Mumble Intercom worked great
  • check your devices SDI specs
  • don't decentralize too much to fast
  • testing, testing, testing

Conclusions

Thank you for your attention

Anton Schubert

anton.schubert@fem.tu-ilmenau.de

cookie-factory.de

Alexander Votteler

alexander.votteler@fem.tu-ilmenau.de

alexandervotteler.de

This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License