AIS: Vision, Graphics and AI for Streaming

Schedule Details - Arch 3A, 17th June

• 09:00 - 09:15: Opening
• 09:15 - 10:00: "Event-based Eye-Tracking Challenge & Papers Session" , Qinyu Chen (Leiden University)
  

  ---

  "Retina : Low-Power Eye Tracking with Event Camera and Spiking Hardware", Pietro Bonazzi (ETH Zurich)
  "A Hybrid ANN-SNN Architecture for Low-Power and Low-Latency Visual Perception", Asude Aydin
  "A Lightweight Spatiotemporal Network for Online Eye Tracking with Event Camera", Yan Ru Pei
  "Co-designing a Sub-millisecond Latency Event-based Eye Tracking System with Submanifold Sparse CNN", Baoheng Zhang
  

---

• 10:00 - 10:45: "AI and Machine Learning for Video Compression" by Ryan Lei (Video Codec Specialist, Meta)

  Over the past 30 years, significant advances have been achieved in the video compression domain, which has made it become an absolutely indispensable technology that powers today’s Internet. Meanwhile, the past decade has also witnessed the great success of machine learning in many areas, especially in computer vision and image processing. It is very natural to ask if and how machine learning techniques can be leveraged to improve video compression. In this talk, the speaker will first provide a high level overview of progress in the conventional video coding standard development and the challenges that the community is facing. Then the speaker will focus on a few trends on how machine learning can be leveraged for video compression. The first topic is how machine learning is used to further optimize coding tools of traditional video coding, such as intra/inter prediction, loop filtering, etc. The second topic is how machine learning is used to improve the coding efficiency of the overall compression system, such as pre/post filtering, super resolution, layered coding, etc. The last topic is how neural networks are used to develop end-to-end learned video coding frameworks and fully replace the conventional video coding system. Over the talk, the speaker will also present examples of techniques that have worked and techniques that may not work in the near future.

• 10:45 - 11:15: "Video Quality Assessment Challenge and Results"
  "COVER: A Comprehensive Video Quality Evaluator", Zhengzhong Tu (Texas A&M University)

---

• 11:15 - 11:40: Paper Presentation Session #1
  "Deep Video Codec Control for Vision Models", Christoph Reich (Technical University of Munich, TU Darmstadt, NEC Laboratories America, Inc., and University of Oxford)
  "A Perspective on Deep Vision Performance with Standard Image and Video Codecs", Christoph Reich (Technical University of Munich, TU Darmstadt, NEC Laboratories America, Inc., and University of Oxford)
  "Adaptive render-visual (REVI) streaming for virtual environments", Matthias Treder (Sony PlayStation)

---

• 11:40 - 12:30: "Neural Compression & Realism" by Lucas Theis (Google DeepMind)

  Neural compression has the potential to transform the streaming industry by enabling highly realistic outputs at very low rates. Although these techniques have yet to break into mainstream codecs, recent advances in ML and information theory continue to bring this goal closer into view. This talk will focus on how generative AI and novel coding methods allow us to generate realistic outputs at arbitrarily low rates. In particular, we will explore two approaches based on diffusion generative models. Time permitting, we will also look at recent advances in making neural compression much more computationally efficient.

• 12:30 - 13:45: Lunch & Poster Session
• 13:45 - 14:35: "On the Visual Quality of Pictures, Games, and GenAI", by Prof. Alan Bovik (The University of Texas)

  Predicting the perceptual quality of pictures and videos is a hard problem that has been successfully addressed in many scenarios, such as quality control of streaming videos and sharing of social media pictures. In this talk I will address how visual quality perception can be understood using principles of neuroscience and neuro-statistical models of distortion. In particular I will review some basic vision science that makes accurate perceptual visual quality prediction possible, and how algorithms are designed that are now used worldwide. However, generated content like gaming videos and GenAI pictures are fundamentally different from photographs, and those differences may be statistically testable. I will also discuss a recent approach to gaming content quality prediction called GAMIVAL that combines neuro-statistical models adapted to gaming distortions with deep features that capture gaming semantics. I will also discuss a very early exploration into the underlying perceptual statistics of GenAI pictures, and how these might be used to gauge quality and detectibility.

---

The "Super-session" starts!
• 14:40 - 15:00: On the Edge Real-time Image Super-Resolution Challenge and News!!
• 15:00 - 15:40: "Video Super-Resolution Using Deep Learning" by Kelvin C.K. Chan (Google DeepMind)

  The demand for high-resolution content continues to grow, necessitating advanced video super-resolution (VSR) techniques. However, existing VSR methods often struggle with efficiency and generalization to real-world scenes. This talk explores the benefits of leveraging temporal information in VSR, highlighting how recurrent networks can effectively propagate and align information to improve performance in both synthetic and real-world settings. Additionally, we will delve into the potential of generative models, such as diffusion models, for VSR, motivated by their recent successes in image super-resolution.

• 15:40 - 16:10: VideoGigaGAN: "Towards Detail-rich Video Super-Resolution", Yiran Xu (University of Maryland)
• 16:10 - 16:40: FMA-Net: "Flow-Guided Dynamic Filtering and Iterative Feature Refinement with Multi-Attention for Joint Video Super-Resolution and Deblurring", Jihyong Oh (Chung-Ang University (CAU))
• 16:45 - 17:30: "Deploying neural networks for video downscaling: why and how" by Christos Bampis (Netflix)

  Video downscaling is an important component of video processing. Within adaptive streaming, it tailors streaming to different device resolutions and optimizes picture quality under varying network conditions. Video downscaling is typically done by a conventional resampling filter like Lanczos. We have deployed neural networks that perform video downscaling within the Netflix adaptive streaming pipeline. This talk describes why we embarked on this journey, how it was done and lessons learned. We also discuss possible pathways moving forward.

---

• 17:30 - 17:50: Paper Presentation Session #2
  "Depth Compression and Upsampling", Jinseong Kim (RGA Inc.)
  "One-Click Upgrade from 2D to 3D: Sandwiched RGB-D Video Compression for Stereoscopic Teleconferencing", Yueyu Hu (New York University)
  "Low Latency Point Cloud Rendering with Learned Splatting", Yueyu Hu (New York University)
  
• 17:50 - 18:00: Closing Remarks & Award Ceremony [Challenge Diplomas]

Virtual Presentations