IMPROVED targets studies and tuning of a new generation of tools dedicated to image and video enhancement, for police investigations, while keeping their ability to be used as evidence and respecting individual liberties. The project aims at bridging the gaps encountered by Scientific Police in the field of video enhancement for getting relevant evidence. New video sources, in addition to video surveillance cameras are now available, such as images from smartphones, dashcams or wearable cameras. These devices bring their own sensor characteristics, and video coding features and parameters, thus increasing the difficulties. Enhancement algorithms, especially those relying on Artificial Intelligence, set questions regarding their receivability as legal evidence, due to their black box like design. Image enhancements must consider rights and fundamental principles, in particular loyalty of the legal evidence, and guaranty explainability of the used algorithms. For this, in IMPROVED an interdisciplinary work will be performed between algorithmists and lawyers of the project, with the aim to check, for each enhancement algorithm, whether the processing did not alter the capacity of images to be considered as legal evidence. The project will study new algorithmic directions, such as explainable AI, for making system behaviour more understandable, while providing explanations at each processing step. These algorithmic studies target going beyond the state-of-the-art in image enhancement, while addressing challenges related to new generations of video coding and image sources.

Video-based services involved in tactile internet (vehicle teleoperation, remote surgery…) require data exchange with an end-to-end latency of a few milliseconds. While 5G provides significantly reduced latency for the access network, video acquisition, coding, long-range transmission, and decoding delays become prevailing. ZL-LVC aims at proposing low-to-Zero Latency (ZL) video coding and delivery schemes, giving an end-user the feeling to watch pictures at the time they are remotely acquired. For that purpose, latency will be compensated thanks to temporal frame extrapolation at the coder or/and at the decoder. At the encoder, future frames are obtained via extrapolation, encoded and transmitted. They are then decoded and displayed while the corresponding frames are acquired at encoder. At the decoder, frames are extrapolated from previously received frames to compensate latency. With time-varying channels, encoding rate adaptation induces delivery delay jitter. Linear Video Coding (LVC) only relies on linear operators and delivers video to the decoder with a quality commensurate to that of the transmission channel. Moreover, LVC does not require any adaptation of the coding parameters to the channel characteristics, nor retransmission of corrupted data, which is beneficial in terms of latency. ZL-LVC will combine LVC and extrapolation schemes. This will, however, require a reduction of the encoding latency of LVC systems, which exploit temporal redundancy via 2D+T DCT over a group of pictures (GoP). This leads to a latency of the duration of the GoP and requires extrapolation on a relatively long horizon with a reduced overall quality. The considered use-case consists in sport cars driven on tracks and streaming video to base stations around the track. The aim is to propose a ZL video coding scheme compatible with the tele-operation of vehicles. In work package (WP) 1, baseline ZL schemes will be developed. The conventional video encoder design choices are explored that would best fit with a ZL scheme. Extrapolation at the encoder and/or at the decoder will be compared in point-to-point communication. To reduce the encoding latency of LVC schemes, while preserving their efficiency, image extrapolation combined with source coding techniques with side information at the decoder will be considered. WP2 addresses the design of extrapolation techniques accounting for compression and channel noise. A joint end-to-end optimization of the LVC video codec and frame extrapolation will be performed by representing them with an auto-encoder trained using perceptual loss measures. This approach will benefit from our previous experience on video compression using deep learning techniques. WP3 considers the design of an hybrid ZL scheme benefiting from the best of both conventional and LVC worlds. The adaptation of coding parameters of ZL schemes will be considered in response to variation of channel characteristics, so as to optimize the latency-rate-distortion trade-off. WP4 is dedicated to the ZL-LVC software library and demonstrators. ZL schemes will be compared, based on conventional video codecs, involving a 4G/5G access network, or involving an LVC implemented on a software-defined radio platform. WP0 will specify the simulation and demonstration conditions. Finally, WP5 is devoted to results dissemination & consortium management. ZL-LVC will provide original solutions for ZL video coding and transmission. This will be an ideal solution for applications where delays and robustness are critical, such as vehicle teleoperation. Results of ZL-LVC will be published on relevant scientific venues, presented in MPEG meetings, and patented. They will also lead to the development of solutions by the industrial partner to better address the market of low-delay and high reliability video delivery. Experience acquired within ZL-LVC will provide us the opportunity to initiate new collaborative projects of broader scope.