Personalisation of media experiences for the individual is vital for audience engagement of young and old, allowing more meaningful encounters tailored to their interest, making them part of the story, and increasing accessibility. The goal of the BBC Prosperity Partnership is to realise a transformation to future personalised content creation and delivery at scale for the public at home or on the move. Evolution of mass-media audio-visual 'broadcast' content (news, sports, music, drama) has moved increasingly towards Internet delivery, which creates exciting potential for hyper-personalised media experiences delivered at scale to mass audiences. This radical new user-centred approach to media creation and delivery has the potential to disrupt the media landscape by directly engaging individuals at the centre of their experience, rather than predefining the content as with existing media formats (radio, TV, film). This will allow a new form of user-centred media experience which dynamically adapts to the individual, their location, the media content and producer storytelling intent, together with the platform/device and the network/compute resources available for rendering the content.The BBC Prosperity Partnership will position the BBC at the forefront of this 'Personalised Media' revolution enabling the creation and delivery of new services, and positioning the UK creative industry to lead future personalised media creation and intelligent network distribution to render personalised experiences for everyone anywhere. Realisation of personalised experiences at scale presents three fundamental research challenges: capture of object-based representations of the content to enable dynamic adaption for personalisation at the point of rendering; production to create personalised experiences which enhance the perceived quality of experience for each user; and delivery at scale with intelligent utilisation of the available network, edge and device resources for mass audiences. The BBC Prosperity Partnership will address the major technical and creative challenges to delivering user-centred personalised audience experiences at scale. Advances in audio-visual AI for machine understanding of captured content will enable the automatic transformation of captured 2D video streams to an object-based media (OBM) representation. OBM will allow adaptation for efficient production, delivery and personalisation of the media experience whilst maintaining the perceived quality of the captured video content. To deliver personalised experiences to audiences of millions requires transformation of media processing and distribution architectures into a hybrid and distributed low-latency computation platform, allowing flexible deployment of compute-intensive tasks across the network. This will achieve efficiency in terms of cost and energy use, while providing optimal quality of experience for the audience within the technical constraints of the system.

The COVID-19 pandemic has changed our lifestyle and caused high demand for remote communication and experience. Many organizations have had to set up remote work systems with video conferencing platforms. However, current video conferencing systems do not meet basic requirements for remote collaboration due to the lack of eye contact, gaze awareness and spatial audio synchronisation. Reproduction of a real space as an audio-visual 3D model allows users to remotely experience real-time interaction in real environments, thus it can be widely utilised in various applications such as healthcare, teleconferencing, education, entertainments, etc. The goal of this project is to develop a simple and practical solution to estimate geometrical structure and acoustic properties of general scenes allowing spatial audio to be adapted to the environment and listener location to give an immersive rendering of the scene to improve user experience. Existing 3D scene reproduction systems have two problems. (i) Audio and vision systems have been researched separately. Computer vision research has mainly focused on improving the visual side of scene reconstruction. In an immersive display, such as a VR system, the experience is not perceived as "realistic" by users if sound is not matched with the visual cues. On the other hand, audio researches have been using only audio sensors to measure acoustic properties without considering the complementary effect with visual sensors. (ii) Current capture and recording systems for 3D scene reproduction require too invasive set up and professional process to be deployed by users in their private places. A LiDAR sensor is expensive and requires long scanning time. Perspective images require large number of photos to cover the whole scene. The objective of this research is to develop an end-to-end audio-visual 3D scene reproduction pipeline using a single shot from a consumer 360 (panoramic) camera. In order to make the system easily accessible by common users in their own private spaces, automatic solution using computer vision and artificial intelligence algorithms should be included in the back-end. A deep neural network (DNN) jointly trained for semantic scene reconstruction and acoustic property prediction for the captured environments will be developed. This process includes inference for invisible regions from the camera. Impulse Responses (IRs) characterising acoustic attributes of an environment allow to reproduce the acoustics of the space with any sound sources. It also allows to extract the original (dry) sound by eliminating acoustic effects from recorded sound so that this source can be re-rendered in new environments with different acoustic effects. A simple and efficient method to estimate acoustic IRs from the captured single 360 photo will be investigated. This semantic scene data is used to provide immersive audio-visual experience to users. Two types of display scenarios will be considered: personalised display system such as a VR headset with headphones and communal display system (e.g., TV or projector) with loudspeakers. Real-time 3D human pose tracking using a single 360 camera will be developed to accurately render 3D audio-visual scene at the locations of users. Delivering binaural sound to listeners using loudspeakers is a challenging task. Audio beam-forming techniques aligned with human-pose tracking for multiple loudspeakers will be investigated in collaboration with the project partners in audio processing. The resulting system would have a significant impact on innovation of VR and multimedia systems, and open up new and interesting applications for their deployment. This award should provide the foundation for the PI to establish and lead a group with a unique research direction which is aligned with national priorities and will address a major long-term research challenge.