The digital transformation and the availability of more diverse and cost-effective methods for 3D capture has led to the creation of digitised representations of parts of our public spaces, machinery and processes, so-called Digital Twins. These Digital Twins are an important basis for building advanced XR applications for cityscapes and the industrial environments. These XR applications need high-fidelity models for accurate localization, which are kept in-sync with the real world. However, currently (i) creating high-fidelity Digital Twins is costly, (ii) integrating data from heterogeneous sensors is still challenging and (iii) the update of Digital Twins requires explicit manual intervention. The vision of DIDYMOS-XR is to enable advanced, more realistic and more dynamic extended reality (XR) applications, powered through artificial intelligence. The project thus focuses on advancing technologies for creating large-scale digital twins, synchronised with the real world. DIDYMOS-XR will research and develop methods for reconstruction and mapping from heterogeneous inputs, including static and mobile sensors, AI-based data fusion, scene understanding and rendering. Digtal twin based XR applications also require means of accurately positioning in the environment, aware of the dynamics of the scene, also addresed by the project. The capture of scenes at scale, as well as using cameras and other sensor data for synchronising the digital representation, bears the risk of capturing personal and sensitive data. Hence, the technologies resulting from DIDYMOS-XR must be and will be ethical and privacy-aware by design. In order to demonstrate and validate the enabling technologies researched and developed in DIDYMOS-XR, the project will address use cases of XR applications in two domains that differ in their scale and characteristics. One is urban planning and smart mobility, the other is the collaboration with autonomous mobile robots at an industrial production facility.

The concept of presence can be understood as a synthesis of interrelated psychophysical ingredients where multiple perceptual dimensions intervene. A better understanding on how specific aspects, such as plausibility (the illusion that virtual events are really happening), co-presence (the illusion of being with others), or place illusion (the feeling of being there) impact XR experiences is key to improve their quality. The availability and performance of current technologies do not reach high levels of presence in XR, which is essential to get us closer than ever to the perennial VR dream: to be anywhere, doing anything, together with others, from any place. PRESENCE will impact multiple dimensions of presence in physical-digital worlds, addressing three main challenges: i) how to create realistic visual interactions among remote humans, delivering high-end holoportation based on live volumetric capturing, compression and optimization techniques, under heterogeneous computation and network conditions; ii) how to provide realistic touch among remote users and synthetic objects, developing novel haptic systems and enabling spatial multi device synchronisation in multi user scenarios; iii) how to produce realistic social interactions among avatars and agents, generating AI virtual humans, representing actual users or AI agents. PRESENCE will ensure the future uptake of research results following a threefold evaluation method: 1) each technology will be independently evaluated to understand its impact on the illusion of presence; 2) each component will be evaluated by the integration team, providing scientific and technical feedback in order to facilitate their use in each project iteration as well as beyond the project scope, towards technology transfer and exploitation; 3) all components will be integrated in two demonstrators (professional and social setups), following a human-centred design approach and ultimately evaluating the user experience.

While media organisations increasingly support non-linear experiences for the consumer, those are still limited to single channels and media domains. Although several media organisations have recently succeeded in breaking data silos, data sharing is mostly limited to the organisation. Thus, there are challenges for producing content feeding multiple channels with different granularities and structures, mainly related to data discovery, management and (re-)use. XRECO will create a new data-driven ecosystem for the media industry, focusing on facilitating data sharing, search and discovery and supporting creation of news and entertainment content, in particular, the creation and (re-)use of location-related 2D and 3D assets and the creation of XR experiences. The ecosystem core, represented by a Neural Media Repository (NMR), will foster inter-organisation content sharing and provide increased access to content for media creators, considering novel data monetization and rights management policies. A set of AI-based media transformation services are built around the NMR to produce novel media- and XR experiences, including 3D neural reconstructions, neural based device localisation, image stitching, de-/re-lighting and holoportation. The developed technology will be validated in use case scenarios for (i) the news media for XR-based broadcasting and automatic and customized multitarget news publishing, and for (ii) location-based information and entertainment content, with applications in tourism and the automotive industry.