The increasing quantities of data generated by modern industrial and business processes pose enormous challenges for organizations seeking to glean knowledge and understanding from the data. Combinations of HPC, Cloud and Big Data technologies are key to meeting the increasingly diverse needs of large and small organizations alike. Critically, access to powerful compute platforms for SMEs - which has been difficult due to both technical and financial reasons - may now be possible. LEXIS (Large-scale EXecution for Industry & Society) project will build an advanced engineering platform at the confluence of HPC, Cloud and Big Data which will leverage large-scale geographically-distributed resources from existing HPC infrastructure, employ Big Data analytics solutions and augment them with Cloud services. Driven by the requirements of the pilots, the LEXIS platform will build on best of breed data management solutions (EUDAT) and advanced, distributed orchestration solutions (TOSCA), augmenting them with new, efficient hardware capabilities in the form of Data Nodes and federation, usage monitoring and accounting/billing supports to realize an innovative solution. The consortium will develop a demonstrator with a significant Open Source dimension including validation, test and documentation. It will be validated in the pilots - in the industrial and scientific sectors (Aeronautics, Earthquake and Tsunami, Weather and Climate) – where significant improvements in KPIs including job execution time and solution accuracy are anticipated. LEXIS will promote the solution to the HPC, Cloud and Big Data sectors maximizing impact through targeted and qualified communications. LEXIS brings together a consortium with the skills and experience to deliver a complex multi-faceted project, spanning a range of complex technologies across seven European countries, including large industry, flagship HPC centres, industrial and scientific compute pilot users, technology providers and SMEs.

The distributed and heterogeneous nature of the data sources in High Performance Big Data Analytics (HPDA) applications, as well as the required computational power, is pushing designers towards novel computing systems that combine HPC, Cloud, and IoT solutions (for efficient and distributed computation closer to the data) with Artificial Intelligence (AI) algorithms (for knowledge extraction and decision making). In this context, the EVEREST project addresses the matching problem between application (and data) requirements, and the characteristics of the underlying heterogeneous hardware. Only an optimal match leads to efficient computation. In particular, we forecast that the creation of future Big Data systems will be of course data-driven, but also featuring complex heterogeneous and reconfigurable architectures that must be redesigned or customized based on the nature and locality of the data, and the type of learning/decisions to be performed. The EVEREST project aims at developing a holistic approach for co-designing computation and communication in a heterogeneous, distributed, scalable and secure system for HPDA. This is achieved by simplifying the programmability of heterogeneous and distributed architectures through a “data-driven” design approach, the use of hardware-accelerated AI, and through an efficient monitoring of the execution with a unified hardware/software paradigm. EVEREST proposes a design environment that combines state-of-the-art, stable programming models, and emerging communication standards, with novel and dedicated domain-specific extensions. Three industry-relevant application scenarios are used to validate the EVEREST approach and act as business cases for the project exploitation: (i) a weather analysis-based prediction model for the renewable energy trading market, (ii) an application for air-quality monitoring of industrial sites, and (iii) a real-time traffic modeling framework for intelligent transportation in smart cities.