Projet eXtreme DataCloud XDC

Présentation au sujet: "Projet eXtreme DataCloud XDC"— Transcription de la présentation:

1 Projet eXtreme DataCloud XDC
Vincent Poireau, Nadine Neyroud Vincent Poireau

2 Appel d’offre e-infra21 Réponse à l’appel d’offre e-infra21 de la communauté européenne (H2020, horizon 2020) Appel d’offre: Computing e-infrastructure with extreme large datasets Develop service prototypes to cope with very large data resources Open source Common interfaces to access and analyse underlying data collected/stored in different platforms, formats, locations and e-infrastructures and be tested against requirements of very large or highly heterogeneous research data sets High-performance computing and data targeting 'data factory' requirements of research communities and large scale facilities (e.g. ESFRI projects) The successful prototyping of this action services will support the evolution of e-infrastructure services based on exascale data resources. It will prepare data and computing infrastructure to absorb needs of communities that push the envelope in terms of data and computing intensive science while softening the learning curve for scientific communities that will be using new services. Budget maximum : 3 millions d’euros Vincent Poireau

3 XDC Réunion à Bologne le 28 février – 1er mars pour définir le projet
Dossier rendu fin mars Les partenaires AGH-UST, CERN, CNRS-IN2P3, DESY, ECRIN, EGI, INFN, Université de Cantabria Au CNRS LAPP et CCIN2P3 Projet accepté en août (avec des corrections à apporter) Devrait commencer en novembre 2017 pour une durée de 27 mois Kickoff meeting : fin janvier 2018 Vincent Poireau

4 XDC XDC will develop scalable technologies for federating storage resources and managing data in highly distributed computing environments XDC will be based on existing tools that the project will enrich with new functionalities and plugins already available as prototypes Targeted platforms European Open Science Cloud (EOSC) The European Grid Infrastructure (EGI) The Worldwide LHC Computing Grid (WLCG) Topics addressed by the project Federation of storage resources with standard protocols Smart caching solutions Policy driven data management based on Quality of Service Data lifecycle management Metadata handling and manipulation Data preprocessing and encryption during ingestion Optimized data management based on access patterns All the developments driven and tested against real life use cases Life Science Astrophysics High Energy Physics Photon Science and Clinical Research Vincent Poireau

5 XDC use cases for CTA Two different use cases proposed
Intelligent and automated dataset distribution 4 data centers Different levels of data reconstruction with complex policies (disk vs tape, nb of versions, …) => XDC should be able to handle these constraints Data ingestion pre-processing Ingest file must extract metadata from file and build metadata database Metadata will be used for the further query of archive Able to manage replicas, tapes, disks Vincent Poireau

6 XDC au CNRS CCIN2P3 et LAPP y participent
LAPP : développer le use case pour CTA Un CDD (ingénieur en informatique) embauché au 6 novembre 2017 CCIN2P3 : e-infrastructure pilote pour XDC Vincent Poireau

7 Budget XDC Vincent Poireau