Nadine Neyroud , Cécile Barbier Data Management “L’Ingénieur Système” pour le Computing CTA Nadine Neyroud , Cécile Barbier Réunion de Service Janvier 2014

Le LAPP et le projet CTA Data Management

L'ingénieur système est un ingénieur pluridisciplinaire (génie informatique, génie civil, électronique, automatique, productique, etc.) chargé de spécifier ou de concevoir des systèmes complexes. Cette présentation concerne les activités de « System Engineering » pour la phase de pré-construction de CTA (=>2016-2017)

Process Elaborer les “Requirements fonctionnels et techniques” Définir les “Product & Work Breakdown Structures” Converger vers un “Design technique” pour tous les composants CTA Identifier les technologies critiques et les besoins en prototype Converger vers le choix des sites Investiguer l’implémentation, les coûts, le planning, l’organisation, la construction, les opérations, la maintenance et le decommissioning de CTA Identifier et quantifier toutes les activités et ressources nécessaires pour la construction et les opérations de CTA Spécifier les constributions individuelles des différents partenaires

Définir le PBS/WBS Tout le projet s’appuie sur cette nomenclature. Product Breakdown Structure: Décomposition du projet en sous projets logiques: Work Breakdown Structure: Décomposition de chaque sous-projet en tâches. Tout le projet s’appuie sur cette nomenclature.

Converger vers le Design technique Etape 1: Définir les requirements & Spécifications Etape 2: Le process de “Verification et de Validation” USER PROJET

Verification & Validation process Vérification: Comment démontrer que le Design est conforme aux spécifications? Validation: Comment démontrer que les spécifications sont conformes aux requirements? DATA Management est en charge de deux types de deliverables: Des Centre de Calcul et du Computing model associé pour le traitement des données off-line Des logiciels spécifiques à développer Pour la partie développement software: Data Management a retenu le standard de l’ESA PSS-05-0 Issue 2

La méthodologie ESA PSS-05-0 Issue 2 Validation Verification => Un document de 87 pages avec 200 specifications qui valident 67 Requirements

Validation

Verification

CTA Quality : RAMS Plan Dans cette phase de pré-construction: identification des risques et implémentation du process FMECA (Failure Mode, Effects and Criticality Analysis)

CTA Quality : RAMS Plan Dans cette phase de pré-construction: identification des risques et implémentation du process FMECA (Failure Mode, Effects and Criticality Analysis)

Conclusion Les ingénieurs systèmes du Project Office CTA n’ont aucune expérience en System Engineering pour la partie Computing et pas beaucoup pour le reste => Il a fallu essayer de comprendre, beaucoup inventer et refaire plusieurs fois le travail au fur et à mesure de l’évolution de la compréhension C’est un travail en concertation avec chaque responsable de sous-projet pour définir les specifications – pas toujours facile quand ils sont distribués sur toute l’Europe Une méthodologie contraignante mais qui permet de se poser quelques bonnes questions qui seraient arrivées très tard: Performance, Disponibilité, Modularité….. En cours: Interface Control Document, Software Policy

Réunion service informatique – 21 janvier 2014 Software Policy and Services Cécile BARBIER LAPP Réunion service informatique – 21 janvier 2014

CTA collaboration services Unique support contact (5 people): cta-support@cta-observatory.org Existing software tools: SVN (code repository) and Redmine (forge) in production at CC-IN2P3 User documentation in CTA SharePoint Directory structure organized by Work Packages (ACTL, DATA, LST, …) Read for all / Write allowed per project (validation by project managers) SVN (svn.in2p3.fr/cta) SSH access with public keys: 54 active users Redmine (forge.in2p3.fr/cta) Access with your CTA credentials (same as CTA Indico and SharePoint): 150 active users Internal ticketing system, wikis, forums SVN Read access

Redmine: CTA SVN repository

CTA collaboration services Future software tools: Jenkins (continuous integration) and Sonar (software quality): most popular at the moment Both already used at INSU/IRAP for ctools/GammaLib Will be integrated in Redmine Jenkins: See https://cta-jenkins.irap.omp.eu/ Used to build and test the code with various OS, Python, compilers … Test bed for CTA at CC-IN2P3 available with cloud for multi-OS support: https://ci.in2p3.fr Sonar: See https://cta-sonar.irap.omp.eu/ Used to measure quality of code: several metrics gathered in a dashboard and tips provided to improve the code

Jenkins build pipeline for GammaLib/ctools

Sonar dashboard for GammaLib

Software Policy Started after DATA&ACTL Review meeting (14/15 May 2013): Rec. 8. Software Engineering resources should be allocated in common between DATA and ACTL to establish and maintain SW engineering standards and practices. These include: coding standards, unit and integration test systems, common SW building and distribution tools, documentation, harmonization of external libraries and dependencies. First Software Policy meeting (11 September 2013): A survey was sent early August to all ACTL and DATA software managers: questions about currently used and future coding, documentation, testing and deployment tools, licenses … The 11 responses were used as a guideline for the meeting Existing wiki site in Redmine for discussion

Software Policy topics Languages, libraries and OS Coding rules Licensing IDE (Integrated Development Environment) Documentation Testing and deployment tools Logging and errors

Software Policy meeting decisions and next steps General: Write a first document from the minutes of Sept 11th meeting and produce an associated action plan Languages: Java/C++/Python emerging as main languages for future software developments Libraries: Issue new survey to see what kind of libraries are currently used in order to set up a common list of libraries Official running environment (target platform): Best effort from each team (reconstruction, on-site analysis …) to have something homogeneous (OS, compilers and common libraries) Frequency of change in production environment: at least 1 year between 2 versions

Software Policy meeting decisions and next steps Coding rules: Sonar includes predefined coding rules that could be used as a reference Common definition of terms required = CTA glossary under construction People already using coding rules (for example for HESS) could drive working groups Ask the software managers to fill in the wiki and then gather the common rules and the subprojects specific rules Licensing: Impossible to define a unique licensing solution Produce as much open-source software as possible Define a CTA header IDE: Eclipse recommended CTA configuration, users support and tutorials required

Software Policy meeting decisions and next steps Documentation: doxygen/javadoc tools for code documentation Rules and templates to be defined Testing tools: Common tool compatible with Jenkins required Software Managers must investigate what is compatible with Jenkins and start discussion Packaging/deployment tools: Automatic deployment process is required Each project must define their deployment tool(s) and directory structure(s) Errors/Logging: Already a huge topic for on-site computing (Current Big Data prototypes) Try to use common libraries

Software Policy conclusion To verify work packages compliance, it has been agreed to plan specific and periodic code reviews for each software product Official role in CTA but: Existing code and code under development that don’t follow yet specific rules Guideline required for developers but not too many constraints either Need to converge to common solutions and convince developers to follow them