Grilles de Calcul - Formation utilisateurs Lyon, 19 octobre 2010 Sylvain Reynaud

JSAGA : une implémentation de SAGA Applications utilisant JSAGA Plan Contexte exemples de grilles interopérabilité : motivations Standards de grille Open Grid Forum la spécification SAGA JSAGA : une implémentation de SAGA Applications utilisant JSAGA Perspectives JSAGA

Quelques grilles dans le monde GOS NAREGI middleware infrastructures JSAGA

Quelques grilles dans le monde middleware GOS NAREGI infrastructures JSAGA

Quelques grilles en Europe middleware GOS NAREGI infrastructures N JSAGA

Quelques grilles en Europe middleware GOS NAREGI infrastructures JSAGA

Quelques grilles en France JSAGA

Interopérabilité : motivations j'ai besoin de beaucoup de ressources batch SSH cluster JSAGA

Interopérabilité : motivations certains de mes jobs ont besoin de logiciels ou services particuliers batch SSH cluster JSAGA

Interopérabilité : motivations certains de mes jobs doivent tourner sur un supercalculateur batch SSH cluster JSAGA

Interopérabilité : motivations je n'utilise la grille que pour absorber des pics de charge batch SSH cluster JSAGA

Interopérabilité : motivations je veux éviter l'overhead de la grille pour certains jobs batch SSH cluster JSAGA

Interopérabilité : motivations je débugue en local avant de soumettre sur la grille fork batch SSH cluster JSAGA

Standards de grille Besoins et concepts communs Une communauté d'industriels et de chercheurs définit des standards pour l'architecture globale d'une grille (Open Grid Service Architecture) pour chaque composant d'un middleware de grille JSAGA

Sécurité Authentification Autorisation (décision partagée) grid-proxy-init VOMS CE, SE… Authentification Autorisation (décision partagée) Interopérer avec les infrastructures de sécurité locales (sans les compromettre) Délégation Négociation Isolation des utilisateurs Traçabilité, détection d’intrusion LCAS / LCMAPS JSAGA Source: The Open Grid Services Architecture, Version 1.5

Systèmes d'information Architectures et interfaces dépendent de… type d'information (fréquence de rafraîchissement, durée de vie des infos) type d'utilisation (critères de QoS différents) BDII Information Services Execution management Resource reservation Problem determination Accounting Application monitoring Load balancing Service discovery Registry Asynchronous notification Producers Consumers Retrieval Reliable Secure Efficient Logger APEL LB JSAGA Source: Hiro Kishimoto

Non-OGSA client APIs & other services Gestion des données Non-OGSA client APIs & other services FTS LFC, AMGA Gestion de stockage Accès aux données Gestion de réplicats Catalogues de méta- -données Transfert de données Gestion de cache Transfer Lookup Transfer Registries SRM Sink/ Source Storage Management Storage Data Management Other Data Services Description Access Access Description Sink/ Source Service interface Managed Storage Data Resources Data Resources Transfer Protocols Resource interface GridFTP JSAGA Source: The Open Grid Services Architecture, Version 1.5

Gestion de l'exécution Description du job Sélection de la ressource Soumission, suivi et contrôle du job Déploiement JDL Application Contents Service Config Desc, Deployment & Lifecycle Mgmt JSDL LCG-CE, CREAM-CE JSDL WMS Basic Execution Service Job Manager resource user BDII Information Service ? Resource Selection Service JSAGA Source: Keisuke Fukui (ACS-WG)

Interface utilisateur gLite-UI VOMS, MyProxy user WMS, LCG-CE, CREAM-CE SRM, gsiftp LFC, AMGA BDII JSAGA Source: S Jha, T Kielmann, A Merzky

Simple API for Grid Applications SAGA est une interface standard spécification de l' orientée objet simple pour 80% des besoins, 20% des fonctionnalités suffisent intégrée "look & feel" commun à tous les packages de l'API high-level les détails du fonctionnement du middleware sont cachés indépendant du middleware même interface pour tous les middlewares indépendant du langage Java C++ python (wrappers) stable spécification SAGA 1.0 (15 janvier 2008) extensible Service Discovery, Message… JSAGA

Simple API for Grid Applications Exemple de code en Java (data management) Session session = SessionFactory.createSession(true); URL url = URLFactory.createURL("gsiftp://cclcgseli01.in2p3.fr/tmp/"); // instanciation de l'objet SAGA à l'aide d'une fabrique NSDirectory dir = NSFactory.createNSDirectory(session, url); // utilisation de l'objet SAGA List<URL> result = dir.list(); for (URL r : result) System.out.println(r); liste le contenu du répertoire gsiftp://… JSAGA

Simple API for Grid Applications Exemple de code en Java (execution management) Session session = SessionFactory.createSession(true); URL url = URLFactory.createURL("wms://lapp-wms02.in2p3.fr/…"); // instanciation de l'objet SAGA à l'aide d'une fabrique JobService svc = JobFactory.createJobService(session, url); // utilisation de l'objet SAGA Job job = svc.runJob("/bin/date --utc"); job.waitFor(); System.out.println(job.getState()); soumet un job et attend la fin de son exécution JSAGA

Simple API for Grid Applications JSAGA

Simple API for Grid Applications Execution Data Security control monitor protocol security JSAGA

The JSAGA implementation a Java implementation of the SAGA specification focuses on uniform usage of middleware…. (beyond what they natively support) ease of extension………………. efficiency and scalability………. control of the API behavior........ (configure binding rather than trying) operating-system independency (tested on ) Under LGPL license (thanks to the design of adaptor interfaces) JSAGA

Uniform usage of existing grids JSAGA

Example – uniform usage SAGA job SAGA job user applications JSAGA VOMS WMS gsiftp Gatekeep. Globus GridFTP out.txt out.txt JSAGA

Example – what happens behind SAGA job user applications EGEE proxy + attributes translate submit job poll status delegate transfer DGrid proxy adapt + translate submit job receive status transfer to any protocol modified job JSAGA wrapper script VOMS WMS gsiftp Gatekeep. Globus EGEE D-Grid JDL RSL VOMS WMS GridFTP LCG-CE CREAM-CE Globus GK Unicore out.txt out.txt JSAGA

Layered software architecture interface Layered software architecture implementation JSAGA JSAGA

Layered software architecture interface Layered software architecture implementation Used by end users web portals, GUI, CLI… Used by application developers a single uniform interface object oriented, high-level Implemented by adaptors each "way" to implement a given feature has 1 interface service oriented, low-level Implemented by middleware each technology has its own interfaces user applications SAGA Java binding core engine JSAGA adaptors interfaces JSAGA adaptors JSAGA middleware APIs JSAGA

Layered software architecture Used by end users web portals, GUI, CLI… Used by application developers a single uniform interface object oriented, high-level Implemented by adaptors each "way" to implement a given feature has 1 interface service oriented, low-level Implemented by middleware each technology has its own interfaces user applications transfer output listen status changes JSAGA core engine JSAGA adaptor interfaces adaptor adaptor GridFTP Globus GK upload data get output get status JSAGA

Layered software architecture Used by end users web portals, GUI, CLI… Used by application developers a single uniform interface object oriented, high-level Implemented by adaptors each "way" to implement a given feature has 1 interface service oriented, low-level Implemented by middleware each technology has its own interfaces user applications transfer output listen status changes JSAGA core engine JSAGA adaptor interfaces adaptor adaptor GridFTP Globus GK upload data get output get status JSAGA

Layered software architecture Middleware API can be used more efficiently engine selects best interface for each user request Developing new adaptors is easier just wrap functionalities supported by middleware API user applications transfer output listen status changes JSAGA core engine JSAGA adaptor interfaces adaptor adaptor GridFTP Globus GK upload data get output get status JSAGA

Design of adaptors interfaces As many interfaces as ways to implement each functionality Example: ways to monitor jobs API mode poll job status listen to notifications about job status changes API granularity individual jobs list of jobs jobs filter (e.g. by user, by date, by tag…) SAGA getState waitFor adaptors interfaces JSAGA query status for individual job listen status for individual job query status for filtered jobs Monitoring: polling/notification, requête individuelle/liste/filtre Streaming: avant/pendant/après l'exécution, direct/buffered/redirigé Job monitoring gatekeeper gLite-WMS wsgram unicore6 cream naregi ssh local planned construction done JSAGA

Design of adaptors interfaces Optional interfaces for optimization Example: ways to copy a file data read/write methods either stream methods or get/put methods data copy e.g. enable third-party transfer delegated transfer SAGA copy adaptors interfaces JSAGA data reader data writer data copy Physical rbyteio gsiftp srm srb irods http https sftp ftp file zip cache planned construction done JSAGA

Supported technologies Grid techno. Others Execution gatekeeper gLite-WMS wsgram unicore6 cream naregi ssh local Grid techno. Others Data Logical Physical rns lfn srb irods catalog rbyteio gsiftp srm srb irods http https sftp ftp file zip cache Grid techno. Others Minimal dependencies on external libraries external tools e.g. no gLite-UI operating system tested on Security InMemCred Globus G. Legacy G. RFC820 MyProxy VOMS JKS SSH Login / pwd X509 planned construction done JSAGA

Software quality Adaptors Automated build process validated by a middleware-independent SAGA test suite maven 'archetype' to generate skeleton of new adaptor project Automated build process download and install build tools external libraries generate source code execute test-suites unitary tests integration tests generate project web site documentation reports generate installer GUI extract library dependencies from maven project description # SAGA protocols test-suite configuration gsiftp.base=gsiftp://ccrugceli01.in2p3.fr/tmp/ gsiftp.base2=gsiftp://agena.c-s.fr/grid/tmp/ gsiftp.context=OpenPlast_proxy https.base=http://grid.in2p3.fr/html/Private/ https.context=Web_X509 file.base=file:///c:/tmp/ file.base2=file:///c:/ JSAGA

Installer GUI JSAGA

Licenses LGPL license Optional licenses for the core engine and for most adaptors Optional licenses for adaptors having external dependencies, which license is not compatible with LGPL then, end-user must… either accept the terms of the license agreement, or go back to previous screen and uncheck the adaptor(s) JSAGA

Simplicité versus contrôle Essayer tous les contextes attractif pour les débutants problème d'efficacité gaspillage de temps gaspillage de ressources (risque d'échec à la fin du job) problème de fiabilité risque de créer les fichiers avec le mauvais propriétaire risque de bloquer un compte (si trop de tentatives échouent) Associer les URL aux contextes contrôle du comportement nécessite qq. connaissances sur les infrastructures utilisées la configuration de JSAGA Le nouveau moteur de configuration permet configuration dynamique supporte 1 configuration/session détecte les conflits à la config. JSAGA

N Démonstration Contextes de sécurité utilisés SRB **** iRods https Password: X509 **** VO Rhône-Alpes VO dteam CA allemande CA perso SSH N iRods SRB https JSAGA

Coexistence of SAGA implementations several implementations different languages (sometimes) different engines different context selection mechanisms (trying / configuring) different adaptor interfaces (high-level / low-level interfaces) … different supported technologies (i.e. adaptors) how to use all these technologies from a grid application ? Application Application C++ Bindings SAGA S A G A Java Bindings for SAGA SAGA-C++ JSAGA Java SAGA JSAGA

Coexistence of SAGA implementations C Python Jython Application Application S A G A Python Bindings for SAGA (PySAGA) CppySAGA JPySAGA JySAGA C++ Bindings SAGA Java Bindings for SAGA SAGA-C++ JSAGA Java SAGA JSAGA

Example applications DIRAC (Distributed Infrastructure with Remote Agent Control) the pilot jobs system of LHCb experiment Pandora Gateway the grid platform of the company a Grid/Cloud Computing Business Services Provider JUX (Java Universal eXplorer) a multi-protocols extensible file explorer JSAGA

/ Example applications Elis@ JJS (Java Job Submission) SimExplorer web portal for industrial and academic production grids JJS (Java Job Submission) tools for submitting jobs to EGI resource selection based on QoS observed while submitting jobs SimExplorer software for simulation experiment management includes a workflow engine that uses JSAGA to run simulations on grids jCAE (Java CAE) Computer Aided Engineering with support for distributed computing facilities / JSAGA

Example applications JSAGA provides command line interfaces for… security jsaga-context-init jsaga-context-info jsaga-context-destroy execution management jsaga-job-run jsaga-job-status jsaga-job-cancel jsaga-job-output Also useful as code examples to start with SAGA data management jsaga-cat jsaga-cp jsaga-ls jsaga-mkdir jsaga-mv jsaga-rm jsaga-rmdir jsaga-stat jsaga-test jsaga-logical JSAGA

Perspectives Develop new plug-ins Develop API extensions NorduGrid middleware ARC plug-in in progress… LRMS (DRMAA-based) Grid Engine LRMS (CLI-based/ssh) torque Develop API extensions Grid-RPC (from SAGA core) Service Discovery API (a SAGA extension) with plug-ins for technologies BDII with plug-ins for schemas GLUE CIM JSAGA

Perspectives Job submission Remote Procedure Call (RPC) job Grid-RPC principle describe user's job submit it to the grid middleware select the execution host OGF standards BES, DRMAA, SAGA examples Remote Procedure Call (RPC) principle deploy user's service on grid call a procedure middleware select the service instance OGF standards Grid-RPC, SAGA examples JSAGA

Questions ? JSAGA