Grilles de Calcul - Formation utilisateurs

Grilles de Calcul - Formation utilisateurs
Lyon, 19 octobre 2010 Sylvain Reynaud

JSAGA : une implémentation de SAGA Applications utilisant JSAGA
Plan Contexte exemples de grilles interopérabilité : motivations Standards de grille Open Grid Forum la spécification SAGA JSAGA : une implémentation de SAGA Applications utilisant JSAGA Perspectives JSAGA

Quelques grilles dans le monde
GOS NAREGI middleware infrastructures JSAGA

Quelques grilles dans le monde
middleware GOS NAREGI infrastructures JSAGA

Quelques grilles en Europe
middleware GOS NAREGI infrastructures N JSAGA

Quelques grilles en Europe
middleware GOS NAREGI infrastructures JSAGA

Quelques grilles en France
JSAGA

Interopérabilité : motivations
j'ai besoin de beaucoup de ressources batch SSH cluster JSAGA

certains de mes jobs ont besoin de logiciels ou services particuliers batch SSH cluster JSAGA

certains de mes jobs doivent tourner sur un supercalculateur batch SSH cluster JSAGA

je n'utilise la grille que pour absorber des pics de charge batch SSH cluster JSAGA

je veux éviter l'overhead de la grille pour certains jobs batch SSH cluster JSAGA

je débugue en local avant de soumettre sur la grille fork batch SSH cluster JSAGA

Standards de grille Besoins et concepts communs
Une communauté d'industriels et de chercheurs définit des standards pour l'architecture globale d'une grille (Open Grid Service Architecture) pour chaque composant d'un middleware de grille JSAGA

Sécurité Authentification Autorisation (décision partagée)
grid-proxy-init VOMS CE, SE… Authentification Autorisation (décision partagée) Interopérer avec les infrastructures de sécurité locales (sans les compromettre) Délégation Négociation Isolation des utilisateurs Traçabilité, détection d’intrusion LCAS / LCMAPS JSAGA Source: The Open Grid Services Architecture, Version 1.5

Systèmes d'information
Architectures et interfaces dépendent de… type d'information (fréquence de rafraîchissement, durée de vie des infos) type d'utilisation (critères de QoS différents) BDII Information Services Execution management Resource reservation Problem determination Accounting Application monitoring Load balancing Service discovery Registry Asynchronous notification Producers Consumers Retrieval Reliable Secure Efficient Logger APEL LB JSAGA Source: Hiro Kishimoto

Non-OGSA client APIs & other services
Gestion des données Non-OGSA client APIs & other services FTS LFC, AMGA Gestion de stockage Accès aux données Gestion de réplicats Catalogues de méta- -données Transfert de données Gestion de cache Transfer Lookup Transfer Registries SRM Sink/ Source Storage Management Storage Data Management Other Data Services Description Access Access Description Sink/ Source Service interface Managed Storage Data Resources Data Resources Transfer Protocols Resource interface GridFTP JSAGA Source: The Open Grid Services Architecture, Version 1.5

Gestion de l'exécution Description du job Sélection de la ressource
Soumission, suivi et contrôle du job Déploiement JDL Application Contents Service Config Desc, Deployment & Lifecycle Mgmt JSDL LCG-CE, CREAM-CE JSDL WMS Basic Execution Service Job Manager resource user BDII Information Service ? Resource Selection Service JSAGA Source: Keisuke Fukui (ACS-WG)

Interface utilisateur
gLite-UI VOMS, MyProxy user WMS, LCG-CE, CREAM-CE SRM, gsiftp LFC, AMGA BDII JSAGA Source: S Jha, T Kielmann, A Merzky

Simple API for Grid Applications
SAGA est une interface standard spécification de l' orientée objet simple pour 80% des besoins, 20% des fonctionnalités suffisent intégrée "look & feel" commun à tous les packages de l'API high-level les détails du fonctionnement du middleware sont cachés indépendant du middleware même interface pour tous les middlewares indépendant du langage Java C++ python (wrappers) stable spécification SAGA 1.0 (15 janvier 2008) extensible Service Discovery, Message… JSAGA

Exemple de code en Java (data management) Session session = SessionFactory.createSession(true); URL url = URLFactory.createURL("gsiftp://cclcgseli01.in2p3.fr/tmp/"); // instanciation de l'objet SAGA à l'aide d'une fabrique NSDirectory dir = NSFactory.createNSDirectory(session, url); // utilisation de l'objet SAGA List<URL> result = dir.list(); for (URL r : result) System.out.println(r); liste le contenu du répertoire gsiftp://… JSAGA

Exemple de code en Java (execution management) Session session = SessionFactory.createSession(true); URL url = URLFactory.createURL("wms://lapp-wms02.in2p3.fr/…"); // instanciation de l'objet SAGA à l'aide d'une fabrique JobService svc = JobFactory.createJobService(session, url); // utilisation de l'objet SAGA Job job = svc.runJob("/bin/date --utc"); job.waitFor(); System.out.println(job.getState()); soumet un job et attend la fin de son exécution JSAGA

JSAGA

Execution Data Security control monitor protocol security JSAGA

The JSAGA implementation
a Java implementation of the SAGA specification focuses on uniform usage of middleware…. (beyond what they natively support) ease of extension………………. efficiency and scalability………. control of the API behavior (configure binding rather than trying) operating-system independency (tested on ) Under LGPL license (thanks to the design of adaptor interfaces) JSAGA

Uniform usage of existing grids
JSAGA

Example – uniform usage
SAGA job SAGA job user applications JSAGA VOMS WMS gsiftp Gatekeep. Globus GridFTP out.txt out.txt JSAGA

Example – what happens behind
SAGA job user applications EGEE proxy + attributes translate submit job poll status delegate transfer DGrid proxy adapt + translate submit job receive status transfer to any protocol modified job JSAGA wrapper script VOMS WMS gsiftp Gatekeep. Globus EGEE D-Grid JDL RSL VOMS WMS GridFTP LCG-CE CREAM-CE Globus GK Unicore out.txt out.txt JSAGA

Layered software architecture
interface Layered software architecture implementation JSAGA JSAGA

interface Layered software architecture implementation Used by end users web portals, GUI, CLI… Used by application developers a single uniform interface object oriented, high-level Implemented by adaptors each "way" to implement a given feature has 1 interface service oriented, low-level Implemented by middleware each technology has its own interfaces user applications SAGA Java binding core engine JSAGA adaptors interfaces JSAGA adaptors JSAGA middleware APIs JSAGA

Used by end users web portals, GUI, CLI… Used by application developers a single uniform interface object oriented, high-level Implemented by adaptors each "way" to implement a given feature has 1 interface service oriented, low-level Implemented by middleware each technology has its own interfaces user applications transfer output listen status changes JSAGA core engine JSAGA adaptor interfaces adaptor adaptor GridFTP Globus GK upload data get output get status JSAGA

Middleware API can be used more efficiently engine selects best interface for each user request Developing new adaptors is easier just wrap functionalities supported by middleware API user applications transfer output listen status changes JSAGA core engine JSAGA adaptor interfaces adaptor adaptor GridFTP Globus GK upload data get output get status JSAGA

Design of adaptors interfaces
As many interfaces as ways to implement each functionality Example: ways to monitor jobs API mode poll job status listen to notifications about job status changes API granularity individual jobs list of jobs jobs filter (e.g. by user, by date, by tag…) SAGA getState waitFor adaptors interfaces JSAGA query status for individual job listen status for individual job query status for filtered jobs Monitoring: polling/notification, requête individuelle/liste/filtre Streaming: avant/pendant/après l'exécution, direct/buffered/redirigé Job monitoring gatekeeper gLite-WMS wsgram unicore6 cream naregi ssh local planned construction done JSAGA

Design of adaptors interfaces
Optional interfaces for optimization Example: ways to copy a file data read/write methods either stream methods or get/put methods data copy e.g. enable third-party transfer delegated transfer SAGA copy adaptors interfaces JSAGA data reader data writer data copy Physical rbyteio gsiftp srm srb irods http https sftp ftp file zip cache planned construction done JSAGA

Supported technologies
Grid techno. Others Execution gatekeeper gLite-WMS wsgram unicore6 cream naregi ssh local Grid techno. Others Data Logical Physical rns lfn srb irods catalog rbyteio gsiftp srm srb irods http https sftp ftp file zip cache Grid techno. Others Minimal dependencies on external libraries external tools e.g. no gLite-UI operating system tested on Security InMemCred Globus G. Legacy G. RFC820 MyProxy VOMS JKS SSH Login / pwd X509 planned construction done JSAGA

Software quality Adaptors Automated build process
validated by a middleware-independent SAGA test suite maven 'archetype' to generate skeleton of new adaptor project Automated build process download and install build tools external libraries generate source code execute test-suites unitary tests integration tests generate project web site documentation reports generate installer GUI extract library dependencies from maven project description # SAGA protocols test-suite configuration gsiftp.base=gsiftp://ccrugceli01.in2p3.fr/tmp/ gsiftp.base2=gsiftp://agena.c-s.fr/grid/tmp/ gsiftp.context=OpenPlast_proxy https.base= https.context=Web_X509 file.base=file:///c:/tmp/ file.base2=file:///c:/ JSAGA

Installer GUI JSAGA

Licenses LGPL license Optional licenses
for the core engine and for most adaptors Optional licenses for adaptors having external dependencies, which license is not compatible with LGPL then, end-user must… either accept the terms of the license agreement, or go back to previous screen and uncheck the adaptor(s) JSAGA

Simplicité versus contrôle
Essayer tous les contextes attractif pour les débutants problème d'efficacité gaspillage de temps gaspillage de ressources (risque d'échec à la fin du job) problème de fiabilité risque de créer les fichiers avec le mauvais propriétaire risque de bloquer un compte (si trop de tentatives échouent) Associer les URL aux contextes contrôle du comportement nécessite qq. connaissances sur les infrastructures utilisées la configuration de JSAGA Le nouveau moteur de configuration permet configuration dynamique supporte 1 configuration/session détecte les conflits à la config. JSAGA

N Démonstration Contextes de sécurité utilisés SRB **** iRods https
Password: X509 **** VO Rhône-Alpes VO dteam CA allemande CA perso SSH N iRods SRB https JSAGA

Coexistence of SAGA implementations
several implementations different languages (sometimes) different engines different context selection mechanisms (trying / configuring) different adaptor interfaces (high-level / low-level interfaces) … different supported technologies (i.e. adaptors) how to use all these technologies from a grid application ? Application Application C++ Bindings SAGA S A G A Java Bindings for SAGA SAGA-C++ JSAGA Java SAGA JSAGA

Coexistence of SAGA implementations
C Python Jython Application Application S A G A Python Bindings for SAGA (PySAGA) CppySAGA JPySAGA JySAGA C++ Bindings SAGA Java Bindings for SAGA SAGA-C++ JSAGA Java SAGA JSAGA

Example applications DIRAC (Distributed Infrastructure with Remote Agent Control) the pilot jobs system of LHCb experiment Pandora Gateway the grid platform of the company a Grid/Cloud Computing Business Services Provider JUX (Java Universal eXplorer) a multi-protocols extensible file explorer JSAGA

/ Example applications Elis@ JJS (Java Job Submission) SimExplorer
web portal for industrial and academic production grids JJS (Java Job Submission) tools for submitting jobs to EGI resource selection based on QoS observed while submitting jobs SimExplorer software for simulation experiment management includes a workflow engine that uses JSAGA to run simulations on grids jCAE (Java CAE) Computer Aided Engineering with support for distributed computing facilities / JSAGA

Example applications JSAGA provides command line interfaces for…
security jsaga-context-init jsaga-context-info jsaga-context-destroy execution management jsaga-job-run jsaga-job-status jsaga-job-cancel jsaga-job-output Also useful as code examples to start with SAGA data management jsaga-cat jsaga-cp jsaga-ls jsaga-mkdir jsaga-mv jsaga-rm jsaga-rmdir jsaga-stat jsaga-test jsaga-logical JSAGA

Perspectives Develop new plug-ins Develop API extensions
NorduGrid middleware ARC plug-in in progress… LRMS (DRMAA-based) Grid Engine LRMS (CLI-based/ssh) torque Develop API extensions Grid-RPC (from SAGA core) Service Discovery API (a SAGA extension) with plug-ins for technologies BDII with plug-ins for schemas GLUE CIM JSAGA

Perspectives Job submission Remote Procedure Call (RPC) job Grid-RPC
principle describe user's job submit it to the grid middleware select the execution host OGF standards BES, DRMAA, SAGA examples Remote Procedure Call (RPC) principle deploy user's service on grid call a procedure middleware select the service instance OGF standards Grid-RPC, SAGA examples JSAGA

Questions ? JSAGA

Grilles de Calcul - Formation utilisateurs

Présentations similaires

Présentation au sujet: "Grilles de Calcul - Formation utilisateurs"— Transcription de la présentation:

Présentations similaires

Notre projet

Feed-back

Entrer

S'autoriser via un réseau social:

Grilles de Calcul - Formation utilisateurs

Présentations similaires

Présentation au sujet: "Grilles de Calcul - Formation utilisateurs"— Transcription de la présentation:

Présentations similaires

Notre projet

Feed-back