Présentation des grilles de calcul et initiation à la grille de calcul EGEE Lyon, 17 mars 2010 Sylvain Reynaud.

Présentation au sujet: "Présentation des grilles de calcul et initiation à la grille de calcul EGEE Lyon, 17 mars 2010 Sylvain Reynaud."— Transcription de la présentation:

1 Présentation des grilles de calcul et initiation à la grille de calcul EGEE Lyon, 17 mars 2010 Sylvain Reynaud

2 JSAGA2 La grille dans le monde middleware infrastructures GOSNAREGI

3 JSAGA3 La grille en Europe (bientôt) middleware infrastructures GOSNAREGI

4 JSAGA4 La grille dans les NGI (exemples) middleware infrastructures GOSNAREGI

5 JSAGA5 Standardisation Besoins et concepts communs Une communauté d'industriels et de chercheurs définit des standards –pour l'architecture globale d'une grille (Open Grid Service Architecture) –pour chaque composant d'un middleware de grille

6 JSAGA6 Sécurité Source: The Open Grid Services Architecture, Version 1.5 grid- proxy-init VOMS LCAS / LCMAP S CE, SE… Authentification Autorisation (décision partagée) Interopérer avec les infrastructures de sécurité locales (sans les compromettre) DélégationNégociation Isolation des utilisateurs Traçabilité, détection d’intrusion

7 JSAGA7 Systèmes d'information Execution management Resource reservation Problem determination Accounting Application monitoring Load balancing Service discovery Consumers Information Services Reliable Secure Efficient Producers Asynchronous notification Retrieval Registry Logger Source: Hiro Kishimoto BDII R-GMA LB Architectures et interfaces dépendent de… type d'information (statique/dynamique, fréquence de rafraîchissement) type d'utilisation (critères de QoS différents)

8 JSAGA8 Gestion des données Data Resources Managed Storage Data Resources SRM Transfer Protocols Storage Managemen t Data Managemen t Other Data Services Transfer Registries Service interface Resource interface Source: The Open Grid Services Architecture, Version 1.5 Access Sink/ Source Description Access Sink/ Source Description TransferLookup Storage Non-OGSA client APIs & other services Gestion de stockage Accès aux données Gestion de réplicats Catalogues de méta- -données Transfert de données Gestion de cache FTS LFC,AMGA GridFTP

9 JSAGA9 Gestion de l'exécution WMS Job Manager Application Contents Service Config Desc, Deployment & Lifecycle Mgmt Resource Selection Service Basic Execution Service resourceuser Information Service ? ? ? JSDL Description du job Sélection de la ressource Soumission, suivi et contrôle du job Déploiement Source: Keisuke Fukui (ACS-WG) LCG-CE,CREAM-CE BDII JDL

10 JSAGA10 Interface utilisateur Source: S Jha, T Kielmann, A Merzky user gLite-UI VOM S WMS,LCG-CE,CREAM-CE SRM,gsiftpLFC, AMG A BDII

11 JSAGA11 Interface utilisateur SAGA est une interface –standard spécification de l' –orientée objet –simple pour 80% des besoins, 20% des fonctionnalités suffisent –intégrée "look & feel" commun à tous les packages de l'API –high-level les détails du fonctionnement du middleware sont cachés –uniforme indépendante du middleware sous-jacent –multi-langages implémentations Java et C++ wrappers pour python –stable spécification SAGA 1.0 (15 janvier 2008) bientôt un 1 er update mineur (ne casse pas la compatibilité ascendante)

12 JSAGA12 Interface utilisateur Exemple de code en Java (data management) Session session = SessionFactory.createSession(true); URL url = URLFactory.createURL(" gsiftp://cclcgseli01.in2p3.fr/tmp/ "); // instanciation de l'objet SAGA à l'aide d'une fabrique NSDirectory dir = NSFactory.createNSDirectory(session, url); // utilisation de l'objet SAGA List result = dir.list(); for (URL r : result) System.out.println(r); liste le contenu du répertoire gsiftp://…

13 JSAGA13 Interface utilisateur Exemple de code en Java (execution management) Session session = SessionFactory.createSession(true); URL url = URLFactory.createURL(" wms://lapp-wms02.in2p3.fr/… "); // instanciation de l'objet SAGA à l'aide d'une fabrique JobService svc = JobFactory.createJobService(session, url); // utilisation de l'objet SAGA Job job = svc.runJob("/bin/date --utc"); job.waitFor(); System.out.println(job.getState()); soumet un job et attend la fin de son exécution

14 JSAGA14 JSAGA cluster ( ) JSAGA disponibilité de logiciels payants confidentialit é pour certains traitements faible latence (e.g. job de consolidation), interactivité débuguage, mise au point du job accès grille de super-calculateurs augmenter le nombre de ressources accessibles

15 JSAGA15 VOMS WMS gsiftp JSAGA Gatekeep. Globus JSAGA GridFTP CREAM-CELCG-CE Unicore WMS WMS Gatekeeper SAGA job D-Grid VOMS VOMS jsaga-context-init EGEE

16 JSAGA16 VOMS WMS gsiftp JSAGA Gatekeep. Globus JSAGA CREAM-CELCG-CE Unicore WMS WMS Gatekeeper SAGA job JDL D-Grid VOMS VOMS jsaga-job-run–Executable /bin/hostname –Output out.txt -FileTransfer gsiftp://…/out.txt<out.txt -r wms://lapp-wms02.in2p3.fr:7443/glite_wms_wmproxy_server -r wms://lapp-wms02.in2p3.fr:7443/glite_wms_wmproxy_server EGEE out.txtGridFTP

17 JSAGA17 VOMS WMS gsiftp JSAGA Gatekeep. Globus JSAGA CREAM-CELCG-CE Unicore WMS WMS Gatekeeper SAGA job RSL D-Grid VOMS VOMS jsaga-job-run–Executable /bin/hostname –Output out.txt -FileTransfer gsiftp://…/out.txt<out.txt -r gatekeeper://gt4-fzk.gridka.de:2119/jobmanager-pbs -r gatekeeper://gt4-fzk.gridka.de:2119/jobmanager-pbs modified job EGEE wrapper scriptout.txtGridFTP

18 JSAGA18 Architecture logicielle JSAGA interface implementation

19 JSAGA19 Architecture logicielle Ready to use software, adapted to targeted scientific field Ready to use software, adapted to targeted scientific field A single interface for using all middlewares A single interface for using all middlewares As many interfaces as ways to implement each functionality As many interfaces as ways to implement each functionality –easy to develop new plug-in –efficient usage of legacy APIs As many interfaces as used technologies As many interfaces as used technologies user applications legacy APIs core engine JSAGA adaptors JSAGA adaptors interfaces adaptors interfacesJSAGA interface implementation SAGA SAGA Java binding end user application developer adaptor developer

20 JSAGA20 LogicalPhysical Job monitoring Job control Provided adaptors Minimal dependencies on external libraries external tools –e.g. no gLite-UI operating system –tested on planned construction done Data Execution Security InMemCredGlobusG. LegacyG. RFC820MyProxyVOMSJKSSSHLogin / pwdX509gatekeepergLite-WMSwsgramunicore6creamnaregisshlocalgatekeepergLite-LBwsgramunicore6creamnaregisshlocal rnslfnsrbirodscatalogrbyteiogsiftpsrmsrbirodshttphttpssftpftpfilezipcache Grid techno. Others

21 JSAGA21 Software quality Adaptors –validated by a middleware- independent SAGA test suite –maven 'archetype' to generate skeleton of new adaptor project Automated build process –download and install build tools external libraries –generate source code –execute test-suites unitary tests integration tests –generate project web site documentation reports –generate installer GUI extract library dependencies from maven project description # SAGA protocols test-suite configuration gsiftp.base=gsiftp://ccrugceli01.in2p3.fr/tmp/ gsiftp.base2=gsiftp://agena.c-s.fr/grid/tmp/ gsiftp.context=OpenPlast_proxy https.base=http://grid.in2p3.fr/html/Private/ https.context=Web_X509 file.base=file:///c:/tmp/ file.base2=file:///c:/

22 JSAGA22 Installer GUI

23 JSAGA23 Licenses LGPL license –for the core engine and for most adaptors Optional licenses –for adaptors having external dependencies, which license is not compatible with LGPL –then, end-user must… either accept the terms of the license agreement, or go back to previous screen and uncheck the adaptor(s)

24 JSAGA24 Applications – Commandes en ligne JSAGA provides command line interfaces for… –security jsaga-context-init jsaga-context-info jsaga-context-destroy –execution management jsaga-job-run jsaga-job-status jsaga-job-cancel jsaga-job-output Also useful as code examples to start with SAGA –data management jsaga-cat jsaga-cp jsaga-ls jsaga-mkdir jsaga-mv jsaga-rm jsaga-rmdir jsaga-stat jsaga-test jsaga-logical

25 JSAGA25 Applications – JUX

26 JSAGA26 Applications – autres exemples Elis@ a web portal for submitting jobs to industrial and research grid infrastructures JJS (Java Job Submission) a tool for running efficiently short-life jobs on EGEE SimExplorer a set of tools, including a workflow engine providing distributed computing facilities, for managing simulation experiments jCAE (Java CAE) Computer Aided Engineering on distributed computing environment/

27 JSAGA27 Thank you !

28 JSAGA28 Design of adaptors interfaces Close to application developer needs –object-oriented –high-level –uniform interface for all the supported technologies Design objectives –easy to use … but > (T. Kielmann) Close to existing middleware APIs –service-oriented –low-level –as many interfaces as ways to implement each functionality + optional interfaces Design objectives –easy to extend wrap m/w functionalities only code size: engine= 2x adaptors –enable efficient usage of middleware APIs engine choose best interface SAGA SAGAadaptorsinterfacesJSAGA

29 JSAGA29adaptorsinterfacesJSAGA As many interfaces as ways to implement each functionality Example: ways to monitor jobs –API mode poll job status listen to notifications about job status changes –API granularity individual jobs list of jobs jobs filter (e.g. by user, by date, by tag…) planned construction done Job monitoring gatekeepergLite-WMSwsgramunicore6creamnaregi Design of adaptors interfaces sshlocal SAGA SAGA query status for individual job listen status for individual job query status for filtered jobs getState waitFor

30 JSAGA30adaptorsinterfacesJSAGA Design of adaptors interfaces Optional interfaces for optimization Example: ways to copy a file –data read/write methods either stream methods or get/put methods –data copy e.g. enable third-party transfer –delegated transfer planned construction done Physical rbyteiogsiftpsrmsrbirodshttphttpssftpftpfilezipcache SAGA SAGA data reader data writer data copy

31 JSAGA31 Provided adaptors SAGA hides most of the differences between the technologies Information on remaining differences is available –on JSAGA web site, for each adaptor module Frequently Asked Questions Configuration example –with command line jsaga-help supported protocols associated security contexts context attr. usage/defaults usage: jsaga-help -a. | --config | -d | -h | -j | -s | -v [-D. = where: -a,--attribute. Output the value of security context attribute --config Output the effective configuration -d,--data Information about data protocols. = service | context -D. = Set context instance attribute (e.g. -DVOMS[0].UserVO=dteam) -h,--help Display this help and exit -j,--job Information about job services. = service | context -s,--security Information about security context instances. = usage | default | missing -v,--version Output version information and exit