1 HPC et Windows 2003 CCS pour les développeurs Visual Studio 2005

Introduction au HPC avec Windows CCS pour les développeurs et Visual Studio 2005 Démonstration(s) sur le développement dune application parallèle

RéseauMPI Réseauprivé Réseaupublic Infrastructure IT Infrastructure IT Nœud Nœud maître ADDNSDHCP Windows Update Supervision MOMMOM 3 rd party3 rd party Administration des systèmes SMS SMS 3 rd party 3 rd party Compute Cluster Admin / User Cons RIS(optional) Job Scheduler Management NAT (optional) Node Manager MPI Management Nœud MPI Management

Le Job Scheduler Soumettre des jobs en mode interactif (interface graphique et/ou batch) Possibilité de soumettre et de contrôler le Scheduler via des APIs COM ou.NET Application MPI Dérivée de limplémentation MPI-2 dArgonne National Lab (MPICH2) MS MPI cest 2 parties distinctes Pour les éditeurs de logiciels: + de 160 Fonctions (mais on peut sen tirer avec environ 10 Fonctions) Pour les utilisateurs: en ligne de commande (mpiexec) ou avec loutil graphique qui lance les Jobs. Avec CCS, il est possible dutiliser nimporte quelle implémentation MPI Visual Studios 2005 Nouveau débogueur parallèle. Supporte la Pile Message Passing Interface (MS-MPI) Supporte Open Message Passing (OpenMP).

DébogeurP arallèle SupportDOpenMP

User App MPI Node Manager Job Execution DB/FS User Cmd line Desktop App Job Mgr UI Admin Admin Console Cmd line Head Node Job Mgmt Resource Mgmt Cluster Mgmt Scheduling High speed, low latency interconnect Tasks Management Jobs Policy, reports Active Directory Web Services Data Input Domain\UserA

ConceptsConcepts ObjetsDescriptions Cluster Unité de plus haut niveau dun CCP; comprenant un ensemble de nœuds, une queue et des jobs. Queue Unité fournissant le queuing et le scheduling des jobs. Il y a une queue dans un cluster CCP et elle contient les jobs (en attente, sexécutant et terminés). Les jobs terminés sont purgés régulièrement de la queue. Job Une collection de tâches, les tâches sont utilisées pour effectuer les calculs, à linitiative dun utilisateur. Les jobs permettent la réservation des ressources nécessaires à une ou plusieurs tâches. Task Une tâche est lexécution dun programme sur un nœud du cluster. Une tâche peut être séquentielle ou parallèle avec lusage de MPI. Node Un compute node, avec un ou plusieurs processeurs.

ConceptsConcepts ObjectsDescriptions Cluster The top-level organizational unit of CCP; comprising a set of nodes, a queue and jobs. Queue An organization unit which provides queuing and scheduling of jobs. There is only one queue in a CCP cluster and it contains pending, running and completed jobs. Completed jobs are purged periodically from the queue. Job A collection of tasks, using a task to perform a computation, initiated by a user. Jobs are used to reserve resources for subsequent use by one or more tasks. Task A task represents the execution of a program on some compute nodes. A task can be a serial program (single process) or an MPI program with multiple processes Node A single compute node, with one or more processors

Le Job Scheduler Head Node Client Node Compute Node Admission Allocation Activation Jobs/Tasks

Quelle Interface utiliser Interfaces Meilleur usage pour CLI Tout langage: e.g. Perl, Python, C/C++, Fortran Job Manager Utilisateur final C# API Managed code COM E.g.: C/C++, VB Script and Perl Pas pour du managed code

Which Interface to use Interfaces Best Suited for CLI Any language: e.g. Perl, Python, C/C++, Fortran Job Manager End user C# API Managed code COM E.g.: C/C++, VB Script and Perl (because the COM interface is based on managed code) Not for managed code (because the COM interface is based on managed code)

Task Terms Task terms Descriptions cmdline Command line of the task env Environment variables for the task depend Inter-task dependencies rerunnable Specifies that a task can be rerun after failure workdir Start-up folder of the task (or current working directory) runtime Specify the runtime of the task numprocessors Number of processors a task requires requirednodes Specific list of nodes the task runs on exclusive exclusive allocation of nodes a task stdin Redirect standard input of the task from a file stdout Redirect standard output of the task to a file stderr Redirect standard error of the task to a file

Batch scripts rem Run a serial job job submit /stdout:%FSHARE%\pi10.out %PDCSHARE%\pi.exe 10 rem Run a parallel job job submit /stdout:%FSHARE%\pi100.out /numprocessors:2 mpiexec %FSHARE%\pi.exe 100 rem Run a parametric sweep job for /F "usebackq tokens=4" %j in (`job new`) do set JOBID=%j for /L %i in (100, 100, 1000) do job add %JOBID% /stdout:%FSHARE%\pi%i.out %FSHARE%\pi.exe %i job submit /id:%JOBID% rem Run a task flow job for /F "usebackq tokens=4" %j in (`job new`) do set JOBID=%j job add %JOBID% /name:setup /stdout:%FSHARE%\setup.out cmd.exe /c echo Running Setup Task %TIME% for /L %i in (100, 100, 1000) do job add %JOBID% /name:compute /depend:setup /stdout:%FSHARE%\pi%i.out %FSHARE%\pi.exe %i job add %JOBID% /name:finish /depend:compute /stdout:%FSHARE%\finish.out cmd.exe /c echo Running Finish Task %TIME% job submit /id:%JOBID% rem Run a serial job job submit /stdout:%FSHARE%\pi10.out %PDCSHARE%\pi.exe 10 rem Run a parallel job job submit /stdout:%FSHARE%\pi100.out /numprocessors:2 mpiexec %FSHARE%\pi.exe 100 rem Run a parametric sweep job for /F "usebackq tokens=4" %j in (`job new`) do set JOBID=%j for /L %i in (100, 100, 1000) do job add %JOBID% /stdout:%FSHARE%\pi%i.out %FSHARE%\pi.exe %i job submit /id:%JOBID% rem Run a task flow job for /F "usebackq tokens=4" %j in (`job new`) do set JOBID=%j job add %JOBID% /name:setup /stdout:%FSHARE%\setup.out cmd.exe /c echo Running Setup Task %TIME% for /L %i in (100, 100, 1000) do job add %JOBID% /name:compute /depend:setup /stdout:%FSHARE%\pi%i.out %FSHARE%\pi.exe %i job add %JOBID% /name:finish /depend:compute /stdout:%FSHARE%\finish.out cmd.exe /c echo Running Finish Task %TIME% job submit /id:%JOBID% setup finish compute

Microsoft.ComputeCluster Connexion au cluster _cluster = new Cluster(); _cluster.Connect(Nom du Cluster); Lister les noeuds du cluster IClusterEnumerable enumerable = _cluster.ComputeNodes; Démarrer un Job IJob job = _cluster.CreateJobFromXmlFile(templatePath); int JobID=_cluster.AddJob(job); _cluster.SubmitJob(JobID, User, Pwd, true, 0); Lister tous les Jobs IClusterEnumerable enumerable = _cluster.ListAllJobs (); Lister les jobs IClusterEnumerable enumerable = _cluster.ListJobs(S, JobStatus.Failed) IClusterEnumerable enumerable = _cluster.ListJobs(S, JobStatus.Running) IClusterEnumerable enumerable = _cluster.ListJobs(S, JobStatus.Finished) Lister les tâches IClusterEnumerable enumerable=_cluster.ListTasks (JobID); Etc….

Networking Hardware Virtual Bus Driver IP TCP User Host Channel Adapter Driver IB WinSock Provider DLL Miniport (GigE) Miniport (IPoIB) Kernel API (verbs based) User API (verbs based) Host Channel Adapter Driver GigE RDMA WinSock Provider DLL IHV-provided component OS component NDIS WinSock DLL IB w/ RDMA GigE w/ RDMA Ethernet User Mode Kernel Mode Switch traffic based on sub-net Manage hardware resources in user space (eg., Send and receive queues) HPC Application MPI Winsock Switch

Microsoft HPC web site Argonne National Labs MPI website Tutorial from Lawrence Livermoore National Lab Microsoft HPC newsgroup microsoft.public.windows.hpc Microsoft HPC community portal

© 2007 Microsoft France Votre potentiel, notre passion TM