Utilitaires VISION

Outils de profiling

visualisation de la charge

  • gr_systat

placement des processus

  • dplace
    Forcer la répartition des coeurs et l'allocation en local de la mémoire :
dplace -e -n nom_programme -c 0-16 numactl -l nom_programme

ici le programme "nom_programme" et a été lancée sur 16 coeurs (code openmp par exemple)
l'utilisation de dplace évite le load balancing des nouveaux noyaux Linux : la mémoire est allouée sur la même lame que le coeur de calcul alloué

  • omplace
    De la même manière que dplace, place les process de manières contigues sur les coeurs pour optimisé les codes ;
    *Nécessite de charger au préalable le module mpt
    [homer@vision ~]$ module add mpi/hpempi-1.6/hmpt/2.22
    [homer@vision ~]$ omplace
    Usage:
        mpiexec_mpt -np numproc \
            omplace [-bs] [-b cpu] [-c cpulist] [-nt numthreads] \
            [-r replicate] [-s skip] [-tm none|intel|open64|pthreads] \
            [-ht compact|spread] \
            [-v] [-vv] prog [args ...]
    [homer@vision ~]$ man omplace
    omplace(1)                            General Commands Manual                            omplace(1)
    
    NAME
           omplace - a tool for controlling the placement of MPI processes and various types of
           threads,
    
    SYNOPSIS
           omplace [ OPTIONS ] program args...
           mpiexec_mpt -np nranks omplace [ OPTIONS ] program args...
           mpirun -np nranks omplace [ OPTIONS ] program args...
    
    DESCRIPTION
           The omplace command causes the successive threads in a threaded or in a hybrid MPI/threaded
           job to be pinned to unique CPUs.  The CPUs are assigned in order from the effective CPU list
           within the containing cpuset. This command is layered on dplace, and can be easier to use
           with MPI application launch commands because it hides the details associated with process
           skip counts, nested MPI and OpenMP processes and threads, and complex CPU lists.
    
           The CPU placement is performed by dynamically generating a placement file and invoking
           dplace with the MPI job launch.
    
    

files PBS disponibles

scripts localisés sous /softs/tools/bin. Pour les utiliser, il faut ajouter ce chemin au PATH dans votre .bashrc

PATH=$PATH:/softs/tools/bin

[homer@vision ~]$ pbs_queue
=====================================================================================================
     Queue        Max       MaxRun    MaxRun  MaxQueued MaxQueued Priority  Current  Running  Queued
               Walltime     /user    forall     /user    forall             Jobs     Jobs     Jobs
=====================================================================================================
     workq   50:00:00      3         0         0         0        20         0         0         0
  calcul_small90:00:00      0         0         0         0        80         6         3         3
     gpuq    40:00:00      0         0         0         0        140        0         0         0
    default     --         0         0         0         0         -         0         0         0
     visuq   24:00:00      0         0         0         0        140        2         2         0
    interq   24:00:00      0         0         0         0        140        0         0         0
  calcul_big 48:00:00      1         0         0         0        120        1         1         0
Sorting Jobs policy use : 3*queue_priority+2*eligible_time/3600+custom_prio+50*fai
Max queued per user  : 20
Max run per user  : 4
Max cpus per user : 376
Max gpus per user : 4

accounting

scripts localisés sous /softs/tools/bin. Pour les utiliser, il faut ajouter ce chemin au PATH dans votre .bashrc

PATH=$PATH:/softs/tools/bin

[@vision vision-git]$ qacct -u homer  -d Hours
BEGIN: 2020-11-20 10:05:05
END  : 2021-08-17 18:30:09
user        count(jobid)  Ucput_Hour  Uwalltime_Hour  PBSWallTime_Hour  EffWallTime_Hour  UsageEfcy   AvgCoreUsed  MeanCoreUsed  AvgWait_s
----------  ------------  ----------  --------------  ----------------  ----------------  ----------  -----------  ------------  ----------
homer       1050          28299       1960            1962              1962              0.24        51.96        18.06         10173.99

scripts utilitaires

ces scripts sont localisés sous /softs/tools/bin. Pour les utiliser, il faut ajouter ce chemin au PATH dans votre .bashrc

PATH=$PATH:/softs/tools/bin

  • pbsn : liste les noeuds et leurs états (down, job-exclusive, free) et les partitions attribuées (@gpu, compute, inter, visu) aux noeuds
  • connaitre les noeuds libres :
    [homer@vision ~]$ pbsn
    
     VNODES          STATE        MEM       CPUS   Gpu-Visu        PARTITION              JOB
                        ============================================================
    
     vision           free         0b          0          0
     vision[0]    job-busy   189320mb         24          4             visu  20905.vision/0,
     vision[1]    job-busy   190386mb         24          4             visu  20905.vision/0,
     vision[2]        free   190386mb         24          4       visu,inter
     vision[3]    job-busy   190386mb         24          4       visu,inter  20966.vision/0,
     vision[4]        free   190386mb         24          0        gpu,inter   20966.vision/0
     vision[5]        free   190386mb         24          0        gpu,inter   20966.vision/0
     vision[6]    job-busy   190386mb         24          0 gpu,inter,compute  20906.vision/0,
     vision[7]    job-busy   190386mb         24          0 gpu,inter,compute  20906.vision/0,
     vision[8]    job-busy   190386mb         24          0    compute,inter  20933.vision/0,
     vision[9]    job-busy   190386mb         24          0    compute,inter  20933.vision/0,
     vision[10]   job-busy   190386mb         24          0    compute,inter  20933.vision/0,
     vision[11]   job-busy   190386mb         24          0    compute,inter  20933.vision/0,
     vision[12]   job-busy   190386mb         24          0          compute  20933.vision/0,
     vision[13]       free   190386mb         24          0          compute  20933.vision/0,
     vision[14]   job-busy   190386mb         24          0          compute  20967.vision/0,
     vision[15]       free   190386mb         24          0          compute
     
  • get_mem_usage -all|me : connaitre l'utilisation mémoire de la machine pour l'ensemble des processus ou pour ses propres processus.
    [homer@bision ~]$ get_mem_usage -all
    VISION Mem Total 135.647 GB
    [homer@vision ~]$ get_mem_usage -me
       PID STAT   RSS %CPU COMMAND
    336776 S+    1364  0.0 /bin/bash /sw/tools/bin/get_mem_usage -me
    336778 R+    1188 82.0 ps -u homer -o pid,stat,rss,pcpu,command
    336779 S+     980  0.0 awk {print $0}{sum+=$3} END {print "Mem Total for $USER", sum/(1024**2), "GB"}
    421274 S     1980  0.0 sshd: homer@pts/9
    421275 Ss    2392  0.0 -bash
    Mem Total for $USER 0.00753784 GB
    
  • qmem : connaitre l'utilisation mémoire d'un job :
    [homer@hulk ~]$qmem --job=5618
    Memory Usage for job : 5618
    -------------------------------
    PBS Hosts used : vision[0] vision[1] vision[2]
    -> number of host requested : 3
    node number : 0 1 2
         user : homer,JID : 5618
        -> Memory used from numactl : -6.7646484375 -24.4560546875 -152.767578125 (GB) for nodes 0 1 2
        -> Mem Nodes used for cpuset : mems 0-2
        -> RAM mem = 268.017951965332     (GB) ; TOT mem = 276.91187286377 (GB)
        -> nb user pid : 51
        -> pids : 62326 62574 62582 62615 65566 65567 65568 65569 65570 65571 65572 65573 65574 65575 65576 65577 65578 65579 65580 65581 65582 65583 65584 65585 65586 65587 65588 65589 65590 65591 65592 65593 65594 65595 65596 65597 65598 65599 65600 65601 65602 65603 65604 65605 65606 65607 65608 65609 65610 65611 65612