Monitor Charlie
There are several ways to monitor the status of Charlie. The following commands are used from cfe1.
The command
ctop allows users to monitor the current load on the compute nodes of Charlie. From cfe1, the ctop command can be used directly as follows:-bash-4.2$ ctop
REQUESTED RESOURCES BY NODE
node cpu %cpu mem(gb) %mem #jobs
----- ------- ---- --------- ---- -----
c1 16/160 10% 100/1219 8% 2
c2 104/256 40% 130/1826 7% 3
c3-0 0/24 0% 0/31 0% 0
...
...
...
c3-14 0/24 0% 0/31 0% 0
The command also offers other options. Option
-j shows additionally the jobs that are running on the compute nodes.As mentioned in the Job status page, the
qstat command offers a variety of ways to monitor the jobs that are running. For details on the available options and possible job status, please refer to the Job status page. Here, we describe some common, useful commands.We created a shortcut command
qwatch, which calls qstat -an1 every second, so it creates an real-time view of the status of the jobs in Charlie:-bash-4.2$ qwatch
Every 1.0s: qstat -an1 Wed Jul 29 08:51:34 2020
cfe1:
Req'd Req'd Elap
Job ID Username Queue Jobname SessID NDS TSK Memory Time S Time
--------------- -------- -------- ---------- ------ --- --- ------ ----- - -----
387996.cfe1 my_user1 normal my_job_1 31816 1 8 50gb 20:00 R 16:35 c1/1*8
388017.cfe1 my_user1 normal my_job_2 48712 1 8 50gb 20:00 R 13:14 c1/18*8
388020.cfe1 my_user2 normal my_job_4 49772 1 8 50gb 20:00 R 13:02 c1/9*8
The
qstat command itself offers snapshots of different aspects of Charlie depending on the options. Here provide a list of common option combinations used in qstat [option], such as qstat -an1.Option | Description |
-an1 | Display job information, including the node it is running on |
-Jtan1 | Using the same format as -an1, display only job arrays, and expand the sub-jobs in the job arrays |
-u <username> | Display the running jobs of a user |
-xu <username> | Display all (running and finished) jobs of a user |
-xf <job_id> | Display detailed information about a job (running or finished) |
The
qstat command also offers options to monitor the status of the queues of Charlie. For example,-bash-4.2$ qstat -q
server: cfe1
Queue Memory CPU Time Walltime Node Run Que Lm State
---------------- ------ -------- -------- ---- ----- ----- ---- -----
scgc-normal -- -- 336:00:0 -- 3 0 -- E R
normal -- -- 576:00:0 -- 3 156 -- E R
route -- -- -- -- 0 0 -- E R
...
scgc-high -- -- -- -- 0 0 -- E R
workq -- -- -- -- 0 0 -- D R
----- -----
7 156
Last modified 1yr ago