Search…
Monitor Charlie
There are several ways to monitor the status of Charlie. The following commands are used from cfe1.

Compute Nodes

The command ctop allows users to monitor the current load on the compute nodes of Charlie. From cfe1, the ctop command can be used directly as follows:
1
-bash-4.2$ ctop
2
REQUESTED RESOURCES BY NODE
3
4
node cpu %cpu mem(gb) %mem #jobs
5
----- ------- ---- --------- ---- -----
6
c1 16/160 10% 100/1219 8% 2
7
c2 104/256 40% 130/1826 7% 3
8
c3-0 0/24 0% 0/31 0% 0
9
...
10
...
11
...
12
c3-14 0/24 0% 0/31 0% 0
Copied!
The command also offers other options. Option -j shows additionally the jobs that are running on the compute nodes.

Jobs

As mentioned in the Job status page, the qstat command offers a variety of ways to monitor the jobs that are running. For details on the available options and possible job status, please refer to the Job status page. Here, we describe some common, useful commands.
We created a shortcut command qwatch, which calls qstat -an1 every second, so it creates an real-time view of the status of the jobs in Charlie:
1
-bash-4.2$ qwatch
2
3
Every 1.0s: qstat -an1 Wed Jul 29 08:51:34 2020
4
5
cfe1:
6
Req'd Req'd Elap
7
Job ID Username Queue Jobname SessID NDS TSK Memory Time S Time
8
--------------- -------- -------- ---------- ------ --- --- ------ ----- - -----
9
387996.cfe1 my_user1 normal my_job_1 31816 1 8 50gb 20:00 R 16:35 c1/1*8
10
388017.cfe1 my_user1 normal my_job_2 48712 1 8 50gb 20:00 R 13:14 c1/18*8
11
388020.cfe1 my_user2 normal my_job_4 49772 1 8 50gb 20:00 R 13:02 c1/9*8
Copied!
The qstat command itself offers snapshots of different aspects of Charlie depending on the options. Here provide a list of common option combinations used in qstat [option], such as qstat -an1.
Option
Description
-an1
Display job information, including the node it is running on
-Jtan1
Using the same format as -an1, display only job arrays, and expand the sub-jobs in the job arrays
-u <username>
Display the running jobs of a user
-xu <username>
Display all (running and finished) jobs of a user
-xf <job_id>
Display detailed information about a job (running or finished)

Queues

The qstat command also offers options to monitor the status of the queues of Charlie. For example,
1
-bash-4.2$ qstat -q
2
3
server: cfe1
4
5
Queue Memory CPU Time Walltime Node Run Que Lm State
6
---------------- ------ -------- -------- ---- ----- ----- ---- -----
7
scgc-normal -- -- 336:00:0 -- 3 0 -- E R
8
normal -- -- 576:00:0 -- 3 156 -- E R
9
route -- -- -- -- 0 0 -- E R
10
...
11
scgc-high -- -- -- -- 0 0 -- E R
12
workq -- -- -- -- 0 0 -- D R
13
----- -----
14
7 156
Copied!
Last modified 9mo ago
Export as PDF
Copy link