slurm - What does the state 'drain' mean? -

when use sinfo see following:

$ sinfo partition avail  timelimit  nodes  state nodelist [...] rg3          28-00:00:0      1  drain rg3hpc4 [...] 

what state 'drain' mean?

it means no further job scheduled on node, running jobs keep running (by contrast setting node down kills jobs running on node).

nodes set state maintenance operation can take place once running jobs finished.

from manpage of scontrol command:

if want remove node service, typically want set it's state "drain"

note system administrator gave reason why node drained, , can see reason

sinfo -r