MGHPCC will be performing power upgrades on Pod 8A in order to increase density and allow more nodes to be added in that Pod's rows.  Similar to the May 13th work, this means that we will be idling half the nodes in 8A on two dates: June 17 and June 24th.

These are all day events, meaning that the nodes in question will not be available for the 24 hours of that day.  This is being accomplished via reservations. So no jobs will be canceled but nodes will be drained and users may notice that their jobs may pend longer than normal as the scheduler idles these nodes.

Where possible, please use or include other partitions in your job scripts and plan accordingly for any new or long-running jobs during that period: https://docs.rc.fas.harvard.edu/kb/running-jobs/#Slurm_partitions

This affects the Cannon cluster. FASSE is not affected.

Impacted partitions are:

arguelles_delgado_gpu

bigmem_intermediate

bigmem

blackhole_gpu

eddy

enos

gershman gpu

hejazi hernquist_ice

hoekstra hsph

huce_ice

iaifi_gpu

iaifi_gpu_priority

iaifi_gpu_requeue

intermediate

itc_gpu

itc_gpu_requeue

joonholee

jshapiro

jshapiro_priority

jshapiro_sapphire

kempner

kempner_dev

kempner_h100

kempner_requeue

kempner_reservation

kovac

kozinsky

kozinsky_gpu

kozinsky_priority

kozinsky_requeue

murphy_ice

ortegahernandez_ice

sapphire

seas_compute

seas_gpu siag

siag_combo

siag_gpu

sur test

yao

yao_priority

zhuang

The 2024 MGHPCC data center annual power downtime will take place May 21-24, 2024.

We will begin our shutdown on Tuesday May 21st and expect a return to service by 5PM Friday May 24th.

- Jobs: Please plan ahead as all still running jobs on the morning of May 21st will be stopped and canceled and will need to be resubmitted after the downtime. Pending jobs will remain in the queue until the cluster returns to regular service on May 24th.

- Access: The cluster, scheduler, login, and OoD nodes will be unavailable for the duration of the downtime. New lab and account requests should wait until after the downtime.

- Storage: All Holyoke storage will be powered down and unavailable for the duration of the downtime. Boston storage will remain online, but your ability to access it may be impacted and network changes may briefly affect its availability.

Further details, an explanation for this year's change in scheduling, a visual timeline, and a list of maintenance tasks overview can be found at:

https://www.rc.fas.harvard.edu/blog/2024-mghpcc-power-downtime/

Progress of the downtime will be posted here on our status page during the event. Note that you can subscribe to receive updates as they happen. Click Get Updates in the upper right.

MAJOR TASK OVERVIEW

• OS upgrade to Rocky 8.9 - Point upgrade, no code rebuilds will be required. Switch from system OFED to Mellanox OFED on nodes for improved performance

• Infiniband (network) upgrades

• BIOS updates (various)

• Storage firmware updates

• Network Maintenance

• Decommission old nodes (targets contacted)

• Additional minor one-off updates and maintenance (cable swap, reboots, etc.)

Thanks,

FAS Research Computing

https://www.rc.fas.harvard.edu/

https://docs.rc.fas.harvard.edu/

https://status.rc.fas.harvard.edu/