FAS Research Computing - Historique des incidents

FASRC will be doing OS upgrades to the latest version of Rocky 8.10 from August 24-27th. These rolling upgrades will improve cluster security and install the latest Cuda 13.2 driver.

Upgrades will happen in stages with each stage occurring between 8a-5p each day. During that period the nodes being upgraded will be unavailable. No jobs will be canceled but jobs will stay pending if all nodes in the partition are down.

Note that during this week the cluster will be in a partially upgraded state, thus jobs that span multiple nodes may become unstable due to mismatching libraries. Users concerned about this should wait until August 28th to restart runs.

Users should plan their work accordingly.

The upgrade schedule with impacted partitions is as follows:

August 24th:

FASSE
boslogin
private login nodes

August 25th: Cannon Part 1

holylogin
arguelles_delgado
blackhole
davies_gpu
davies
desai
eddy
holy-cow
holy-smokes
huce_bigmem
huce_cascade
huttenhower
jacobsen2
janson_bigmem
janson_cascade
janson
ke
lukin
nguyen
olveczky_gpu
remoteviz
seas_compute
shared
sompolinsky_gpu
tambe
vishwanath
whipple
xlin
xlin_ice
zhuang_gpu
zhuang

August 26th: Cannon Part 2
arguelles_delgado
conroy
davies
doshi-velez
dsouza
edwards
geophysics
giribet
gpu_test
hernquist
huce_cascadeimasc
janson
kempner_h200
kempner_rtx
murphy
ni_lab
olveczky
ortegahernandez
pehlevan
seas_compute
shakhnovich
shared
unrestricted
xlin
yin
zon

August 27th: Cannon Part 3

arguelles_delgado_gpu_a100
arguelles_delgado_gpu_mixed
arguelles_delgado_h100
bigmem_intermediate
bigmem
blackhole_gpu
dvorkin
eddy
enos
gershman
gpu
gpu_h200
hejazi
hernquist_ice
hoekstra
hsph_gpu
hsph
huce_ice
iaifi_gpu
intermediate
itc_cluster
itc_gpu
janson_sapphire
joonholee
jshapiro
kempner_h100
kempner_h200
kempner
kempner_interactive
kovac
kozinsky_gpu
kozinsky
murphy_ice
mweber_compute
mweber_gpu
olveczky_sapphire
ortegahernandez_ice
rivas
sapphire
seas_compute
seas_gpu
seas_gpu_perf
siag_gpu
siag_combo
siag
sur
test
yao_alphatns
yao_gpu
yao
zhuang

FASRC monthly maintenance will take place on July 6th 2026. Our maintenance tasks should be completed between 9am-1pm.

Cannon cluster will be paused during this maintenance?: NO
FASSE cluster will be paused during this maintenance?: NO

NOTICES:

• Friday July 3rd is a university holiday (independence Day observed)

• Training: Upcoming training from FASRC and other sources can be found on our Training Calendar. at https://www.rc.fas.harvard.edu/upcoming-training/

• Status Page: You can subscribe to our status to receive notifications of maintenance, incidents, and their resolution at https://status.rc.fas.harvard.edu/ (click Get Updates for options).

• We'd love to hear success stories about your or your lab's use of FASRC. Submit your story here.

MAINTENANCE TASKS

• Domain controller replacement

  • Audience: Internal

  • Impact: None. End users should not see any impact.

• Reboot drained nodes in error state

  • Audience: Cluster nodes with errors.

  • Impact: These nodes will have been drained already in preparation. No impact on jobs on the day and the affected nodes will return to service in their respective partitions after the maintenance period.

• OOD/Open OnDemand reboots

  • Audience: All OOD users, reboot of the head nodes.

  • Impact: Running sessions will not be affected.

• Login node reboots

  • Audience; All login node users.

  • Impact: Login nodes will reboot during the maintenance window.

• Netscratch 90-day retention cleanup

  • Audience; All netscratch users

  • Impact: Files older than 90 days will be removed per our scratch policy. Please note that this cleanup can happen at any time, not just during maintenance.

Thank you,
FAS Research Computing
https://docs.rc.fas.harvard.edu/
https://www.rc.fas.harvard.edu/

The yearly power downtime at our Holyoke data center, MGHPCC, has been scheduled by the facility. This year's power downtime will take place on Tuesday June  15th - 18th, 2025.  There will be no June monthly maintenance as a result.

Since the facility will be powered down for two days this year, we will not be performing the usual maintenance tasks. 
That said, networking and other key infrastructure will be doing maintenance.

IMPORTANT NOTE: FASRC storage at both Holyoke and Boston will be affected and should not be expected to be available throughout the downtime. Please plan ahead accordingly.

• Monday June 15th -  Power-down begins at 9AM

• Tuesday June 16th - Power out at MGHPCC

• Wednesday June 17th - Power out at MGHPCC

• Thursday June 18th - Expected return to full service by 5PM

• Friday June 19th - Please note that June 19th is a university holiday

For more detailed information and follow-up, please see:
https://www.rc.fas.harvard.edu/mghpcc-yearly-shutdown or this Status Page