SLURM Scheduler - Cannon - Працює
SLURM Scheduler - Cannon
Cannon Compute Cluster (Holyoke) - Працює
Cannon Compute Cluster (Holyoke)
Boston Compute Nodes - Працює
Boston Compute Nodes
GPU nodes (Holyoke) - Працює
GPU nodes (Holyoke)
seas_compute - Працює
seas_compute
SLURM Scheduler - FASSE - Працює
SLURM Scheduler - FASSE
FASSE Compute Cluster (Holyoke) - Працює
FASSE Compute Cluster (Holyoke)
Kempner Cluster CPU - Працює
Kempner Cluster CPU
Kempner Cluster GPU - Працює
Kempner Cluster GPU
FASSE login nodes - Працює
FASSE login nodes
Cannon Open OnDemand - Працює
Cannon Open OnDemand
FASSE Open OnDemand - Працює
FASSE Open OnDemand
Netscratch (Global Scratch) - Працює
Netscratch (Global Scratch)
Home Directory Storage - Boston - Працює
Home Directory Storage - Boston
Tape - (Tier 3) - Працює
Tape - (Tier 3)
Holylabs - Працює
Holylabs
Isilon Storage Holyoke (Tier 1) - Працює
Isilon Storage Holyoke (Tier 1)
Holystore01 (Tier 0) - Працює
Holystore01 (Tier 0)
HolyLFS04 (Tier 0) - Працює
HolyLFS04 (Tier 0)
HolyLFS05 (Tier 0) - Працює
HolyLFS05 (Tier 0)
HolyLFS06 (Tier 0) - Повністю недоступно
HolyLFS06 (Tier 0)
Holyoke Tier 2 NFS - Працює
Holyoke Tier 2 NFS
Holyoke Specialty Storage - Працює
Holyoke Specialty Storage
holECS - Працює
holECS
Isilon Storage Boston (Tier 1) - Працює
Isilon Storage Boston (Tier 1)
BosLFS02 (Tier 0) - Працює
BosLFS02 (Tier 0)
Boston Tier 2 NFS - Працює
Boston Tier 2 NFS
CEPH Storage Boston (Tier 2) - Працює
CEPH Storage Boston (Tier 2)
Boston Specialty Storage - Працює
Boston Specialty Storage
bosECS - Працює
bosECS
Samba Cluster - Працює
Samba Cluster
Globus Data Transfer - Працює
Globus Data Transfer
Історія сповіщень
Переглянути поточний статуссерп. 2026
- Завершено · серпня 27, 2026 у 20:06UTC
OS upgrade work is mostly completed.
There are a few remaining nodes in the cluster that did not successfully receive the upgrade.
If you see nodes that are 'DOWN' for the following reasons, they are all nodes that need manual intervention for OS upgrade. FASRC staff are aware of them and will upgrade these nodes over the next week.
"wrong kernel"
"Not responding"
"unexpected reboot"
"gres/gpu count lower than configured"
"/n/sw not mounted"
We appreciate your continued patience.
- Оновлення · серпня 26, 2026 у 20:01UTC
Today's work is complete. The final round, Cannon Part 3, takes place tomorrow.
August 26th: Cannon Part 2
arguelles_delgado
conroy
davies
doshi-velez
dsouza
edwards
geophysics
giribet
gpu_test
hernquist
huce_cascadeimasc
janson
kempner_h200
kempner_rtx
murphy
ni_lab
olveczky
ortegahernandez
pehlevan
seas_compute
shakhnovich
shared
unrestricted
xlin
yin
zon - Оновлення · серпня 25, 2026 у 19:37UTC
Today's work is complete for Cannon Part 1 and will resume tomorrow for Cannon Part 2
August 25th: Cannon Part 1
holylogin
arguelles_delgado
blackhole
davies_gpu
davies
desai
eddy
holy-cow
holy-smokes
huce_bigmem
huce_cascade
huttenhower
jacobsen2
janson_bigmem
janson_cascade
janson
ke
lukin
nguyen
olveczky_gpu
remoteviz
seas_compute
shared
sompolinsky_gpu
tambe
vishwanath
whipple
xlin
xlin_ice
zhuang_gpu
zhuang - Оновлення · серпня 24, 2026 у 19:07UTC
Today's portion of the upgrades are completed. Tomorrow's list (Cannon part 1) can be found in this maintenance event.
August 24th:
FASSE
boslogin
private login nodes - Триває · серпня 24, 2026 у 13:00UTCЗараз триває технічне обслуговування
- Заплановано · серпня 03, 2026 у 19:35UTC
FASRC will be doing OS upgrades to the latest version of Rocky 8.10 from August 24-27th. These rolling upgrades will improve cluster security and install the latest Cuda 13.2 driver.
Upgrades will happen in stages with each stage occurring between 8a-5p each day. During that period the nodes being upgraded will be unavailable. No jobs will be canceled but jobs will stay pending if all nodes in the partition are down.
Note that during this week the cluster will be in a partially upgraded state, thus jobs that span multiple nodes may become unstable due to mismatching libraries. Users concerned about this should wait until August 28th to restart runs.
Users should plan their work accordingly.
The upgrade schedule with impacted partitions is as follows:
August 24th:
FASSE
boslogin
private login nodesAugust 25th: Cannon Part 1
holylogin
arguelles_delgado
blackhole
davies_gpu
davies
desai
eddy
holy-cow
holy-smokes
huce_bigmem
huce_cascade
huttenhower
jacobsen2
janson_bigmem
janson_cascade
janson
ke
lukin
nguyen
olveczky_gpu
remoteviz
seas_compute
shared
sompolinsky_gpu
tambe
vishwanath
whipple
xlin
xlin_ice
zhuang_gpu
zhuangAugust 26th: Cannon Part 2
arguelles_delgado
conroy
davies
doshi-velez
dsouza
edwards
geophysics
giribet
gpu_test
hernquist
huce_cascadeimasc
janson
kempner_h200
kempner_rtx
murphy
ni_lab
olveczky
ortegahernandez
pehlevan
seas_compute
shakhnovich
shared
unrestricted
xlin
yin
zonAugust 27th: Cannon Part 3
arguelles_delgado_gpu_a100
arguelles_delgado_gpu_mixed
arguelles_delgado_h100
bigmem_intermediate
bigmem
blackhole_gpu
dvorkin
eddy
enos
gershman
gpu
gpu_h200
hejazi
hernquist_ice
hoekstra
hsph_gpu
hsph
huce_ice
iaifi_gpu
intermediate
itc_cluster
itc_gpu
janson_sapphire
joonholee
jshapiro
kempner_h100
kempner_h200
kempner
kempner_interactive
kovac
kozinsky_gpu
kozinsky
murphy_ice
mweber_compute
mweber_gpu
olveczky_sapphire
ortegahernandez_ice
rivas
sapphire
seas_compute
seas_gpu
seas_gpu_perf
siag_gpu
siag_combo
siag
sur
test
yao_alphatns
yao_gpu
yao
zhuang
- Оновлення · UTC
A number of jobs appear to have terminated during the upgrade for some reason. Those jobs will re-queue or, if not, will need to be re-submitted,
We apologize for the inconvenience. - Вирішено · UTC
The update has been applied and the scheduler is returned to service. Jobs are unpaused.
- Вивчається · UTC
We have been manually dealing with a Slurm memory leak bug behind the scenes and now have an official fix in the form of a Slurm upgrade (26.05.3). It is necessary to get this update in place as soon as possible.
At 2pm today we will pause the cluster. Jobs will be paused, not terminated, and the scheduler will be unavailable for querying or submitting new jobs until the update is completed. Once complete, jobs will resume.
Technical information about the bug:
лип. 2026
- Завершено · липня 06, 2026 у 17:00UTCТехнічне обслуговування успішно завершено
- Триває · липня 06, 2026 у 13:00UTCЗараз триває технічне обслуговування
- Заплановано · червня 26, 2026 у 13:45UTC
FASRC monthly maintenance will take place on July 6th 2026. Our maintenance tasks should be completed between 9am-1pm.
Cannon cluster will be paused during this maintenance?: NO
FASSE cluster will be paused during this maintenance?: NONOTICES:
Friday July 3rd is a university holiday (independence Day observed)
Training: Upcoming training from FASRC and other sources can be found on our Training Calendar. at https://www.rc.fas.harvard.edu/upcoming-training/
Status Page: You can subscribe to our status to receive notifications of maintenance, incidents, and their resolution at https://status.rc.fas.harvard.edu/ (click Get Updates for options).
We'd love to hear success stories about your or your lab's use of FASRC. Submit your story here.
MAINTENANCE TASKS
Domain controller replacement
Audience: Internal
Impact: None. End users should not see any impact.
Reboot drained nodes in error state
Audience: Cluster nodes with errors.
Impact: These nodes will have been drained already in preparation. No impact on jobs on the day and the affected nodes will return to service in their respective partitions after the maintenance period.
OOD/Open OnDemand reboots
Audience: All OOD users, reboot of the head nodes.
Impact: Running sessions will not be affected.
Login node reboots
Audience; All login node users.
Impact: Login nodes will reboot during the maintenance window.
Netscratch 90-day retention cleanup
Audience; All netscratch users
Impact: Files older than 90 days will be removed per our scratch policy. Please note that this cleanup can happen at any time, not just during maintenance.
Thank you,
FAS Research Computing
https://docs.rc.fas.harvard.edu/
https://www.rc.fas.harvard.edu/
черв. 2026
- Завершено · червня 18, 2026 у 21:15UTC
The yearly power downtime at our Holyoke data center, MGHPCC, has completed.
The clusters and storage are back online and login nodes and OOD nodes are now available.
If you have an issue/need help, please send a ticket to rchelp@rc.fas.harvard.edu with details.
IMPORTANT NOTE: Tomorrow, June 19th is a university holiday. FASRC staff will return Monday to address any lingering issues and any new tickets.
- Оновлення · червня 18, 2026 у 20:45UTC
Power-up is nearly complete, but a delay earlier in the day has us slightly behind.
New ETA is 6PM.
- Оновлення · червня 18, 2026 у 12:16UTC
MGHPCC has completed their maintenance and restored power to the facility.
FASRC will now begin the power-up process. Please be aware that this takes several hours.
We will update this status once complete.
NOTE: A reminder that tomorrow (Friday) is a university holiday.
- Триває · червня 15, 2026 у 13:00UTCMaintenance is now in progress
- Заплановано · червня 15, 2026 у 13:00UTC
The yearly power downtime at our Holyoke data center, MGHPCC, has been scheduled by the facility. This year's power downtime will take place on Tuesday June 15th - 18th, 2025. There will be no June monthly maintenance as a result.
Since the facility will be powered down for two days this year, we will not be performing the usual maintenance tasks.
That said, networking and other key infrastructure will be doing maintenance.IMPORTANT NOTE: FASRC storage at both Holyoke and Boston will be affected and should not be expected to be available throughout the downtime. Please plan ahead accordingly.
Monday June 15th - Power-down begins at 9AM
Tuesday June 16th - Power out at MGHPCC
Wednesday June 17th - Power out at MGHPCC
Thursday June 18th - Expected return to full service by 5PM
Friday June 19th - Please note that June 19th is a university holiday

For more detailed information and follow-up, please see:
https://www.rc.fas.harvard.edu/mghpcc-yearly-shutdown or this Status Page

