SLURM Scheduler - Cannon - Đang hoạt động
SLURM Scheduler - Cannon
Cannon Compute Cluster (Holyoke) - Đang hoạt động
Cannon Compute Cluster (Holyoke)
Boston Compute Nodes - Đang hoạt động
Boston Compute Nodes
GPU nodes (Holyoke) - Đang hoạt động
GPU nodes (Holyoke)
seas_compute - Đang hoạt động
seas_compute
SLURM Scheduler - FASSE - Đang hoạt động
SLURM Scheduler - FASSE
FASSE Compute Cluster (Holyoke) - Đang hoạt động
FASSE Compute Cluster (Holyoke)
Kempner Cluster CPU - Đang hoạt động
Kempner Cluster CPU
Kempner Cluster GPU - Đang hoạt động
Kempner Cluster GPU
FASSE login nodes - Đang hoạt động
FASSE login nodes
Cannon Open OnDemand - Đang hoạt động
Cannon Open OnDemand
FASSE Open OnDemand - Đang hoạt động
FASSE Open OnDemand
Netscratch (Global Scratch) - Đang hoạt động
Netscratch (Global Scratch)
Home Directory Storage - Boston - Đang hoạt động
Home Directory Storage - Boston
Tape - (Tier 3) - Đang hoạt động
Tape - (Tier 3)
Holylabs - Đang hoạt động
Holylabs
Isilon Storage Holyoke (Tier 1) - Đang hoạt động
Isilon Storage Holyoke (Tier 1)
Holystore01 (Tier 0) - Đang hoạt động
Holystore01 (Tier 0)
HolyLFS04 (Tier 0) - Đang hoạt động
HolyLFS04 (Tier 0)
HolyLFS05 (Tier 0) - Đang hoạt động
HolyLFS05 (Tier 0)
HolyLFS06 (Tier 0) - Đang hoạt động
HolyLFS06 (Tier 0)
Holyoke Tier 2 NFS - Đang hoạt động
Holyoke Tier 2 NFS
Holyoke Specialty Storage - Đang hoạt động
Holyoke Specialty Storage
holECS - Đang hoạt động
holECS
Isilon Storage Boston (Tier 1) - Đang hoạt động
Isilon Storage Boston (Tier 1)
BosLFS02 (Tier 0) - Đang hoạt động
BosLFS02 (Tier 0)
Boston Tier 2 NFS - Đang hoạt động
Boston Tier 2 NFS
CEPH Storage Boston (Tier 2) - Sự cố lớn
CEPH Storage Boston (Tier 2)
Boston Specialty Storage - Đang hoạt động
Boston Specialty Storage
bosECS - Đang hoạt động
bosECS
Samba Cluster - Đang hoạt động
Samba Cluster
Globus Data Transfer - Sự cố lớn
Globus Data Transfer
Lịch sử thông báo
Xem trạng thái hiện tạithg 8 2026
- Hoàn thành · tháng 08 27, 2026 tại 20:06UTC
OS upgrade work is mostly completed.
There are a few remaining nodes in the cluster that did not successfully receive the upgrade.
If you see nodes that are 'DOWN' for the following reasons, they are all nodes that need manual intervention for OS upgrade. FASRC staff are aware of them and will upgrade these nodes over the next week.
"wrong kernel"
"Not responding"
"unexpected reboot"
"gres/gpu count lower than configured"
"/n/sw not mounted"
We appreciate your continued patience.
- Cập nhật · tháng 08 26, 2026 tại 20:01UTC
Today's work is complete. The final round, Cannon Part 3, takes place tomorrow.
August 26th: Cannon Part 2
arguelles_delgado
conroy
davies
doshi-velez
dsouza
edwards
geophysics
giribet
gpu_test
hernquist
huce_cascadeimasc
janson
kempner_h200
kempner_rtx
murphy
ni_lab
olveczky
ortegahernandez
pehlevan
seas_compute
shakhnovich
shared
unrestricted
xlin
yin
zon - Cập nhật · tháng 08 25, 2026 tại 19:37UTC
Today's work is complete for Cannon Part 1 and will resume tomorrow for Cannon Part 2
August 25th: Cannon Part 1
holylogin
arguelles_delgado
blackhole
davies_gpu
davies
desai
eddy
holy-cow
holy-smokes
huce_bigmem
huce_cascade
huttenhower
jacobsen2
janson_bigmem
janson_cascade
janson
ke
lukin
nguyen
olveczky_gpu
remoteviz
seas_compute
shared
sompolinsky_gpu
tambe
vishwanath
whipple
xlin
xlin_ice
zhuang_gpu
zhuang - Cập nhật · tháng 08 24, 2026 tại 19:07UTC
Today's portion of the upgrades are completed. Tomorrow's list (Cannon part 1) can be found in this maintenance event.
August 24th:
FASSE
boslogin
private login nodes - Đang tiến hành · tháng 08 24, 2026 tại 13:00UTCHiện đang tiến hành bảo trì
- Đã lên kế hoạch · tháng 08 03, 2026 tại 19:35UTC
FASRC will be doing OS upgrades to the latest version of Rocky 8.10 from August 24-27th. These rolling upgrades will improve cluster security and install the latest Cuda 13.2 driver.
Upgrades will happen in stages with each stage occurring between 8a-5p each day. During that period the nodes being upgraded will be unavailable. No jobs will be canceled but jobs will stay pending if all nodes in the partition are down.
Note that during this week the cluster will be in a partially upgraded state, thus jobs that span multiple nodes may become unstable due to mismatching libraries. Users concerned about this should wait until August 28th to restart runs.
Users should plan their work accordingly.
The upgrade schedule with impacted partitions is as follows:
August 24th:
FASSE
boslogin
private login nodesAugust 25th: Cannon Part 1
holylogin
arguelles_delgado
blackhole
davies_gpu
davies
desai
eddy
holy-cow
holy-smokes
huce_bigmem
huce_cascade
huttenhower
jacobsen2
janson_bigmem
janson_cascade
janson
ke
lukin
nguyen
olveczky_gpu
remoteviz
seas_compute
shared
sompolinsky_gpu
tambe
vishwanath
whipple
xlin
xlin_ice
zhuang_gpu
zhuangAugust 26th: Cannon Part 2
arguelles_delgado
conroy
davies
doshi-velez
dsouza
edwards
geophysics
giribet
gpu_test
hernquist
huce_cascadeimasc
janson
kempner_h200
kempner_rtx
murphy
ni_lab
olveczky
ortegahernandez
pehlevan
seas_compute
shakhnovich
shared
unrestricted
xlin
yin
zonAugust 27th: Cannon Part 3
arguelles_delgado_gpu_a100
arguelles_delgado_gpu_mixed
arguelles_delgado_h100
bigmem_intermediate
bigmem
blackhole_gpu
dvorkin
eddy
enos
gershman
gpu
gpu_h200
hejazi
hernquist_ice
hoekstra
hsph_gpu
hsph
huce_ice
iaifi_gpu
intermediate
itc_cluster
itc_gpu
janson_sapphire
joonholee
jshapiro
kempner_h100
kempner_h200
kempner
kempner_interactive
kovac
kozinsky_gpu
kozinsky
murphy_ice
mweber_compute
mweber_gpu
olveczky_sapphire
ortegahernandez_ice
rivas
sapphire
seas_compute
seas_gpu
seas_gpu_perf
siag_gpu
siag_combo
siag
sur
test
yao_alphatns
yao_gpu
yao
zhuang
- Cập nhật · UTC
A number of jobs appear to have terminated during the upgrade for some reason. Those jobs will re-queue or, if not, will need to be re-submitted,
We apologize for the inconvenience. - Đã khắc phục · UTC
The update has been applied and the scheduler is returned to service. Jobs are unpaused.
- Đang điều tra · UTC
We have been manually dealing with a Slurm memory leak bug behind the scenes and now have an official fix in the form of a Slurm upgrade (26.05.3). It is necessary to get this update in place as soon as possible.
At 2pm today we will pause the cluster. Jobs will be paused, not terminated, and the scheduler will be unavailable for querying or submitting new jobs until the update is completed. Once complete, jobs will resume.
Technical information about the bug:
thg 7 2026
- Hoàn thành · tháng 07 06, 2026 tại 17:00UTCBảo trì đã hoàn tất thành công
- Đang tiến hành · tháng 07 06, 2026 tại 13:00UTCHiện đang tiến hành bảo trì
- Đã lên kế hoạch · tháng 06 26, 2026 tại 13:45UTC
FASRC monthly maintenance will take place on July 6th 2026. Our maintenance tasks should be completed between 9am-1pm.
Cannon cluster will be paused during this maintenance?: NO
FASSE cluster will be paused during this maintenance?: NONOTICES:
Friday July 3rd is a university holiday (independence Day observed)
Training: Upcoming training from FASRC and other sources can be found on our Training Calendar. at https://www.rc.fas.harvard.edu/upcoming-training/
Status Page: You can subscribe to our status to receive notifications of maintenance, incidents, and their resolution at https://status.rc.fas.harvard.edu/ (click Get Updates for options).
We'd love to hear success stories about your or your lab's use of FASRC. Submit your story here.
MAINTENANCE TASKS
Domain controller replacement
Audience: Internal
Impact: None. End users should not see any impact.
Reboot drained nodes in error state
Audience: Cluster nodes with errors.
Impact: These nodes will have been drained already in preparation. No impact on jobs on the day and the affected nodes will return to service in their respective partitions after the maintenance period.
OOD/Open OnDemand reboots
Audience: All OOD users, reboot of the head nodes.
Impact: Running sessions will not be affected.
Login node reboots
Audience; All login node users.
Impact: Login nodes will reboot during the maintenance window.
Netscratch 90-day retention cleanup
Audience; All netscratch users
Impact: Files older than 90 days will be removed per our scratch policy. Please note that this cleanup can happen at any time, not just during maintenance.
Thank you,
FAS Research Computing
https://docs.rc.fas.harvard.edu/
https://www.rc.fas.harvard.edu/
thg 6 2026
- Hoàn thành · tháng 06 18, 2026 tại 21:15UTC
The yearly power downtime at our Holyoke data center, MGHPCC, has completed.
The clusters and storage are back online and login nodes and OOD nodes are now available.
If you have an issue/need help, please send a ticket to rchelp@rc.fas.harvard.edu with details.
IMPORTANT NOTE: Tomorrow, June 19th is a university holiday. FASRC staff will return Monday to address any lingering issues and any new tickets.
- Cập nhật · tháng 06 18, 2026 tại 20:45UTC
Power-up is nearly complete, but a delay earlier in the day has us slightly behind.
New ETA is 6PM.
- Cập nhật · tháng 06 18, 2026 tại 12:16UTC
MGHPCC has completed their maintenance and restored power to the facility.
FASRC will now begin the power-up process. Please be aware that this takes several hours.
We will update this status once complete.
NOTE: A reminder that tomorrow (Friday) is a university holiday.
- Đang tiến hành · tháng 06 15, 2026 tại 13:00UTCMaintenance is now in progress
- Đã lên kế hoạch · tháng 06 15, 2026 tại 13:00UTC
The yearly power downtime at our Holyoke data center, MGHPCC, has been scheduled by the facility. This year's power downtime will take place on Tuesday June 15th - 18th, 2025. There will be no June monthly maintenance as a result.
Since the facility will be powered down for two days this year, we will not be performing the usual maintenance tasks.
That said, networking and other key infrastructure will be doing maintenance.IMPORTANT NOTE: FASRC storage at both Holyoke and Boston will be affected and should not be expected to be available throughout the downtime. Please plan ahead accordingly.
Monday June 15th - Power-down begins at 9AM
Tuesday June 16th - Power out at MGHPCC
Wednesday June 17th - Power out at MGHPCC
Thursday June 18th - Expected return to full service by 5PM
Friday June 19th - Please note that June 19th is a university holiday

For more detailed information and follow-up, please see:
https://www.rc.fas.harvard.edu/mghpcc-yearly-shutdown or this Status Page

