Duration: 19 hours and 9 minutes

Affected Components: HolyLFS06 (Tier 0)

Jun 19, 19:00:00 GMT+0
調査中 - holylfs06 storage may be slow or unresponsive. We are investigating this issue..

Jun 20, 14:09:15 GMT+0
解決済み - This incident has been resolved..

Duration: 3 days, 8 hours and 16 minutes

Affected Components: Globus Data Transfer, Coldfront, Boston Data Center, GPU nodes (Holyoke), Infiniband - Holyoke/MGHPCC, FASSE Compute Cluster (Holyoke), SLURM Scheduler - FASSE, Boston Tier 2 NFS (new), Holyoke Tier 2 NFS (new), Holylabs, Holyoke Firewall, Network - Holyoke/MGHPCC, Holyoke-Boston fiber link (short path), Infiniband - Boston, HolyLFS06 (Tier 0), bosECS, Boston Specialty Storage, Software & Modules, Holyoke/MGHPCC Data Center, Cannon Compute Cluster (Holyoke), Network - Boston, Cambridge firewall and other redundancy, License Servers, seas_compute, FASSE login nodes, Starfish, Web Proxies, Authentication, Virtual Infrastructure - Holyoke, FIINE billing portal, CEPH Storage Boston (Tier 2), NESE (NorthEast Storage Exchange), Isilon Storage Holyoke (Tier 1), holECS, Holyoke Specialty Storage, FASRC Downloads Site, Citrix, HolyLFS05 (Tier 0), Virtual Infrastructure - Boston, Network - Cambridge, FASRC VPN (Cambridge) , FASRC VPN (Boston), BosLFS02 (Tier 0), Isilon Storage Boston (Tier 1), Home Directory Storage - Boston, Netscratch (Global Scratch), Tape - (Tier 3), Boston Compute Nodes, portal.rc.fas.harvard.edu, Login Nodes - Holyoke, Login Nodes - Boston, Samba Cluster, HolyLFS04 (Tier 0), Holystore01 (Tier 0), FASRC Two-Factor (OpenAuth), Holyoke-Boston fiber link (long path), Grafana Cloud (FASRC), SLURM Scheduler - Cannon, Kempner Cluster GPU, Kempner Cluster CPU, Cannon Open OnDemand/VDI, FASSE Open OnDemand/VDI

Jun 15, 13:00:01 GMT+0
原因判明済み - Maintenance is now in progress.

Jun 15, 13:00:00 GMT+0
原因判明済み - The yearly power downtime at our Holyoke data center, MGHPCC, has been scheduled by the facility. This year's power downtime will take place on Tuesday June 15th - 18th, 2025\. There will be no June monthly maintenance as a result. Since the facility will be powered down for two days this year, we will not be performing the usual maintenance tasks. That said, networking and other key infrastructure will be doing maintenance. **IMPORTANT NOTE**: FASRC storage at both Holyoke and Boston **will be** affected and should not be expected to be available throughout the downtime. Please plan ahead accordingly. * **Monday June 15th** \- Power-down begins at 9AM * **Tuesday June 16th** \- Power out at MGHPCC * **Wednesday June 17th** \- Power out at MGHPCC * **Thursday June 18th** \- Expected return to full service by 5PM * **Friday June 19th** \- Please note that June 19th is a university holiday  **For more detailed information and follow-up, please see:** <https://www.rc.fas.harvard.edu/mghpcc-yearly-shutdown> **or this** [**Status Page**](https://status.rc.fas.harvard.edu/).

Jun 18, 12:16:49 GMT+0
原因判明済み - MGHPCC has completed their maintenance and restored power to the facility. FASRC will now begin the power-up process. Please be aware that this takes several hours. We will update this status once complete. NOTE: A reminder that tomorrow (Friday) is a university holiday. .

Jun 18, 20:45:50 GMT+0
原因判明済み - Power-up is nearly complete, but a delay earlier in the day has us slightly behind. New ETA is 6PM..

Jun 18, 21:15:42 GMT+0
完了 - The yearly power downtime at our Holyoke data center, MGHPCC, has completed. The clusters and storage are back online and login nodes and OOD nodes are now available. If you have an issue/need help, please send a ticket to [rchelp@rc.fas.harvard.edu](mailto:rchelp@rc.fas.harvard.edu) with details. IMPORTANT NOTE: Tomorrow, June 19th is a university holiday. FASRC staff will return Monday to address any lingering issues and any new tickets..

Duration: 25 minutes

Affected Components: HolyLFS06 (Tier 0)

Jun 10, 20:07:18 GMT+0
解決済み - This incident has been resolved..

Jun 10, 19:41:54 GMT+0
調査中 - Holylfs06 storage may be slow or unresponsive We are currently investigating this incident..

Duration: 3 hours and 16 minutes

Affected Components: portal.rc.fas.harvard.edu

Jun 9, 19:38:55 GMT+0
解決済み - The Portal website should be accessible for all now. This incident has been resolved..

Jun 9, 16:22:32 GMT+0
調査中 - There is an SSL issue with the Portal which will cause an error for anyone attempting to connect. Investigating..

Duration: 5 minutes

Affected Components: Authentication

Jun 8, 20:24:54 GMT+0
調査中 - Authentication issues with openauth/radius. This incident was created by an automated monitoring service..

Jun 8, 20:29:54 GMT+0
解決済み - Openauth/radius is now operational. This update was created by an automated monitoring service..

Duration: 12 minutes

Affected Components: Coldfront

Jun 8, 20:02:18 GMT+0
調査中 - Coldfront is showing a server error. Investigating..

Jun 8, 20:14:34 GMT+0
解決済み - This incident has been resolved. Coldfront is now available..

Duration: 9 minutes

Affected Components: Login Nodes - Holyoke, Login Nodes - Boston

Jun 8, 13:16:28 GMT+0
調査中 - [lofin.rc.fas.harvard.edu](http://lofin.rc.fas.harvard.edu) is refusing new connections and will error out when attempting to SSH in. Investigating..

Jun 8, 13:25:32 GMT+0
解決済み - This incident has been resolved and SSH login to [login.rc.fas.harvard.edu](http://login.rc.fas.harvard.edu) is working again. Thanks for your patience..

Affected Components: FASRC VPN (Cambridge)

Jun 5, 04:06:55 GMT+0
調査中 - FASRC VPN (Cambridge) is down at the moment. This incident was automatically created by Instatus monitoring..

Jun 5, 04:16:32 GMT+0
解決済み - FASRC VPN (Cambridge) is back up. This incident was automatically resolved by Instatus monitoring..

Duration: 22 minutes

Affected Components: Authentication, portal.rc.fas.harvard.edu

Jun 2, 20:42:13 GMT+0
調査中 - Password reset emails are not being sent out at this time. We are currently investigating this incident..

Jun 2, 21:04:09 GMT+0
解決済み - Password reset emails are now being sent out. This incident has been resolved..

Duration: 2 days, 18 hours and 24 minutes

Affected Components: GPU nodes (Holyoke), Cannon Compute Cluster (Holyoke), seas_compute, SLURM Scheduler - Cannon, Kempner Cluster GPU, Kempner Cluster CPU, Cannon Open OnDemand/VDI, FASSE Open OnDemand/VDI, Boston Compute Nodes

Jun 1, 14:45:41 GMT+0
解決済み - Slurm crashed on 4:30p on Friday due to a user running a large sacct query against the Slurm database. This caused the database host to run out of memory and crash the scheduler. To prevent this from reoccurring we are reducing the time range that users are permitted to query at one time to 7 days. Thus if you need to cover a month you would need to query in four 7 day increments. We do ask users to be judicious in their querying of the Slurm. Only ask for those fields that you require. Please also ensure any AI agents you have running limit their queries appropriately..

May 29, 20:21:17 GMT+0
調査中 - The Slurm scheduler is experiencing an error which is impacting jobs. The Cannon cluster will be inaccessible while we troubleshoot. We are currently investigating this incident..

May 29, 21:13:09 GMT+0
原因判明済み - To temporarily stabilize the situation, we have reduced the maximum query time for sacct and other Slurm commands to be 1 day. We have filed a ticket with SchedMD to further analyze the issue. The cluster is back up and the scheduler is accepting new jobs. We will continue to monitor for emergencies over the weekend, and resume in-depth troubleshooting on Monday. .

Duration: 3 days, 23 hours and 2 minutes

Affected Components: portal.rc.fas.harvard.edu

May 22, 15:18:39 GMT+0
調査中 - Account approvals through Portal are not available at this time. We are currently investigating this incident..

May 26, 14:20:10 GMT+0
解決済み - This incident has been resolved. Login is again available for approvals/onboarding..

Affected Components: FASRC VPN (Cambridge)

May 22, 04:50:10 GMT+0
調査中 - FASRC VPN (Cambridge) is down at the moment. This incident was automatically created by Instatus monitoring..

May 22, 05:09:48 GMT+0
解決済み - FASRC VPN (Cambridge) is back up. This incident was automatically resolved by Instatus monitoring..

Affected Components: FASRC VPN (Cambridge)

May 22, 03:35:48 GMT+0
調査中 - FASRC VPN (Cambridge) is down at the moment. This incident was automatically created by Instatus monitoring..

May 22, 03:55:19 GMT+0
解決済み - FASRC VPN (Cambridge) is back up. This incident was automatically resolved by Instatus monitoring..

Affected Components: FASRC VPN (Cambridge)

May 22, 03:05:05 GMT+0
調査中 - FASRC VPN (Cambridge) is down at the moment. This incident was automatically created by Instatus monitoring..

May 22, 03:14:35 GMT+0
解決済み - FASRC VPN (Cambridge) is back up. This incident was automatically resolved by Instatus monitoring..

Affected Components: , Login Nodes - Holyoke, Login Nodes - Boston, , Login Nodes → login.rc.fas.harvard.edu →

May 22, 02:40:38 GMT+0
調査中 - login.rc.fas.harvard.edu is not responding normally. This incident was automatically created..

May 22, 03:40:12 GMT+0
解決済み - \\\[login.rc.fas.harvard.edu\\\](http://login.rc.fas.harvard.edu) is responding normally. This incident was automatically resolved..

Affected Components: FASRC VPN (Cambridge)

May 22, 02:24:22 GMT+0
調査中 - FASRC VPN (Cambridge) is down at the moment. This incident was automatically created by Instatus monitoring..

May 22, 02:53:53 GMT+0
解決済み - FASRC VPN (Cambridge) is back up. This incident was automatically resolved by Instatus monitoring..

Affected Components: FASRC VPN (Cambridge)

May 22, 02:13:11 GMT+0
解決済み - FASRC VPN (Cambridge) is back up. This incident was automatically resolved by Instatus monitoring..

May 22, 01:43:38 GMT+0
調査中 - FASRC VPN (Cambridge) is down at the moment. This incident was automatically created by Instatus monitoring..

Duration: 45 minutes

Affected Components: FIINE billing portal

May 21, 15:04:16 GMT+0
解決済み - This incident has been resolved. Fiine is back online..

May 21, 14:18:48 GMT+0
調査中 - The FIINE financial tool is down. This affects only a small number of people, but we are noting here out of caution..

Affected Components: portal.rc.fas.harvard.edu

May 21, 13:17:12 GMT+0
調査中 - portal.rc.fas.harvard.edu is down at the moment. This incident was created automatically..

May 21, 13:27:23 GMT+0
解決済み - portal.rc.fas.harvard.edu is back up. his incident was resolved automatically..

Affected Components: FASRC VPN (Cambridge)

May 21, 04:43:03 GMT+0
調査中 - FASRC VPN (Cambridge) is down at the moment. This incident was automatically created by Instatus monitoring..

May 21, 05:29:42 GMT+0
解決済み - FASRC VPN (Cambridge) is back up. This incident was automatically resolved by Instatus monitoring..

Duration: 3 days, 13 hours and 15 minutes

Affected Components: GPU nodes (Holyoke), FASSE Compute Cluster (Holyoke), SLURM Scheduler - FASSE, Cannon Compute Cluster (Holyoke), seas_compute, Kempner Cluster GPU, Kempner Cluster CPU, SLURM Scheduler - Cannon, Boston Compute Nodes

May 18, 11:00:00 GMT+0
原因判明済み - Rescheduled to May 18.

May 18, 11:00:00 GMT+0
原因判明済み - Our Holyoke data center, MGHPCC, will be doing power work on Row 8A. This work, which is being completed over the course of 2 weeks, will bring online another power feed which will increase power capacity. In order to do this work, it will require us to idle half the nodes in 8a for the duration of the week. This means all partitions in this row will be at half capacity. Existing jobs should drain naturally and no job should need to be canceled. The impacted partitions are: ``` arguelles_delgado_h100 bigmem bigmem_intermediate blackhole_gpu dvorkin eddy enos gershman gpu gpu_h200 gpu_requeue hejazi hernquist_ice hoekstra hsph hsph_gpu huce_ice iaifi_gpu_requeue intermediate itc_cluster itc_gpu janson_sapphire joonholee jshapiro kempner kempner_priority kempner_dev kempner_eng kempner_h200_priority kempner_h100 kempner_h100_priority kempner_h100_priority2 kempner_h100_priority3 kempner_h100_priority4 kempner_interactive kovac kozinsky kozinsky_gpu kozinsky_requeue murphy_ice mweber_compute mweber_gpu olveczky_sapphire ortegahernandez_ice rivas sapphire seas_compute seas_gpu siag siag_combo test yao yao_priority zhuang ```.

May 18, 11:00:01 GMT+0
原因判明済み - Maintenance is now in progress.

May 22, 00:14:55 GMT+0
完了 - The power work has completed successfully. All nodes have been returned to normal service..

Duration: 5 days and 12 hours

Affected Components: GPU nodes (Holyoke), FASSE Compute Cluster (Holyoke), SLURM Scheduler - FASSE, Cannon Compute Cluster (Holyoke), seas_compute, Kempner Cluster GPU, Kempner Cluster CPU, SLURM Scheduler - Cannon, Boston Compute Nodes

May 11, 11:00:00 GMT+0
原因判明済み - Rescheduled to May 11.

May 11, 11:00:00 GMT+0
原因判明済み - Our Holyoke data center, MGHPCC, will be doing power work on Row 8A. This work, which will occur this week and next week, will bring online another power feed which will increase power capacity. In order to do this work, it will require us to idle half the nodes in 8a for the duration of the week. This means all partitions in this row will be at half capacity. Existing jobs should drain naturally and no job should need to be canceled. The impacted partitions are: ``` arguelles_delgado_h100 bigmem bigmem_intermediate blackhole_gpu dvorkin eddy enos gershman gpu gpu_h200 gpu_requeue hejazi hernquist_ice hoekstra hsph hsph_gpu huce_ice iaifi_gpu_requeue intermediate itc_cluster itc_gpu janson_sapphire joonholee jshapiro kempner kempner_priority kempner_dev kempner_eng kempner_h200_priority kempner_h100 kempner_h100_priority kempner_h100_priority2 kempner_h100_priority3 kempner_h100_priority4 kempner_interactive kovac kozinsky kozinsky_gpu kozinsky_requeue murphy_ice mweber_compute mweber_gpu olveczky_sapphire ortegahernandez_ice rivas sapphire seas_compute siag siag_combo test yao yao_priority zhuang ```.

May 16, 23:00:00 GMT+0
完了 - Maintenance has completed successfully.

May 11, 11:00:01 GMT+0
原因判明済み - Maintenance is now in progress.

Affected Components: www.rc.fas.harvard.edu

May 6, 16:20:55 GMT+0
調査中 - www.rc.fas.harvard.edu is down at the moment. This incident was created automatically..

May 6, 16:30:56 GMT+0
解決済み - www.rc.fas.harvard.edu is back up. This incident was resolved automatically..

Duration: 4 hours

Affected Components: FASRC Two-Factor (OpenAuth), GPU nodes (Holyoke), FASSE Compute Cluster (Holyoke), SLURM Scheduler - FASSE, Cannon Compute Cluster (Holyoke), seas_compute, Cannon Open OnDemand/VDI, FASSE login nodes, Kempner Cluster GPU, Kempner Cluster CPU, FASSE Open OnDemand/VDI, Login Nodes - Holyoke, SLURM Scheduler - Cannon, Login Nodes - Boston, Netscratch (Global Scratch), Boston Compute Nodes

May 4, 13:00:00 GMT+0
原因判明済み - FASRC monthly maintenance will take place on May 4th 2026\. Our maintenance tasks should be completed between 9am-1pm. **NOTICES:** * Annual data center power downtime: The annual downtime at MGHPCC will take place June 15 - June 18\. This year's downtime will be one day longer. More details will be sent to all users next month. * Training: Upcoming training from FASRC and other sources can be found on our Training Calendar. at <https://www.rc.fas.harvard.edu/upcoming-training/> * Status Page: You can subscribe to our status to receive notifications of maintenance, incidents, and their resolution at <https://status.rc.fas.harvard.edu/> (click Get Updates for options). **MAINTENANCE TASKS** Cannon cluster will be paused during this maintenance?: **YES** FASSE cluster will be paused during this maintenance?: **YES** * Slurm 25.11.5 Upgrade * Audience: All cluster users * Impact: Jobs will be paused during the upgrade * Reboot remaining stuck nodes from power outage * Audience: N/A * Impact: No visible impact to user * Two-Factor/OpenAuth ([two-factor.rc.fas.harvard.edu](http://two-factor.rc.fas.harvard.edu)) replacement * Audience: All account holders * Impact: The server will be unavailable during maintenance. You will be unable to obtain a new or replacement OpenAuth token during this period. * Domain controller replacement * Audience: Internal * Impact: End users should not see any impact * OOD/Open OnDemand reboots * Audience: All OOD users, reboot of the head nodes * Impact: Running sessions will _not_ be affected * Login node reboots * Audience; All login node users * Impact: Login nodes will reboot during the maintenance window * Netscratch 90-day retention cleanup * Audience; All netscratch users * Impact: Files older than 90 days will be removed per our [scratch policy](https://docs.rc.fas.harvard.edu/kb/policy-scratch/). Please note that this cleanup can happen at any time, not just during maintenance. Thank you, FAS Research Computing <https://docs.rc.fas.harvard.edu/> <https://www.rc.fas.harvard.edu/>.

May 4, 13:00:01 GMT+0
原因判明済み - Maintenance is now in progress.

May 4, 14:56:02 GMT+0
原因判明済み - The scheduler is re-opened and jobs un-paused. Other, non-impacting, work continues..

May 4, 17:00:00 GMT+0
完了 - Maintenance has completed successfully.

Duration: 1 hour

Affected Components: Starfish

May 1, 20:00:00 GMT+0
原因判明済み - Starfish will be upgraded to the latest version on Friday, May 1st from 4pm-5pm. The service and dashboard will be down during this time. .

May 1, 20:00:01 GMT+0
原因判明済み - Maintenance is now in progress.

May 1, 21:00:00 GMT+0
完了 - Maintenance has completed successfully.

Duration: 2 hours

Affected Components: Cannon Open OnDemand/VDI, FASSE Open OnDemand/VDI

Apr 30, 12:00:00 GMT+0
原因判明済み - At 8am on Thursday April 30th we will be upgrading from Open OnDemand version 4.0.7 to 4.1.4 on both the Cannon and FASSE clusters. This is not expected to impact running jobs. This upgrade adds the Jobs->Project Manager menu item and fixes an issue that affected access to the Clusters->Shell Access menu item when using Firefox..

Apr 30, 12:00:01 GMT+0
原因判明済み - Maintenance is now in progress.

Apr 30, 14:00:00 GMT+0
完了 - Maintenance has completed successfully.

Affected Components: , Login Nodes - Holyoke, Login Nodes - Boston, , Login Nodes → login.rc.fas.harvard.edu →

Apr 30, 02:31:26 GMT+0
調査中 - login.rc.fas.harvard.edu is not responding normally. This incident was automatically created..

Apr 30, 15:43:53 GMT+0
解決済み - \\\[login.rc.fas.harvard.edu\\\](http://login.rc.fas.harvard.edu) is responding normally. This incident was automatically resolved..

Duration: 16 hours and 2 minutes

Affected Components: Cannon Open OnDemand/VDI, FASSE login nodes, FASSE Open OnDemand/VDI, Login Nodes - Holyoke, Login Nodes - Boston

Apr 30, 02:11:01 GMT+0
原因判明済み - **Due to a serious in-the-wild exploit which can compromise Fedora-based Linux distributions including Rocky, which is used on the cluster, we need to restrict access. All login and OOD nodes are shut down until a fix can be put in place. Jobs running on the cluster will continue running.** **No ETA, There is not fix at this time. We will update our status page in the morning once we have more information or a fix to roll out.** **This is a serious exploit and we do not take this measure lightly. Please follow this status page for updates and eventual resolution.**.

Apr 30, 14:44:58 GMT+0
原因判明済み - We are developing a plan of attack to mitigate this exploit. Please know that this is a very serious issue and so we are treating it as such. Thank you for your understanding. We are currently awaiting further information from the Redhat/Fedora/Rocky community but building a plan in the meantime with the information we have. More details to follow as we can share them. If you need to access storage (except scratch and home directories), [Globus ](https://docs.rc.fas.harvard.edu/kb/globus-file-transfer/)is still online and available. But again, login nodes and OOD are not available..

Apr 30, 15:40:41 GMT+0
原因判明済み - To mitigate this exploit we will need to restart -all nodes- on the cluster. This **will begin at 1PM** and run until all nodes have restarted (no ETA). This **will mean** any un-finished jobs will be **terminated**. There is no way to avoid this. We will then be validating the fix before re-opening the login. OOD nodes, and scheduler. Next steps and updates will be posted here..

Apr 30, 18:12:41 GMT+0
解決済み - The cluster has been rebooted and all nodes, including login and OOD, have been patched. The scheduler is re-opened and jobs which were preempted/requeued have priority for re-scheduling. Some non-standard, lab-owned nodes may still require patching. The owners of these machines may be contacted about this. Thank you for your patience. This is a global issue and is being addressed at centers everywhere..

Duration: 33 minutes

Affected Components: HolyLFS06 (Tier 0)

Apr 29, 16:00:11 GMT+0
原因判明済み - Holylfs06 storage is down. We are investigating. More details as they are known..

Apr 29, 16:32:45 GMT+0
解決済み - Holylfs06 is accessible again. This incident has been resolved..

Duration: 17 minutes

Affected Components: docs.rc.fas.harvard.edu, www.rc.fas.harvard.edu

Apr 28, 17:00:00 GMT+0
原因判明済み - Security updates are required for [www.rc.fas.harvard.edu](http://www.rc.fas.harvard.edu) and [docs.rc.fas.harvard.edu](http://docs.rc.fas.harvard.edu) This work will take place today between 1pm and 2pm Both sites will be down for very short periods during the updates..

Apr 28, 17:00:01 GMT+0
原因判明済み - Maintenance is now in progress.

Apr 28, 17:16:58 GMT+0
完了 - Website maintenance has completed successfully..

Duration: 7 days and 18 hours

Affected Components: Starfish

Apr 20, 21:03:38 GMT+0
原因判明済み - Some starfish scans restarted following earlier issue and may be up to a week out of date due to the delayed scans Two scanning agents are still down. So please bear this in mind when viewing Starfish data..

Apr 21, 13:48:40 GMT+0
監視中 - The down agents are back in service. Scans are on-going but data will still be out of date until they catch up..

Apr 28, 15:03:28 GMT+0
解決済み - Scans are running normally. Any out of date stats will catch up shortly..

Affected Components: , Login Nodes - Holyoke, Login Nodes - Boston, , Login Nodes → login.rc.fas.harvard.edu →

Apr 19, 11:55:47 GMT+0
調査中 - login.rc.fas.harvard.edu is not responding normally. This incident was automatically created..

Apr 19, 12:55:25 GMT+0
解決済み - \\\[login.rc.fas.harvard.edu\\\](http://login.rc.fas.harvard.edu) is responding normally. This incident was automatically resolved..

Duration: 4 hours

Affected Components: FASRC Two-Factor (OpenAuth), , , Cannon Open OnDemand/VDI, FASSE login nodes, FASSE Open OnDemand/VDI, Login Nodes - Holyoke, Login Nodes - Boston, Netscratch (Global Scratch), , Login Nodes → VDI/OpenOnDemand → login.rc.fas.harvard.edu →

Apr 6, 13:00:01 GMT+0
原因判明済み - Maintenance is now in progress.

Apr 6, 17:00:00 GMT+0
完了 - Maintenance has completed successfully.

Apr 6, 13:00:00 GMT+0
原因判明済み - FASRC monthly maintenance will take place on April 6th 2026\. Our maintenance tasks should be completed between 9am-1pm. **NOTICES:** * Training: Upcoming training from FASRC and other sources can be found on our Training Calendar. at <https://www.rc.fas.harvard.edu/upcoming-training/> * Status Page: You can subscribe to our status to receive notifications of maintenance, incidents, and their resolution at <https://status.rc.fas.harvard.edu/> (click Get Updates for options). * We'd love to hear success stories about your or your lab's use of FASRC. Submit your story [here](https://www.rc.fas.harvard.edu/user-stories/). **MAINTENANCE TASKS** Cannon cluster will be paused during this maintenance?: **NO** FASSE cluster will be paused during this maintenance?: **NO** * [two-factor.rc.fas.harvard.edu](http://two-factor.rc.fas.harvard.edu) [OpenAuth](https://docs.rc.fas.harvard.edu/kb/openauth/) cut-over to new server * Audience: New accounts or anyone requesting an OpenAuth token * Impact: two-factor will be unavailable while moving to a new server * RStudio Server (Open OnDemand) * Audience: RStudio Server users on Cannon and FASSE * Impact: We will be decommissioning some versions of RStudio Server so we can properly maintain all production versions. Versions to be decommissioned: * R 4.1.3 (Bioconductor 3.14, RStudio 2022.02.0) * R 4.1.0 (Bioconductor 3.13, RStudio 1.4.1717) * R 4.0.3 (Bioconductor 3.12, Rstudio 1.3.1093) * R 4.0.0 (Bioconductor 3.11, Rstudio 1.3.1093) * If you use one of these versions, we recommend replacing it with the most recent version, R 4.4.2 (Bioconductor 3.20, RStudio 2024.12.0). You must reinstall previously installed libraries. * Domain controller replacement * Audience: Internal * Impact: End users should not see any impact * OOD/Open OnDemand reboots * Audience: All OOD users, reboot of the head nodes * Impact: Running sessions will _not_ be affected * Login node reboots * Audience; All login node users * Impact: Login nodes will reboot during the maintenance window * Netscratch 90-day retention cleanup * Audience; All netscratch users * Impact: Files older than 90 days will be removed per our [scratch policy](https://docs.rc.fas.harvard.edu/kb/policy-scratch/). Please note that this cleanup can happen at any time, not just during maintenance. Thank you, FAS Research Computing <https://docs.rc.fas.harvard.edu/> <https://www.rc.fas.harvard.edu/>.

Duration: 4 hours

Affected Components: Login Nodes - Holyoke, FASSE Compute Cluster (Holyoke), SLURM Scheduler - Cannon, , , GPU nodes (Holyoke), , SLURM Scheduler - FASSE, , Cannon Compute Cluster (Holyoke), , seas_compute, Cannon Open OnDemand/VDI, FASSE login nodes, Kempner Cluster CPU, Kempner Cluster GPU, FASSE Open OnDemand/VDI, Login Nodes - Boston, Netscratch (Global Scratch), , Boston Compute Nodes, Login Nodes → Cannon Cluster → VDI/OpenOnDemand → Kempner Cluster → FASSE Cluster → login.rc.fas.harvard.edu →

Mar 2, 14:00:01 GMT+0
原因判明済み - Maintenance is now in progress.

Mar 2, 14:00:00 GMT+0
原因判明済み - Monthly maintenance will take place on Monday March 2nd, 2026\. Our maintenance tasks should be completed between 9am-1pm. **NOTICES:** * Training: Upcoming training from FASRC and other sources can be found on our Training Calendar. at <https://www.rc.fas.harvard.edu/upcoming-training/> * Status Page: You can subscribe to our status to receive notifications of maintenance, incidents, and their resolution at <https://status.rc.fas.harvard.edu/> (click Get Updates for options). * We'd love to hear success stories about your or your lab's use of FASRC. Submit your story [here](https://www.rc.fas.harvard.edu/user-stories/). **MAINTENANCE TASKS** Cannon cluster will be paused during this maintenance?: **YES** FASSE cluster will be paused during this maintenance?: **YES** * Slurm scheduler update * Audience: All cluster users * Impact: Jobs will be paused during maintenance * OOD node reboots * Audience; All Open OnDemand users * Impact: OOD nodes will reboot during the maintenance window * Login node reboots * Audience: All login node users * Impact: Login nodes will reboot during the maintenance window * Netscratch retention purge * Audience: All users of Netscratch * Impact: Files older than 90 days will be removed. Please note that retention cleanup can and does run at any time, not just during the maintenance window. Thank you, FAS Research Computing <https://docs.rc.fas.harvard.edu/> [https://www.rc.fas.harvard.edu/](https://www.rc.fas.harvard.edu/upcoming-training/).

Mar 2, 18:00:00 GMT+0
完了 - Maintenance has completed successfully.

Duration: 1 day

Affected Components: Starfish

Feb 25, 14:00:00 GMT+0
原因判明済み - Starfish will be unavailable starting Wednesday, February 25th at 9AM until Thursday, February 26th at 9AM, for routine maintenance. The online dashboard will be inaccessible during this time..

Feb 26, 14:00:00 GMT+0
完了 - Maintenance has completed successfully.

Feb 25, 14:00:01 GMT+0
原因判明済み - Maintenance is now in progress.

Duration: 9 hours

Affected Components: NESE (NorthEast Storage Exchange)

Feb 19, 13:00:01 GMT+0
原因判明済み - Maintenance is now in progress.

Feb 19, 13:00:00 GMT+0
原因判明済み - From our partners at NESE. Details follow: We are installing four new tape frames, which will bring the tape system raw storage capacity to 253 petabytes. **Service Affected:** NESE Tape Service **Maintenance Window:** 8:00 AM - 5:00 PM (EST) * The tape service will be unavailable. * All upgrade activities are expected to be completed on the same day. NOTES: * Monitor the MGHPCC Slack #nese channel for status updates and announcements * Monitor <https://nese.instatus.com/> for real-time updates on progress Subscribe to <https://nese.instatus.com/subscribe/email> for updates and announcements.

Feb 19, 22:00:00 GMT+0
完了 - Maintenance has completed successfully.

Duration: 8 minutes

Affected Components: docs.rc.fas.harvard.edu, www.rc.fas.harvard.edu

Feb 9, 21:10:00 GMT+0
原因判明済み - Security updates will require a brief interruption for our primary websites [www.rc.fas.harvard.edu](http://www.rc.fas.harvard.edu) and [docs.rc.fas.harvard.edu](http://docs.rc.fas.harvard.edu) We will endeavour to keep this update as short as possible. Each site may be unavailable for a few minutes..

Feb 9, 21:10:01 GMT+0
原因判明済み - Maintenance is now in progress.

Feb 9, 21:17:51 GMT+0
完了 - Maintenance has completed successfully..

Duration: 9 hours

Affected Components: NESE (NorthEast Storage Exchange)

Feb 9, 13:00:00 GMT+0
原因判明済み - From our partners at NESE. Details follow: In the process of the tape front-end file caching system upgrade, we will be installing a new IBM Storage Scale System 6000\. We will provide an additional update for when the software integration and data transfer from the current IBM Elastic Storage System 5000 will be performed. **Service Affected:** NESE Tape Service **Maintenance Window: No Downtime expected** NOTES: * Monitor the MGHPCC Slack #nese channel for status updates and announcements * Monitor <https://nese.instatus.com/> for real-time updates on progress * Subscribe to <https://nese.instatus.com/subscribe/email> for updates and announcements.

Feb 9, 13:00:01 GMT+0
原因判明済み - Maintenance is now in progress.

Feb 9, 22:00:00 GMT+0
完了 - Maintenance has completed successfully.

Duration: 4 hours

Affected Components: Cannon Open OnDemand/VDI, Login Nodes - Holyoke, , , FASSE Compute Cluster (Holyoke), GPU nodes (Holyoke), , SLURM Scheduler - FASSE, , SLURM Scheduler - Cannon, Cannon Compute Cluster (Holyoke), , FASSE Open OnDemand/VDI, seas_compute, FASSE login nodes, Kempner Cluster CPU, Kempner Cluster GPU, Login Nodes - Boston, Netscratch (Global Scratch), , Boston Compute Nodes, Login Nodes → Cannon Cluster → VDI/OpenOnDemand → Kempner Cluster → FASSE Cluster → login.rc.fas.harvard.edu →

Feb 2, 14:00:00 GMT+0
原因判明済み - Monthly maintenance will take place on Monday February 2nd, 2026\. Our maintenance tasks should be completed between 9am-1pm. **NOTICES:** * Training: Upcoming training from FASRC and other sources can be found on our Training Calendar. at <https://www.rc.fas.harvard.edu/upcoming-training/> * Status Page: You can subscribe to our status to receive notifications of maintenance, incidents, and their resolution at <https://status.rc.fas.harvard.edu/> (click Get Updates for options). * We'd love to hear success stories about your or your lab's use of FASRC. Submit your story [here](https://www.rc.fas.harvard.edu/user-stories/). **MAINTENANCE TASKS** Cannon cluster will be paused during this maintenance?: **YES** FASSE cluster will be paused during this maintenance?: **YES** * MaxTime change * Audience: Cluster users * Impact: In order to improve scheduling efficiency and stability, we will be setting a maximum run time on all partitions that have MaxTime set to UNLIMITED to a MaxTime of 3 days. The unrestricted partition will be set to 365 days. Partitions that already have MaxTime set will retain their current setting. Partition owners wishing to set a different MaxTime for their partition should contact FASRC. Note that we do no guarantee uptime and so users should utilize checkpointing to save state in case of node failure. * Slurm upgrade to 25.11.2 * Audience: All cluster users * Impact: Jobs will be paused during maintenance * OOD node reboots * Audience; All Open OnDemand users * Impact: OOD nodes will reboot during the maintenance window * Login node reboots * Audience; All login node users * Impact: Login nodes will reboot during the maintenance window Thank you, FAS Research Computing <https://docs.rc.fas.harvard.edu/> [https://www.rc.fas.harvard.edu/](https://www.rc.fas.harvard.edu/upcoming-training/).

Feb 2, 14:00:01 GMT+0
原因判明済み - Maintenance is now in progress.

Feb 2, 18:00:00 GMT+0
完了 - Maintenance has completed successfully.

Duration: 4 hours

Affected Components: , , FASSE Compute Cluster (Holyoke), SLURM Scheduler - Cannon, GPU nodes (Holyoke), , SLURM Scheduler - FASSE, , Cannon Compute Cluster (Holyoke), , Cannon Open OnDemand/VDI, FASSE Open OnDemand/VDI, seas_compute, Login Nodes - Holyoke, FASSE login nodes, Kempner Cluster CPU, Kempner Cluster GPU, Login Nodes - Boston, , Boston Compute Nodes, Login Nodes → Cannon Cluster → VDI/OpenOnDemand → Kempner Cluster → FASSE Cluster → login.rc.fas.harvard.edu →

Jan 12, 14:00:00 GMT+0
原因判明済み - Monthly maintenance will take place on January 12th, 2026\. Our maintenance tasks should be completed between 9am-1pm. **NOTICES:** * Changes to SEAS partitions, please see tasks below. * Changes to job age priority weighting, please see tasks below. * Status Page: You can subscribe to our status to receive notifications of maintenance, incidents, and their resolution at <https://status.rc.fas.harvard.edu/> (click Get Updates for options). * We'd love to hear success stories about your or your lab's use of FASRC. Submit your story [here](https://www.rc.fas.harvard.edu/user-stories/). **MAINTENANCE TASKS** Cannon cluster will be paused during this maintenance?: **YES** FASSE cluster will be paused during this maintenance?:**YES** * Slurm upgrade to 25.11.1 * Audience: All cluster users (Cannon and FASSE) * Impact: Jobs will be paused during maintenance * In conjunction with SEAS we will modify seas\_gpu and seas\_compute time limits * Audience: SEAS users * Impact: seas\_gpu: will be set to 2 days maximum seas\_compute: will be set to 3 days maximum Existing pending jobs longer than these limits will be set to 2 day and 3 day run times depending on partition. * Job Age Priority Weight Change * Audience: Cluster users * Impact: We will be adjusting the weight applied to the priority earned by jobs by virtue of their age. Currently job priority is made up of two factors, Fairshare and Job Age. The Job Age factor is currently set such that jobs gain priority over 3 days with a maximum priority equivalent to jobs with Fairshare of 0.5\. This keeps low fairshare jobs from languishing at the bottom of the queue. With the current settings though, users with low fairshare can gain significant advantage over users with higher relative fairshare. To remedy this we will be adjusting the Job Age weight to cap out at an equivalent Fairshare of 0.1\. This will still allow jobs with 0 fairshare to gain priority and thus not languish while letting fairshare govern a wider range of higher priority jobs. * Login node reboots * Audience; All login node users * Impact: Login nodes will reboot during the maintenance window * Open OnDemand (OOD) node reboots * Audienc:; All OOD users * Impact: OOD nodes will reboot during the maintenance window * Netscratch retention will run * Audience: All cluster netscratch users * Impact: Files older than 90 days will be removed. Please note that retention cleanup can and does run at any time, not just during the maintenance window. Thank you, FAS Research Computing <https://docs.rc.fas.harvard.edu/> [https://www.rc.fas.harvard.edu/](https://www.rc.fas.harvard.edu/upcoming-training/).

Jan 12, 14:00:01 GMT+0
原因判明済み - Maintenance is now in progress.

Jan 12, 18:00:00 GMT+0
完了 - Maintenance has completed successfully.

Duration: 12 hours

Affected Components: Login Nodes - Holyoke, , GPU nodes (Holyoke), FASSE Compute Cluster (Holyoke), Isilon Storage Holyoke (Tier 1), , SLURM Scheduler - FASSE, Virtual Infrastructure - Holyoke, , Cannon Compute Cluster (Holyoke), , Cannon Open OnDemand/VDI, SLURM Scheduler - Cannon, FASSE Open OnDemand/VDI, License Servers, seas_compute, , FASSE login nodes, Kempner Cluster CPU, Kempner Cluster GPU, Login Nodes - Boston, Virtual Infrastructure - Boston, Isilon Storage Boston (Tier 1), , Boston Compute Nodes, Login Nodes → VDI/OpenOnDemand → Kempner Cluster → FASSE Cluster → Cannon Cluster → login.rc.fas.harvard.edu →

Dec 8, 11:00:00 GMT+0
原因判明済み - Monthly maintenance will take place on December 8th. Our maintenance tasks should be completed between 9am-1pm. However: _Additionally_, MGHPCC will be performing power upgrades on the odd side of Row 8A where much of our computer resides. This is the final upgrade for this row. Current estimate for this work is a 12 hour window 6am-6pm. A list of the affected partitions is provided at the bottom of this notice. The nodes in those partitions will be drained prior to the work and will be powered down. Once the work is completed, those nodes will be returned to service. **Notices:** * New FASSE partition `fasse_gpu_h200`. This partitions has 2 H200 nodes and a 3day limit. It is available now. * 11/26 - 11/28 are university holidays (Thanksgiving). No on-site support, FASRC staff will return on 12/1. * Training: Upcoming training from FASRC and other sources can be found on our Training Calendar. at <https://www.rc.fas.harvard.edu/upcoming-training/> * Status Page: You can subscribe to our status to receive notifications of maintenance, incidents, and their resolution at <https://status.rc.fas.harvard.edu/> (click Get Updates for options). * We'd love to hear success stories about your or your lab's use of FASRC. Submit your story [here](https://www.rc.fas.harvard.edu/user-stories/). **MAINTENANCE TASKS** Cannon cluster will be paused during this maintenance?: **PARTIAL OUTAGE/YES** FASSE cluster will be paused during this maintenance?: **PARTIAL OUTAGE/YES** * Power work on Row 8A odd * Audience: Users of the partitions listed below * Impact: These nodes and partitions will be fully or partially down all day * OneFS (Isilon) upgrade * Audience: All Isilon (Tier 1) shares * Impact: Some VMs will be impacted including Cannon OOD, CBScentral, MCZapps/MCZbase, Portal, and Rclic1 (license server) * Slurm upgrade to 25.05.5 * Audience: All cluster users * Impact: Jobs will be paused during maintenance * Login node reboots * Audience: All login node users * Impact: Login nodes will reboot during the maintenance window **Impacted Cannon Partitions (Full or Partial Outage):** * arguelles\_delgado\_gpu\_a100 * arguelles\_delgado\_gpu\_mixed * bigmem\_intermediate * blackhole\_gpu * eddy * gershman * gpu\_requeue * hejazi * hernquist\_ice * hoekstra * huce\_ice * iaifi\_gpu * iaifi\_gpu\_priority * iaifi\_gpu\_requeue * itc\_gpu * jshapiro * kempner * kempner\_dev * kempner\_priority * kempner\_h100 * kempner\_h100\_priority * kempner\_h100\_priority2 * kempner\_h100\_priority3 * kempner\_interactive * kempner\_requeue * kovac * kozinsky * kozinsky\_gpu * kozinsky\_priority * kozinsky\_requeue * murphy\_ice * ortegahernandez\_ice * rivas * seas\_compute * seas\_gpu * serial\_requeue * siag\_combo * siag\_gpu * sur * zhuang.

Dec 8, 11:00:01 GMT+0
原因判明済み - Maintenance is now in progress.

Dec 8, 23:00:00 GMT+0
完了 - Maintenance has completed successfully.

Duration: 4 days, 1 hour and 10 minutes

Affected Components: HolyLFS04 (Tier 0)

Dec 5, 14:00:00 GMT+0
原因判明済み - The holylfs04 migration to holylfs06 has begun. All holylfs04 folders will be **read-only** for the duration of the migration, from **Friday, December 5th at 9AM until end of day on Monday, December 8th.** All labs with holylfs04 have been informed via email; please email [rdm@rc.fas.harvard.edu](mailto:rdm@rc.fas.harvard.edu) if you have any questions..

Dec 9, 15:09:30 GMT+0
完了 - Maintenance has completed successfully..

Duration: 5 days

Affected Components: NESE (NorthEast Storage Exchange), Tape - (Tier 3)

Dec 1, 11:00:00 GMT+0
原因判明済み - NESE, the Northeast Storage Exchange at MGHPCC which supplies the Tier3 tape service used by FASRC, will be offline for maintenance on the system Dec 1st - 5th. There will be ongoing performance-affecting maintenance until Dec 12th. Please see below for details. WHO: Any lab who has or is moving data to tape. IMPACT: No access 12/1/25 - 12/5/25\. Reduced performance 12/5/25 - 12/12/25. > NESE tape system maintenance and major software upgrade is scheduled to begin on December 1, 2025\. As a result, the NESE Tape service will be offline from December 1 to December 5. > > Starting December 8 through December 12, the service will be back online with reduced performance. All maintenance activities are planned to conclude on December 12, 2025. > > * Monitor: <https://nese.instatus.com/> for real-time updates on progress > * Subscribe to <https://nese.instatus.com/subscribe/email> for updates and announcements.

Dec 1, 11:00:01 GMT+0
原因判明済み - Maintenance is now in progress.

Dec 6, 11:00:00 GMT+0
完了 - Maintenance has completed successfully.

Duration: 1 hour

Affected Components: Starfish

Nov 14, 22:00:01 GMT+0
原因判明済み - Maintenance is now in progress.

Nov 14, 23:00:00 GMT+0
完了 - Maintenance has completed successfully.

Nov 14, 22:00:00 GMT+0
原因判明済み - There is a planned upgrade of the Starfish dashboard scheduled for Friday November 14th starting at 5PM. The dashboard will be down for an hour while the upgrade is performed..

Duration: 9 hours and 4 minutes

Affected Components: , , SLURM Scheduler - FASSE, , Kempner Cluster CPU, Cannon Compute Cluster (Holyoke), , Cannon Open OnDemand/VDI, SLURM Scheduler - Cannon, FASSE Open OnDemand/VDI, seas_compute, Kempner Cluster GPU, Login Nodes - Holyoke, , GPU nodes (Holyoke), FASSE Compute Cluster (Holyoke), FASSE login nodes, Login Nodes - Boston, Netscratch (Global Scratch), Boston Compute Nodes, Login Nodes → VDI/OpenOnDemand → Kempner Cluster → FASSE Cluster → Cannon Cluster →

Nov 3, 11:00:00 GMT+0
原因判明済み - Monthly maintenance will take place on November 3rd. Additionally, MGHPCC will be performing power upgrades on the even side of Row 8A where much of our computer resides. A further upgrade will take place Dec. 8th on the odd side. A list of the affected partitions is provided at the bottom of this notice. The nodes in those partitions will be drained prior to the work and will be powered down. Once the work is completed, those nodes will be returned to service. Current estimate is a 12 hour window. We will adjust as we know more. **MAINTENANCE TASKS** Cannon cluster will be paused during this maintenance?: **PARTIAL OUTAGE/YES** FASSE cluster will be paused during this maintenance?: **PARTIAL OUTAGE/YES** * Power work on Row 8A Even * Audience: Users of the partitions listed below * Impact: These nodes and partitions will be fully or partially down all day * Slurm upgrade to 25.05.4 * Audience: All cluster users * Impact: Jobs will be paused during maintenance * Block repo.anaconda.com cluster wide * Audience: Anyone attempting to use repo.anaconda.com * Impact: This change should not impact your Python workflow on the cluster. But if it does, consider using the open-source channel, `conda-forge`, through Miniforge distribution to install Python packages. This can be done by following our instructions on <https://docs.rc.fas.harvard.edu/kb/python-package-installation/> * Change Slurm User to Local User * Audience: All cluster users * Impact: Behind the scenes. No impact to users * Login node reboots (morning) * Audience: Anyone logged into a FASRC Cannon or FASSE login node * Impact: All login nodes will rebooted during this maintenance window * Netscratch cleanup ( <https://docs.rc.fas.harvard.edu/kb/policy-scratch/> ) * Audience: Cluster users * Impact: Files older than 90 days will be removed. Please note that retention cleanup can and does run at any time, not just during the maintenance window. **AFFECTED PARTITIONS** Nov. 3, 2025 - All Day Power Work Partial or Full Outage Apples to: arguelles\_delgado\_h100 bigmem dvorkin eddy enos gpu gpu\_h200 gpu\_requeue hsph hsph\_gpu intermediate itc\_cluster joonholee jshapiro kempner\_dev kemkpner\_eng kempner\_requeue mweber\_compute mweber\_gpu olveczky\_sapphire sapphire seas\_compute seas\_gpu serial\_requeue yao yao\_gpu yao\_priority test.

Nov 3, 11:00:01 GMT+0
原因判明済み - Maintenance is now in progress.

Nov 3, 20:04:13 GMT+0
完了 - Maintenance has completed successfully including power work at MGHPCC. A reminder that additional all-day power work will take place on Dec 8th, along with our maintenance from 9am-1pm.

Duration: 4 hours

Affected Components: , Network - Cambridge, Network - Boston, Network - Holyoke/MGHPCC, Login Nodes - Boston, Netscratch (Global Scratch), Login Nodes - Holyoke, FASSE login nodes, Login Nodes →

Oct 6, 13:00:01 GMT+0
原因判明済み - Maintenance is now in progress.

Oct 6, 13:00:00 GMT+0
原因判明済み - FASRC monthly maintenance will take place Monday October 6th, 2025 from 9am-1pm **NOTICES** * Training: Upcoming training from FASRC and other sources can be found on our Training Calendar. at <https://www.rc.fas.harvard.edu/upcoming-training/> * Status Page: You can subscribe to our status to receive notifications of maintenance, incidents, and their resolution at <https://status.rc.fas.harvard.edu/> (click Get Updates for options). * Upcoming holidays: Columbus / Indigenous Peoples’ Day - October 13 **MAINTENANCE TASKS** Cannon cluster will be paused during this maintenance?: **NO** FASSE cluster will be paused during this maintenance?: **NO** * DNS server reboots * Audience: All FASRC services * Impact: Rolling reboot should have no impact * Login node reboots * Audience: Anyone logged into a FASRC Cannon or FASSE login node * Impact: All login nodes will rebooted during this maintenance window * Netscratch cleanup ( <https://docs.rc.fas.harvard.edu/kb/policy-scratch/> ) * Audience: Cluster users * Impact: Files older than 90 days will be removed. Please note that retention cleanup can and does run at any time, not just during the maintenance window. Thank you, FAS Research Computing <https://docs.rc.fas.harvard.edu/> <https://www.rc.fas.harvard.edu/>.

Oct 6, 17:00:00 GMT+0
完了 - Maintenance is now in progress.

Sep 30, 07:00:00 GMT+0
完了 - Maintenance has completed successfully.

Sep 30, 06:00:00 GMT+0
完了 - Networking will be patching the VPN concentrators overnight. This will be done in a rolling order so that one ore more are always online. This may cause active VPN connections to drop, but they can be re-connected shortly after. ETA is one hour total..

Sep 30, 06:00:00 GMT+0
原因判明済み - Networking will be patching the VPN concentrators overnight. This will be done in a rolling order so that one ore more are always online. This may cause active VPN connections to drop, but they can be re-connected shortly after. ETA is one hour total..

Duration: 4 hours

Affected Components: Kempner Cluster CPU, Kempner Cluster GPU, , Cannon Compute Cluster (Holyoke), Boston Compute Nodes, Netscratch (Global Scratch), , , SLURM Scheduler - FASSE, , GPU nodes (Holyoke), Login Nodes - Boston, FASSE login nodes, Login Nodes - Holyoke, , FASSE Open OnDemand/VDI, Cannon Open OnDemand/VDI, SLURM Scheduler - Cannon, FASSE Compute Cluster (Holyoke), seas_compute, Login Nodes → Cannon Cluster → FASSE Cluster → VDI/OpenOnDemand → Kempner Cluster →

Sep 8, 13:00:01 GMT+0
原因判明済み - Maintenance is now in progress.

Sep 8, 17:00:00 GMT+0
完了 - Maintenance has completed successfully.

Sep 8, 13:00:00 GMT+0
原因判明済み - FASRC monthly maintenance will take place Monday September 8th, 2025 from 9am-1pm **NOTICES** * Training: Upcoming training from FASRC and other sources can be found on our Training Calendar. at <https://www.rc.fas.harvard.edu/upcoming-training/> * Status Page: You can subscribe to our status to receive notifications of maintenance, incidents, and their resolution at <https://status.rc.fas.harvard.edu/> (click Get Updates for options). * Upcoming holidays: Labor Day, Monday September 1st **MAINTENANCE TASKS** Cannon cluster will be paused during this maintenance?: **YES** FASSE cluster will be paused during this maintenance?: **YES** * Slurm Upgrade to 25.05.2 * Audience: All cluster users * Impact: Jobs and the scheduler will be paused during this upgrade * Domain controller work * Audience: Internal network * Impact: No impact expected * Login node reboots * Audience: Anyone logged into a FASRC Cannon or FASSE login node * Impact: All login nodes will rebooted during this maintenance window * Netscratch cleanup ( <https://docs.rc.fas.harvard.edu/kb/policy-scratch/> ) * Audience: Cluster users * Impact: Files older than 90 days will be removed. Please note that retention cleanup can and does run at any time, not just during the maintenance window. Thank you, FAS Research Computing <https://docs.rc.fas.harvard.edu/> <https://www.rc.fas.harvard.edu/>.

Duration: 2 hours

Affected Components: Starfish

Aug 20, 21:00:01 GMT+0
原因判明済み - Maintenance is now in progress.

Aug 20, 21:00:00 GMT+0
原因判明済み - Starfish will be performing an upgrade on Wednesday, August 20th from 5PM-7PM. The web interface will be unavailable during that timeframe..

Aug 20, 23:00:00 GMT+0
完了 - Maintenance has completed successfully.

Duration: 4 days, 9 hours and 39 minutes

Affected Components: seas_compute

Aug 11, 10:00:00 GMT+0
原因判明済み - Between 8/11/25 6AM - 8/14/25 5PM FASRC will be upgrading 14 of the H100 GPU nodes in the `seas_gpu `partition to H200 GPUs. This sill also affect `mweber_gpu` A reservation has been set which will drain the nodes of jobs prior to the maintenance. The SEAS GPU partition will be running at 75% capacity during these updates. FASRC has hundreds of GPU, so users should feel free to utilize `gpu_requeue` if needed for their jobs. Affected nodes: `mweber_gpu` nodes (13): ``` holygpu8a[18204,18301-18304,18401-18404,18501-18502,18601-18602] ``` seas\_gpu nodes (14): ``` holygpu8a[16101-16104,16201-16204,16301-16304,16401-16402] ``` Please reach out to [rchelp@rc.fas.harvard.edu](mailto:rchelp@rc.fas.harvard.edu) if you have any questions or concerns..

Aug 15, 14:13:24 GMT+0
原因判明済み - Maintenance is still in progress - imaging of the new H200 nodes is ongoing. Current ETA is end of day Friday. For further questions, please contact rchelp@rc.fas.harvard.edu.

Aug 15, 19:39:27 GMT+0
完了 - Maintenance has completed successfully. SEAS H200 nodes have been imaged and are back in service. .

Aug 11, 10:00:01 GMT+0
原因判明済み - Maintenance is now in progress.