Affected Components: FASSE Compute Cluster (Holyoke), Cannon Compute Cluster (Holyoke), seas_compute, SLURM Scheduler - FASSE, Boston Compute Nodes, Login Nodes - Boston, Login Nodes - Holyoke, SLURM Scheduler - Cannon, Kempner Cluster GPU, Kempner Cluster CPU, Cannon Open OnDemand, FASSE Open OnDemand, GPU nodes (Holyoke), FASSE login nodes

Aug 3, 19:35:00 GMT+0
Identified - FASRC will be doing OS upgrades to the latest version of Rocky 8.10 from August 24-27th. These rolling upgrades will improve cluster security and install the latest Cuda 13.2 driver. Upgrades will happen in stages with each stage occurring between 8a-5p each day. During that period the nodes being upgraded will be unavailable. No jobs will be canceled but jobs will stay pending if all nodes in the partition are down. Note that during this week the cluster will be in a partially upgraded state, thus jobs that span multiple nodes may become unstable due to mismatching libraries. Users concerned about this should wait until August 28th to restart runs. Users should plan their work accordingly. ### The upgrade schedule with impacted partitions is as follows: **August 24th:** FASSE boslogin private login nodes **August 25th: Cannon Part 1** holylogin arguelles\_delgado blackhole davies\_gpu davies desai eddy holy-cow holy-smokes huce\_bigmem huce\_cascade huttenhower jacobsen2 janson\_bigmem janson\_cascade janson ke lukin nguyen olveczky\_gpu remoteviz seas\_compute shared sompolinsky\_gpu tambe vishwanath whipple xlin xlin\_ice zhuang\_gpu zhuang **August 26th: Cannon Part 2** arguelles\_delgado conroy davies doshi-velez dsouza edwards geophysics giribet gpu\_test hernquist huce\_cascadeimasc janson kempner\_h200 kempner\_rtx murphy ni\_lab olveczky ortegahernandez pehlevan seas\_compute shakhnovich shared unrestricted xlin yin zon **August 27th: Cannon Part 3** arguelles\_delgado\_gpu\_a100 arguelles\_delgado\_gpu\_mixed arguelles\_delgado\_h100 bigmem\_intermediate bigmem blackhole\_gpu dvorkin eddy enos gershman gpu gpu\_h200 hejazi hernquist\_ice hoekstra hsph\_gpu hsph huce\_ice iaifi\_gpu intermediate itc\_cluster itc\_gpu janson\_sapphire joonholee jshapiro kempner\_h100 kempner\_h200 kempner kempner\_interactive kovac kozinsky\_gpu kozinsky murphy\_ice mweber\_compute mweber\_gpu olveczky\_sapphire ortegahernandez\_ice rivas sapphire seas\_compute seas\_gpu seas\_gpu\_perf siag\_gpu siag\_combo siag sur test yao\_alphatns yao\_gpu yao zhuang.

Affected Components: Authentication, FASRC Two-Factor (OpenAuth)

Aug 12, 16:25:20 GMT+0
Identified - We have identified an issue which keeps new accounts from using their two-factor/openauth token for authentication. If you have a **new account** and are unable to authenticate to the cluster, FASRC VPN, or other FASRC services, this is why. Existing accounts are not affected. We are working to resolve this as quickly as possible, but no ETA at this time. We will update this status as things change. Thanks for your understanding and patience..

Duration: 32 minutes

Affected Components: Holystore01 (Tier 0)

Aug 12, 13:36:07 GMT+0
Identified - holystore01 is wedged. We are failing over OST..

Aug 12, 14:07:58 GMT+0
Resolved - This incident has been resolved..

Affected Components: Grafana Cloud (FASRC)

Aug 5, 09:19:23 GMT+0
Resolved - Grafana Cloud (FASRC) is back up. This incident was automatically resolved by Instatus monitoring..

Aug 5, 09:14:42 GMT+0
Investigating - Grafana Cloud (FASRC) is down at the moment. This incident was automatically created by Instatus monitoring..

Duration: 1 day, 2 hours and 59 minutes

Affected Components: SLURM Scheduler - Cannon

Aug 4, 16:00:00 GMT+0
Investigating - jobstats is not working due to an issue with the cgroup exporter. We are working on updating this tool, but there is no ETA at this time. .

Aug 5, 18:59:09 GMT+0
Resolved - Jobstats has been rebuilt and should be functional again for both CPU and GPU jobs This incident has been resolved..

Duration: 1 minute

Affected Components: FASSE login nodes

Aug 3, 19:57:13 GMT+0
Investigating - We are currently investigating a problem with the FASSE login nodes. No ETA at this time..

Aug 3, 19:58:08 GMT+0
Resolved - FASSE login nodes are back up..

Duration: 30 minutes

Affected Components: Authentication

Aug 3, 16:54:53 GMT+0
Investigating - Authentication issues with openauth/radius. This incident was created by an automated monitoring service..

Aug 3, 17:24:53 GMT+0
Resolved - Openauth/radius is now operational. This update was created by an automated monitoring service..

Duration: 6 hours and 5 minutes

Affected Components: Globus Data Transfer, FASSE Compute Cluster (Holyoke), Boston Tier 2 NFS (new), Holyoke Tier 2 NFS (new), Holylabs, HolyLFS06 (Tier 0), bosECS, Boston Specialty Storage, Cannon Compute Cluster (Holyoke), seas_compute, SLURM Scheduler - FASSE, CEPH Storage Boston (Tier 2), Holyoke Specialty Storage, Isilon Storage Holyoke (Tier 1), holECS, BosLFS02 (Tier 0), Isilon Storage Boston (Tier 1), Home Directory Storage - Boston, Netscratch (Global Scratch), Tape - (Tier 3), Boston Compute Nodes, Login Nodes - Boston, Samba Cluster, HolyLFS04 (Tier 0), Holystore01 (Tier 0), FASRC Two-Factor (OpenAuth), HolyLFS05 (Tier 0), Login Nodes - Holyoke, SLURM Scheduler - Cannon, Kempner Cluster GPU, Kempner Cluster CPU, Cannon Open OnDemand, FASSE Open OnDemand, GPU nodes (Holyoke), FASSE login nodes

Jul 15, 17:57:35 GMT+0
Identified - FASRC monthly maintenance will take place on August 3rd, 2026\. Our maintenance tasks should be completed between **9am-3pm**. **Note**: This maintenance takes place at the same time as the [**firewall cutover \[details\]**](https://dashboard.instatus.com/fasrc/fasrc/maintenances/cmrjgmzwb01780rmp8pxkqwr0) and all running jobs **_will be canceled on the morning of August 3rd_**. Note the longer duration of this maintenance period: 9am-3pm **NOTICES:** * Training: Upcoming training from FASRC and other sources can be found on our Training Calendar. at <https://www.rc.fas.harvard.edu/upcoming-training/> * Status Page: You can subscribe to our status to receive notifications of maintenance, incidents, and their resolution at <https://status.rc.fas.harvard.edu/> (click Get Updates for options). **MAINTENANCE TASKS** Cannon cluster will be paused during this maintenance?: **YES** FASSE cluster will be paused during this maintenance?: **YES** * Slurm Upgrade to 26.05.2 * Audience: Cluster * Impact: The cluster will be paused during this maintenance * [two-factor.rc.fas.harvard.edu](http://two-factor.rc.fas.harvard.edu) [OpenAuth](https://docs.rc.fas.harvard.edu/kb/openauth/) cut-over to new server * Audience: New accounts or anyone requesting an OpenAuth token * Impact: two-factor will be unavailable while moving to a new server * Login node down/reboots * Audience: All login nodes * Impact: Login nodes will be unavailable until after maintenance * OOD/Open OnDemand down/reboots * Audience: All OOD users * Impact: OOD will be unavailable until after maintenance * Lab storage cutover * Audience; Anyone who was contacted about this * Impact: See email(s) from RDM to affected users * Netscratch 90-day retention cleanup * Audience; All netscratch users * Impact: Files older than 90 days will be removed per our [scratch policy](https://docs.rc.fas.harvard.edu/kb/policy-scratch/). Please note that this cleanup can happen at any time, not just during maintenance. Thank you, FAS Research Computing <https://docs.rc.fas.harvard.edu/> <https://www.rc.fas.harvard.edu/>.

Aug 3, 13:05:01 GMT+0
Identified - Maintenance is now in progress.

Aug 3, 19:10:00 GMT+0
Completed - Maintenance has completed successfully.

Duration: 6 hours

Affected Components: Globus Data Transfer, Infiniband - Holyoke/MGHPCC, FASSE Compute Cluster (Holyoke), Boston Tier 2 NFS (new), Holyoke Tier 2 NFS (new), Holylabs, Holyoke Firewall, Network - Holyoke/MGHPCC, Holyoke-Boston fiber link (short path), Infiniband - Boston, HolyLFS06 (Tier 0), bosECS, Boston Specialty Storage, Software & Modules, Cannon Compute Cluster (Holyoke), Network - Boston, Cambridge firewall and other redundancy, License Servers, seas_compute, Web Proxies, SLURM Scheduler - FASSE, CEPH Storage Boston (Tier 2), Holyoke Specialty Storage, Isilon Storage Holyoke (Tier 1), holECS, Network - Cambridge, BosLFS02 (Tier 0), Isilon Storage Boston (Tier 1), Home Directory Storage - Boston, Netscratch (Global Scratch), Tape - (Tier 3), Boston Compute Nodes, Login Nodes - Boston, Samba Cluster, HolyLFS04 (Tier 0), Holystore01 (Tier 0), Holyoke-Boston fiber link (long path), HolyLFS05 (Tier 0), Login Nodes - Holyoke, SLURM Scheduler - Cannon, Kempner Cluster GPU, Kempner Cluster CPU, Cannon Open OnDemand, FASSE Open OnDemand, GPU nodes (Holyoke), FASSE login nodes

Jul 13, 16:52:17 GMT+0
Identified - ### Data Center Firewall Replacement August 3rd 9am-3pm NOTE: The following is in addition to our [regular monthly maintenance](https://status.rc.fas.harvard.edu/cmrmduopd01qg0kpnmrd8zylp) at the same time. On Monday August 3rd in addition to our monthly maintenance, the primary firewall in one of the data centers will be replaced. This will interrupt storage and home directory connectivity to the cluster and therefore the cluster must be idle for this cutover. General storage access will also be affected during this period. This has been scheduled with networking to coincide with our monthly maintenance but will run longer. **9AM - 3PM** **\- All jobs still running on August 3rd will be canceled -** On the morning of August 3rd: * Any running jobs will be canceled * All partitions will be closed * Login nodes will be shut down * The Networking group will commence work Once the firewall work is complete and tested, we will re-open the partitions and boot the login nodes. ETA 3PM Due to the disruptive nature of this work, we want to advertise this with a longer advance notice. Normal maintenance emails will be sent per usual starting next week and will contain a condensed version of this information Thank you, FAS Research Computing [rchelp@rc.fas.harvard.edu](mailto:rchelp@rc.fas.harvard.edu) <https://docs.rc.fas.harvard.edu/> <https://www.rc.fas.harvard.edu/>.

Aug 3, 13:00:01 GMT+0
Identified - Maintenance is now in progress.

Aug 3, 19:00:00 GMT+0
Completed - Maintenance has completed successfully.

Duration: 59 minutes

Affected Components: HolyLFS06 (Tier 0)

Jul 30, 13:50:04 GMT+0
Identified - We are rebooting holylfs06\. holylfs06 storage may be slow or unresponsive. We are investigating this issue..

Jul 30, 14:49:27 GMT+0
Resolved - This incident has been resolved..

Duration: 33 minutes

Affected Components: Coldfront

Jul 22, 14:17:31 GMT+0
Investigating - Coldfront is temporarily inaccessible. We are currently investigating this incident..

Jul 22, 14:50:28 GMT+0
Resolved - This incident has been resolved. Coldfront is now accessible..

Jul 22, 14:50:10 GMT+0
Resolved - Coldfront is back up This incident has been resolved..

Duration: 4 hours

Affected Components: FASSE login nodes, Netscratch (Global Scratch), Login Nodes - Holyoke, Login Nodes - Boston, Cannon Open OnDemand, FASSE Open OnDemand

Jun 26, 13:45:03 GMT+0
Identified - FASRC monthly maintenance will take place on July 6th 2026\. Our maintenance tasks should be completed between 9am-1pm. Cannon cluster will be paused during this maintenance?: **NO** FASSE cluster will be paused during this maintenance?: **NO** **NOTICES:** * Friday July 3rd is a university holiday (independence Day observed) * Training: Upcoming training from FASRC and other sources can be found on our Training Calendar. at <https://www.rc.fas.harvard.edu/upcoming-training/> * Status Page: You can subscribe to our status to receive notifications of maintenance, incidents, and their resolution at <https://status.rc.fas.harvard.edu/> (click Get Updates for options). * We'd love to hear success stories about your or your lab's use of FASRC. Submit your story [here](https://www.rc.fas.harvard.edu/user-stories/). **MAINTENANCE TASKS** * Domain controller replacement * Audience: Internal * Impact: None. End users should not see any impact. * Reboot drained nodes in error state * Audience: Cluster nodes with errors. * Impact: These nodes will have been drained already in preparation. No impact on jobs on the day and the affected nodes will return to service in their respective partitions after the maintenance period. * OOD/Open OnDemand reboots * Audience: All OOD users, reboot of the head nodes. * Impact: Running sessions will _not_ be affected. * Login node reboots * Audience; All login node users. * Impact: Login nodes will reboot during the maintenance window. * Netscratch 90-day retention cleanup * Audience; All netscratch users * Impact: Files older than 90 days will be removed per our [scratch policy](https://docs.rc.fas.harvard.edu/kb/policy-scratch/). Please note that this cleanup can happen at any time, not just during maintenance. Thank you, FAS Research Computing <https://docs.rc.fas.harvard.edu/> <https://www.rc.fas.harvard.edu/>.

Jul 6, 13:00:01 GMT+0
Identified - Maintenance is now in progress.

Jul 6, 17:00:00 GMT+0
Completed - Maintenance has completed successfully.

Duration: 34 minutes

Affected Components: Login Nodes - Holyoke, Login Nodes - Boston

Jul 2, 14:20:29 GMT+0
Investigating - We are currently investigating an issue where users cannot log into login nodes or cannot log in consistently..

Jul 2, 14:54:47 GMT+0
Resolved - The underlying cause has been identified and fixed. Logins should work as expected now. This incident has been resolved..

Duration: 4 days, 14 hours and 51 minutes

Affected Components: GPU nodes (Holyoke), Cannon Compute Cluster (Holyoke), seas_compute, Boston Compute Nodes, SLURM Scheduler - Cannon

Jun 24, 14:06:30 GMT+0
Monitoring - We have rolled out a fix and are currently monitoring the result. Any nodes still in **Kill Task Fail** will need to fully drain of running jobs first before being reopened to ensure there are no orphaned processes. **PLEASE NOTE: This will take quite some time to fully resolve as jobs _are_ still running on these nodes. But the nodes cannot accept new jobs until cleared.** **Currently # of potentially impacted nodes: 18 (of an original 600 - number will update periodically)**.

Jun 29, 04:31:34 GMT+0
Resolved - This incident has been resolved. Less than 18 nodes remain in this state and will be cleaned up soon..

Jun 24, 13:40:49 GMT+0
Investigating - We are currently investigating this incident. Due to a CVMFS mount issue, some nodes are being closed and labelled as "Kill task fail" WIP.

Duration: 19 hours and 9 minutes

Affected Components: HolyLFS06 (Tier 0)

Jun 19, 19:00:00 GMT+0
Investigating - holylfs06 storage may be slow or unresponsive. We are investigating this issue..

Jun 20, 14:09:15 GMT+0
Resolved - This incident has been resolved..

Duration: 3 days, 8 hours and 16 minutes

Affected Components: Globus Data Transfer, Coldfront, Boston Data Center, GPU nodes (Holyoke), Infiniband - Holyoke/MGHPCC, FASSE Compute Cluster (Holyoke), SLURM Scheduler - FASSE, Boston Tier 2 NFS (new), Holyoke Tier 2 NFS (new), Holylabs, Holyoke Firewall, Network - Holyoke/MGHPCC, Holyoke-Boston fiber link (short path), Infiniband - Boston, HolyLFS06 (Tier 0), bosECS, Boston Specialty Storage, Software & Modules, Holyoke/MGHPCC Data Center, Cannon Compute Cluster (Holyoke), Network - Boston, Cambridge firewall and other redundancy, License Servers, seas_compute, FASSE login nodes, Starfish, Web Proxies, Authentication, Virtual Infrastructure - Holyoke, FIINE billing portal, CEPH Storage Boston (Tier 2), NESE (NorthEast Storage Exchange), Isilon Storage Holyoke (Tier 1), holECS, Holyoke Specialty Storage, FASRC Downloads Site, Citrix, HolyLFS05 (Tier 0), Virtual Infrastructure - Boston, Network - Cambridge, FASRC VPN (Cambridge) , FASRC VPN (Boston), BosLFS02 (Tier 0), Isilon Storage Boston (Tier 1), Home Directory Storage - Boston, Netscratch (Global Scratch), Tape - (Tier 3), Boston Compute Nodes, portal.rc.fas.harvard.edu, Login Nodes - Holyoke, Login Nodes - Boston, Samba Cluster, HolyLFS04 (Tier 0), Holystore01 (Tier 0), FASRC Two-Factor (OpenAuth), Holyoke-Boston fiber link (long path), Grafana Cloud (FASRC), SLURM Scheduler - Cannon, Kempner Cluster GPU, Kempner Cluster CPU, Cannon Open OnDemand, FASSE Open OnDemand

Jun 15, 13:00:01 GMT+0
Identified - Maintenance is now in progress.

Jun 15, 13:00:00 GMT+0
Identified - The yearly power downtime at our Holyoke data center, MGHPCC, has been scheduled by the facility. This year's power downtime will take place on Tuesday June 15th - 18th, 2025\. There will be no June monthly maintenance as a result. Since the facility will be powered down for two days this year, we will not be performing the usual maintenance tasks. That said, networking and other key infrastructure will be doing maintenance. **IMPORTANT NOTE**: FASRC storage at both Holyoke and Boston **will be** affected and should not be expected to be available throughout the downtime. Please plan ahead accordingly. * **Monday June 15th** \- Power-down begins at 9AM * **Tuesday June 16th** \- Power out at MGHPCC * **Wednesday June 17th** \- Power out at MGHPCC * **Thursday June 18th** \- Expected return to full service by 5PM * **Friday June 19th** \- Please note that June 19th is a university holiday  **For more detailed information and follow-up, please see:** <https://www.rc.fas.harvard.edu/mghpcc-yearly-shutdown> **or this** [**Status Page**](https://status.rc.fas.harvard.edu/).

Jun 18, 12:16:49 GMT+0
Identified - MGHPCC has completed their maintenance and restored power to the facility. FASRC will now begin the power-up process. Please be aware that this takes several hours. We will update this status once complete. NOTE: A reminder that tomorrow (Friday) is a university holiday. .

Jun 18, 20:45:50 GMT+0
Identified - Power-up is nearly complete, but a delay earlier in the day has us slightly behind. New ETA is 6PM..

Jun 18, 21:15:42 GMT+0
Completed - The yearly power downtime at our Holyoke data center, MGHPCC, has completed. The clusters and storage are back online and login nodes and OOD nodes are now available. If you have an issue/need help, please send a ticket to [rchelp@rc.fas.harvard.edu](mailto:rchelp@rc.fas.harvard.edu) with details. IMPORTANT NOTE: Tomorrow, June 19th is a university holiday. FASRC staff will return Monday to address any lingering issues and any new tickets..

Duration: 25 minutes

Affected Components: HolyLFS06 (Tier 0)

Jun 10, 20:07:18 GMT+0
Resolved - This incident has been resolved..

Jun 10, 19:41:54 GMT+0
Investigating - Holylfs06 storage may be slow or unresponsive We are currently investigating this incident..

Duration: 3 hours and 16 minutes

Affected Components: portal.rc.fas.harvard.edu

Jun 9, 19:38:55 GMT+0
Resolved - The Portal website should be accessible for all now. This incident has been resolved..

Jun 9, 16:22:32 GMT+0
Investigating - There is an SSL issue with the Portal which will cause an error for anyone attempting to connect. Investigating..

Duration: 5 minutes

Affected Components: Authentication

Jun 8, 20:24:54 GMT+0
Investigating - Authentication issues with openauth/radius. This incident was created by an automated monitoring service..

Jun 8, 20:29:54 GMT+0
Resolved - Openauth/radius is now operational. This update was created by an automated monitoring service..

Duration: 12 minutes

Affected Components: Coldfront

Jun 8, 20:02:18 GMT+0
Investigating - Coldfront is showing a server error. Investigating..

Jun 8, 20:14:34 GMT+0
Resolved - This incident has been resolved. Coldfront is now available..

Duration: 9 minutes

Affected Components: Login Nodes - Holyoke, Login Nodes - Boston

Jun 8, 13:16:28 GMT+0
Investigating - [lofin.rc.fas.harvard.edu](http://lofin.rc.fas.harvard.edu) is refusing new connections and will error out when attempting to SSH in. Investigating..

Jun 8, 13:25:32 GMT+0
Resolved - This incident has been resolved and SSH login to [login.rc.fas.harvard.edu](http://login.rc.fas.harvard.edu) is working again. Thanks for your patience..

Affected Components: FASRC VPN (Cambridge)

Jun 5, 04:06:55 GMT+0
Investigating - FASRC VPN (Cambridge) is down at the moment. This incident was automatically created by Instatus monitoring..

Jun 5, 04:16:32 GMT+0
Resolved - FASRC VPN (Cambridge) is back up. This incident was automatically resolved by Instatus monitoring..

Duration: 22 minutes

Affected Components: Authentication, portal.rc.fas.harvard.edu

Jun 2, 20:42:13 GMT+0
Investigating - Password reset emails are not being sent out at this time. We are currently investigating this incident..

Jun 2, 21:04:09 GMT+0
Resolved - Password reset emails are now being sent out. This incident has been resolved..

Duration: 2 days, 18 hours and 24 minutes

Affected Components: GPU nodes (Holyoke), Cannon Compute Cluster (Holyoke), seas_compute, SLURM Scheduler - Cannon, Kempner Cluster GPU, Kempner Cluster CPU, Cannon Open OnDemand, FASSE Open OnDemand, Boston Compute Nodes

Jun 1, 14:45:41 GMT+0
Resolved - Slurm crashed on 4:30p on Friday due to a user running a large sacct query against the Slurm database. This caused the database host to run out of memory and crash the scheduler. To prevent this from reoccurring we are reducing the time range that users are permitted to query at one time to 7 days. Thus if you need to cover a month you would need to query in four 7 day increments. We do ask users to be judicious in their querying of the Slurm. Only ask for those fields that you require. Please also ensure any AI agents you have running limit their queries appropriately..

May 29, 20:21:17 GMT+0
Investigating - The Slurm scheduler is experiencing an error which is impacting jobs. The Cannon cluster will be inaccessible while we troubleshoot. We are currently investigating this incident..

May 29, 21:13:09 GMT+0
Identified - To temporarily stabilize the situation, we have reduced the maximum query time for sacct and other Slurm commands to be 1 day. We have filed a ticket with SchedMD to further analyze the issue. The cluster is back up and the scheduler is accepting new jobs. We will continue to monitor for emergencies over the weekend, and resume in-depth troubleshooting on Monday. .

Duration: 3 days, 23 hours and 2 minutes

Affected Components: portal.rc.fas.harvard.edu

May 22, 15:18:39 GMT+0
Investigating - Account approvals through Portal are not available at this time. We are currently investigating this incident..

May 26, 14:20:10 GMT+0
Resolved - This incident has been resolved. Login is again available for approvals/onboarding..

Affected Components: FASRC VPN (Cambridge)

May 22, 04:50:10 GMT+0
Investigating - FASRC VPN (Cambridge) is down at the moment. This incident was automatically created by Instatus monitoring..

May 22, 05:09:48 GMT+0
Resolved - FASRC VPN (Cambridge) is back up. This incident was automatically resolved by Instatus monitoring..

Affected Components: FASRC VPN (Cambridge)

May 22, 03:35:48 GMT+0
Investigating - FASRC VPN (Cambridge) is down at the moment. This incident was automatically created by Instatus monitoring..

May 22, 03:55:19 GMT+0
Resolved - FASRC VPN (Cambridge) is back up. This incident was automatically resolved by Instatus monitoring..

Affected Components: FASRC VPN (Cambridge)

May 22, 03:05:05 GMT+0
Investigating - FASRC VPN (Cambridge) is down at the moment. This incident was automatically created by Instatus monitoring..

May 22, 03:14:35 GMT+0
Resolved - FASRC VPN (Cambridge) is back up. This incident was automatically resolved by Instatus monitoring..

Affected Components: , Login Nodes - Holyoke, Login Nodes - Boston, , Login Nodes → login.rc.fas.harvard.edu →

May 22, 02:40:38 GMT+0
Investigating - login.rc.fas.harvard.edu is not responding normally. This incident was automatically created..

May 22, 03:40:12 GMT+0
Resolved - \\\[login.rc.fas.harvard.edu\\\](http://login.rc.fas.harvard.edu) is responding normally. This incident was automatically resolved..

Affected Components: FASRC VPN (Cambridge)

May 22, 02:24:22 GMT+0
Investigating - FASRC VPN (Cambridge) is down at the moment. This incident was automatically created by Instatus monitoring..

May 22, 02:53:53 GMT+0
Resolved - FASRC VPN (Cambridge) is back up. This incident was automatically resolved by Instatus monitoring..

Duration: 3 days, 13 hours and 15 minutes

Affected Components: GPU nodes (Holyoke), FASSE Compute Cluster (Holyoke), SLURM Scheduler - FASSE, Cannon Compute Cluster (Holyoke), seas_compute, Kempner Cluster GPU, Kempner Cluster CPU, SLURM Scheduler - Cannon, Boston Compute Nodes

May 18, 11:00:00 GMT+0
Identified - Rescheduled to May 18.

May 18, 11:00:00 GMT+0
Identified - Our Holyoke data center, MGHPCC, will be doing power work on Row 8A. This work, which is being completed over the course of 2 weeks, will bring online another power feed which will increase power capacity. In order to do this work, it will require us to idle half the nodes in 8a for the duration of the week. This means all partitions in this row will be at half capacity. Existing jobs should drain naturally and no job should need to be canceled. The impacted partitions are: ``` arguelles_delgado_h100 bigmem bigmem_intermediate blackhole_gpu dvorkin eddy enos gershman gpu gpu_h200 gpu_requeue hejazi hernquist_ice hoekstra hsph hsph_gpu huce_ice iaifi_gpu_requeue intermediate itc_cluster itc_gpu janson_sapphire joonholee jshapiro kempner kempner_priority kempner_dev kempner_eng kempner_h200_priority kempner_h100 kempner_h100_priority kempner_h100_priority2 kempner_h100_priority3 kempner_h100_priority4 kempner_interactive kovac kozinsky kozinsky_gpu kozinsky_requeue murphy_ice mweber_compute mweber_gpu olveczky_sapphire ortegahernandez_ice rivas sapphire seas_compute seas_gpu siag siag_combo test yao yao_priority zhuang ```.

May 18, 11:00:01 GMT+0
Identified - Maintenance is now in progress.

May 22, 00:14:55 GMT+0
Completed - The power work has completed successfully. All nodes have been returned to normal service..

Duration: 5 days and 12 hours

Affected Components: GPU nodes (Holyoke), FASSE Compute Cluster (Holyoke), SLURM Scheduler - FASSE, Cannon Compute Cluster (Holyoke), seas_compute, Kempner Cluster GPU, Kempner Cluster CPU, SLURM Scheduler - Cannon, Boston Compute Nodes

May 11, 11:00:00 GMT+0
Identified - Rescheduled to May 11.

May 11, 11:00:00 GMT+0
Identified - Our Holyoke data center, MGHPCC, will be doing power work on Row 8A. This work, which will occur this week and next week, will bring online another power feed which will increase power capacity. In order to do this work, it will require us to idle half the nodes in 8a for the duration of the week. This means all partitions in this row will be at half capacity. Existing jobs should drain naturally and no job should need to be canceled. The impacted partitions are: ``` arguelles_delgado_h100 bigmem bigmem_intermediate blackhole_gpu dvorkin eddy enos gershman gpu gpu_h200 gpu_requeue hejazi hernquist_ice hoekstra hsph hsph_gpu huce_ice iaifi_gpu_requeue intermediate itc_cluster itc_gpu janson_sapphire joonholee jshapiro kempner kempner_priority kempner_dev kempner_eng kempner_h200_priority kempner_h100 kempner_h100_priority kempner_h100_priority2 kempner_h100_priority3 kempner_h100_priority4 kempner_interactive kovac kozinsky kozinsky_gpu kozinsky_requeue murphy_ice mweber_compute mweber_gpu olveczky_sapphire ortegahernandez_ice rivas sapphire seas_compute siag siag_combo test yao yao_priority zhuang ```.

May 16, 23:00:00 GMT+0
Completed - Maintenance has completed successfully.

May 11, 11:00:01 GMT+0
Identified - Maintenance is now in progress.

Duration: 4 hours

Affected Components: FASRC Two-Factor (OpenAuth), GPU nodes (Holyoke), FASSE Compute Cluster (Holyoke), SLURM Scheduler - FASSE, Cannon Compute Cluster (Holyoke), seas_compute, Cannon Open OnDemand, FASSE login nodes, Kempner Cluster GPU, Kempner Cluster CPU, FASSE Open OnDemand, Login Nodes - Holyoke, SLURM Scheduler - Cannon, Login Nodes - Boston, Netscratch (Global Scratch), Boston Compute Nodes

May 4, 13:00:00 GMT+0
Identified - FASRC monthly maintenance will take place on May 4th 2026\. Our maintenance tasks should be completed between 9am-1pm. **NOTICES:** * Annual data center power downtime: The annual downtime at MGHPCC will take place June 15 - June 18\. This year's downtime will be one day longer. More details will be sent to all users next month. * Training: Upcoming training from FASRC and other sources can be found on our Training Calendar. at <https://www.rc.fas.harvard.edu/upcoming-training/> * Status Page: You can subscribe to our status to receive notifications of maintenance, incidents, and their resolution at <https://status.rc.fas.harvard.edu/> (click Get Updates for options). **MAINTENANCE TASKS** Cannon cluster will be paused during this maintenance?: **YES** FASSE cluster will be paused during this maintenance?: **YES** * Slurm 25.11.5 Upgrade * Audience: All cluster users * Impact: Jobs will be paused during the upgrade * Reboot remaining stuck nodes from power outage * Audience: N/A * Impact: No visible impact to user * Two-Factor/OpenAuth ([two-factor.rc.fas.harvard.edu](http://two-factor.rc.fas.harvard.edu)) replacement * Audience: All account holders * Impact: The server will be unavailable during maintenance. You will be unable to obtain a new or replacement OpenAuth token during this period. * Domain controller replacement * Audience: Internal * Impact: End users should not see any impact * OOD/Open OnDemand reboots * Audience: All OOD users, reboot of the head nodes * Impact: Running sessions will _not_ be affected * Login node reboots * Audience; All login node users * Impact: Login nodes will reboot during the maintenance window * Netscratch 90-day retention cleanup * Audience; All netscratch users * Impact: Files older than 90 days will be removed per our [scratch policy](https://docs.rc.fas.harvard.edu/kb/policy-scratch/). Please note that this cleanup can happen at any time, not just during maintenance. Thank you, FAS Research Computing <https://docs.rc.fas.harvard.edu/> <https://www.rc.fas.harvard.edu/>.

May 4, 13:00:01 GMT+0
Identified - Maintenance is now in progress.

May 4, 14:56:02 GMT+0
Identified - The scheduler is re-opened and jobs un-paused. Other, non-impacting, work continues..

May 4, 17:00:00 GMT+0
Completed - Maintenance has completed successfully.

Duration: 1 hour

Affected Components: Starfish

May 1, 20:00:00 GMT+0
Identified - Starfish will be upgraded to the latest version on Friday, May 1st from 4pm-5pm. The service and dashboard will be down during this time. .

May 1, 20:00:01 GMT+0
Identified - Maintenance is now in progress.

May 1, 21:00:00 GMT+0
Completed - Maintenance has completed successfully.

Duration: 2 hours

Affected Components: Cannon Open OnDemand, FASSE Open OnDemand

Apr 30, 12:00:00 GMT+0
Identified - At 8am on Thursday April 30th we will be upgrading from Open OnDemand version 4.0.7 to 4.1.4 on both the Cannon and FASSE clusters. This is not expected to impact running jobs. This upgrade adds the Jobs->Project Manager menu item and fixes an issue that affected access to the Clusters->Shell Access menu item when using Firefox..

Apr 30, 12:00:01 GMT+0
Identified - Maintenance is now in progress.

Apr 30, 14:00:00 GMT+0
Completed - Maintenance has completed successfully.

Duration: 17 minutes

Affected Components: docs.rc.fas.harvard.edu, www.rc.fas.harvard.edu

Apr 28, 17:00:00 GMT+0
Identified - Security updates are required for [www.rc.fas.harvard.edu](http://www.rc.fas.harvard.edu) and [docs.rc.fas.harvard.edu](http://docs.rc.fas.harvard.edu) This work will take place today between 1pm and 2pm Both sites will be down for very short periods during the updates..

Apr 28, 17:00:01 GMT+0
Identified - Maintenance is now in progress.

Apr 28, 17:16:58 GMT+0
Completed - Website maintenance has completed successfully..

Duration: 4 hours

Affected Components: FASRC Two-Factor (OpenAuth), , , Cannon Open OnDemand, FASSE login nodes, FASSE Open OnDemand, Login Nodes - Holyoke, Login Nodes - Boston, Netscratch (Global Scratch), , Login Nodes → OpenOnDemand/OOD → login.rc.fas.harvard.edu →

Apr 6, 13:00:01 GMT+0
Identified - Maintenance is now in progress.

Apr 6, 17:00:00 GMT+0
Completed - Maintenance has completed successfully.

Apr 6, 13:00:00 GMT+0
Identified - FASRC monthly maintenance will take place on April 6th 2026\. Our maintenance tasks should be completed between 9am-1pm. **NOTICES:** * Training: Upcoming training from FASRC and other sources can be found on our Training Calendar. at <https://www.rc.fas.harvard.edu/upcoming-training/> * Status Page: You can subscribe to our status to receive notifications of maintenance, incidents, and their resolution at <https://status.rc.fas.harvard.edu/> (click Get Updates for options). * We'd love to hear success stories about your or your lab's use of FASRC. Submit your story [here](https://www.rc.fas.harvard.edu/user-stories/). **MAINTENANCE TASKS** Cannon cluster will be paused during this maintenance?: **NO** FASSE cluster will be paused during this maintenance?: **NO** * [two-factor.rc.fas.harvard.edu](http://two-factor.rc.fas.harvard.edu) [OpenAuth](https://docs.rc.fas.harvard.edu/kb/openauth/) cut-over to new server * Audience: New accounts or anyone requesting an OpenAuth token * Impact: two-factor will be unavailable while moving to a new server * RStudio Server (Open OnDemand) * Audience: RStudio Server users on Cannon and FASSE * Impact: We will be decommissioning some versions of RStudio Server so we can properly maintain all production versions. Versions to be decommissioned: * R 4.1.3 (Bioconductor 3.14, RStudio 2022.02.0) * R 4.1.0 (Bioconductor 3.13, RStudio 1.4.1717) * R 4.0.3 (Bioconductor 3.12, Rstudio 1.3.1093) * R 4.0.0 (Bioconductor 3.11, Rstudio 1.3.1093) * If you use one of these versions, we recommend replacing it with the most recent version, R 4.4.2 (Bioconductor 3.20, RStudio 2024.12.0). You must reinstall previously installed libraries. * Domain controller replacement * Audience: Internal * Impact: End users should not see any impact * OOD/Open OnDemand reboots * Audience: All OOD users, reboot of the head nodes * Impact: Running sessions will _not_ be affected * Login node reboots * Audience; All login node users * Impact: Login nodes will reboot during the maintenance window * Netscratch 90-day retention cleanup * Audience; All netscratch users * Impact: Files older than 90 days will be removed per our [scratch policy](https://docs.rc.fas.harvard.edu/kb/policy-scratch/). Please note that this cleanup can happen at any time, not just during maintenance. Thank you, FAS Research Computing <https://docs.rc.fas.harvard.edu/> <https://www.rc.fas.harvard.edu/>.

Duration: 4 hours

Affected Components: Login Nodes - Holyoke, FASSE Compute Cluster (Holyoke), SLURM Scheduler - Cannon, , , GPU nodes (Holyoke), , SLURM Scheduler - FASSE, , Cannon Compute Cluster (Holyoke), , seas_compute, Cannon Open OnDemand, FASSE login nodes, Kempner Cluster CPU, Kempner Cluster GPU, FASSE Open OnDemand, Login Nodes - Boston, Netscratch (Global Scratch), , Boston Compute Nodes, Login Nodes → Cannon Cluster → OpenOnDemand/OOD → Kempner Cluster → FASSE Cluster → login.rc.fas.harvard.edu →

Mar 2, 14:00:01 GMT+0
Identified - Maintenance is now in progress.

Mar 2, 14:00:00 GMT+0
Identified - Monthly maintenance will take place on Monday March 2nd, 2026\. Our maintenance tasks should be completed between 9am-1pm. **NOTICES:** * Training: Upcoming training from FASRC and other sources can be found on our Training Calendar. at <https://www.rc.fas.harvard.edu/upcoming-training/> * Status Page: You can subscribe to our status to receive notifications of maintenance, incidents, and their resolution at <https://status.rc.fas.harvard.edu/> (click Get Updates for options). * We'd love to hear success stories about your or your lab's use of FASRC. Submit your story [here](https://www.rc.fas.harvard.edu/user-stories/). **MAINTENANCE TASKS** Cannon cluster will be paused during this maintenance?: **YES** FASSE cluster will be paused during this maintenance?: **YES** * Slurm scheduler update * Audience: All cluster users * Impact: Jobs will be paused during maintenance * OOD node reboots * Audience; All Open OnDemand users * Impact: OOD nodes will reboot during the maintenance window * Login node reboots * Audience: All login node users * Impact: Login nodes will reboot during the maintenance window * Netscratch retention purge * Audience: All users of Netscratch * Impact: Files older than 90 days will be removed. Please note that retention cleanup can and does run at any time, not just during the maintenance window. Thank you, FAS Research Computing <https://docs.rc.fas.harvard.edu/> [https://www.rc.fas.harvard.edu/](https://www.rc.fas.harvard.edu/upcoming-training/).

Mar 2, 18:00:00 GMT+0
Completed - Maintenance has completed successfully.

Duration: 1 day

Affected Components: Starfish

Feb 25, 14:00:00 GMT+0
Identified - Starfish will be unavailable starting Wednesday, February 25th at 9AM until Thursday, February 26th at 9AM, for routine maintenance. The online dashboard will be inaccessible during this time..

Feb 26, 14:00:00 GMT+0
Completed - Maintenance has completed successfully.

Feb 25, 14:00:01 GMT+0
Identified - Maintenance is now in progress.

Duration: 9 hours

Affected Components: NESE (NorthEast Storage Exchange)

Feb 19, 13:00:01 GMT+0
Identified - Maintenance is now in progress.

Feb 19, 13:00:00 GMT+0
Identified - From our partners at NESE. Details follow: We are installing four new tape frames, which will bring the tape system raw storage capacity to 253 petabytes. **Service Affected:** NESE Tape Service **Maintenance Window:** 8:00 AM - 5:00 PM (EST) * The tape service will be unavailable. * All upgrade activities are expected to be completed on the same day. NOTES: * Monitor the MGHPCC Slack #nese channel for status updates and announcements * Monitor <https://nese.instatus.com/> for real-time updates on progress Subscribe to <https://nese.instatus.com/subscribe/email> for updates and announcements.

Feb 19, 22:00:00 GMT+0
Completed - Maintenance has completed successfully.

Duration: 8 minutes

Affected Components: docs.rc.fas.harvard.edu, www.rc.fas.harvard.edu

Feb 9, 21:10:00 GMT+0
Identified - Security updates will require a brief interruption for our primary websites [www.rc.fas.harvard.edu](http://www.rc.fas.harvard.edu) and [docs.rc.fas.harvard.edu](http://docs.rc.fas.harvard.edu) We will endeavour to keep this update as short as possible. Each site may be unavailable for a few minutes..

Feb 9, 21:10:01 GMT+0
Identified - Maintenance is now in progress.

Feb 9, 21:17:51 GMT+0
Completed - Maintenance has completed successfully..

Duration: 9 hours

Affected Components: NESE (NorthEast Storage Exchange)

Feb 9, 13:00:00 GMT+0
Identified - From our partners at NESE. Details follow: In the process of the tape front-end file caching system upgrade, we will be installing a new IBM Storage Scale System 6000\. We will provide an additional update for when the software integration and data transfer from the current IBM Elastic Storage System 5000 will be performed. **Service Affected:** NESE Tape Service **Maintenance Window: No Downtime expected** NOTES: * Monitor the MGHPCC Slack #nese channel for status updates and announcements * Monitor <https://nese.instatus.com/> for real-time updates on progress * Subscribe to <https://nese.instatus.com/subscribe/email> for updates and announcements.

Feb 9, 13:00:01 GMT+0
Identified - Maintenance is now in progress.

Feb 9, 22:00:00 GMT+0
Completed - Maintenance has completed successfully.

Duration: 4 hours

Affected Components: Cannon Open OnDemand, Login Nodes - Holyoke, , , FASSE Compute Cluster (Holyoke), GPU nodes (Holyoke), , SLURM Scheduler - FASSE, , SLURM Scheduler - Cannon, Cannon Compute Cluster (Holyoke), , FASSE Open OnDemand, seas_compute, FASSE login nodes, Kempner Cluster CPU, Kempner Cluster GPU, Login Nodes - Boston, Netscratch (Global Scratch), , Boston Compute Nodes, Login Nodes → Cannon Cluster → OpenOnDemand/OOD → Kempner Cluster → FASSE Cluster → login.rc.fas.harvard.edu →

Feb 2, 14:00:00 GMT+0
Identified - Monthly maintenance will take place on Monday February 2nd, 2026\. Our maintenance tasks should be completed between 9am-1pm. **NOTICES:** * Training: Upcoming training from FASRC and other sources can be found on our Training Calendar. at <https://www.rc.fas.harvard.edu/upcoming-training/> * Status Page: You can subscribe to our status to receive notifications of maintenance, incidents, and their resolution at <https://status.rc.fas.harvard.edu/> (click Get Updates for options). * We'd love to hear success stories about your or your lab's use of FASRC. Submit your story [here](https://www.rc.fas.harvard.edu/user-stories/). **MAINTENANCE TASKS** Cannon cluster will be paused during this maintenance?: **YES** FASSE cluster will be paused during this maintenance?: **YES** * MaxTime change * Audience: Cluster users * Impact: In order to improve scheduling efficiency and stability, we will be setting a maximum run time on all partitions that have MaxTime set to UNLIMITED to a MaxTime of 3 days. The unrestricted partition will be set to 365 days. Partitions that already have MaxTime set will retain their current setting. Partition owners wishing to set a different MaxTime for their partition should contact FASRC. Note that we do no guarantee uptime and so users should utilize checkpointing to save state in case of node failure. * Slurm upgrade to 25.11.2 * Audience: All cluster users * Impact: Jobs will be paused during maintenance * OOD node reboots * Audience; All Open OnDemand users * Impact: OOD nodes will reboot during the maintenance window * Login node reboots * Audience; All login node users * Impact: Login nodes will reboot during the maintenance window Thank you, FAS Research Computing <https://docs.rc.fas.harvard.edu/> [https://www.rc.fas.harvard.edu/](https://www.rc.fas.harvard.edu/upcoming-training/).

Feb 2, 14:00:01 GMT+0
Identified - Maintenance is now in progress.

Feb 2, 18:00:00 GMT+0
Completed - Maintenance has completed successfully.

Duration: 4 hours

Affected Components: , , FASSE Compute Cluster (Holyoke), SLURM Scheduler - Cannon, GPU nodes (Holyoke), , SLURM Scheduler - FASSE, , Cannon Compute Cluster (Holyoke), , Cannon Open OnDemand, FASSE Open OnDemand, seas_compute, Login Nodes - Holyoke, FASSE login nodes, Kempner Cluster CPU, Kempner Cluster GPU, Login Nodes - Boston, , Boston Compute Nodes, Login Nodes → Cannon Cluster → OpenOnDemand/OOD → Kempner Cluster → FASSE Cluster → login.rc.fas.harvard.edu →

Jan 12, 14:00:00 GMT+0
Identified - Monthly maintenance will take place on January 12th, 2026\. Our maintenance tasks should be completed between 9am-1pm. **NOTICES:** * Changes to SEAS partitions, please see tasks below. * Changes to job age priority weighting, please see tasks below. * Status Page: You can subscribe to our status to receive notifications of maintenance, incidents, and their resolution at <https://status.rc.fas.harvard.edu/> (click Get Updates for options). * We'd love to hear success stories about your or your lab's use of FASRC. Submit your story [here](https://www.rc.fas.harvard.edu/user-stories/). **MAINTENANCE TASKS** Cannon cluster will be paused during this maintenance?: **YES** FASSE cluster will be paused during this maintenance?:**YES** * Slurm upgrade to 25.11.1 * Audience: All cluster users (Cannon and FASSE) * Impact: Jobs will be paused during maintenance * In conjunction with SEAS we will modify seas\_gpu and seas\_compute time limits * Audience: SEAS users * Impact: seas\_gpu: will be set to 2 days maximum seas\_compute: will be set to 3 days maximum Existing pending jobs longer than these limits will be set to 2 day and 3 day run times depending on partition. * Job Age Priority Weight Change * Audience: Cluster users * Impact: We will be adjusting the weight applied to the priority earned by jobs by virtue of their age. Currently job priority is made up of two factors, Fairshare and Job Age. The Job Age factor is currently set such that jobs gain priority over 3 days with a maximum priority equivalent to jobs with Fairshare of 0.5\. This keeps low fairshare jobs from languishing at the bottom of the queue. With the current settings though, users with low fairshare can gain significant advantage over users with higher relative fairshare. To remedy this we will be adjusting the Job Age weight to cap out at an equivalent Fairshare of 0.1\. This will still allow jobs with 0 fairshare to gain priority and thus not languish while letting fairshare govern a wider range of higher priority jobs. * Login node reboots * Audience; All login node users * Impact: Login nodes will reboot during the maintenance window * Open OnDemand (OOD) node reboots * Audienc:; All OOD users * Impact: OOD nodes will reboot during the maintenance window * Netscratch retention will run * Audience: All cluster netscratch users * Impact: Files older than 90 days will be removed. Please note that retention cleanup can and does run at any time, not just during the maintenance window. Thank you, FAS Research Computing <https://docs.rc.fas.harvard.edu/> [https://www.rc.fas.harvard.edu/](https://www.rc.fas.harvard.edu/upcoming-training/).

Jan 12, 14:00:01 GMT+0
Identified - Maintenance is now in progress.

Jan 12, 18:00:00 GMT+0
Completed - Maintenance has completed successfully.

Duration: 12 hours

Affected Components: Login Nodes - Holyoke, , GPU nodes (Holyoke), FASSE Compute Cluster (Holyoke), Isilon Storage Holyoke (Tier 1), , SLURM Scheduler - FASSE, Virtual Infrastructure - Holyoke, , Cannon Compute Cluster (Holyoke), , Cannon Open OnDemand, SLURM Scheduler - Cannon, FASSE Open OnDemand, License Servers, seas_compute, , FASSE login nodes, Kempner Cluster CPU, Kempner Cluster GPU, Login Nodes - Boston, Virtual Infrastructure - Boston, Isilon Storage Boston (Tier 1), , Boston Compute Nodes, Login Nodes → OpenOnDemand/OOD → Kempner Cluster → FASSE Cluster → Cannon Cluster → login.rc.fas.harvard.edu →

Dec 8, 11:00:00 GMT+0
Identified - Monthly maintenance will take place on December 8th. Our maintenance tasks should be completed between 9am-1pm. However: _Additionally_, MGHPCC will be performing power upgrades on the odd side of Row 8A where much of our computer resides. This is the final upgrade for this row. Current estimate for this work is a 12 hour window 6am-6pm. A list of the affected partitions is provided at the bottom of this notice. The nodes in those partitions will be drained prior to the work and will be powered down. Once the work is completed, those nodes will be returned to service. **Notices:** * New FASSE partition `fasse_gpu_h200`. This partitions has 2 H200 nodes and a 3day limit. It is available now. * 11/26 - 11/28 are university holidays (Thanksgiving). No on-site support, FASRC staff will return on 12/1. * Training: Upcoming training from FASRC and other sources can be found on our Training Calendar. at <https://www.rc.fas.harvard.edu/upcoming-training/> * Status Page: You can subscribe to our status to receive notifications of maintenance, incidents, and their resolution at <https://status.rc.fas.harvard.edu/> (click Get Updates for options). * We'd love to hear success stories about your or your lab's use of FASRC. Submit your story [here](https://www.rc.fas.harvard.edu/user-stories/). **MAINTENANCE TASKS** Cannon cluster will be paused during this maintenance?: **PARTIAL OUTAGE/YES** FASSE cluster will be paused during this maintenance?: **PARTIAL OUTAGE/YES** * Power work on Row 8A odd * Audience: Users of the partitions listed below * Impact: These nodes and partitions will be fully or partially down all day * OneFS (Isilon) upgrade * Audience: All Isilon (Tier 1) shares * Impact: Some VMs will be impacted including Cannon OOD, CBScentral, MCZapps/MCZbase, Portal, and Rclic1 (license server) * Slurm upgrade to 25.05.5 * Audience: All cluster users * Impact: Jobs will be paused during maintenance * Login node reboots * Audience: All login node users * Impact: Login nodes will reboot during the maintenance window **Impacted Cannon Partitions (Full or Partial Outage):** * arguelles\_delgado\_gpu\_a100 * arguelles\_delgado\_gpu\_mixed * bigmem\_intermediate * blackhole\_gpu * eddy * gershman * gpu\_requeue * hejazi * hernquist\_ice * hoekstra * huce\_ice * iaifi\_gpu * iaifi\_gpu\_priority * iaifi\_gpu\_requeue * itc\_gpu * jshapiro * kempner * kempner\_dev * kempner\_priority * kempner\_h100 * kempner\_h100\_priority * kempner\_h100\_priority2 * kempner\_h100\_priority3 * kempner\_interactive * kempner\_requeue * kovac * kozinsky * kozinsky\_gpu * kozinsky\_priority * kozinsky\_requeue * murphy\_ice * ortegahernandez\_ice * rivas * seas\_compute * seas\_gpu * serial\_requeue * siag\_combo * siag\_gpu * sur * zhuang.

Dec 8, 11:00:01 GMT+0
Identified - Maintenance is now in progress.

Dec 8, 23:00:00 GMT+0
Completed - Maintenance has completed successfully.

Duration: 4 days, 1 hour and 10 minutes

Affected Components: HolyLFS04 (Tier 0)

Dec 5, 14:00:00 GMT+0
Identified - The holylfs04 migration to holylfs06 has begun. All holylfs04 folders will be **read-only** for the duration of the migration, from **Friday, December 5th at 9AM until end of day on Monday, December 8th.** All labs with holylfs04 have been informed via email; please email [rdm@rc.fas.harvard.edu](mailto:rdm@rc.fas.harvard.edu) if you have any questions..

Dec 9, 15:09:30 GMT+0
Completed - Maintenance has completed successfully..

Duration: 5 days

Affected Components: NESE (NorthEast Storage Exchange), Tape - (Tier 3)

Dec 1, 11:00:00 GMT+0
Identified - NESE, the Northeast Storage Exchange at MGHPCC which supplies the Tier3 tape service used by FASRC, will be offline for maintenance on the system Dec 1st - 5th. There will be ongoing performance-affecting maintenance until Dec 12th. Please see below for details. WHO: Any lab who has or is moving data to tape. IMPACT: No access 12/1/25 - 12/5/25\. Reduced performance 12/5/25 - 12/12/25. > NESE tape system maintenance and major software upgrade is scheduled to begin on December 1, 2025\. As a result, the NESE Tape service will be offline from December 1 to December 5. > > Starting December 8 through December 12, the service will be back online with reduced performance. All maintenance activities are planned to conclude on December 12, 2025. > > * Monitor: <https://nese.instatus.com/> for real-time updates on progress > * Subscribe to <https://nese.instatus.com/subscribe/email> for updates and announcements.

Dec 1, 11:00:01 GMT+0
Identified - Maintenance is now in progress.

Dec 6, 11:00:00 GMT+0
Completed - Maintenance has completed successfully.

Duration: 1 hour

Affected Components: Starfish

Nov 14, 22:00:01 GMT+0
Identified - Maintenance is now in progress.

Nov 14, 23:00:00 GMT+0
Completed - Maintenance has completed successfully.

Nov 14, 22:00:00 GMT+0
Identified - There is a planned upgrade of the Starfish dashboard scheduled for Friday November 14th starting at 5PM. The dashboard will be down for an hour while the upgrade is performed..

Duration: 9 hours and 4 minutes

Affected Components: , , SLURM Scheduler - FASSE, , Kempner Cluster CPU, Cannon Compute Cluster (Holyoke), , Cannon Open OnDemand, SLURM Scheduler - Cannon, FASSE Open OnDemand, seas_compute, Kempner Cluster GPU, Login Nodes - Holyoke, , GPU nodes (Holyoke), FASSE Compute Cluster (Holyoke), FASSE login nodes, Login Nodes - Boston, Netscratch (Global Scratch), Boston Compute Nodes, Login Nodes → OpenOnDemand/OOD → Kempner Cluster → FASSE Cluster → Cannon Cluster →

Nov 3, 11:00:00 GMT+0
Identified - Monthly maintenance will take place on November 3rd. Additionally, MGHPCC will be performing power upgrades on the even side of Row 8A where much of our computer resides. A further upgrade will take place Dec. 8th on the odd side. A list of the affected partitions is provided at the bottom of this notice. The nodes in those partitions will be drained prior to the work and will be powered down. Once the work is completed, those nodes will be returned to service. Current estimate is a 12 hour window. We will adjust as we know more. **MAINTENANCE TASKS** Cannon cluster will be paused during this maintenance?: **PARTIAL OUTAGE/YES** FASSE cluster will be paused during this maintenance?: **PARTIAL OUTAGE/YES** * Power work on Row 8A Even * Audience: Users of the partitions listed below * Impact: These nodes and partitions will be fully or partially down all day * Slurm upgrade to 25.05.4 * Audience: All cluster users * Impact: Jobs will be paused during maintenance * Block repo.anaconda.com cluster wide * Audience: Anyone attempting to use repo.anaconda.com * Impact: This change should not impact your Python workflow on the cluster. But if it does, consider using the open-source channel, `conda-forge`, through Miniforge distribution to install Python packages. This can be done by following our instructions on <https://docs.rc.fas.harvard.edu/kb/python-package-installation/> * Change Slurm User to Local User * Audience: All cluster users * Impact: Behind the scenes. No impact to users * Login node reboots (morning) * Audience: Anyone logged into a FASRC Cannon or FASSE login node * Impact: All login nodes will rebooted during this maintenance window * Netscratch cleanup ( <https://docs.rc.fas.harvard.edu/kb/policy-scratch/> ) * Audience: Cluster users * Impact: Files older than 90 days will be removed. Please note that retention cleanup can and does run at any time, not just during the maintenance window. **AFFECTED PARTITIONS** Nov. 3, 2025 - All Day Power Work Partial or Full Outage Apples to: arguelles\_delgado\_h100 bigmem dvorkin eddy enos gpu gpu\_h200 gpu\_requeue hsph hsph\_gpu intermediate itc\_cluster joonholee jshapiro kempner\_dev kemkpner\_eng kempner\_requeue mweber\_compute mweber\_gpu olveczky\_sapphire sapphire seas\_compute seas\_gpu serial\_requeue yao yao\_gpu yao\_priority test.

Nov 3, 11:00:01 GMT+0
Identified - Maintenance is now in progress.

Nov 3, 20:04:13 GMT+0
Completed - Maintenance has completed successfully including power work at MGHPCC. A reminder that additional all-day power work will take place on Dec 8th, along with our maintenance from 9am-1pm.

Duration: 4 hours

Affected Components: , Network - Cambridge, Network - Boston, Network - Holyoke/MGHPCC, Login Nodes - Boston, Netscratch (Global Scratch), Login Nodes - Holyoke, FASSE login nodes, Login Nodes →

Oct 6, 13:00:01 GMT+0
Identified - Maintenance is now in progress.

Oct 6, 13:00:00 GMT+0
Identified - FASRC monthly maintenance will take place Monday October 6th, 2025 from 9am-1pm **NOTICES** * Training: Upcoming training from FASRC and other sources can be found on our Training Calendar. at <https://www.rc.fas.harvard.edu/upcoming-training/> * Status Page: You can subscribe to our status to receive notifications of maintenance, incidents, and their resolution at <https://status.rc.fas.harvard.edu/> (click Get Updates for options). * Upcoming holidays: Columbus / Indigenous Peoples’ Day - October 13 **MAINTENANCE TASKS** Cannon cluster will be paused during this maintenance?: **NO** FASSE cluster will be paused during this maintenance?: **NO** * DNS server reboots * Audience: All FASRC services * Impact: Rolling reboot should have no impact * Login node reboots * Audience: Anyone logged into a FASRC Cannon or FASSE login node * Impact: All login nodes will rebooted during this maintenance window * Netscratch cleanup ( <https://docs.rc.fas.harvard.edu/kb/policy-scratch/> ) * Audience: Cluster users * Impact: Files older than 90 days will be removed. Please note that retention cleanup can and does run at any time, not just during the maintenance window. Thank you, FAS Research Computing <https://docs.rc.fas.harvard.edu/> <https://www.rc.fas.harvard.edu/>.

Oct 6, 17:00:00 GMT+0
Completed - Maintenance is now in progress.