FAS Research Computing - स्थिति पृष्ठ

Our maintenance tasks should be completed between 9am-1pm.
Some power work on row 7c will run 9-5 but will not affect running jobs or new jobs (see below).

NOTICES:

• Monday, October 12 is a university holiday (Indigenous Peoples/Columbus)

• Training: Upcoming training from FASRC and other sources can be found on our Training Calendar. at https://www.rc.fas.harvard.edu/upcoming-training/

• Status Page: You can subscribe to our status to receive notifications of maintenance, incidents, and their resolution at https://status.rc.fas.harvard.edu/ (click Get Updates for options).

MAINTENANCE TASKS

Cannon cluster will be paused during this maintenance?: YES
FASSE cluster will be paused during this maintenance?: YES

• Slurm Upgrade to 26.05.4

  • Audience: Cluster

  • Impact: The cluster will be paused during this maintenance

• New: Enable Termination of User Processes on Logout

  • Audience: Cluster

  • Impact: Going forward all user processes will be terminated upon login session exit on the login nodes, excluding things running in screen and tmux. Users should leverage the cluster for non-interactive processes. This is to clean up after AI agents which tend to create orphaned processes which drag down login node performance.

• New: watch command cadence limit

  • Audience: Cluster

  • Impact:The watch command will have a minimum cadence of 60s when used on commands talking to the slurm scheduler (i.e. squeue, showq, sinfo, scontrol, sdiag, lsload). Users desiring faster polling should leverage the sacct command that talks to the slurm database. In general users should not poll the scheduler more than once every minute, ideally once every 5-10 minutes. Polling more often slows the scheduler. FASRC reserves the right to ban users who who tax the scheduler with queries. For more on cluster customs and responsibilities see: https://docs.rc.fas.harvard.edu/kb/responsibilities/

• Holyoke/MGHPCC row 7c power work - 9am-5pm (CANCELLED)

  • Audience: Cluster

  • Impact: MGHPCC will be upgrading power on Pod 7c Even Side on September 14th 7am-5pm. This necessitates idling half the nodes on that side of the pod. A blocking reservation has been put in place to accomplish this. No jobs will be canceled but users will notice degraded scheduling throughput due to half the nodes being closed in the following partitions: 

    arguelles_delgado, blackhole, conroy, davies, desai, doshi-velez, dsouza, eddy, edwards, geophysics, giribet, gpu_test, hernquist, huce_cascade, huttenhower, imasc, jacobsen2, janson_cascade, janson, ke, lukin, murphy, nguyen, ni_lab, olveczky, ortegahernandez, pehlevan, seas_compute, shared, shakhnovich, tambe, unrestricted, vishwanath, whipple, xlin, yin, zon

• Login node reboots 

  • Audience: All login nodes

  • Impact: Login nodes will be unavailable until after maintenance

• OOD/Open OnDemand down/reboots

  • Audience: All OOD users

  • Impact: OOD will be unavailable until after maintenance

• Gurobi license key update

  • Audience: Anyone who uses Gurobi software on the clusters.

  • Impact: Jobs running Gurobi may fail. FASRC recommends waiting until maintenance is over to run Gurobi jobs.

• Netscratch 90-day retention cleanup

  • Audience; All netscratch users

  • Impact: Files older than 90 days will be removed per our scratch policy. Please note that this cleanup can happen at any time, not just during maintenance.

Thank you,
FAS Research Computing
https://docs.rc.fas.harvard.edu/
https://www.rc.fas.harvard.edu/