SLURM Scheduler - Cannon - Працює
SLURM Scheduler - Cannon
Cannon Compute Cluster (Holyoke) - Працює
Cannon Compute Cluster (Holyoke)
Boston Compute Nodes - Працює
Boston Compute Nodes
GPU nodes (Holyoke) - Працює
GPU nodes (Holyoke)
seas_compute - Працює
seas_compute
SLURM Scheduler - FASSE - Працює
SLURM Scheduler - FASSE
FASSE Compute Cluster (Holyoke) - Працює
FASSE Compute Cluster (Holyoke)
Kempner Cluster CPU - Працює
Kempner Cluster CPU
Kempner Cluster GPU - Працює
Kempner Cluster GPU
FASSE login nodes - Працює
FASSE login nodes
Cannon Open OnDemand - Працює
Cannon Open OnDemand
FASSE Open OnDemand - Працює
FASSE Open OnDemand
Netscratch (Global Scratch) - Працює
Netscratch (Global Scratch)
Home Directory Storage - Boston - Працює
Home Directory Storage - Boston
Tape - (Tier 3) - Працює
Tape - (Tier 3)
Holylabs - Працює
Holylabs
Isilon Storage Holyoke (Tier 1) - Працює
Isilon Storage Holyoke (Tier 1)
Holystore01 (Tier 0) - Працює
Holystore01 (Tier 0)
HolyLFS04 (Tier 0) - Працює
HolyLFS04 (Tier 0)
HolyLFS05 (Tier 0) - Працює
HolyLFS05 (Tier 0)
HolyLFS06 (Tier 0) - Повністю недоступно
HolyLFS06 (Tier 0)
Holyoke Tier 2 NFS - Працює
Holyoke Tier 2 NFS
Holyoke Specialty Storage - Працює
Holyoke Specialty Storage
holECS - Працює
holECS
Isilon Storage Boston (Tier 1) - Працює
Isilon Storage Boston (Tier 1)
BosLFS02 (Tier 0) - Працює
BosLFS02 (Tier 0)
Boston Tier 2 NFS - Працює
Boston Tier 2 NFS
CEPH Storage Boston (Tier 2) - Працює
CEPH Storage Boston (Tier 2)
Boston Specialty Storage - Працює
Boston Specialty Storage
bosECS - Працює
bosECS
Samba Cluster - Працює
Samba Cluster
Globus Data Transfer - Працює
Globus Data Transfer
Історія сповіщень
Переглянути поточний статустрав. 2024
- Вирішено · UTC
NESE maintenance has resolved and is now back in service
- Моніторинг · UTC
A note from NESE. Their maintenance has been delayed by hardware issues. ETA now Monday 6/3
Dear All,
NESE Tape system upgrade is currently in progress. While IBM hardware
team works on TS4500 library tape frame expansion and IBM software team
works on ESS and Archive software and firmware upgrades, the work
progress has been slowed down due to unforeseen hardware issues.We now expect to bring the tape service back into production this Monday
morning. We apologize for any inconvenience caused by the delay. - Визначено · UTC
Due to maintenance at our tape partner, NESE (Northeast Storage Exchange), access to tape allocations will be unavailable until at least late Thursday (5/30). Normal operations will resume by Friday (5/31).
If you continue to have issues with a Globus tape endpoint on Friday, please contact FASRC or NESE
- Завершено · травня 24, 2024 у 21:50UTC
2024 MGHPCC downtime complete
DOWNTIME COMPLETE
The annual multi-day power downtime at MGHPCC (https://www.rc.fas.harvard.edu/blog/2024-mghpcc-power-downtime/) is complete (with any exceptions noted below). Normal service resumes today (Friday May 24th) at 5pm.
The cluster has been updated to Rocky Linux 8.9. Several network, InfiniBand, computer, and storage firmware updates were installed. Available security updates were also installed.
CANNON NODES
More than 90% of nodes are up and all partitions are enabled. If your specialty partition has a downed node, we will attend to this on Tuesday.
FASSE OOD
Some updates are still propagating. If your FASSE Open OnDemand/VDI session does not work initially, please wait or retry your job/session.
POST-DOWNTIME SUPPORT
If you have any further concerns or unanswered questions please submit a help ticket (https://portal.rc.fas.harvard.edu/rcrt/submit_ticket) and we will do our best to respond quickly. Please bear in mind it is a long weekend, so lingering issues may not be dealt with until Tuesday.
Also, have a good long Memorial Day weekend!
Thanks,
FAS Research Computing
https://www.rc.fas.harvard.edu/
https://docs.rc.fas.harvard.edu/
- Оновлення · травня 24, 2024 у 21:08UTC
We are currently delayed opening the cluster due to some lingering issues.
We will re-open as soon as possible or update again at 6pm.
- Оновлення · травня 24, 2024 у 13:47UTC
Power work completed by facility. Currently on schedule for powerup and return to service. ETA 5pm.
- Триває · травня 21, 2024 у 13:00UTCMaintenance is now in progress
- Заплановано · травня 21, 2024 у 13:00UTC
The 2024 MGHPCC data center annual power downtime will take place May 21-24, 2024.
We will begin our shutdown on Tuesday May 21st and expect a return to service by 5PM Friday May 24th.
- Jobs: Please plan ahead as all still running jobs on the morning of May 21st will be stopped and canceled and will need to be resubmitted after the downtime. Pending jobs will remain in the queue until the cluster returns to regular service on May 24th.
- Access: The cluster, scheduler, login, and OoD nodes will be unavailable for the duration of the downtime. New lab and account requests should wait until after the downtime.
- Storage: All Holyoke storage will be powered down and unavailable for the duration of the downtime. Boston storage will remain online, but your ability to access it may be impacted and network changes may briefly affect its availability.
Further details, an explanation for this year's change in scheduling, a visual timeline, and a list of maintenance tasks overview can be found at:
https://www.rc.fas.harvard.edu/blog/2024-mghpcc-power-downtime/
Progress of the downtime will be posted here on our status page during the event. Note that you can subscribe to receive updates as they happen. Click Get Updates in the upper right.
MAJOR TASK OVERVIEW
OS upgrade to Rocky 8.9 - Point upgrade, no code rebuilds will be required. Switch from system OFED to Mellanox OFED on nodes for improved performance
Infiniband (network) upgrades
BIOS updates (various)
Storage firmware updates
Network Maintenance
Decommission old nodes (targets contacted)
Additional minor one-off updates and maintenance (cable swap, reboots, etc.)
Thanks,
FAS Research Computing
https://www.rc.fas.harvard.edu/
- Вирішено · UTCThis incident has been resolved.
- Вивчається · UTC
We are still unable to resolve the issue with these nodes and are working with the facility, networking, and our staff to find a solution. The affected partitions (noted in previous update below) will be resource-constrained and continue to be slow or unable to queue new jobs.
If you are using a partition that cannot queue new jobs, please consider adding additional partitions to your job: https://docs.rc.fas.harvard.edu/kb/running-jobs/#Slurm_partitions
Also, a reminder that the data center power downtime will begin Tuesday morning. So any new jobs scheduled for more than 3 days will not complete: https://www.rc.fas.harvard.edu/blog/2024-mghpcc-power-downtime/ - Визначено · UTC
We are still working on the root cause and resolution for these downed nodes.
Partitions with one or more affected nodes [involves multiple nodes unless denoted as (1) ]:
arguelles_delgado_gpu (1)
hsph
joonholee
jshapiro_sapphire
lichtmandce01
bigmem
gpu_requeue (1)
intermediate
sapphire
serial_requeue (1)
shared (1)
test
yao / yao_priority
use 'sinfo -p [partition name]' if you wish to see the down nodes in particular queue
- Вивчається · UTC
We are currently investigating this incident. An unknown outage has downed many nodes in row 8A of our data center. More information to follow.
Includes nodes from the sapphire, test, gpu, and other partitions.
квіт. 2024
берез. 2024
- Вирішено · UTCThis incident has been resolved.
- Моніторинг · UTC
Informational Notice
The Slurm upgrade to 23.11.4 was completed successfully during maintenance. However a complication with the automation of Slurm's cryptographic keys occurred during the upgrade which caused nodes to lose the ability to talk to the Slurm master. The Slurm master therefore viewed those nodes as down and requeued their jobs.
All jobs on Cannon and FASSE were requeued.
This is deeply regrettable but the chain of events which caused this could not be foreseen.
To check the status of your jobs, see the common Slurm commands at:
https://docs.rc.fas.harvard.edu/kb/convenient-slurm-commands/#Information_on_jobs
FAS Research Computing

