FAS Research Computing - Lịch sử thông báo

Globus Data Transfer Trải qua sự cố ngừng hoạt động lớn

Status page for the Harvard FAS Research Computing cluster and other resources.

Cluster Utilization (VPN and FASRC login required): Cannon | FASSE


Please scroll down to see details on any Incidents or maintenance notices.
Monthly maintenance occurs on the first Monday of the month (except holidays).

GETTING HELP
Documentation: https://docs.rc.fas.harvard.edu | Account Portal https://portal.rc.fas.harvard.edu
Email: rchelp@rc.fas.harvard.edu | Support Hours

The colors shown in the bars below were chosen to increase visibility for color-blind visitors.
For higher contrast, switch to light mode at the bottom of this page if the background is dark and colors are muted.
Đang hoạt động

SLURM Scheduler - Cannon - Đang hoạt động

Cannon Compute Cluster (Holyoke) - Đang hoạt động

Boston Compute Nodes - Đang hoạt động

GPU nodes (Holyoke) - Đang hoạt động

seas_compute - Đang hoạt động

Đang hoạt động

SLURM Scheduler - FASSE - Đang hoạt động

FASSE Compute Cluster (Holyoke) - Đang hoạt động

Đang hoạt động

Kempner Cluster CPU - Đang hoạt động

Kempner Cluster GPU - Đang hoạt động

Đang hoạt động

FASSE login nodes - Đang hoạt động

Đang hoạt động

Cannon Open OnDemand - Đang hoạt động

FASSE Open OnDemand - Đang hoạt động

Sự cố lớn

Netscratch (Global Scratch) - Đang hoạt động

Home Directory Storage - Boston - Đang hoạt động

Tape - (Tier 3) - Đang hoạt động

Holylabs - Đang hoạt động

Isilon Storage Holyoke (Tier 1) - Đang hoạt động

Holystore01 (Tier 0) - Đang hoạt động

HolyLFS04 (Tier 0) - Đang hoạt động

HolyLFS05 (Tier 0) - Đang hoạt động

HolyLFS06 (Tier 0) - Đang hoạt động

Holyoke Tier 2 NFS - Đang hoạt động

Holyoke Specialty Storage - Đang hoạt động

holECS - Đang hoạt động

Isilon Storage Boston (Tier 1) - Đang hoạt động

BosLFS02 (Tier 0) - Đang hoạt động

Boston Tier 2 NFS - Đang hoạt động

CEPH Storage Boston (Tier 2) - Đang hoạt động

Boston Specialty Storage - Đang hoạt động

bosECS - Đang hoạt động

Samba Cluster - Đang hoạt động

Globus Data Transfer - Sự cố lớn

Lịch sử thông báo

Xem trạng thái hiện tại

thg 5 2025

Coldfront service disruption
Đã khắc phụcSự cố một phần3 giờ 26 phút
  • Đã khắc phụcUTC
    This incident has been resolved.
  • Đang điều traUTC

    Coldfront is undergoing maintenance and may experience service disruption in the course of this process

    ETA for resolution is end of business on Friday.

THG 529
Starfish upgrade Thursday, May 29th from 5PM-6PM
Hoàn thànhBảo trì17 giờ 27 phút
  • Hoàn thànhtháng 05 30, 2025 tại 14:27UTC
    Maintenance has completed successfully.
  • Đã lên kế hoạchtháng 05 29, 2025 tại 21:00UTC
    Starfish upgrade Thursday, May 29th from 5PM-6PM. Starfish will be unavailable during that time
THG 528
Websites maintenance www.rc / docs.rc
Hoàn thànhBảo trì19 phút
  • Hoàn thànhtháng 05 28, 2025 tại 18:34UTC
    Maintenance has completed successfully.
  • Đã lên kế hoạchtháng 05 28, 2025 tại 18:15UTC
    In preparation for the upcoming downtime, we will be performing pre-emptive maintenance on the FASRC websites today to ensure they and their backup host are up to date. No interruption is expected, however the sites may be slower during the updates. https://www.rc.fas.harvard.edu https://docs.rc.fas.harvard.edu
Coldfront is undergoing necessary maintenance. WIP
Đã khắc phụcHiệu suất giảm sút5 giờ 23 phút
  • Đã khắc phụcUTC
    maintenance complete
  • Đã nhận diệnUTC
    Coldfront is in need of database maintenance. Expect return to service by 5PM.
THG 521
MGHPCC power work 5/21 - 5/23 - Some partitions will be at half capacity
Hoàn thànhBảo trì56 giờ
  • Hoàn thànhtháng 05 23, 2025 tại 19:00UTC
    Maintenance has completed successfully
  • Đang tiến hànhtháng 05 21, 2025 tại 11:00UTC
    Maintenance is now in progress
  • Đã lên kế hoạchtháng 05 21, 2025 tại 11:00UTC

    The MGHPCC Holyoke data center will be performing power work on May 21st -23rd. This work will take out one half (or one 'side') of the power capacity for certain rows/racks including our compute rows. Because of our power draw, one side is not enough power to keep each full rack running.

    As such, we will be adding a reservation to idle half the nodes in the partitions listed below. A reservation will cause nodes to drain as jobs complete and stop scheduling new jobs on those nodes if they cannot be completed before the outage. This will allow us to idle and power down those nodes prior to the work and avoid potential blackout/brownout on those racks.

    This will mean that these partitions will be up and available, but that half the nodes from each will be down (assuming an even number of nodes).

    This work is part of an on-going power capacity upgrade at MGHPCC. We expect this will be the last power work needed and the facility will then provide enough additional power for future expansion as well adding overhead for the current load.

    The affected partitions are:

    • arguelles_delgado

    • bigmem_intermediate

    • blackhole_gpu

    • eddy gershman

    • hejazi

    • hernquist

    • hoekstra

    • huce_ice

    • iaifi_gpu

    • iaifi_gpu_requeue

    • iaifi_priority

    • jshapiro

    • jshapiro_priority

    • kempner

    • kempner_requeue

    • kempner_h100

    • kempner_h100_priority

    • kempner_h100_priority2

    • kovac kozinsky

    • kozinsky_gpu

    • kozinsky_requeue

    • ortegahernandez_ice

    • rivas

    • seas_compute

    • seas_gpu

    • siag_combo

    • siag_gpu

    • sur

    • zhuang

thg 4 2025

Starfish login unavailable
Đã khắc phụcSự cố một phần6 giờ 10 phút
  • Đã khắc phụcUTC
    Starfish is once again accepting logins.
  • Đang điều traUTC

    Starfish is not allowing login due to a database issue. We have contacted the vendor and expect to resolve this soon.

Power Loss at MGHPCC
Đã khắc phụcSự cố lớn1 giờ 39 phút
  • Đã khắc phụcUTC

    Most nodes have been returned to service, both Cannon and FASSE are operable. Remaining down nodes will be remediated by FASRC staff over the next several days.

  • Đang điều traUTC

    At 1:05am EDT MGHPCC lost power to compute due to cooling failure. The chiller has been restored and power is back as of 5:43am. As a result of this all compute is powered off and all running jobs were requeued. FASRC staff is triaging the down hardware and will bring it back online when it is ready. Jobs will start once FASRC staff have confirmed that everything is in good state.

holylabs reboot to resolve instability
Đã khắc phụcHiệu suất giảm sút12 phút
  • Đã khắc phụcUTC

    holylabs is back up

  • Đã nhận diệnUTC
    holylabs must be rebooted to resolved a stability/funciton issue.
Login nodes temporarily down
Đã khắc phụcSự cố lớn19 phút
  • Đã khắc phụcUTC

    Cannon boslogin and FASSE login nodes are back up and operational.

    All holylogin nodes are still down for repair, please see our posted incident for more updates: https://status.rc.fas.harvard.edu/cm97gyay90013dturk7fxg5pb

    We apologize for the unexpected disruption.

  • Đang điều traUTC

    Due to a configuration error, all cluster login nodes are rebooting and are temporarily unavailable. Please save any work immediately.

holylogin[05-08] down
Đã khắc phục167 giờ 15 phút
  • Đã khắc phụcUTC
    Hardware has been repaired and holyoke login nodes are back online. Thanks for your patience.
  • Đang theo dõiUTC

    Holylogin chassis repair during maintenance was unsuccessful and replacement parts have been ordered.

    • holyoke login nodes (holylogin05-08) are down for hardware repair

    • Only Boston login nodes available (ie, boslogin[05-08])

    If you have holylogin hard-coded in your scripts, please update to login.rc.fas.harvard.edu or boslogin.rc.fas.harvard.edu for the time being, which will redirect you to an available login node.

    As always, the best method for obtaining a login node is usinglogin.rc.fas.harvard.edu which will pick a node for you.

    If you require a login node in a specific data center, use boslogin.rc.fas.harvard.edu (Boston) or (once they are back in service) holylogin.rc.fas.harvard.edu (Holyoke).

    See also: Command line access with Terminal (login nodes) – FASRC DOCS

  • Đã khắc phụcUTC

    This incident was posted by mistake.

    holylogin01-04 were replaced by holylogin05-08 some time back.

    As always, the best method for obtaining a login node is usinglogin.rc.fas.harvard.edu which will pick a node for you.

    Or if you require a login node in a specific data center, use boslogin.rc.fas.harvard.edu (Boston) or holylogin.rc.fas.harvard.edu (Holyoke).

    See also: Command line access with Terminal (login nodes) – FASRC DOCS

  • Đang điều traUTC

    Holylogin chassis repair during maintenance was unsuccessful and replacement parts have been ordered.

    Audience:

    • All cluster users

    Impact:

    • All holylogin** servers will be down till further notice

    • Only Boston login nodes available (ie, boslogin[05-08])

    If you have holylogin hard-coded in your scripts, please update to login.rc.fas.harvard.edu or boslogin.rc.fas.harvard.edu for the time being, which will redirect you to an available login node.

    Updates to follow as we have them.

thg 3 2025

seas_gpu partition occasional pre-emption issue
Đã khắc phục1224 giờ 12 phút
  • Đã khắc phụcUTC

    This issue has not reoccurred and appears to be due to unique cluster state. We are resolving this but will continue to monitor for recurrence.

  • Đã nhận diệnUTC
    We are investigating an issue with the seas_gpu queue where backfill jobs sometimes are not pre-empted. We have filed a bug with SchedMD and are monitoring.
THG 321
Starfish maintenance - Friday March 21, 2025 5pm-7pm
Hoàn thànhBảo trì2 giờ
  • Hoàn thànhtháng 03 21, 2025 tại 23:00UTC
    Maintenance has completed successfully
  • Đang tiến hànhtháng 03 21, 2025 tại 21:00UTC
    Maintenance is now in progress
  • Đã lên kế hoạchtháng 03 21, 2025 tại 21:00UTC

    Starfish will be under maintenance Friday March 21, 2025 5pm-7pm

    Starfish will be inaccessible during this period.

boslfs02 hanging issues - rebooting
Đã khắc phụcHiệu suất giảm sút18 phút
  • Đã khắc phụcUTC
    boslfs02 has been restarted
  • Đã nhận diệnUTC
    boslfs02 is experiencing issues. We are restarting the filesystem.
THG 313
Open OnDemand update March 12th 2025, 8:30AM-11:30AM
Hoàn thànhBảo trì3 giờ
  • Hoàn thànhtháng 03 13, 2025 tại 03:30UTC
    Maintenance has completed successfully
  • Đang tiến hànhtháng 03 13, 2025 tại 00:30UTC
    Maintenance is now in progress
  • Đã lên kế hoạchtháng 03 13, 2025 tại 00:30UTC

    OOD will be updated to include a Metrics page, reachable from the Help menu

    • Audience: Affects all production Open OnDemand nodes on FASSE and Cannon

    • Impact: There will be minimal impact to users. Some users may encounter an error on the dashboard. Simply refresh the page a few times and the error should go away.

    Logins and jobs on OOD should not be affected during this maintenance.

THG 33
FASRC monthly maintenance - Monday March 3rd, 2025 from 9am-1pm
Hoàn thànhBảo trì4 giờ
  • Cập nhậttháng 03 03, 2025 tại 18:00UTC
    Maintenance has completed successfully
  • Hoàn thànhtháng 03 03, 2025 tại 18:00UTC
    Maintenance has completed successfully
  • Đang tiến hànhtháng 03 03, 2025 tại 14:00UTC
    Maintenance is now in progress
  • Đã lên kế hoạchtháng 03 03, 2025 tại 14:00UTC

    PLEASE NOTE - New time window going forward - 9am-1pm

    FASRC monthly maintenance will take place Monday March 3rd, 2025 from 9am-1pm

    NOTICES

    • Training: Upcoming training from FASRC and other sources can be found on our Training Calendar. at https://www.rc.fas.harvard.edu/upcoming-training/

    • Status Page: You can subscribe to our status to receive notifications of maintenance, incidents, and their resolution at https://status.rc.fas.harvard.edu/ (click Get Updates for options).

    • Upcoming holidays: Memorial Day - Monday, May 26

    • You can subscribe to our status page using the Get Updates button in the upper right

    MAINTENANCE TASKS
    Cannon cluster will be paused during this maintenance?: YES
    FASSE cluster will be paused during this maintenance?: YES

    • Slurm Upgrade to 24.11.2 - Crucial Update

      • Audience: All cluster users

      • Impact: Jobs and the scheduler will be paused during this upgrade

    • Open Ondemand (OOD) reboots

      • Audience: All OOD users

      • Impact: All Open OnDemand (aka OOD/VDI/RCOOD) nodes will be rebooted

    • Login node reboots

      • Audience: Anyone logged into a FASRC Cannon or FASSE login node

      • Impact: Login nodes will rebooted during this maintenance window

    • bos-Isilon firmware updates

      • Audience: bos-isilon users

      • Impact: No noticeable impact for storage users

    • Netscratch retention/cleanup ( https://docs.rc.fas.harvard.edu/kb/policy-scratch/ )

      • Audience: Cluster users

      • Impact: Files older than 90 days will be removed. Please note that retention cleanup can and does run at any time, not just during the maintenance window.

    Thank you,
    FAS Research Computing
    https://www.rc.fas.harvard.edu/
    https://docs.rc.fas.harvard.edu/

Trước

thg 3 2025 đến thg 5 2025

Sau

For issues not shown here, please contact FASRC via
https://portal.rc.fas.harvard.edu or email rchelp@rc.fas.harvard.edu