FAS Research Computing - Hinweisverlauf

Bei Globus Data Transfer ist ein schwerer Ausfall aufgetreten

Status page for the Harvard FAS Research Computing cluster and other resources.

Cluster Utilization (VPN and FASRC login required): Cannon | FASSE


Please scroll down to see details on any Incidents or maintenance notices.
Monthly maintenance occurs on the first Monday of the month (except holidays).

GETTING HELP
Documentation: https://docs.rc.fas.harvard.edu | Account Portal https://portal.rc.fas.harvard.edu
Email: rchelp@rc.fas.harvard.edu | Support Hours

The colors shown in the bars below were chosen to increase visibility for color-blind visitors.
For higher contrast, switch to light mode at the bottom of this page if the background is dark and colors are muted.
Betriebsbereit

SLURM Scheduler - Cannon - Betriebsbereit

Cannon Compute Cluster (Holyoke) - Betriebsbereit

Boston Compute Nodes - Betriebsbereit

GPU nodes (Holyoke) - Betriebsbereit

seas_compute - Betriebsbereit

Betriebsbereit

SLURM Scheduler - FASSE - Betriebsbereit

FASSE Compute Cluster (Holyoke) - Betriebsbereit

Betriebsbereit

Kempner Cluster CPU - Betriebsbereit

Kempner Cluster GPU - Betriebsbereit

Betriebsbereit

FASSE login nodes - Betriebsbereit

Betriebsbereit

Cannon Open OnDemand - Betriebsbereit

FASSE Open OnDemand - Betriebsbereit

Schwerer Ausfall

Netscratch (Global Scratch) - Betriebsbereit

Home Directory Storage - Boston - Betriebsbereit

Tape - (Tier 3) - Betriebsbereit

Holylabs - Betriebsbereit

Isilon Storage Holyoke (Tier 1) - Betriebsbereit

Holystore01 (Tier 0) - Betriebsbereit

HolyLFS04 (Tier 0) - Betriebsbereit

HolyLFS05 (Tier 0) - Betriebsbereit

HolyLFS06 (Tier 0) - Betriebsbereit

Holyoke Tier 2 NFS - Betriebsbereit

Holyoke Specialty Storage - Betriebsbereit

holECS - Betriebsbereit

Isilon Storage Boston (Tier 1) - Betriebsbereit

BosLFS02 (Tier 0) - Betriebsbereit

Boston Tier 2 NFS - Betriebsbereit

CEPH Storage Boston (Tier 2) - Betriebsbereit

Boston Specialty Storage - Betriebsbereit

bosECS - Betriebsbereit

Samba Cluster - Betriebsbereit

Globus Data Transfer - Schwerer Ausfall

Hinweisverlauf

Aktuellen Status anzeigen

Nov. 2025

Coldfront outage
GelöstSchwerer Ausfall1 Stunde 11 Minuten
  • GelöstUTC

    This alert was a false alarm. Coldfront is operating normally.

  • AnalyseUTC
    Coldfront cannot be accessed at the moment. This incident was created by an automated monitoring service.
Authentication outage
GelöstSchwerer Ausfall1 Minute
  • GelöstUTC

    This test incident has been resolved.

  • AnalyseUTC

    Authentication issues with openauth/radius. This incident was created by an automated monitoring service.

NOV18
Slurm scheduler unplanned maintenance
GelöstWartung6 Minuten
  • GelöstUTC
    The Slurm scheduler is back online.
  • IdentifizierungUTC
    The Slurm scheduler is experiencing a problem and we need to take it down briefly to clear this up. Should be short duration, but no ETA yet. This will not affect running jobs but will mean you cannot talk to the scheduler or submit new jobs.
NOV14
Starfish dashboard maintenance Nov. 14th 5-6PM
AbgeschlossenWartung1 Stunde
  • AbgeschlossenNovember 14, 2025 um 23:00UTC
    Maintenance has completed successfully
  • In BearbeitungNovember 14, 2025 um 22:00UTC
    Maintenance is now in progress
  • GeplantNovember 14, 2025 um 22:00UTC

    There is a planned upgrade of the Starfish dashboard scheduled for Friday November 14th starting at 5PM.
    The dashboard will be down for an hour while the upgrade is performed.

Slurm scheduler issues - WIP
GelöstBeeinträchtigte Leistung15 Minuten
  • GelöstUTC

    The Slurm scheduler is back up and new jobs can be scheduled at this time.

    This incident has been resolved.

  • IdentifizierungUTC
    The Slurm job scheduler will be offline temporarily while we resolve some minor issues. Jobs will continue to run but you will be unable to access the scheduler or submit new jobs until we return it to service.

Okt. 2025

Holystore01 is degraded.
Gelöst32 Minuten
  • GelöstUTC
    This incident has been resolved.
  • AnalyseUTC

    We are currently investigating an issue with holystore01. Service may be degraded. Updates will be posted here as we identify the root cause.

License servers down
GelöstTeilausfall49 Minuten
  • GelöstUTC

    The license server is responding normally again.

  • UpdateUTC
    We are currently investigating this incident.
  • AnalyseUTC

    rclic is currently down. Software licenses may not be accessible, particularly for the following:

    • Matlab

    • Gurobi

    • Mathematica

    We are currently investigating this incident.

OKT6
FASRC monthly maintenance Monday October 6th, 2025 9am-1pm
AbgeschlossenWartung4 Stunden
  • AbgeschlossenOktober 06, 2025 um 17:00UTC
    Maintenance is now in progress
  • In BearbeitungOktober 06, 2025 um 13:00UTC
    Maintenance is now in progress
  • GeplantOktober 06, 2025 um 13:00UTC

    FASRC monthly maintenance will take place Monday October 6th, 2025 from 9am-1pm

    NOTICES

    MAINTENANCE TASKS
    Cannon cluster will be paused during this maintenance?: NO
    FASSE cluster will be paused during this maintenance?: NO

    • DNS server reboots

      • Audience: All FASRC services

      • Impact: Rolling reboot should have no impact

    • Login node reboots

      • Audience: Anyone logged into a FASRC Cannon or FASSE login node

      • Impact: All login nodes will rebooted during this maintenance window

    • Netscratch cleanup ( https://docs.rc.fas.harvard.edu/kb/policy-scratch/ )

      • Audience: Cluster users

      • Impact: Files older than 90 days will be removed. Please note that retention cleanup can and does run at any time, not just during the maintenance window.

    Thank you,
    FAS Research Computing
    https://docs.rc.fas.harvard.edu/
    https://www.rc.fas.harvard.edu/

OKT3
OOD, FASRC authentication down
GelöstWartung20 Minuten
  • GelöstUTC

    This incident has been resolved and login to FASRC sites and services, including OOD and Spinal, is restored.

  • IdentifizierungUTC

    Some FASRC webpages and services (such as OOD) that require authentication may not load.

    We have identified the cause and are working on a solution

Sep. 2025

SEP30
VPN concentrator rolling updates overnight
AbgeschlossenWartung
  • UpdateSeptember 30, 2025 um 07:00UTC
    Maintenance has completed successfully
  • AbgeschlossenSeptember 30, 2025 um 06:00UTC

    Networking will be patching the VPN concentrators overnight. This will be done in a rolling order so that one ore more are always online.

    This may cause active VPN connections to drop, but they can be re-connected shortly after. ETA is one hour total.

  • GeplantSeptember 30, 2025 um 06:00UTC

    Networking will be patching the VPN concentrators overnight. This will be done in a rolling order so that one ore more are always online.

    This may cause active VPN connections to drop, but they can be re-connected shortly after. ETA is one hour total.

holylfs05 changelog errors
GelöstSchwerer Ausfall3 Stunden 4 Minuten
  • GelöstUTC

    The root cause was found and a fix has been implemented.

  • AnalyseUTC

    The changelog (which tracks changes on the filesystem) on holylfs05 is throwing errors creating instability on the filesystem. We are investigating.

gpu_test partition down
GelöstTeilausfall2 Stunden 18 Minuten
  • GelöstUTC

    The gpu_test partition is back to normal service.

    This incident has been resolved.

  • AnalyseUTC

    The gpu_test partition is currently down. Please use our other public gpu partitions in the meanwhile.

    We are currently investigating this incident. incident.

gpu_test down
GelöstTeilausfall1 Stunde 52 Minuten
  • GelöstUTC

    The gpu_test partition is back in normal service.

  • AnalyseUTC

    The gpu_test partition is currently down due to a networking issue in that row. Please use our other public gpu partitions in the meanwhile.

    We are currently investigating this incident.

holylfs05 issues
GelöstSchwerer Ausfall64 Stunden 38 Minuten
  • GelöstUTC

    holylfs05 is back online. Thank you for your patience.

  • UpdateUTC

    holylfs05 is undergoing filesystem checks. Due to the size of the system, this process will take the rest of the weekend at a minimum.

    No ETA at this time.

    We are continuing to work on a fix for this incident. Our sincere apologies for the unexpected disruption.

  • IdentifizierungUTC

    holylfs05 will be inaccessible for an extended period of time while our staff continues to troubleshoot the underlying cause.

    We are continuing to work on a fix for this incident.

  • AnalyseUTC

    Writing to holylfs05 may result in errors.

    We are currently investigating this incident.

Zurück

Sep. 2025 zu Nov. 2025

Nächste

For issues not shown here, please contact FASRC via
https://portal.rc.fas.harvard.edu or email rchelp@rc.fas.harvard.edu