FAS Research Computing - Intermittent node failure – Incident details

All systems operational

Status page for the Harvard FAS Research Computing cluster and other resources.

Cluster Utilization (VPN and FASRC login required): Cannon | FASSE


Please scroll down to see details on any Incidents or maintenance notices.
Monthly maintenance occurs on the first Monday of the month (except holidays).

GETTING HELP
https://docs.rc.fas.harvard.edu | https://portal.rc.fas.harvard.edu | Email: rchelp@rc.fas.harvard.edu

The colors shown in the bars below were chosen to increase visibility for color-blind visitors.
For higher contrast, switch to light mode at the bottom of this page if the background is dark and colors are muted.

Intermittent node failure

Resolved
Operational
Started about 3 years agoLasted 17 days

Affected

Cannon Cluster

Operational from 3:56 PM to 2:16 PM

Cannon Compute Cluster (Holyoke)

Operational from 3:56 PM to 2:16 PM

Updates
  • Resolved
    Resolved
  • Monitoring
    Monitoring

    The occurrence of node failures has subsided. We believe it may have been caused by the removal of too many end-of-life compute nodes at once. We are still working with SchedMD to prevent this from happening again in the future, but the risk of node failure has passed.

  • Investigating
    Investigating

    Due to the following Slurm bug, compute nodes on the Cannon cluster have been noticed to fail seemingly randomly. If this occurs on a node where your job is running, your job will be requeued on a different node. We are working with SchedMD (the maintainers of the Slurm job scheduler) to resolve this issue.

    https://bugs.schedmd.com/show_bug.cgi?id=13805

For issues not shown here, please contact FASRC via
https://portal.rc.fas.harvard.edu or email rchelp@rc.fas.harvard.edu