FAS Research Computing - Slurm responding slowly - WIP – Incident details

All systems operational

Status page for the Harvard FAS Research Computing cluster and other resources.

Cluster Utilization (VPN and FASRC login required): Cannon | FASSE


Please scroll down to see details on any Incidents or maintenance notices.
Monthly maintenance occurs on the first Monday of the month (except holidays).

GETTING HELP
https://docs.rc.fas.harvard.edu | https://portal.rc.fas.harvard.edu | Email: rchelp@rc.fas.harvard.edu

The colors shown in the bars below were chosen to increase visibility for color-blind visitors.
For higher contrast, switch to light mode at the bottom of this page if the background is dark and colors are muted.

Slurm responding slowly - WIP

Resolved
Degraded performance
Started about 1 month agoLasted 7 days

Affected

Cannon Cluster

Operational from 3:06 PM to 3:16 PM, Degraded performance from 3:16 PM to 4:30 PM

SLURM Scheduler - Cannon

Operational from 3:06 PM to 3:16 PM, Degraded performance from 3:16 PM to 4:30 PM

Cannon Compute Cluster (Holyoke)

Operational from 3:06 PM to 3:16 PM, Degraded performance from 3:16 PM to 4:30 PM

Boston Compute Nodes

Operational from 3:06 PM to 3:16 PM, Degraded performance from 3:16 PM to 4:30 PM

GPU nodes (Holyoke)

Operational from 3:06 PM to 3:16 PM, Degraded performance from 3:16 PM to 4:30 PM

seas_compute

Operational from 3:06 PM to 3:16 PM, Degraded performance from 3:16 PM to 4:30 PM

Updates
  • Resolved
    Resolved

    After another patch of the scheduler, Slurm should be much more stable now

  • Update
    Update

    We are still investigating this incident, we are working with SchedMD to find a solution.

  • Investigating
    Investigating
    We are currently investigating this incident.
  • Identified
    Identified
    We are continuing to work on a fix for this incident.
  • Investigating
    Investigating
    We are currently investigating this incident.
For issues not shown here, please contact FASRC via
https://portal.rc.fas.harvard.edu or email rchelp@rc.fas.harvard.edu