FAS Research Computing - Power Loss at MGHPCC – Incident details

All systems operational

Status page for the Harvard FAS Research Computing cluster and other resources.

Cluster Utilization (VPN and FASRC login required): Cannon | FASSE


Please scroll down to see details on any Incidents or maintenance notices.
Monthly maintenance occurs on the first Monday of the month (except holidays).

GETTING HELP
Documentation: https://docs.rc.fas.harvard.edu | Account Portal https://portal.rc.fas.harvard.edu
Email: rchelp@rc.fas.harvard.edu | Support Hours

The colors shown in the bars below were chosen to increase visibility for color-blind visitors.
For higher contrast, switch to light mode at the bottom of this page if the background is dark and colors are muted.

Power Loss at MGHPCC

Resolved
Major outage
Started about 1 year agoLasted about 2 hours

Affected

Cannon Cluster

Operational from 11:02 AM to 12:41 PM

SLURM Scheduler - Cannon

Operational from 11:02 AM to 12:41 PM

Cannon Compute Cluster (Holyoke)

Operational from 11:02 AM to 12:41 PM

Boston Compute Nodes

Operational from 11:02 AM to 12:41 PM

GPU nodes (Holyoke)

Operational from 11:02 AM to 12:41 PM

seas_compute

Operational from 11:02 AM to 12:41 PM

Updates
  • Resolved
    UTC
    Resolved

    Most nodes have been returned to service, both Cannon and FASSE are operable. Remaining down nodes will be remediated by FASRC staff over the next several days.

  • Investigating
    UTC
    Investigating

    At 1:05am EDT MGHPCC lost power to compute due to cooling failure. The chiller has been restored and power is back as of 5:43am. As a result of this all compute is powered off and all running jobs were requeued. FASRC staff is triaging the down hardware and will bring it back online when it is ready. Jobs will start once FASRC staff have confirmed that everything is in good state.

For issues not shown here, please contact FASRC via
https://portal.rc.fas.harvard.edu or email rchelp@rc.fas.harvard.edu