FAS Research Computing - holylabs - New data missing – Incident details

Holylabs experiencing degraded performance

Status page for the Harvard FAS Research Computing cluster and other resources.

Cluster Utilization (VPN and FASRC login required): Cannon | FASSE


Please scroll down to see details on any Incidents or maintenance notices.
Monthly maintenance occurs on the first Monday of the month (except holidays).

GETTING HELP
https://docs.rc.fas.harvard.edu | https://portal.rc.fas.harvard.edu | Email: rchelp@rc.fas.harvard.edu

The colors shown in the bars below were chosen to increase visibility for color-blind visitors.
For higher contrast, switch to light mode at the bottom of this page if the background is dark and colors are muted.

holylabs - New data missing

Identified
Degraded performance
Started 7 days ago

Affected

Storage

Degraded performance from 2:07 PM to 12:00 AM

Holylabs

Degraded performance from 2:07 PM to 12:00 AM

Updates
  • Identified
    Identified

    While attempting to correct the over-quota/extra data issue on holylabs, an error in the sync command caused the deletion of newly created files since the re-open of the cluster (6/5/25 9AM) for 54 lab directories. We see no evidence that any other lab directories were affected.


    Due to the large nature of the original cleanup and the error being discovered after the fact, regretfully these deleted files cannot be recovered.

    A list follows of affected /n/holylabs lab directories. If your lab is not on that list, then it is not identified as being affected but this error:
    acc_lab
    alvarez_lab
    avillar_lab
    barnett_lab
    bertoldi_lab
    bol_lab
    brenner_lab
    cgolden_lab
    charbonneau_lab
    charrison_lab
    chetty_lab
    cnelya_lab
    dam_lab
    doshi-velez_lab
    eisenstein_lab
    enos_lab
    eps_preceptors
    glassman_lab
    hanson_lab
    hekstra_lab
    holbrook_lab
    iaifi_lab
    idreos_lab
    iebecker_lab
    imai_lab
    jacobsen_lab
    jialiu_lab
    junweil_lab
    kaxiras_lab
    kdbrantley_lab
    kempner_dev
    king_lab
    kiyoul_lab
    konkle_lab
    koumoutsakos_lab
    kozinsky_lab
    kramer_lab
    maustern_lab
    nliu_lab
    pallais_lab
    park_lab
    pierce_lab
    protopapas_lab
    pslade_lab
    shro_lab
    sitanc_lab
    smousavih_lab
    sneel_lab
    snyder_lab
    sompolinsky_lab
    tamano_lab
    ylei_lab
    zickler_lab

  • Investigating
    Investigating

    We are currently investigating an issue on holylabs where some labs have noticed newly created files are missing.

    We will update this incident with more info as soon as possible.

For issues not shown here, please contact FASRC via
https://portal.rc.fas.harvard.edu or email rchelp@rc.fas.harvard.edu