FAS Research Computing - holylfs02 performance issues – Incident details

The unrepairable volume on holylfs02 is isolated to two labs and they have been informed of next steps. This issue does not affect other areas on holylfs02, so we are closing this incident.

Recently we noticed an uptick in bad blocks on a RAID6 disk volume that is part of the entire filesystem. Generally speaking, the operating system will vector these out so no data is written there. During this period, we had to replace a number of drives due to failures; they are part of RAID 6 multi-disk set with dual parity and will rebuild with minimal impact on performance.

What we believe happened is that, during the rebuild process, bad data was copied to the replacement disks and the filesystem got corrupted. One of the staff ran a read-only, non-destructive repair on the volume in question and noted quite a few errors.

So far, we have 1) contacted the vendor (who gave us a command to clear additional bad blocks)
2) NOT run the actual repair command to "fix" the filesystem (which would delete data) 3) contacted the vendor to see if they had any partners that might be able to assist. We are planning to meet with one of those partners early next week, but are not confident they will have a solution. At this point (and depending on how that meeting goes), the next step would be to run the repair, note the extent of loss, and attempt to get the volume remounted.

Updates to follow when we have more information.

Note that these shelves are covered under warranty support for another year. We see no need at present to replace the 3PB allocation with new hardware (this is planned to take place Q3-4 FY23)