We are seeing continued performance issues with Regal after the long weekend. This is related to one of the OST (Object Storage Targets) nodes reaching a stuck state. We are actively trying to identify the cause and correcting this, likely the OST node will need to restarted which may cause further performance issues with Regal during this time. Likely this should be done around 12pm or 1pm today, barring any unexpected issues. No data loss is expected from this. We will also continue to update this page as progress is made.
Thank you, FASRC
Update @ 12:16pm: The engineer working with Regal has been able to restore functionality. Performance should be returning to normal. We have opened all the nodes that were previously closed to Regal. Additionally we are also running a retention of regal, this should help get the usage down that helps alleviate further performance issues (use 95% and over causes issues).
Update @ 2:57pm: Since this morning Regal has crept up past 95%, this is causing ongoing performance issues for the time being. We are currently running retention to free up more space. The retention should finish some time this evening or possibly tomorrow morning. Feel free to check back here for status updates.
Update @ 9/5 12:00pm: Retention finished overnight and Regal seems to be returning to a more normal state. We will continue to watch usage and report back.