UPDATE 5:55pm: This was a case of an errant set of jobs overwhelming the filesystem. The jobs have been identified and stopped. Regal is back to normal.
Regal scratch is experiencing performance issues. We are actively working on the problem.
Additional information about future of Regal:
The Regal scratch filesystem has served us venerably since 2013, processing billions of jobs and numerous petabytes of data. But as recent issues involving object identifier errors and aging hardware have shown, it is time to retire it and implement a new fast scratch system. To that end we are in the planning stages of a Regal replacement for the coming months. We are taking knowledge gained from Regal and from other technologies that we've since had the opportunity to evaluate and will be designing, and later building, a replacement. We thank you for your patience and want to assure you there is light at the end of the tunnel . We look forward to presenting the next step in fast scratch storage to replace this venerable but aging filesystem as soon as possible.