holylfs06 has been experiencing ongoing performance and latency issues. This has impacted workflows and made it difficult for jobs to effectively complete. Our engineers have been troubleshooting the underlying causes, and identified that the general load on holylfs06 is greater than what the filesystem can process and deliver. It is not the misuse of any particular job or group, but rather the combination of all jobs and groups. In a shared filesystem, the high load ends up impacting all users and leading to a unresponsive server.
A more technical explanation: Jobs are reading very large volumes of data, keeping the storage arrays continuously busy with big requests. The ldiskfs journal has to make a small synchronous write every time an object is created or destroyed, and it sits on the same disks as the data. A journal bound operation that should take ~1ms is taking ~1s, with some taking over 12s. OSTs can't open new transactions until the journal commits, so service threads pile up meaning hundreds are stuck in uninterruptible wait, some for 15+ minutes. The node comes up as unresponsive, and times out. This results in the slow/stuck filesystem that many of you have experienced.
In order to mitigate this, we have been resetting holylfs06 servers and rebooting when possible, but this is not a sustainable solution. We are asking all groups, particularly those with large sequential reads and small-file workloads to shift their workflow to netscratch if possible.
A visual flowchart of an optimal workflow is depicted here: https://docs.rc.fas.harvard.edu/kb/data-storage-workflow-rdm/#Data_Storage_Workflow
We also have recommendations on job efficiency and best practices to be kinder to fileystems here: https://docs.rc.fas.harvard.edu/kb/job-efficiency-and-optimization-best-practices/
A long term solution will be the upcoming Compute Storage which uses a different hardware (nvme) than holylfs06 (Lustre). We expect to migrate your holylfs06 data to Compute Storage in the coming months, and will send out additional communication at that time.
Please reach out to us at rchelp@rc.fas.harvard.edu if your group needs additional help adjusting your workflow.
Thank you again for your understanding.
