FAS Research Computing - gpu_test queue slow - note on use – Детали инцидента

This issue has become one of supply and availability. As such, we are changing this issue to Maintenance as there is nothing further we can do until the vendor replaces the hardware. Please continue to use other GPU resources. We will notify the community once we have new hardware to replenish the gpu_test queue.

Please see Running Jobs for other available resources.

Difficulty obtaining replacement mainboards has this issue stalled. More information when we have it.

Correction to first part of previous update:

Currently only 2 of the nodes in gpu_test are working and the single remoteviz node is down. These will require help from the vendor to revive. We are also investigating with them the root cause for multiple node failures. No ETA

Currently only 2 of the nodes in gpu_test are working and the single remoteviz node is down. These will require help from the vendor to revive. We are also investigating with them the root cause for multiple node failures. No ETA

As a result of this, we are pre-emptively modifying the QoS for gpu_test now. We had planned to make a change on Thursday after announcing it in tomorrow's maintenance email, but this forces our hand.

QoS changes for gpu_test effective now: Limited to 1 job per user. That job is limited to a maximum 16 cores and 90GB memory.

See Running Jobs for a list of available partitions, including the gpu partition.

Please note that this issue also affects the remoteviz partition.

Updates as we have them.