EDIT: As of 1pm, this issue should be resolved. Users should now be able to request new 2 factor tokens.
Users are unable to obtain new OpenAuth tokens (software.rc.fas.harvard.edu/oa)
We are working to resolve this issue as quickly as possible. NO ETA currently.
3rd December 2019
Winter Maintenance Dec. 3rd, 2019 (all day), scheduled 1 day ago
Date: December 3rd, 2019
Start Time: 7:00AM
End Time: 5:00PM estimated
This is an all day maintenance event and involves all data centers.
FASRC's Winter maintenance will take place on Tuesday December 3rd, 2019 starting at 7am with an end ETA of 5pm. This maintenance downtime is an all day maintenance event and involves all data centers and nearly every service. Please plan accordingly.
All running jobs will be terminated by 7am start of maintenance.
Any pending jobs will remain pending until the scheduler resumes at end of maintenance.
Home directories and login servers will be unavailable all day.
The help ticket system will be unavailable all day.
List of maintenance tasks:
User home directories - We will cut over to a new home directory server. This will be transparent to users once complete, but will require several hours for final sync.
Login nodes will be offline due to home directories mounts moving (home directories are necessary for login)
Authentification, especially OpenAuth tokens, will be periodically interrupted
ALL custromer/lab VMs will be rebooted
SLURM and Slurm master server upgrade - Scheduler will be offline until all work is completed.
Lustre filesystem upgrades - Affects numerous LFS storage shares during the day
Add $SCRATCH global variable - See details at https://www.rc.fas.harvard.edu/policy-scratch/
HDR Infiniband updates - Affects numerous network connections in data centers
Authentication and Domain Controller updates and changes
Upgrade seasfs01, 02, 03 servers and upgrade networking on SeasDGX
seas_dgx1 partition max timelimit reduced from 3days to 1day - Use the gpu partition for longer jobs
new Module search added to Portal
Ticket system database move - Ticket system (rchelp@rc/Portal) will be unavailable most of the day