System Updates Archive

As of today, September 14, the HPC team has completed restoration of files to /home from tape backup and has recovered a significant number of files (under 1 MB in size) from /scratch. The team has successfully run confidence tests and benchmarks on both the storage system and the InfiniBand network, with entirely satisfactory results. However, we are continuing to seek additional assurances from the Lustre file system development team and our vendors to confirm that we have taken all reasonable steps to ensure that the storage system is in the best possible condition.  At this point, unless the Lustre developers or our vendors inform us of additional recommended actions, we plan to return Omega to service by tomorrow afternoon. At that time, we will provide users with both detailed information about files affected by the storage corruption, and guidance regarding the potential for recovery of additional files that may be of particular importance. 

We are keenly aware of the impact that this disruption may have on your research, and we sincerely apologize for the inconvenience. We will provide all possible assistance as you recover from this unfortunate event.  Please email hpc@yale.edu to request help, or with any questions, concerns or comments.

Please refer to this page for the latest updates and please note that logins will be disabled until Omega is brought back into service.

As of today, September 11, the team is continuing to restore files to /home from tape backup, and to recover as many small files (under 1 MB in size) as possible from /scratch.  We will continue these processes until Sunday, September 13, when we will suspend them (even if they are incomplete) and begin file system confidence tests, so that we can bring Omega back into normal service Monday morning.   We will continue to restore files from the /home backup tapes next week, if necessary, even after Omega is back in service. (These files will be placed in a separate disk area to avoid disruption.) We may also be able to recover additional corrupted files from /scratch (including some files larger than 1 MB) after Omega returns to normal service, but we will need to coordinate with individual principal investigators to do so.

We believe that the confidence tests on Omega will proceed without issue on Sunday, but their successful completion will be necessary before we can return Omega to normal service.  When the tests are complete, we will send out a communication indicating Omega’s status and readiness for service Monday morning.  At that time, we will provide users with lists of files that were damaged, including the status of each one (i.e., recovered, potentially recoverable, or lost).  Logins will continue to remain disabled until Omega is returned to normal service.

We are deeply sympathetic to the impact of this issue on your research, and we sincerely apologize for the inconvenience. Once Omega is back in normal operation, we will provide any assistance we can to help you recover from this unfortunate disruption.

To check for the latest updates, please see that status page on the Yale Center for Research Computing website.  If you have any questions, concerns or comments please contact us at hpc@yale.edu

During file system maintenance on Omega last week, the HPC team discovered file corruption on both the home and scratch partitions. We anticipate that the damaged files on the home partition will be restored from backup tapes, so the HPC team and the hardware vendor have focused their efforts on recovering as many scratch files as possible. We believe that we can recover a significant number of files, but there will still be data loss. To date, the diligent efforts of the HPC team have recovered roughly 25% of the corrupt files, and we believe that it will be fruitful to continue the recovery effort throughout the holiday weekend. 

Our current plan is to start a number of confidence tests on Tuesday to ensure that the file system is healthy. After the tests are complete, we will then bring Omega back online for regular usage.  Since we do not know exactly how long the recovery and testing processes will take, we are not able to provide a firm estimate of when Omega will be returned to regular service.  Our aim is to recover as much data as reasonably possible and to ensure that future usage will be entirely safe.  When we return Omega to service, we will provide users with lists of files that were damaged and recovered, as well as lists of files that we believe were lost.  Please note that until Omega is returned to service, all logins will be disabled.

We are very mindful of the serious impact this may have on your research, and we apologize for any inconvenience this issue has caused.

We will be updating this status page with current information, so please check for updates.  As always, if you have any questions, concerns or comments, please email hpc@yale.edu.

An extended maintenance window is scheduled for the Omega Cluster, which includes both the Geology and Geophysics (GEO) and High Energy Physics (HEP) clusters.  Maintenance will begin starting at 8 am on Monday, August 24th.  All systems will be available for use again by the morning of Monday, August 31st.