Computing Systems Status

Scheduled Bouchet and Hopper Maintenance, September 15th 

We will perform scheduled maintenance on the Bouchet and Hopper clusters on Tuesday, September 15th.

This will not be a full downtime, but will instead have limited disruptions. The clusters and storage will remain online and available throughout the maintenance period. Compute node availability will be reduced during and for a few days after the maintenance, so users may experience temporary increases in wait times.

The login and Open OnDemand nodes for each cluster will be rebooted one at a time early in the morning of the maintenance. This may interrupt login sessions but should not impact running batch jobs.

We will send an email once maintenance is complete.

Bouchet critically low on storage and ways you can help 

Due to overwhelming demand for storage and significantly more interest than anticipated in Bouchet from new and current users, we are critically low on storage. As of July 31, 2026, Roberts (Bouchet’s storage system) is at 92% with only 274 TiB of available capacity. At the current rate of data growth, Roberts will be out of space in a matter of weeks. If Roberts runs out of free space, all jobs across the cluster will fail, and we will need to close the queues until more storage is available. We write to ask for your assistance.

As part of our efforts to deal with this situation, we have modified the retention period for files in /scratch effective immediately. Files that are older than 30 days (rather than the prior 60 days) will be automatically purged starting this Sunday, August 9, 2026.

Bouchet critical low storage details 

  • If you have access to another cluster, please hold off on any data transfers from that cluster to Bouchet and continue to use that cluster for your workflows until additional storage is available.
  • If you have data duplicated on another cluster and Bouchet, please delete the Bouchet copy until additional storage is available.
  • If you have any data on Bouchet that you do not immediately need for computation (in home, project, scratch or purchased storage), please consider copying it off the cluster (if applicable) and deleting it.
  • In general, please hold off on any data transfers onto Bouchet unless that data is immediately needed for computation.
  • We have added a cap on Scratch (as a whole) as it is the most elastic storage space on Bouchet. Only use scratch for temporary files and please delete those files as soon as they are no longer needed instead of waiting for the 30-day purge.
  • Let us know if people have left your group and their data are no longer needed and therefore can be deleted.

Thank you for your assistance. The YCRC staff are available if you need any help with data management. We are closely monitoring the situation and may need to take additional measures to temporarily constrain data growth.

Additional HPC Storage. We are purchasing several additional petabytes of HPC storage and hope it will be available this fall.

Alternative Storage Options. If you have any data in paid allocations on Roberts that is not immediately needed for computation and you are open to moving it to other (potentially cheaper) storage, please contact us, and we can assist with finding alternative storage options.

Thank you again for your assistance with this situation. If you have questions, comments, or concerns, please contact us at ycrc@yale.edu.

Scheduled Cluster Maintenance 


The next scheduled maintenance for Bouchet and Hopper will start on the morning of Tuesday, September 15th, and is expected to conclude by the end of day.

Previous System Status Updates

Latest updates:

Thursday, July 30, 9.45 am: All of the a1114 nodes suffering from performance degradation have rebooted and are fully operational again

Friday, July 24, 2.30 pm: The electrical work required for Bouchet expansion has been completed. 172 out of 180 Bouchet nodes affected by the expansion work are now back online and performing as normal. Eight nodes in the week partition are still operating at reduced performance. These nodes have been set to not accept new jobs, and each of these nodes will be rebooted once currently running jobs on that node finish.  As the nodes are rebooted, they will return to normal operation at full performance.  The last of these reboots should complete by the end of the day on July 29.

Description

We are working to increase overall compute capacity on Bouchet, which involves electrical work at the data center during the week of July 20-24, 2026. This work may impact the performance of Bouchet compute nodes, as it will temporarily reduce power feeds to the compute racks. Users should expect jobs to run more slowly than usual during the planned work. 

Standard job partitions will still be available, though certain compute nodes may be more affected than others.  Priority-Tier partitions will be disabled during the week.

Scope

180 Bouchet nodes will be impacted by planned data center work July 20-24:

The following will be impacted during the entire week:

  • 10 H200 nodes (gpu_h200, gpu_devel partitions) with names beginning with a1122, a1124, and a1126 
  • 12 RTX 5000 Ada nodes (gpu, gpu_devel, education_gpu partitions) with names beginning with a1128
  • 10 L40S (pi_co54 partition) nodes with names beginning with a1118
  • 100 non-GPU Intel nodes (day, bigmem, devel, education, mpi* partitions) with names beginning with a1130 and a1132

The following will be impacted only on Monday morning (July 20) and Wednesday afternoon (July 22):

  • 7 B200 nodes (gpu_b200 partition) with names beginning with a1116
  • 8 RTX Pro 6000 Blackwell nodes (gpu_rtx6000, gpu_devel partitions) with names beginning with a1112 
  • 33 non-GPU AMD (day, week, devel partitions) nodes with names beginning with a1114

Bouchet nodes outside the scope of this electric work: 60 nodes in the mpi partition and one B200 node in the gpu_devel partition. 

* Since MPI workflows can be sensitive to slow-performing nodes, we will temporarily move some affected nodes (20 non-GPU Intel nodes) from the mpi partition to the day partition to mitigate the impact of slower nodes.  The remaining 60 nodes in the mpi partition will be unaffected by the electrical work.

Performance Impact

We expect minor impact to the RTX Pro 6000 Blackwell, RTX 5000 Ada, and non-GPU AMD nodes during planned electrical work. We estimate a 2x slowdown for the L40S and non-GPU Intel nodes and H200 nodes.

We expect significant performance impact on the B200 nodes. However, the extent of performance degradation will depend on the specific workflow, ranging from minimal to significant. 

Once full power is restored to the B200 nodes, they will need to be restarted to restore normal performance. In preparation, we will create a scheduler reservation to drain all running jobs on the B200 nodes by the end of the work impacting them. During this time, the scheduler will not start jobs with requested run times that overlap with the reservation time. Once the nodes are restarted, they will resume normal operation.

Please adjust your job wall-times when submitting jobs as needed and use checkpointing when possible. Let us know if you would like assistance doing so.

The anticipated completion of scheduled work is end of day, Friday, July 24, 2026. YCRC will send an email communication when the work is completed. We are committed to keeping you as informed as possible at every stage of the work. Updates will be posted on this page. We recognize the reduced performance will impact your work, and we apologize for the inconvenience.  If you have questions, comments, or need assistance with your jobs, please contact us at ycrc@yale.edu.

Helpful tips and resources:
  • Adjust the wall-time with –time SLURM option. For more information, visit Job Scheduling with Slurm documentation page.

8/4/26 : One GPU node (r813u23n03, in commons partition) remains offline. Vendor is scheduled to be on site today to make the necessary repairs.

7/15/26; 4 pm: The faulty CDU has been repaired. 52 out of the 56 nodes affected by it are now back online.

The following four nodes remain offline and require further repair or replacement: 

  • r813u23n01 (a100 node in the commons partition)
  • r813u23n02 (pi_gerstein_gpu GPU node)
  • r813u29n03 (a regular node in the week partition)
  • r813u29n09 (a regular node in the week partition)

The issues with these nodes have been escalated with the vendor.

Please check the status of your jobs and resubmit any jobs that failed due to the outage.  Please contact us if you encounter any issues.

7/15 10 am: The CDU has gone through the overnight testing and test water samples are being collected for analysis to confirm it is functioning as expected. Number of nodes have been powered on and are undergoing testing with the goal of bringing the majority of affected equipment back online in the next several hours.

7/14/26 6 pm: Additional issues discovered during repairs to the faulty CDU required installation of replacement parts. CDU currently in testing. The timeline for completion and bringing nodes back online has been extended to Wednesday, 7/15.

7/14/26 9am: Vendors are on site and continuing with the repair work, as planned. 

7/13/26 6.45 pm: Repairs to the faulty Cooling Distribution Unit (CDU) on McCleary are in progress. 

7/13/26 9am: The vendor has arrived at the West Campus and begun the work to repair the faulty Cooling Distribution Unit (CDU) on McCleary.

7/9/2026:  Reduced capacity on McCleary.  Due to a cooling issue in one of the McCleary racks at West Campus, McCleary nodes that have names beginning with r813 have again shut down. This outage is caused by a faulty Cooling Distribution Unit (CDU), causing equipment in the rack to overheat and shut down. We are working with the CDU vendor to resolve the issue as quickly as possible.

This issue impacts the following McCleary scheduler partitions:

  • 26 of the 33 nodes in the day partition
  • 14 of the 16 nodes in the week partition
  • 3 of the 20 nodes in the gpu partition
  • 1 of the 3 nodes in the pi_gerstein_gpu partition
  • 2 of the 10 nodes in the pi_jetz partition
  • 4 of the 4 nodes in the pi_ohern partition
  • 2 of the 2 nodes in the pi_sestan partition
  • 4 of the 4 nodes in the pi_tsang partition

Jobs running on the affected nodes have been terminated due to the failure. Please check the status of your jobs.

Slurm jobs can be submitted as usual; however, any jobs that would start on the affected nodes won’t start until after the cooling issue is resolved.  

We realize this is an impact to your work and apologize for the inconvenience.  If you have any questions, comments, or concerns, please contact us at research.computing@yale.edu.

8/13/2026:  A power failure occurred at West Campus between 1:00 AM and 1:30 AM.  This impacted the Grace, McCleary, Misha, and Milgram clusters.  Compute nodes on these clusters rebooted, which affected running jobs. Please check the status of your jobs. YCRC is checking status and recovering nodes as needed. Cluster capacity may be reduced in the meantime.

6/15/2026 - 6/18/2026:  Scheduled all-cluster maintenance complete.

6/4/2026 13:00: Hopper experienced a system issue that impacted availability. We engaged the storage vendor to resolve the issue. Hopper availability was restored by 15:30.

5/11/2026:  There was a power drop at the West Campus Data Center on the morning of 5/11/2026.  This impacted running jobs on McCleary, Grace, Misha and Milgram.  Please check your jobs to see how they were impacted.

2/13/2026: Bouchet scheduler issues have been resolved. All systems are operational.