Computing Systems Status

Friday, 10.2.26, 1 pm: Electrical work on Bouchet is complete. All 216 affected Bouchet nodes, including the H200, RTX 5000 Ada, A5000, L40S (pi_co54), and non-GPU Intel nodes, have returned to normal power and performance. You should no longer see the slowdowns that were expected during the planned work.

Priority-tier partitions (priority_gpu*, priority_mpi*) have returned to full capacity, and the affected nodes are again accepting priority-tier jobs.

Monday, 9.28.26, 10.45 am: Electrical work on Bouchet started at 8 am this morning and is progressing as planned. 

Bouchet is critically low on storage and ways you can help  

Due to overwhelming demand for storage and significantly more interest than anticipated in Bouchet from new and current users, we are critically low on storage. As of early August, Roberts (Bouchet’s storage system) is at 92% with only 274 TiB of available capacity. If Roberts runs out of free space, all jobs across the cluster will fail, and we will need to close the queues until more storage is available. 

Ways you can help

If you have access to another cluster, please hold off on any data transfers from that cluster to Bouchet and continue to use that cluster for your workflows until additional storage is available.

If you have data duplicated on another cluster and Bouchet, please delete the Bouchet copy until additional storage is available.

If you have any data on Bouchet that you do not immediately need for computation (in home, project, scratch or purchased storage), please consider copying it off the cluster (if applicable) and deleting it.

In general, please hold off on any data transfers onto Bouchet unless that data is immediately needed for computation.

We have added a cap on Scratch (as a whole) as it is the most elastic storage space on Bouchet. Only use scratch for temporary files and please delete those files as soon as they are no longer needed instead of waiting for the 60-day purge.

Let us know if people have left your group and their data are no longer needed and therefore can be deleted.

Thank you for your assistance. The YCRC staff are available if you need any help with data management. We are closely monitoring the situation and may need to take additional measures to temporarily constrain data growth.

Solution

Additional HPC Storage. We are purchasing several additional petabytes of HPC storage and hope it will be available this fall.

Alternative Storage Options. If you have any data in paid allocations on Roberts that is not immediately needed for computation and you are open to moving it to other (potentially cheaper) storage, please contact us, and we can assist with finding alternative storage options.

Please contact us at ycrc@yale.edu with any questions. 

Scheduled Cluster Maintenance 


The scheduled maintenance on Bouchet and Hopper on Tuesday, September 15th was completed. 
Security updates have been applied to the operating system and related packages to ensure continued compliance with NIST 800-171 for Hopper and with the Yale Minimum Security Standards for Bouchet. No changes have been made to the apps tree.

Previous System Status Updates

Bouchet expansion: Electrical work affecting node performance, September 28-October 2, 2026

We are continuing our efforts to increase overall compute capacity on Bouchet. This requires electrical work at the data center during the week of September 28-October 2, 2026. This work may impact the performance of Bouchet compute nodes, as it will temporarily reduce power feeds to certain compute racks. Users should expect some jobs to run more slowly than usual during the planned work.

Standard job partitions will still be available, though certain compute nodes may be more affected than others. 

Scope

The following 216 Bouchet nodes will be impacted by planned data center work September 28-October 2:

  • All 10 H200 nodes (gpu_h200, gpu_devel, priority_gpu*)
  • All 12 RTX 5000 Ada nodes (gpu, gpu_devel, priority_gpu*, education_gpu)
  • All 12 A5000 nodes (gpu)
  • 10 L40S (pi_co54) nodes with names beginning with a1118
  • 172 non-GPU Intel (day, bigmem, devel, priority_mpi*, education) nodes with names beginning with a1130, a1132, c1102, c1104 & c1106

* Nodes affected by the electrical work will be excluded from accepting priority-tier jobs during the period of reduced performance. Thus, the priority_gpu and priority_tier partitions will effectively have reduced capacity during this time.

Performance Impact

We expect minor impact to RTX 5000 Ada, A5000, and non-GPU Intel 8268 nodes during planned electrical work. We estimate a 2x slowdown for the L40S and non-GPU Intel 8562Y+ nodes.

We expect significant performance impact on the H200 nodes. However, the extent of performance degradation will depend on the specific workflow, ranging from minimal to significant.

Please adjust your job wall-times when submitting jobs as needed and use checkpointing when possible. Let us know if you would like assistance doing so. We recognize that the reduced performance will impact your work, and we apologize for the inconvenience. 

The scheduled work is expected to be completed by the end of the day on October 2, 2026. YCRC will send an email communication when the work is completed. We are committed to keeping you as informed as possible at every stage of the work. Please visit the system status page for the most up-to-date information.

If you have questions, comments, or need assistance with your jobs, please contact us at ycrc@yale.edu.

9/15/26 1.45 pm: All maintenance activities have been completed. Clusters and storage are online and fully operational. Compute node availability will be reduced for a few days after the maintenance; users may experience temporary increases in wait times. 

9/15/26 11.45 am: Reboot of Bouchet login, ondemand, scheduler & monitor service nodes is complete; reboot of Globus & transfer nodes is in progress. Reboot of Bouchet compute nodes is in progress.

9/15/26, 8 am: Bouchet and Hopper login nodes rebooted and back online. Compute nodes will reboot after completing the current running jobs.

9/15/26, 7.30 am: Bouchet login nodes are being rebooted.

Scheduled Bouchet and Hopper Maintenance, September 15th 

We will perform scheduled maintenance on the Bouchet and Hopper clusters on Tuesday, September 15th.

This will not be a full downtime, but will instead have limited disruptions. The clusters and storage will remain online and available throughout the maintenance period. Compute node availability will be reduced during and for a few days after the maintenance, so users may experience temporary increases in wait times.

The login and Open OnDemand nodes for each cluster will be rebooted one at a time early in the morning of the maintenance. This may interrupt login sessions but should not impact running batch jobs.

We will send an email once maintenance is complete.

We are aware of significant networking disruptions to several of our clusters, including Bouchet, Hopper, and Milgram. We are working with ITS to identify the root cause and address it as soon as possible. 

Latest updates:

Thursday, July 30, 9.45 am: All of the a1114 nodes suffering from performance degradation have rebooted and are fully operational again

Friday, July 24, 2.30 pm: The electrical work required for Bouchet expansion has been completed. 172 out of 180 Bouchet nodes affected by the expansion work are now back online and performing as normal. Eight nodes in the week partition are still operating at reduced performance. These nodes have been set to not accept new jobs, and each of these nodes will be rebooted once currently running jobs on that node finish.  As the nodes are rebooted, they will return to normal operation at full performance.  The last of these reboots should complete by the end of the day on July 29.

Description

We are working to increase overall compute capacity on Bouchet, which involves electrical work at the data center during the week of July 20-24, 2026. This work may impact the performance of Bouchet compute nodes, as it will temporarily reduce power feeds to the compute racks. Users should expect jobs to run more slowly than usual during the planned work. 

Standard job partitions will still be available, though certain compute nodes may be more affected than others.  Priority-Tier partitions will be disabled during the week.

Scope

180 Bouchet nodes will be impacted by planned data center work July 20-24:

The following will be impacted during the entire week:

  • 10 H200 nodes (gpu_h200, gpu_devel partitions) with names beginning with a1122, a1124, and a1126 
  • 12 RTX 5000 Ada nodes (gpu, gpu_devel, education_gpu partitions) with names beginning with a1128
  • 10 L40S (pi_co54 partition) nodes with names beginning with a1118
  • 100 non-GPU Intel nodes (day, bigmem, devel, education, mpi* partitions) with names beginning with a1130 and a1132

The following will be impacted only on Monday morning (July 20) and Wednesday afternoon (July 22):

  • 7 B200 nodes (gpu_b200 partition) with names beginning with a1116
  • 8 RTX Pro 6000 Blackwell nodes (gpu_rtx6000, gpu_devel partitions) with names beginning with a1112 
  • 33 non-GPU AMD (day, week, devel partitions) nodes with names beginning with a1114

Bouchet nodes outside the scope of this electric work: 60 nodes in the mpi partition and one B200 node in the gpu_devel partition. 

* Since MPI workflows can be sensitive to slow-performing nodes, we will temporarily move some affected nodes (20 non-GPU Intel nodes) from the mpi partition to the day partition to mitigate the impact of slower nodes.  The remaining 60 nodes in the mpi partition will be unaffected by the electrical work.

Performance Impact

We expect minor impact to the RTX Pro 6000 Blackwell, RTX 5000 Ada, and non-GPU AMD nodes during planned electrical work. We estimate a 2x slowdown for the L40S and non-GPU Intel nodes and H200 nodes.

We expect significant performance impact on the B200 nodes. However, the extent of performance degradation will depend on the specific workflow, ranging from minimal to significant. 

Once full power is restored to the B200 nodes, they will need to be restarted to restore normal performance. In preparation, we will create a scheduler reservation to drain all running jobs on the B200 nodes by the end of the work impacting them. During this time, the scheduler will not start jobs with requested run times that overlap with the reservation time. Once the nodes are restarted, they will resume normal operation.

Please adjust your job wall-times when submitting jobs as needed and use checkpointing when possible. Let us know if you would like assistance doing so.

The anticipated completion of scheduled work is end of day, Friday, July 24, 2026. YCRC will send an email communication when the work is completed. We are committed to keeping you as informed as possible at every stage of the work. Updates will be posted on this page. We recognize the reduced performance will impact your work, and we apologize for the inconvenience.  If you have questions, comments, or need assistance with your jobs, please contact us at ycrc@yale.edu.

Helpful tips and resources:
  • Adjust the wall-time with –time SLURM option. For more information, visit Job Scheduling with Slurm documentation page.

8/4/26 : One GPU node (r813u23n03, in commons partition) remains offline. Vendor is scheduled to be on site today to make the necessary repairs.

7/15/26; 4 pm: The faulty CDU has been repaired. 52 out of the 56 nodes affected by it are now back online.

The following four nodes remain offline and require further repair or replacement: 

  • r813u23n01 (a100 node in the commons partition)
  • r813u23n02 (pi_gerstein_gpu GPU node)
  • r813u29n03 (a regular node in the week partition)
  • r813u29n09 (a regular node in the week partition)

The issues with these nodes have been escalated with the vendor.

Please check the status of your jobs and resubmit any jobs that failed due to the outage.  Please contact us if you encounter any issues.

7/15 10 am: The CDU has gone through the overnight testing and test water samples are being collected for analysis to confirm it is functioning as expected. Number of nodes have been powered on and are undergoing testing with the goal of bringing the majority of affected equipment back online in the next several hours.

7/14/26 6 pm: Additional issues discovered during repairs to the faulty CDU required installation of replacement parts. CDU currently in testing. The timeline for completion and bringing nodes back online has been extended to Wednesday, 7/15.

7/14/26 9am: Vendors are on site and continuing with the repair work, as planned. 

7/13/26 6.45 pm: Repairs to the faulty Cooling Distribution Unit (CDU) on McCleary are in progress. 

7/13/26 9am: The vendor has arrived at the West Campus and begun the work to repair the faulty Cooling Distribution Unit (CDU) on McCleary.

7/9/2026:  Reduced capacity on McCleary.  Due to a cooling issue in one of the McCleary racks at West Campus, McCleary nodes that have names beginning with r813 have again shut down. This outage is caused by a faulty Cooling Distribution Unit (CDU), causing equipment in the rack to overheat and shut down. We are working with the CDU vendor to resolve the issue as quickly as possible.

This issue impacts the following McCleary scheduler partitions:

  • 26 of the 33 nodes in the day partition
  • 14 of the 16 nodes in the week partition
  • 3 of the 20 nodes in the gpu partition
  • 1 of the 3 nodes in the pi_gerstein_gpu partition
  • 2 of the 10 nodes in the pi_jetz partition
  • 4 of the 4 nodes in the pi_ohern partition
  • 2 of the 2 nodes in the pi_sestan partition
  • 4 of the 4 nodes in the pi_tsang partition

Jobs running on the affected nodes have been terminated due to the failure. Please check the status of your jobs.

Slurm jobs can be submitted as usual; however, any jobs that would start on the affected nodes won’t start until after the cooling issue is resolved.  

We realize this is an impact to your work and apologize for the inconvenience.  If you have any questions, comments, or concerns, please contact us at research.computing@yale.edu.

8/13/2026:  A power failure occurred at West Campus between 1:00 AM and 1:30 AM.  This impacted the Grace, McCleary, Misha, and Milgram clusters.  Compute nodes on these clusters rebooted, which affected running jobs. Please check the status of your jobs. YCRC is checking status and recovering nodes as needed. Cluster capacity may be reduced in the meantime.

6/15/2026 - 6/18/2026:  Scheduled all-cluster maintenance complete.

6/4/2026 13:00: Hopper experienced a system issue that impacted availability. We engaged the storage vendor to resolve the issue. Hopper availability was restored by 15:30.

5/11/2026:  There was a power drop at the West Campus Data Center on the morning of 5/11/2026.  This impacted running jobs on McCleary, Grace, Misha and Milgram.  Please check your jobs to see how they were impacted.

2/13/2026: Bouchet scheduler issues have been resolved. All systems are operational.