Newsletter Issue No. 4

Welcome

The Yale Center for Research Computing (YCRC) welcomes Peter Schiffer, the inaugural Vice Provost for Research, and Michael Crair, Deputy Dean for Scientific Affairs, as co-chairs of the YCRC Steering Committee. The YCRC looks forward to working with them to shape the Center’s direction and initiatives. The YCRC is also extremely grateful for the tremendous work of Steve Girvin, former Deputy Provost for Research and Eugene Higgins Professor of Physics & Applied Physics, and the late Carolyn Slayman, former Deputy Dean for Academic & Scientific Affairs. Steve and Carolyn played indispensable roles in the establishment of the Center, and their vision will continue to guide the YCRC as it moves into the next phase of its expansion.

FY17 at the YCRC saw numerous expansions of the cyber-infrastructure, greater advanced support through expanded office hours with research support specialists, an enhanced suite of skills based training workshops, and greater community engagement through events and tours. This year, the YCRC will continue to focus in these areas, and looks forward to hearing from the community regarding future opportunities.

Sincerely,

Kiran D. Keshav
Executive Director

Cyber-Infrastructure

Grace Expansion and Omega Replacement
The Omega cluster has served the Yale research community extremely well since it was purchased at the end of 2010 to support advanced research computing. As it approaches the end of its seventh year of service (over 2 years more than the typical lifetime for similar clusters), the YCRC has begun to implement a multi-year replacement plan. Over the past few months, the YCRC has expanded the Grace cluster by adding approximately 90 new nodes available to all users, including several nodes containing GPUs or up to 1.5 Terabytes (TB) of memory. In addition approximately 90 new nodes to support specific faculty groups across the campus, as well as one Petabyte (PB) of new high-performance GPFS storage, have been added. All these enhancements have made Grace ideal for a broad range of advanced computations, and most users have migrated their work off of the Omega cluster. Looking ahead, Omega’s Lustre storage system will be decommissioned over the next few months. Data will be copied to Grace’s GPFS storage system, where they will be accessible from both clusters. Then, in 2018, nodes and network hardware for a new modest-size cluster designed specifically for tightly-coupled parallel workloads will be purchased.

Milgram Expansion
Over the past few years, the computational demands of researchers in the Department of Psychology’s Neuroscience program have surged as a result of the growth in faculty and student numbers, and the increased computational dependence of the research. Early in the 2015-2016 academic year, Yale deployed the Milgram cluster to help meet the needs of four faculty research groups that needed to perform analysis computations arising from functional neuroimaging (mainly fMRI) studies. The initial Milgram cluster provided 12 compute nodes and space for 150 TB of research data on a medium-performance storage system. This fall, to respond to the needs of what is now 11 faculty research groups, the YCRC has installed a substantial upgrade to the Milgram cluster, which now includes 60 compute nodes and space for 1 PB of research data on a high-performance parallel file system. The upgraded cluster is undergoing final testing by a small group of Psychology researchers and is expected to be in full production soon!

Farnam Purchasing Cycle
The YCRC continues to be impressed with the breadth and depth of life science and biomedical applications that Farnam has supported since it came online in 2016. Recently, the Center put out a call for the PI purchase of additional hardware and storage, including new specialized and cost-effective nodes that have graphics processing units (GPUs) to accelerate certain applications. PIs who opted to purchase new compute nodes and storage will be able to leverage this new infrastructure soon.

Globus
This high-performance data transfer solution is helping researchers move data big and small onto and off of the clusters. There are now dedicated data transfer nodes for Grace, Omega, Farnam and Ruddle. There is also now a pilot program to allow users to directly transfer data to and from Google Drive stores. Users of EliApps will be able to archive and share very large amounts of data through Google Drive. Interested users should contact the YCRC for further information.

High End Support

Cryo and Custom GPU
Various computing fields have seen a recent explosion of interest in using the capabilities of graphics processors (GPUs) for parallel computations. In coordination with groups engaged in cutting-edge Cryo-Electron microscopy research, the YCRC is gearing up for the expanded availability of GPUs on Farnam. The new GPUs are significantly less expensive than previous HPC models. These new nodes should be extremely useful for users leveraging recent advances in deep learning and image processing. Expect to see these nodes, each with 4 GPUs, coming to Farnam soon!

Visualization
A visualization environment is now live and ready for use! Reserve time via the YCRC website and stop in to take advantage of the powerful workstations attached to 80-inch 4K displays, which allow users to view whole datasets on a single screen. The environment also offers a VR workstation with an HTC Vive headset available to the Yale community.
Please contact the Center to work with one of the support specialists to develop VR applications or make use of currently available tools.

Science Research Software Core
Under the leadership of Dr. Misha Guy, YCRC’s Science Research Software Core provides advanced commercial software and modeling support for analytical and experimental research, design, and analysis. Most of the SRSC’s clients come from the physical sciences and engineering, but the SRSC is a resource available to all researchers on campus.

Recently, the SRSC enriched the original cluster integration functionality of Wolfram’s Mathematica with the ability to run parallel kernels on multiple nodes of Yale’s HPC clusters using the Slurm scheduler. To achieve this, Dr. Guy developed custom code that transparently delivers the computational power of Yale parallel clusters to the large group of faculty, students, and research staff who use Mathematica regularly for research computing. According to Dr. Guy, this integration with the Slurm scheduler is uniquely available at Yale.

Containers, Deep Learning and GPUs
The YCRC has been investigating the use of containers solutions to allow greater flexibility in running research computing applications. Container technologies are particularly useful for applications not easily compiled or run natively in an HPC envronment.

Docker is a widely used container solution in many fields, but is unsuitable for a large, shared environment such as an HPC cluster, for security reasons. The YCRC has standardized on using Singularity as the HPC container solution of choice. Docker images can be used without change in the Singularity environment, and can access HPC cluster data.

TensorFlow is an open source project from the Google Brain Team. It is one of the most popular “Deep Learning” packages in use today because it makes machine learning research easier and faster. TensorFlow leverages the Python scripting language, and provides support for GPUs, which can make machine learning applications run hundreds of times faster. Since the Grace and Farnam HPC clusters now have advanced Nvidia GPUs, they are excellent environments for running TensorFlow.

Geo Computation
Dr. Giuseppe Amatulli, a forest scientist and spatial modeler, provides expertise in Geo-Computation and Geo-Science to YCRC users. Dr. Amatulli leads several 2-hour Geo-Computation YCRC workshops each semester, which together provide an overview of the tools and techniques used to manipulate raster and vector datasets. In addition to these workshops, Dr. Amatulli leads an intensive, one week bootcamp each spring. This lays the foundation for running geo-spatial analyses on the YCRC’s HPC clusters and is targeted at researchers with a keen interest in complex spatio-temporal data analyses and modeling.

Dr. Amatulli’s office hours can be found on the YCRC website.

Office Hours
In an effort to better serve the increasingly computational research on campus, the YCRC research support specialists continue to make themselves available through drop in office hours, and by appointment. Stop by if you are in the area!

Community

To visit our facilities, or request a tour to see the HPC data center, please email research.computing@yale.edu.
For additional research related resources on campus, please see the research portal for further details.

We thank those who have submitted their responses to the HPC Usage Survey.  If you have not submitted your responses, please do so.  This information will not only allow us to make a stronger case for additional resources, it will also aid in operating the current resources.