YCRC Newsletter - Summer 2018
Message from the Executive Director
On behalf of the Yale Center for Research Computing (YCRC), I am delighted to share with you activities taking place at Yale in the area of research computing.
The YCRC continues to coordinate its efforts with Yale’s academic priorities, which are consistent with the University Science Strategy Committee Report. The Center is providing support for scientific computing, visualization tools, scientific software, high-speed networking, research storage, and technology consulting. We support university cores such as Cryogenic Electron Microscopy and the Yale Center for Genome Analysis, and we are looking to expand our computational support to include ecological and evolutionary sciences, data science, and quantum computing.
We are focused on expanding technology support opportunities for research by working with various groups to provide research environments for computing and data of all shapes and sizes.
In late spring, under the direction of Peter Schiffer, Vice Provost for Research, co-directors Harlan Krumholz and Daisuke Nagai, and the YCRC Steering Committee, convened faculty focus groups to advance efforts in the areas of National High Performance Computing Facilities Use, Open Science, Grant Opportunities, and Sensitive Data using Cloud Computing Resources. We are pleased that nearly 40 members of our research community have joined one of these focus groups, and we are looking forward to the outcome of the work done by our dedicated colleagues.
Our research community’s successes are supported by an extremely dedicated and talented team who provide technology support and consistently deliver excellent customer service to the community.
An open invitation is extended to you to visit the YCRC and to take advantage of our skills-based workshops/trainings, have an individual consulting session, or use our powerful visualization workstations which allows for viewing of whole datasets on a single screen.
I am continually amazed by the research initiatives taking place on campus and am proud to support Yale’s vibrant and ever-growing research community.
Sincerely,
Kiran D. Keshav
Executive Director
Yale Center for Research Computing
Support
Defense University Research Instrumentation Program
The YCRC congratulates Corey O’Hern, Ph.D. (Associate Professor of Mechanical Engineering & Materials Science, Physics, Applied Physics) and Andrew Sherman, Ph.D. (Senior Research Scientist, Arts & Sciences Research Support), who have been awarded a Defense University Research Instrumentation Program (DURIP) grant for work on the modeling and simulation of fluid-sheared granular beds. This award brings additional Graphics Processing Unit (GPU) resources to both Professor O’Hern’s group and the YCRC computing community.
Consulting Hours
The staff at the YCRC is available to the research community during drop-in office hours. Staff is available Monday through Friday during posted times. We hope that our faculty and students enjoy the one-on-one time as much as our staff does – we especially enjoy the intellectual surprises that arise during visits.
This past fiscal year, we provided over 180 hours of consulting service to researchers on campus. This is in addition to our dedicated departmental support!
Departmental Support
The Research Support team provides in-depth support for computationally and data intensive departments within our research community. Kaylea Nelson, Ph.D., assists researchers in the Department of Geology and Geophysics with everything from ensuring smooth simulation runs to developing analysis pipelines.
Ben Evans, Ph.D., works with Professor Casey Dunn, Ph.D., and his group in the Department of Ecology and Evolutionary Biology on projects involving the internet of things (IoT), lab hardware monitoring, and genome assemblies. Ben is also looking forward to assisting Professors John Lafferty, Ph.D., and Jessi Cisewski, Ph.D., with cloud infrastructure for their upcoming Foundations of Data Science course.
Robert Bjornson, Ph.D., and Jason Ignatius continue their research support for the Yale Center for Genome Analysis.
Andrew Sherman, Misha Guy, Ph.D., Kaylea Nelson and Stephen Weston continue to support the Faculty of Arts & Sciences community at large in areas of scientific computing, scientific software use, and code optimization.
We look forward to strengthening our existing collaborations and joining forces with new research groups in the future.
“Your services and spontaneous follow-ups with us have been exceptional. It is lovely to work with you and your staff.”
BJ Casey, Ph.D.
Director, Fundamentals of the Adolescent Brain (FAB) Lab
and Professor of Psychology and the Interdepartmental Neuroscience Program at Yale University
Cyber Infrastructure & Beyond
Cryo-EM and GPUs
The YCRC has been working closely with Professor Fred Sigworth, Ph.D., Shenping Wu, Ph.D., and Marc Llaguno, Ph.D., at Yale’s Center for Cellular and Molecular Imaging (CCMI) Electron Microscopy Core Facility, which recently purchased a Krios cryo-EM. Processing cryo-EM images requires significant storage and computation and is most efficiently run on Graphics Processing Units (GPUs). The YCRC has purchased additional storage and installed 21 Thinkmate compute nodes on the Farnam cluster, 10 of which are dedicated to cryo-EM processing. Each of the GPU nodes has 4 Nvidia GTX 1080ti GPUs and 128GB of RAM.
In total, the YCRC has rolled out over 90 new GPUs for our clusters this year. This has significantly sped up computations in certain classes of parallel computing. Some are being used for neural network computations and molecular dynamics simulations. Others are being used for analyzing cryo-electron micrographs.
The YCRC’s recent acquisitions also include the ordering of 36 P100 GPUs as part of the DURIP grant.
Cloud Environments
The YCRC, in collaboration with Yale ITS, has been exploring private cloud environments, and is excited to introduce a new OpenStack environment for use by select groups later this summer. This private cloud service will make use of 12 compute nodes and 200TB of storage, providing researchers an exploratory environment to test the latest cloud-based technologies, such as Kubernetes, quickly and efficiently.
In addition, the YCRC and ITS have been collaborating on secure cloud environments for use with sensitive data. Thanks go to Alexander Behzad, Dave Backeberg, Robert Bjornson, Paul Gluhosky, Jason Ignatius, Jay Kubeck, Dave Logie, and the ITS infrastructure and security teams for their continued work.
Cooling Doors
The YCRC is always looking ahead so we can meet the future needs of HPC users at Yale. As part of an expansion of the HPC Data Center at West Campus, the YCRC has purchased 6 Motivair chilled rack doors to allow high-density installations of the latest GPUs and other advanced servers that are essential to support Yale’s efforts in cryo-electron microscopy, computational deep learning, chemistry, engineering, and other computation-based research.
Great work by Dave Logie, Paul Gluhosky, and the ITS data center team for their continued exploration of data center enhancements.
“My group uses compute clusters for the design and analysis of integrated circuits. We use a combination of our own software as well as commercial packages for this purpose. The commercial packages have many moving parts, with complicated licensing requirements and access controls governed by non-disclosure agreements. Our chip simulations generate large data files that have to be post-processed for further analysis. YCRC has made this a painless task, ensuring that my group can focus on research rather than system administration/maintenance. The support we have received has been excellent, and the staff have been extremely responsive to our needs.”
Rajit Manohar
John C. Malone Professor of Electrical Engineering
Professor of Computer Science
Yale University
Skylake Nodes
The YCRC recently installed several new compute nodes that use Intel’s latest scalable processor technology, known as Skylake. Two types of 24-core nodes were installed on Grace: one node contains two Intel Xeon Gold 6126 processors with 192 GB of memory, while each of the others contains two higher performance Intel Xeon Gold 6136 processors with 96 GB of memory. The new nodes are intended to enable users to test, benchmark, and optimize their codes on the Skylake processor technology.
During the 2019 fiscal year, the YCRC will create a new cluster containing Intel Gold 6136 processors and the latest InfiniBand network technology to serve users who run tightly coupled parallel jobs that have previously been run on the Omega cluster, which will be retired in December 2018. In addition, the YCRC plans to upgrade the oldest portion of the Grace cluster with nodes based on scalable processors similar to the Intel Gold 6126.
Visualization
The YCRC welcomes the research community to take advantage of the visualization workstations located at 160 St. Ronan Street. You can reserve time via the YCRC website or by contacting the YCRC to take advantage of the powerful workstations attached to 80-inch 4K displays, which allow users to view whole datasets on a single screen. The environment also offers a VR workstation with an HTC Vive headset available to the Yale community. Reach out to us soon to schedule a time between 9am-5pm Monday through Friday.
Skills-Based Workshops/Trainings
Facts & Figures
This past fiscal year, the YCRC hosted over 60 skills-based workshops/trainings led by YCRC staff and our vendor partners. These were attended by nearly 900 community members across over 40 departments. Offerings included R Optimization, XSEDE, Matlab, HPC, Wolfram Mathematica 11, Git, Geo-Computation, MPI for Py, Python, Linux, Schrodinger Software, Comsol, and Scalable Machine Learning in the AWS Cloud, just to name a few. In the coming year, the YCRC will continue to offer workshops/trainings to benefit new and experienced researchers. Thanks to Paul DiBello, Ph.D. and Carla Mills for their efforts in coordinating our workshops/trainings.
Linux Clusters Institute
The YCRC will be hosting a week-long Linux Clusters Institute Intermediate Workshop in August. Charles Wright and Tyler Trafford, members of the YCRC’s HPC system administration team, will assist with the training.
Forestry & Environmental Studies - 5 Day Immersion Workshop
In May 2018, Giuseppe Amatulli, Ph.D., Research Scientist in Geo-Computation and Spatial Science, held a 5-day Geo-Computation and Environmental Analysis Workshop. This intensive course was focused on enhancing programming skills in Geo-Computation under Linux.
Approximately 20 participants were introduced to the Linux environment and the command-line approach for massive processing of geo-datasets using Bash, AWK, Python, GRASS, QGIS, GDAL/OGR, R, PKtools, and OpenForis. The course guided both experienced GIS users and newbies, who had never previously used a command line interface. Course facilitators included Stephen Weston (YCRC), Benoit Parmentier (The National Socio-Environmental Synthesis Center), and Jacob Bukoski (University of California, Berkeley).
“This is a well-taught, rigorous course on spatial analysis and open-source GIS methods and tools. It has made a significant impact on my graduate career by cementing some methods I was already using with my data and introducing me to a whole new set of skills and their capabilities. After this course I feel more comfortable downloading and processing spatial data and conducting analyses with the command line, gdal, pktools, R, and more. I would recommend this course to others who are enthusiastic about GIS and coding.”
Evlyn Pless
Graduate Student (Jeff Powell Lab)
The Department of Ecology and Evolutionary Biology
Yale University
Helping the Community
Faculty Focus Groups
Faculty and staff in numerous schools, departments and administrative units have formed subgroups and have begun meeting to advance efforts in the areas of National High Performance Computing Facilities Use, Open Science, Grant Opportunities, and Sensitive Data Using Cloud Computing Resources. These subgroups will present their findings to the YCRC Steering Committee, highlighting near-term and long-term recommendations.
Monthly User Groups
At the end of June, more than 50 YCRC users participated in the inaugural meeting of the YCRC User Group organized by Kaylea Nelson. The group will hold monthly meetings, each on a distinct topic of interest to the YCRC user community, such as machine/deep learning, data management strategies, or reproducibility. The meetings will offer users an opportunity to teach and learn from each other via a mixture of panel discussions, formal presentations, lightning talks, and special interest groups. The YCRC User Group meets the first Wednesday of each month at 4pm in the YCRC Auditorium at 160 St. Ronan Street. The next meeting will be Wednesday, August 1, 2018.
School of Management Pilot
The YCRC has begun working with the School of Management (SOM) on a pilot project to assess how faculty and students at SOM might take advantage of the YCRC’s computational expertise and resources. The YCRC has set up a small dedicated partition for the project on the Grace cluster, and SOM staff are working with the YCRC to identify a small number of faculty to participate in the pilot.
National Organizations
Members of YCRC staff are continually collaborating with various national computing organizations, keeping the team on the cutting edge of the latest technologies.
Kaylea Nelson, Ben Evans, and Andrew Sherman are all Campus Champions, which is a group of over 400 HPC support staff at over 200 US colleges and universities. A major role for the Champions is to help campus HPC users make use of the NSF-funded national HPC resources such as the large clusters at the Texas Advanced Computing Center, the National Center for Supercomputing Applications, and the National Center for Atmospheric Research.
Andrew Sherman is a representative to and treasurer of the Coalition for Academic Scientific Computation (CASC), which is an organization of nearly 90 on-campus HPC centers that provides advice on best practices for such centers and allows center representatives to interact regularly with NSF, NIH, and other federal agencies involved in funding and providing HPC facilities.
Andrew and Kaylea also represent Yale in the Campus Research Computing Consortium (CaRCC) that focuses on helping university staff to better support scientists and other researchers on campus who make use of advanced research computing facilities.
Survey Results
In 2017, the YCRC conducted a survey to assess how the community is using the computational and data intensive resources on campus. We thank those who responded and have summarized the survey results.
Day of Data 2018
The Yale Day of Data is an annual, university-wide event for Yale faculty, students, and staff. Researchers discuss questions that have a global impact and learn about worldwide collaborations that are solving problems from the geosciences to public health to environmental justice and beyond.
This year’s theme is ‘Data on Earth’, and the event will be held on Friday, November 30, 2018. Stay tuned for more details!
Staff News
In early 2018, Alexander Behzad and Carla Mills joined the effort to provide technology support to researchers on campus. Alexander is supporting various physics labs on campus. Carla joined the YCRC as our senior administrative assistant, and is supporting office operations, coordinating training, and expanding the YCRC’s community outreach efforts.
This spring, the team also welcomed the addition of three summer interns from the University of Connecticut to join the effort in providing technology support for research on campus.
Christopher Geiger, a rising sophomore majoring in Computer Science, is working with the YCRC’s research support group, Pat Panico, a rising sophomore majoring in Computer Science & Engineering, is working with the engineering team, and Taylor Wade, a rising junior, majoring in Biomedical Engineering and Computer Science, is working with the ITS Research Technology Services group.
The YCRC also welcomes Thomas Langford, Ph.D., a postdoctoral associate with the Heeger Lab, who joined the YCRC on a part-time basis in July and will be providing scientific computing support for the physical sciences.
The YCRC welcomes Eric Peskin, Ph.D., as the Director of High Performance Computing and will report to Kiran Keshav, Executive Director of the YCRC. Eric holds a Ph.D. in Computer Science from the University of Utah, and has worked with high-performance computing systems since 2010, when he joined the NYU Langone Medical Center as Associate Technical Director and later Interim Director of the High-Performance Computing Facility. Prior to his work in HPC, Eric was an Assistant Professor of Electrical Engineering at the Rochester Institute of Technology. His research was in reconfigurable computing, with projects on using field-programmable gate arrays (FPGAs) to accelerate applications in computational fluid dynamics, Bayesian network learning, and image processing.
Left to right: Alexander Behzad, Eric Peskin, Pat Panico, Christopher Geiger, Taylor Wade and Thomas Langford.
Team Training and Certifications
This spring, Charles Wright, Systems Engineer for the YCRC, successfully completed all program requirements and is now a Red Hat Certified Architect. This prestigious certification is awarded to those who have attained Red Hat’s highest level of certification and have passed hands-on exams and kept current in at least five additional Red Hat technologies. Congratulations Charles!
Jason Ignatius, Systems Programmer for the YCRC, attended the OpenStack Summit in Vancouver in May, where he learned more about OpenStack, Kubernetes, and GPUs. Jason will be applying his new knowledge to the development of YCRC’s OpenStack environment.
The Yale Center for Research Computing Team Summer 2018