Previous Meeting Minutes from XSEDE SP Forum

1. Welcome

2. ACCESS Coordinating Office - John Towns and Dina Meek

3. XSEDE and SP Coordination News

4. Next meeting:  August 18, 2022

1. Welcome

2. ACCESS Track 4 (Monitoring & Measurement Services): Tom Furlani and team

3. XSEDE and SP Coordination News

4. Next meeting:  July 21, 2022

1. Welcome

2. Introducing ACCESS's operations and user services - The CONECT project:  Amy Scheule, Tim Boerner, Winona Sapps-Childs, Kathy Benninger and Alex Withers

3. XSEDE and SP Coordination News

4. Next meeting: June 23, 2022

1. Welcome

2. Introducing ACCESS's approach to user services - The MATCH project:  Alan Chalker

3. XSEDE and SP Coordination News

4. Next meeting: June 23, 2022

1. Welcome

2. Introducing ACCESS's approach to allocation services - The RAMPS project:  Dave Hart

3. XSEDE and SP Coordination News

4. Next meeting: June 9, 2022

1. Welcome

2. Open Storage Network current activities and future plans: John Goodhue

3. XSEDE and SP Coordination News

4. Next meeting:  May 25, 2022

Attendees: 21

Agenda:

1. Welcome
   
   

   1. Davide Del Vento is leaving NCAR, welcome Ben Kirk as NCAR's SP Forum representative

2. Introducing ACES, a new NSF-funded resource (Honggao Liu, Texas A&M)

   1. New system ACES program (NSF ACSS grant - #2112356); 

      • $5M + 1.25M for 5 years); program has been archived

      • 2-3 awards/year over 5 years

   2. User Portal — uses Slurm and Liquid Software+Kubernetes

   3. Composable resources

      • 11.500+ CPU cores; dual intel sapphire rapides 48c profs

      • liquid composable infrastructure; 120 nodes;  supports up to 10 PCIe cards (GPU, FPGA, VE)

   4.  Mult. accelerator techs: GPU, FPGA, VE, Intel Optane memory

   5. Deployment: test phase: Sept2022;  early user:  Q4 2022; allocations: Q1/Q2 2023

   6. https://hprc.tami.edu/aces

3. Roundtable

4. Next meeting:  April 14, 2022 - SPs' Training activities and approaches

Attendees:  Tim Boerner, Kevin Colby, Davide Del Vento, Craig Earley, Jeremy Fischer, Jim Griffioen, David Hancock, Doug Jennewein, Ruth Marinshaw, Kenton McHenry, Tim Middelkoop, Sudhakar Pamidighantam, Tabitha Samuel, Sergiu Sanielevici, Eva Siegmann, Bob Sinkovitz, Dan Stanzione, Mary Thomas

Agenda:

1. Welcome

2. Discussion of Campus Champions SP allocations post-XSEDE, with Champions program leadership team member Tim Middelkoop and SP Forum member Doug Jennewein (also part of the CC leadership team)
   
   

   1.  Campus Champion program was introduced in 2009, and it has grown steadily over the years.  Post-XSEDE, the CC program will continue and the leadership team has been crafting a sustainability plan

   2.  Champions find value in the CC allocations (about 40 CCs out of 600 have active allocations) from Service Providers as well as significant value from the XSEDE allocations reports

   3. Champions would like clarity regarding whether the individual SPs will continue to provide allocations post-XSEDE.

   4. SPs noted that the ACCESS awardees should be known soon, but the SPs are committed to continue to provide allocations in some form.  For example, TACC noted that it would be happy to continue allocations going on their systems using local accounts if necessary.  There is hope that the allocations infrastructure and process will continue in ACCESS, rather than each SP having to manage parallel allocations.  In general, CC's usage of the allocations they have received has been quite modestSPs persist across XSEDE and ACCESS, and CCs will too.

   5. Discussion of how best to manage allocations if the number of champions per campus/institution grows significantly, as there IS overhead for the SPs in terms of account management. One recommendation was to create one shared project per institution. 

   6. Another suggestion was perhaps separate the types of CC allocations into (a)kick the tires – low CPU utilization, debugging mostly and (b) testing codes and running science, which are more intensive

3. XSEDE program news:  Tim reported that the XSEDE Quarterly meetings had been held earlier in March, in a hybrid mode.  A future meeting topic for the SP Forum will be:  SP experiences over XSEDE and Lessons Learned, to help inform XSEDE's wrap up reports and insights to the NSF and community.

4. Next meeting:  March 31, 2022

Attendees: Jaime Camboriza, Davide Del Vento, Craig Earley, Jeremy Fischer, Vikram Gazula, Jim Griffioen, Robert Harrison, John Huffman, Doug Jennewein, Honggao Liu, Ruth Marinshaw, Linh Ngo, Sudhakar Pamidighantam, Sheri Sanders, Sergiu Sanielevici, Joe Schoonover, Anita Schwartz, Eva Siegmann, Preston Smith, Carol Song, Dan Stanzione, Mary Thomas, Grigori Yourganov

Agenda:

1. Welcome

2. SP Coordination Updates (Tabitha Samuel) - Ookami will soon be an allocable resource through the XRAC process

3. Ookami presentation and discussion (Robert Harrison and Eva Siegmann)

   • Ookami is a testbed system, up for 2 years, 1 year operating as a testbed; 5 year project, + 1 additional year to ops)

   • mid-2022 move into production, 

   • Move to L2 SP (Oct’22) — 90% allocations dedicated to XSEDE

   • slides on Ookami (Japanese for wolf); motivated by Fugaku

   • located at StonyBrook, NSF funded

   • 1.5 million node-hrs/year; 32GB@1TB/sec

   • 211 users (90% US, 93% academic).   List of the 71 current projects @ https:www.stonybrook.edu//ookami/projtects

   • support: 

     • website https://www.stonybrook.edu//ookami/ ;  

     • email

     • ticketing system (https://iacs.supportsystem.com); 

     • biweekly office hours (Tues 10am, Thur 2); 

     • slack

    

   Qs:

   • SVE compilers are getting better; performance is not what was expected

   • TACC: still waiting for the Fujistu compiler; how does it compare to ARM compiler?

     • Fujitsu and CRAY are generating good SVE code

     • Suggested by Robert: Fujistu ARM compiler is more standard, so better place to start

     • Lack of vectorized math is factor 40x different

     • e.g. memory bound code not looking great

     • ARM compiler getting better for TACC: everything compiles now they are working on optimization

   • apps :which work best? 

     • thread scaling is good for large memory BW

     • issue: short vectors add to instruction latency and power consumption

     • good with apps operating on long vectors

   • how are legacy codes doing (in particular chemistry)?

     • most "out of the box" codes are working fine

     • for well-vectorized codes, they have reasonably competitive code

     • vectorized scalar not doing that well.

   • How is NWChem working?

     • very well; Fujitsu compiler is giving good performance

     • expect different perf between mat-mat-mult and vectorization of FP operations

   • training

     • webinars on ARM vectorization and profiling tools

   • Data and Testbed?

     • each node has SSD (512GB)

     • high-bandwidth/local storage (talking with NSF (Bob Chadduck) to expand to multi-terabytes)

   • Modern (2008) fortran support/what features will Fujitsu support

     • ARM compiler (C/C++) is good

     • Cray is best (C++ 17 support)

     • Fujitsu — challenging

   • Are commercial vendors working on compiling software for the system?

     • they have 1:  Parallelware Analyzer; testing on Ookami

     • VAPS, working with XDMOD team to get “top 10 or 15”

   • Are they expecting a lot of startup allocation requests?

     • Yes. They expect a lot of exploration application

4. Roundtable

   • Joe Schoonover noted that Fluid Numerics is working with AMD to have a co-located "lunch & learn" on HIP and OpenMP at PEARC22

5. Next meeting: March 3, 2022 – seeking volunteers to present.

6. March 17, 2022 meeting - Mary Thomas is organizing a panel or discussion around training