Link to Google Doc for original notes:

Whats Everyone Developing Agenda and Meeting Notes

What’s Everyone Developing

Google Folder: ACCESS "What's everyone developing" Dec 2024

Meeting information

The purpose of this meeting is for all ACCESS teams to share what they are currently working on or are planning to start to work on in the next quarter.

Agenda

Meeting Notes

Google Folder to store presentations: ACCESS "What's everyone developing" Dec 2024 - Google Drive  

Dec 2024 (Virtual) ACCESS Quarterly Meeting  

Allocation team update

Presenters: Matt Yoder & Nathan Tolbert

XRAS

Nathan - 

Much will be items covered in the last meeting but we will be providing status updates.

Focusing on efforts that impact other teams

On-Ramps - widget that lives on other partner websites

• RMACC is now live - Illinois Compute and NewMexico will soon be next

• Don’t need a lot of features requested - more what is there and how to start

Resource Catalog

Variable Marketplace - working on user interface and all the plumbing 

• Goal was beta release at end of year - will be moving to January

• Moving resource controls that are allocation specific to XRAS Admin 

• Provides better interface and variable market controls

Class Registration - with Support

• Allocation worked on backend 

• Goal is making it easier to add in groups of people for classes

• Q - User Profiles and people not filling in NSF user status - how to not miss information

  • Support will talk about this in their session

AMIE Multi-federation for NAIRR

• Still in progress - exploring solutions - no timeline yet

Q - variable market place - how does this play into what PIs see when they apply - will RPs be advertising? 

A - if there is a discount there will be banners 

B - not sure how much actual promotion we will do vs what RPs do

C - maybe a carousel of deals

Q - Puri - NSF can’t log in - Lavanya will fix

Matt Yoder

Resource Catalog

• This is not just an allocations effort

• Review of history

• Working on implementing next Monday

• Almost ready to share!

• Worked on data prep and publishing and have had user testing

• RPs reviewed Beta in November

Resource Data

• 15 AP endpoints from all teams

• Identified common fields for Resource and Groups - so all can be pulled 

• And combined in Browser

Catalog Interface - a few things have changed

• Removed carousel - replaced with pathways typically needed

• Detail page - simplified language of headings - not familiar with terms

• Recent projects was changed to examples of papers of how others are using the resources

• Explained more about Wait Time 

• Bug Fixing and improvements

2 pending changes before launch

• 5 program resources are not in new catalog

• Add breadcrumbs to match ACO Site Navigation

Thank you to all who worked on this.

Q - How to announce to the community that it is available?  And are we collecting feedback?

• Have a call to create blurb in January newsletter

• Need suggestions on how to collect feedback

  • Andrew P will have something - but won’t be ready for January

• They have a standard form 

• Have a call or meeting with everyone who is interested in feedback - in January

  • Comms meeting this Thursday  - they will be talking about this

  • Will let PM know if they need help on getting feedback

Q - what is plan for when an endpoint goes down - how is it being monitored?  With that many APIs there is bound to have one down regularly

JP suggests - work with operations to get the right queues to submit issues or suggestions to the catalog

Cyprus might be able to help with this

Operations has a Nagios that could be used as well

They will follow up on this 

Operations team update

- JP Navarro, Dave, Cathy, and Eric

JP Navarro

Current projects

• Resource Integration Badges - almost to Beta Phase

• Developing resource and staff interface

• They created http://dashboard.operations.access-ci.org - internal only

• Worked with all ACCESS tracks

Upcoming

• RP and staff user beta phase 1 in January

• 3 phases of rollout - improvements and new features 

• 2 includes updating badges

• 3 includes adding new resource

• Hope to be done at end of PY3

Infrastructure is looking at Regional Network and Data Collection integration

Resource Information

• Resource Groups and Group Types in CiDeR - done

  • Used to be in an excel sheet - this was copied and transferred over

  • Just go straight to Cider for the information

  • This is where they need to maintain it

• RG description APIs - done

• Resource Group New API - done

• RP Software publishing, curation and SDS

Planning 

• Badges APIs

• Integration Badges in Resource Catalog 2

• Caches …….

Resource Groups and Group Types affect all the tracks

Reviewed a hierarchy diagram in the presentation

Open

System status news - can get news from central services for outages

Q - Regarding pushing notification to ACCESS users, the current  news posting will only be sent to subscribers. Will any of the current support related efforts resolve this issue.(e.g. sending emails to all users from one RP)?

Open

Andrew Answer: Affinity Groups can send email to all people with an allocation with an RP. The Support Announcements are sent to all people with allocations at any RP. The caveat is that people can remove themselves from this list. Perhaps the wording on that option should be updated? There has been some discussion about another list that would not allow people to opt out.

Guangzhen Jin - Appreciate the options @Andrew Pasquale (he/him). Those would still serve as workarounds for us.

Stephen D - Even if you DID have a list you couldn’t opt out of, you still face the issues of:

• People not reading their mail

• People creating filters

This has been asked for - we should do this - need to negotiate who should do this - Support or Allocations or a combination of teams.

Follow Up - needs more discussion - short meeting with Support, Operations, Allocations and any other track - how to email users - JP will work with Shannon to schedule meeting 30 mins.

• Jira Task: https://access-ci.atlassian.net/browse/ATT-186

Discover and reviews Access/Science Gateways

Resource contacts maintained in CiDeR - done

Will be creating views for RPs and staff  - provide access to all emails for use

Suggested Work

Status for Virtual Host - Syslog feeds into Omni-SOC - under consideration

Derek Simmel

OmniSOC - Security Information and Event Management 

Working on allowing other tracks to submit to OmniSOC

Reviewing instructions for RPs to use this service

SSH Pubkey Service -  in production

• a backend update to the co manage is requiring us to roll out and test a revised version of their database

• When we have it’s ready, we will let folks be aware and invite RPs to try out and working as expected

Active development in incident response procedures

• Asking RPs to participate in exercise 

Dave Wheeler

• Network peering with FABRIC - almost ready to send to EC

• Sending Network Data to NetSage RPs as well as us

  • Continuing to encourage RPs to send updates to NetSage

• Data Transfer Application Evaluation

  • Evaluating data transfer alternatives

JP - 

Policy document exists for data transfers - shared with resource provider forum - recommended it be adopted by RPs as well

• Includes when you should notify re: service status changes

• And How you will notify

• Relies on many teams - effort between operations and support

BREAK

Metrics

XDMoD Development

• Ver 11.0 released

• Working on fixes for bugs related to that

• Including ACCESS Credit Equivalents for Cloud (Jetstream2)

• Integrating job level performance data from Delta and ACES

• Improvements to XDMOD training data with support AI Chatbot

  • Generating new training data with non-technical queries to help with conversational queries

• Starting work on mechanism for embedding users job performance data from xdmod into OnDemand instances

Open XDMoD Development

• Released ver 11.0

  • Swapped HighCharts for Plotly

  • Support Rocky Linux 8

  • New resource specifications realm

• Ver 11.5

  • Support for Rocky linux 9

  • Support for multiple org for resources and people

Data Analytics Framework 

• Hosted

  • Bug fixes

  • On-click access - no additional login

  • Browser based

  • Functioning prototype using JupyterHub

• Software Dec

  • Changes to support Cyber Simulator

  • Feature requests

  • Bug Fixes

  • Analyze network data from Sage

Workload Analysis Reporting

• Reports to NSF senior leadership

  • Supply and demand

  • ACCESS + wider ecosystem

  • Demographics

• Future

  • Auto gen quarterly reports

  • Download queries via portal tab

Reports for RPs and Training Events

• Template showing usage and performance information

• Template report for PIs running training events

CloudBank Integration

• Current status

  • Implemented NAIRR specific data flow

  • Doing final testing - should be done by eOY

• Future

  • Add PI Org by Group

  • Add Fund and Account 

  • Work on Compute Usage

NAIRR Pilot

• Making a basic instance

  • Allocations

  • Public Cloud

  • Academic Cyberinfrastructure (Frontera, Lonestar6, ACCESS/Bridges/Delta

• Initial reports

  • Latency between award and usage

  • Projects with no usage

PATh

• Modules to

  • Display PATh data

  • Installed and tested by team - working well

  • Ingest historical records

  • Integrating MMS team reports to NSF

Doug Southworth

• Netsage - overview of what it is

• There is a resource netsage.access.io 

• We are ingesting Globus data now - every 24 hrs

  • Can be made to look like flow data - looks like completed tasks now

  • Navigation is same

  • Can do confidential tasks as well

Walkthrough

Will be scraping external database information soon and placing in science registry

Q how many sources are in NetSage

A - 6 different regions plus 6 main institutions - soon will bring in tier 2 sites

Q - how researcher facing is NetSage

A - designed to be easy to navigate - so you don’t need to be as technical to use - make it transparent to use

Q - are there interfaces we could point researchers at?

A - Access Dashboard would be first to recommend - other than that we don’t have anything specifically built for that.  The questions on the first page help direct user to look at what they want to know.

Support

- Alan Chalker Andres Pasquale

Andrew

Summary of Current Projects

Summary of Upcoming Projects

OnDemand

Current

• Q&A tool - on access ecosystem

  • Over 1000 Questions answered

  • Will review to make sure answers are correct

  • Potential collaboration with Operations 

  • Looking at what data and how often to update

• Training Registration

  • Approve or deny

  • Waitlist

  • Connect to allocation process to automatically add

  • Report of registrants

  • Upcoming features 

    • Affinity group 

    • Pre and post surveys

    • Metrics (SUs consumed

  • Planning document link in slide deck

ACCESS Widget - Q&A Bot

• Options for opening tickets and providing feedback

Support for Affinity Groups

• Work with Coordinators

• Identify community needs

• Improve documentation

• Provide community info

• Add features to support community engagement

Software Documentation Service

• API currently managed by support used fy the resource calalog to list software installed on Resources

ACCESS Resource Advisor - updates to UI/UX, switch to new software list, new version based on personas

Q - Joe - ARA - when looking at resource allocation - new resources that come online that don’t see high utilization in their early life - make use of recent historical info to promote to the top of the list to get people interested in using them. Can provide that info.

A - Andrew likes this idea

Measuring Topics of Interest

What do they need when they come to our website

• Analytics

• Knowledgebase

Access Video 

• Updating and making new videos

• Working on a process for creating and reviewing content

Alan Chalker

OnDemand - made great progress on metrics for OnDemand

• Most resources have both OnDemand as well as XDMoD

Showed stats for OnDemand Oct 24

Many available stats - examples in powerpoint

This is generally available now

Emily Moffat Sadeghi - Goal is a major release before EOY

4.0 Updates

Walkthrough of what the above changes look like

Open Discussion?

Suggestion - we make updates like this should be 2 hours at future quarterlies

Stephen - one of the best meetings he has been on and really good questions and interaction.

Great job everyone!