Meeting recording available upon request. Please contact contact Cindy Wong (cmw007@ucsd.edu), if interested.
Committee Attendees (19): Abhishek Srivasta, Agbotiname Imoize, Dinadayalane Tandabany, Emre Brookes, Franco Pestilli, Gururaj Kudur Jayaprakash, Harel Weinstein, Jaime Hernandez, Jarek Nabrzysk, Jason Wells, Jason Williams, Jeremy Fischer (RP Chair), Jia-An Yan, Matthew Platt, Parisa Khodabakhshi, Tajendra Vir Singh, Ulf D Schiller, William Lai, and Wladimir Lyra (RAC Chair)
Not in attendance (11) : Brian O'Shea, Cedric Manouan, Dhara Trivedi, George Ostrouchov, Houlong Zhuang, Mohamed Gouiza, Rajamani Narayanan, Steven Santana, Thomas Cheatham, Vojtech Vlcek, Wissam Saidi
ACCESS Attendees (10): Alana Romanella (Support), Amit Majumdar (ACO), Dave Wheeler (Operations), Holly Klinesmith (Allocations), Laura Herriott (Allocations), John Towns (ACO), JP Navarro (Operations), Tom Furlani (Metrics), Winona Snapp-Childs (Operations), and Cindy Wong (ACO),
NSF Attendees: Sharon Geva and Purushotham Bangalore
Summary
Chairperson Wladimir Lyra called the meeting to order and welcomed the committee of 19 in attendance (11not in attendance) to review the agenda:
Welcome and call to order
ACCESS External Advisory Board (EAB) report out
Program-wide Goals and Metrics
Resource Provider Badging Process
Review of actions and close
There were 10 ACCESS staff members in attendance representing all teams.
William Lai and Jason Williams shared the EAB report out including a summary of results from the ACCESS Community survey, chat bot implementation, website personas, ACCESS-wide vision, how ACCESS is internally functioning, and how users are experiencing ACCESS. Further discussion about how ACCESS teams are involved in NAIRR and specific information about the community survey results was requested.
John Towns, ACCESS Coordination Office, shared program-wide goals and metrics, inspiring rich discussion from the committee about the roles campus research units play in the future of ACCESS users and even collaboration opportunities across institutions. Some concerns were mentioned about response times for technical support questions including Resource Provider availability. The publication metrics induced committee input suggesting the current DOI acknowledgement is uncitable by some domains, further consideration should be explored for improvement, see detailed notes.
JP Navarro and Winona Snapp-Childs discussed the resource provider badging process inspiring further discussion of the chatbot functionality. The committee would like further discussion about the chatbot capability for ACCESS.
Laura Herriott, ACCESS Allocations closed the meeting, sharing the following requests for topics at the next meeting: 1) further discussion by the committee via email about the ACCESS-wide goals with the option to request the topic return at the next meeting 2) ACCESS-wide metrics and utilization data including ACCESS community survey results 3) ACCESS involvement in NAIRR 4) ACCESS chatbot discussion 5) the next EAB report out.
The committee will complete a poll for the next meeting in Spring 2025.
Presentation Guidance
Each agenda topic will require a champion or representative to present and take feedback or recommendations back for consideration
These meetings are community-run and community-built, so please be prepared to 1) share information ahead of time 2) seek feedback during your topic 3) arrange for someone to take notes on your behalf 4) enter all topics into the ACCESS spreadsheet
Agenda
Time (Central)
Topic
Notes
Time (Central)
Topic
Notes
10:00 am - 10:10 am
Welcome and Call to Order
10:10 am - 10:25 am
ACCESS External Advisory Board (EAB) report out
Community survey not clear if they were active participants and had received an allocation or if they had just started an account
Positive data overall
Access to services the program provides
Chat bot support concern for all the nuances of systems and services and keeping information up-to-date
General interest in pursuing personas on the website to improve user experience
Coalescing with a vision going forward as we step into year 3
Inward looking – how is ACCESS functioning
Outward looking – how are users gaining access
Community feedback needs being met as understood through surveys
ACCESS support chat bot is available and the feedback is being heard to improve user experience
NAIRR pilot and ACCESS integration playing a role
Agenda request - Summary of community survey results
10:25 am - 10:55 am
Program-Wide Goals & Metrics
Amit Majumdar and Chuck Pavloski as new members to ACO
See slides for ACCESS Goals and annual review metrics
Questions/Comments regarding Goal 1 & 2:
Goal 1: Harel - would like to suggest regarding surveys vs surveying centers of knowledge in the following sense. The valuable stream of info would come from University and research centers who are in charge of research and innovation. They have plans and they know what’s happening at the researcher level. They could be important portals for ongoing information, which would centralize the source of your survey. How can you get the surveys to those who are not known to you as users and those who are in charge of resources and research do.
John: How do we identify who we don't know? Going to researcher leads at institutions is an interesting angle.
Harel: These people also have future plans in front of them and areas of interest and what the plans are or how ACCESS can benefit.
Jason Williams.: Look at a list of NSF’s awards to folks who are trying to do some computational training or active award folks who are trying to bring computational training to their community. You could find a few projects that said they are.
Tom F: Allocation is doing the inverse – widget style of showcasing what resources are available and will be presenting that within the next qtr or two.
Jason Williams: As soon as people’s AWS or cloud related resources expire , CTOs who actually know or Librarians, that their researchers are looking to them for resources, but that is a very specific
Parisa: you noted about publications, looked at documentation on how to acknowledge ACCESS, if we can make that clear to the researchers it would be helpful.
John: this information is available on the website, but message heard it’s hard to find and will make sure to make this more visible.
Dina: We need to have that the technical support questions to be addressed quickly by sharing the knowledge or technical expertise.
John: addressing this has to be done between the ACCESS program and resource provider, it’s a known issue. I don't know that we have a great solution as yet. So I but I do appreciate you raising it because we need to be reminded about things like this.
Dina: maybe it will help for our users, and there it will accelerate the research. Same software used in different locations, I can get a quick response, but in another location I’m not able to get the support in a timely manner (i.e., months), waiting for months on a response.
Franco: a lot of people are using ACCESS, but they may not be cited.. Suggest considering RRIDs as form on citing. (in chat: Would adding a RRID help? https://rrid.site/ )
Comments on acknowledgement - I cannot cite the abstracts. Abstracts are not accepted by journals [specific to his work] . The statement that you have on the website is a little too long and does not include the PIs. Some journals require the PIs, and some are not able to include long statements.
John: citation limit is an interesting challenge. It would be great to get input on the acknowledgement point. I was not aware of the requirement of each PIs on the citations. We could consider longer for those that require it and shorter for those who do not.
Franco in chat: I just submitted and allocation and was an approved in heartbeat I feel like the improvement from the previous types of allocations is apparent and I am pleased by that. Much more streamlined
Goals 2 changes:
Franco in chat: Do you have a publication deposition form?
John: We have done in the past, in their user profile to include relevant resources in their profile. Does that answer your questions?
Franco: I suggest databasing.
John: we can query that information
William: a question about dropping the raw publication counts or modifying it. So are you suggesting doing something like an H index, or, like an I 10, index to to weight the publications by or cross citation
John: We have done measures like that. That is something we can consider going into, but would need to get a supplement from NSF. If you all suggest that,
William: Those are towards longitudinal (long term).
John: We have programmatic data that does back to the mid 2000s and we can conduct an analysis from that but it would with our predecessors XSEDE and TeraGrid
William: would it be tracking ability to get funding after their allocation to see the monetary amount of an allocation award
Tej V. Singh in chat: I guess one item could be added in place of user counts - number of active users, the definition could be a job submitted within a month or some duration that suits
Franco in chat: I overall I agree with John that there might be room for requesting additional funding because we do not have a solution for credit to CI
Jaime: mention it helps to link the awards to some kind of additional grant and it has a dollar value. Would it be helpful to quantify how the resources ACCESS provides?
John: something done in the past with support from XDmod. This was to verify those numbers.
Laura: from allocations, we do have a dollar amount for allocations and can share that more broadly if that is of interest
Jason Williams: there are folks in the research data librarian who are trying to solve these problems and automate. We are already collecting some awarded information. There should be a way to get a large view of what’s happening and if someone wants to drill down, they can. (see his earlier chat)
Harel: I would like to warn and suggest, this is not a business for you and that this kind of statistics comes with uncertainty.. This is the business of the agency (NSF) and not yours.. We advocate for ACCESS and we pride ourselves with ACCESs and that we provide access that yields papers and published science. ACCESS is used for scientific development. The NSF is responsible for connecting important developments of resources it provides and not the productivity. What kind of NSF resources have supported these advances? Collecting these numbers is not going to show anything but numbers, and useful data is worse than having too little data. Definitely request the acknowledgement and make the PI sign a promise to send it that has acknowledged ACCESS and stop at that. If you go deepers it’s all confusing. ACCESS wants to cover a larger range of domains of science. And if the allocation produces a paper that is the victory. And you know that there are fields that produce 10 papers per discovery, and there are other fields that produce one paper per 10 discoveries. And so the number of papers is definitely not a measure. The fact that there are papers from many different people in many different areas. That's the victory of access, not the number of papers from one person and so forth. So please help the people at NSF who understand this and tell them that we want to do the right thing in terms of what we are meant to provide. What we are meant to provide is to provide broad access that produces scientifically publishable work. That's it.
Jason comment in chat, emphasized also by William: Time may not permit my comment. I agree with the wisdom of Harel’s comments. I have been told by NSF POs that NSF is “not an information gathering organization”
Dina: number of citation limitations make it hard to acknowledge access. ACCESS award numbers can be tagged and mentioned. Number of people will reach for the ACCESS.
Franco in chat: This is excellent @Harel Weinstein , there are at least two issues we face as. Gateway with making PI sign forms:
1— Forms are signed early into the lifecycle of a project but papers are published late, often 2-3 years laters
2— There is an onion problem to solve for credit. For example,
—2.2— TACC and other resources that need to be cited,
So, all of the above needs to be cited.
Jason WIlliams: we have to tell the story– if ACCESS wasn't’ there we wouldn't be able to do the science/ research
Dr. Dina Tandabany in chat: It’s true. If ACCESS is not there, it is very difficult to do research many people like me with limited resources available.
Yan, Jia-An in chat: I agree with Dina. I am happy to have the ACCESS available.
Franco Pestilli in chat: I wonder whether we could create an umbrella publication that would encompass all the resources and gateways under ACCSS-CI? I suspect a top journal would be interested under the understanding that citing that publication will be requested as part of an allocation. In the past we have published in such a way so that the last author is the “community” so that new projects using ACCESS-CI are co-authors and can count that towards their value.
10:55 am - 11:25 am
Resource Provider Badging Process
JP to the audience: Does this benefit you or as useful as possible to you?
Emre: I’m interested in the chatbot side and would love to have this as a useful resource? How would this be available or consumed in the chatbot?
JP: Support is leading the chatbot. We would integrate this information with the chatbot.
Tom F: they are refining it and using it internally.
Winona: These are collaborative efforts. There are shared data sources that are going into the badging along with the chatbot. Recommend folks to test out the chatbot.
Emre: if there is a fail on the chatbot question,, how do I report that?
Winona: It's being reported.
Emre: how do you know if I like the answer?
Winona: there is a thumbs up or down that is coming to the chat bot
Alana in chat: Emre I can take these back to the Support team for more information and happy to reach out personally later.
Franco in chat: Great complement to the other metrics
11:25 am - 11:30 am
Recap/Actions
Next steps
Spring 2025
Topics
Close
Agenda topic request:
RAC members agree to continue ACCESS-wide goals via email and let us know if we should bring the topic back again next time.
ACCESS-wide metrics and utilization including a summary of ACCESS community survey results
Some interest in understanding how ACCESS is involved in NAIRR and how ACCESS is receiving and responding to community feedback
