2024-10-10 ACCESS Researcher Advisory Committee - Fall 2024

10:00 am - 10:10 am

Welcome and Call to Order

10:10 am - 10:25 am

ACCESS External Advisory Board (EAB) report out

• Community survey not clear if they were active participants and had received an allocation or if they had just started an account

• Positive data overall

• Access to services the program provides

• Chat bot support concern for all the nuances of systems and services and keeping information up-to-date

• General interest in pursuing personas on the website to improve user experience

• Coalescing with a vision going forward as we step into year 3

• Inward looking – how is ACCESS functioning

• Outward looking – how are users gaining access

• Community feedback needs being met as understood through surveys

• ACCESS support chat bot is available and the feedback is being heard to improve user experience

• NAIRR pilot and ACCESS integration playing a role

• Agenda request - Summary of community survey results

10:25 am - 10:55 am

Program-Wide Goals & Metrics

Open

• Amit Majumdar and Chuck Pavloski as new members to ACO

• See slides for ACCESS Goals and annual review metrics

Questions/Comments regarding Goal 1 & 2:

• Goal 1: Harel - would like to suggest regarding surveys vs surveying centers of knowledge in the following sense. The valuable stream of info would come from University and research centers who are in charge of research and innovation. They have plans and they know what’s happening at the researcher level. They could be important portals for ongoing information, which would centralize the source of your survey. How can you get the surveys to those who are not known to you as users and those who are in charge of resources and research do. 

• John: How do we identify who we don't know? Going to researcher leads at institutions is an interesting angle. 

• Harel: These people also have future plans in front of them and areas of interest and what the plans are or how ACCESS can benefit. 

• Jason Williams.: Look at a list of  NSF’s awards to folks who are trying to do some computational training or active award folks who are trying to bring computational training to their community. You could find a few projects that said they are. 

• Tom F: Allocation is doing the inverse – widget style of showcasing what resources are available and will be presenting that within the next qtr or two.

• Jason Williams: As soon as people’s AWS or cloud related resources expire , CTOs who actually know or Librarians, that their researchers are looking to them for resources, but that is a very specific 

• Parisa: you noted about publications, looked at documentation on how to acknowledge ACCESS, if we can make that clear to the researchers it would be helpful. 

• John: this information is available on the website, but message heard it’s hard to find and will make sure to make this more visible. 

• https://access-ci.org/about/acknowledging-access/  

• Franco in chat: Would adding a RRID help? https://rrid.site/  

• Dina: We need to have that the technical support questions to be addressed quickly by sharing the knowledge or technical expertise.

• John: addressing this has to be done between the ACCESS program and resource provider, it’s a known issue. I don't know that we have a great solution as yet. So I but I do appreciate you raising it because we need to be reminded about things like this.

• Dina: maybe it will help for our users, and there it will accelerate the research. Same software used in different locations, I can get a quick response, but in another location I’m not able to get the support in a timely manner (i.e., months), waiting for months on a response. 

• Franco: a lot of people are using ACCESS, but they may not be cited.. Suggest considering RRIDs as form on citing. (in chat: Would adding a RRID help? https://rrid.site/ ) 

  • Comments on acknowledgement - I cannot cite the abstracts. Abstracts are not accepted by journals [specific to his work] . The statement that you have on the website is a little too long and does not include the PIs. Some journals require the PIs, and some are not able to include long statements. 

• John: citation limit is an interesting challenge. It would be great to get input on the acknowledgement point. I was not aware of the requirement of each PIs on the citations. We could consider longer for those that require it and shorter for those who do not. 

• Franco in chat: I just submitted and allocation and was an approved in heartbeat I feel like the improvement from the previous types of allocations is apparent and I am pleased by that. Much more streamlined

Goals 2 changes: 

• Franco in chat: Do you have a publication deposition form?

• John: We have done in the past, in their user profile to include relevant resources in their profile. Does that answer your questions?

• Franco: I suggest databasing.

• John: we can query that information

• William: a question about dropping the raw publication counts or modifying it. So are you suggesting doing something like an H index, or, like an I 10, index to to weight the publications by or cross citation

• John: We have done measures like that. That is something we can consider going into, but would need to get a supplement from NSF. If you all suggest that, 

• William: Those are towards longitudinal (long term). 

• Jason Williams in chat: Integrating with ROR https://ror.org/   and Crossreff https://www.crossref.org/ may be a way to automate some of this

• John: We have programmatic data that does back to the mid 2000s and we can conduct an analysis from that but it would with our predecessors XSEDE and TeraGrid

• William: would it be tracking ability to get funding after their allocation to see the monetary amount of an allocation award 

• Tej V. Singh in chat: I guess one item could be added in place of user counts - number of active users, the definition could be a job submitted within a month or some duration that suits

• Franco in chat: I overall I agree with John that there might be room for requesting additional funding because we do not have a solution for credit to CI

• Jaime: mention it helps to link the awards to some kind of additional grant and it has a dollar value. Would it be helpful to quantify how the resources ACCESS provides?

• John: something done in the past with support from XDmod. This was to verify those numbers. 

• Laura: from allocations, we do have a dollar amount for allocations and can share that more broadly if that is of interest

• Jason Williams: there are folks in the research data librarian who are trying to solve these problems and automate. We are already collecting some awarded information. There should be a way to get a large view of what’s happening and if someone wants to drill down, they can. (see his earlier chat)

• Harel: I would like to warn and suggest, this is not a business for you and that this kind of statistics comes with uncertainty.. This is the business of the agency  (NSF) and not yours.. We advocate for ACCESS and we pride ourselves with ACCESs and that we provide access that yields papers and published science. ACCESS is used for scientific development.  The NSF is responsible for connecting important developments of resources it provides and not the productivity. What kind of NSF resources have supported these advances?  Collecting these numbers is not going to show anything but numbers, and useful data is worse than having too little data. Definitely request the acknowledgement and make the PI sign a promise to send it that has acknowledged ACCESS and stop at that. If you go deepers it’s all confusing. ACCESS wants to cover a larger range of domains of science. And if the allocation produces a paper that is the victory. And you know that there are fields that produce 10 papers per discovery, and there are other fields that produce one paper per 10 discoveries. And so the number of papers is definitely not a measure. The fact that there are papers from many different people in many different areas. That's the victory of access, not the number of papers from one person and so forth. So please help the people at NSF who understand this and tell them that we want to do the right thing in terms of what we are meant to provide. What we are meant to provide is to provide broad access that produces scientifically publishable work. That's it.

• Jason comment in chat, emphasized also by William: Time may not permit my comment. I agree with the wisdom of Harel’s comments. I have been told by NSF POs that NSF is “not an information gathering organization”

• Dina: number of citation limitations make it hard to acknowledge access. ACCESS award numbers can be tagged and mentioned. Number of people will reach for the ACCESS.

• Franco in chat: This is excellent @Harel Weinstein , there are at least two issues we face as. Gateway with making PI sign forms:

  • 1— Forms are signed early into the lifecycle of a project but papers are published late, often 2-3 years laters

  • 2— There is an onion problem to solve for credit. For example,

  • —2.1—I run brainlife.io and I have grants to cite, brainlife.io, uses Jetstream2,

  • —2.2— TACC and other resources that need to be cited,

  • So, all of the above needs to be cited.

• Jason WIlliams: we have to tell the story–  if ACCESS wasn't’ there we wouldn't be able to do the science/ research

• Dr. Dina Tandabany in chat: It’s true. If ACCESS is not there, it is very difficult to do research many people like me with limited resources available.

• Yan, Jia-An in chat: I agree with Dina. I am happy to have the ACCESS available.

• Franco Pestilli in chat: I wonder whether we could create an umbrella publication that would encompass all the resources and gateways under ACCSS-CI? I suspect a top journal would be interested under the understanding that citing that publication will be requested as part of an allocation. In the past we have published in such a way so that the last author is the “community” so that new projects using ACCESS-CI are co-authors and can count that towards their value.

10:55 am - 11:25 am

Resource Provider Badging Process

Open

• JP to the audience: Does this benefit you or as useful as possible to you?

• Emre: I’m interested in the chatbot side and would love to have this as a useful resource? How would this be available or consumed in the chatbot? 

• JP: Support is leading the chatbot. We would integrate this information with the chatbot. 

• Tom F: they are refining it and using it internally. 

• Winona: These are collaborative efforts. There are shared data sources that are going into the badging along with the chatbot. Recommend folks to test out the chatbot.

• Emre: if there is a fail on the chatbot question,, how do I report that?

• Winona: It's being reported.

• Emre: how do you know if I like the answer? 

• Winona: there is a thumbs up or down that is coming to the chat bot

• Alana in chat: Emre I can take these back to the Support team for more information and happy to reach out personally later. 

• Franco in chat: Great complement to the other metrics

11:25 am - 11:30 am

1. Recap/Actions

2. Next steps

   1. Spring 2025

   2. Topics

3. Close

Agenda topic request:

• RAC members agree to continue ACCESS-wide goals via email and let us know if we should bring the topic back again next time. 

• ACCESS-wide metrics and utilization including a summary of ACCESS community survey results

• Some interest in understanding how ACCESS is involved in NAIRR and how ACCESS is receiving and responding to community feedback

• ACCESS chatbot discussion

Meeting Logistics

• Website: https://access-ci.org/

• Wiki: ACCESS Researcher Advisory Committee

• Charter: https://docs.google.com/document/d/1SjIW8FVfDcUCfIVF_CVcO0rnk5DLa_f9daIa2xCUUuE/edit#heading=h.xb90631xffqb

• Agenda topic requests