ACCESS Coordination Office Response to the PY2 Review Panel Summary
[Submitted July 24, 2024]
| Table of Contents | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
Summary Statement - Rating: Achieved Results
Overall the panel felt that a lot of progress had been made this year, in particular rebooting evaluation efforts, continuing to lead and manage the governance structure, and aiming towards a coalesced ACCESS structure as a whole. There are a number of elements in place that will be helpful in achieving the goals (once finalized), although not all of the results are ready to be reviewed. In particular, the panel would like to see urgency in developing ACCESS goals and measures, to include DEI. The panel is eager to see how future engagements, including RP on the EC and other management integrative steps, help build cohesion among all the ACCESS elements. The panel would like to recognize the willingness, commitment, and capability of the ACO to help ACCESS achieve all of its goals, even when challenging issues must be addressed.
We appreciate the Panel's time, attention, and thoughtful comments provided to this review. The ACO is committed to providing the community with the highest level of service possible and strives to be responsible stewards of the support provided by NSF to the ACO on behalf of ACCESS.
We agree with many of the Panel's comments below. The ACO continues to do more than was scoped for it and has augmented the project with significant contributed resources to compensate for what we believe is an underfunded award for the expected scope. Moving into PY3 we believe it is essential for the EC to be more actively involved in prioritizing work across the program so that available resources can be put to use where they matter most. We will continue to put these important topics before the EC and provide the tools and mechanisms to make progress.
We appreciate the Panel's acknowledgment of the progress made since the last review. The ACO team has worked hard to address issues raised in the prior review as well as those that have emerged since that time. Progress on program-wide goals and metrics, and other program-wide initiatives is paced by the contributions and engagement from the other teams.
We appreciate the Panel's acknowledgment of the human and material resource issues. As presented during the review, it is important to note that the ACO has a total headcount of approximately 2.6 FTE. Additional contributed effort by the NCSA and SDSC partners has been necessary to make progress in key areas such as evaluation workgroup process, general program governance, program-wide strategic planning, and other activities.
The remainder of this response document focuses on the “Opportunities to Improve” indicated by the Panel. This is not to discount the recognition of the Strengths noted in each area and we do appreciate the Panel’s noting of those. In addition, there are some noted Strengths that we also provide comments regarding. Still, it is the case that improvement of the project, and by extension, the ACCESS program as a whole, must focus on the areas of potential improvement. To provide clarity, below the contents of the panel report are italicized and our comments and responses to that are in normal text.
Responses to Observations and Opportunities for Improvement Indicated by the Review Panel
1. Broader Impact - Rating: Partially Achieved
While the potential scope of Broader Impacts can be intimidating, we see the ACCESS program to be in a unique position to have the ability to shape the computational research communities in important and very impactful ways. We suggest the EC, the EAB, the projects and other stakeholders identify what broader impacts ACCESS could achieve that only ACCESS could achieve, and then to focus on one or two of those. The panel had difficulty identifying what broader impacts the ACO project achieved vs ACCESS Program achievements.
Strengths
Strength 1.1: Willingness to have the conversation about DEI and to leverage external expertise when needed - this is the first step and incredibly important.
Strength 1.2: Efforts to increase diversity (including those by communications showcasing the use of ACCESS resources by researchers at non-R1 and minority-serving institutions) seems to be increasing allocations to non-R1 institution researchers.
Strength 1.3: ACO (and ACCESS) have in the last program year put indicators and tools in place that allow for the evaluation, and when needed redirection, of efforts related to Broader Impact. These measures are important and should continue to receive necessary resources and priority.
The ACO appreciates these observations and has no further comment.
Opportunities to Improve
Recommendation 1.1: In terms of DEI in the broader community, ACO can do more to take advantage of the unique and influential position ACCESS has by:
Advocating within the EC to clearly define goals for DEI as they relate to the broader community that ACCESS engages with.
Refining Personas and user journeys to actively invite, and perhaps even prioritize, engagement with researcher communities that have not historically engaged HPC resources.
The ACO concurs with this recommendation but emphasizes for both of the bullets above that while the ACO can and does advocate for these things via the EC, Working Groups, and Standing Committees, it is the the EC that must make decisions regarding prioritization and allocation of staff time to support these program-wide efforts.
...
Action Item
...
Who is Responsible
...
How to address
...
Jira Link
...
EC should clearly define goals for DEI as they relate to the broader community that ACCESS engages with.
...
EC
...
EC should prioritize effort and allocate staff time for refining Personas and user journeys to actively invite, and perhaps even prioritize, engagement with researcher communities that have not historically engaged HPC resources.
...
EC
...
Recommendation 1.2: Work within the EC to develop a coherent program-wide strategy that involves all services to bring under-represented groups into the computational science research community.
The ACO concurs with this recommendation though sees it as a recommendation primarily to the EC and not specifically to the ACO. The ACO is already advocating and expending effort to support this. The EC concurs with this priority and thus chartered the DEI Working Group to address part of this. As we noted during our presentation the Community Building & Engagement (CB&E) efforts have been largely suspended in anticipation of further guidance from NSF on how the program should proceed in this space. The ACO took it upon itself to facilitate discussion during the February Quarterly Meeting to identify the priority events via which ACCESS should engage with the community for PY3. This would normally have been a function of the program-wide CB&E team.
Recommendation 1.3: ACCESS (and ACO) are uniquely placed to make a real impact on DEI in the broader community - if this is taken advantage of. This unique position to shape the composition of the community is a privilege that should be seen as such.
We take the panel’s recommendation very seriously and agree that ACCESS can and should do more to impact DEI in the community. As the panel notes, this is a recommendation to both the EC and the ACO. We will continue to use our role as the EC facilitator and peer member of the team to ensure that DEI remains a top priority of the program. We look forward to the DEI Working Group – in which the ACO participates – bringing recommendations on this to the EC as soon as possible.
Recommendation 1.4: Whereas measures supporting goals within the overall health of ACCESS and how it is enabling greater science to occur were presented and good, it was not clear how these measures were selected and how they will be tracked over a longer period. It would be helpful to explain the methodology for selecting the measures, and show the full picture of all available measures, to better understand how ACCESS is moving science forward.
The ACO appreciates this observation by the review panel, and the opportunity to provide some context. Fundamentally, the metrics are intended to provide indicators of progress toward realizing the program-wide goals that were developed through a highly collaborative process involving all staff across the program. The metrics presented in the shared team slides were put forth by the Evaluation Working Group following an initial brainstorming session at the September 2023 Quarterly meeting facilitated by ACO PI Towns. We view these as v1 of the program-wide metrics and will refine these over time with broad program input. The Evaluation Standing Committee is now reviewing the metrics in light of the results from the 3 ACCESS surveys and other program developments and will bring forth recommendations for consideration. Time is of the essence as this work will impact PY3 surveys.
Recommendation 1.5: While communications that work towards achieving broader impact goals are noted and ongoing, the ACO should pursue more concerted efforts at engaging with traditionally non-HPC communities.
The ACO recognizes the need to focus on communications with traditionally non-HPC communities. In collaboration with the other awardee teams, the ACO continues to seek means by which we can be more effective in this area. Some of those efforts were shared during the review, but others have not been pursued due to resource and staff time constraints. Nonetheless, the ACO will work with the Executive Council to identify additional means to engage the broader community within our resource constraints.
Recommendation 1.6: Showcasing how the science being enabled by ACCESS is contributing in practical ways to society would allow for talking points when approaching non-traditional HPC communities. Perhaps this could be done through metrics or communications stories.
...
https://access-ci.atlassian.net/wiki/spaces/ACP/database/811466766
2. Intellectual Merit - Rating: Achieved Results
Strengths
Strength 2.1: The ACO team leadership includes appropriate expertise and a structure for performing important functions such as project management, communications, and events. These groups seem to be working well together. ACO leadership has shown agility and responsiveness by, for example, shifting the role of the external evaluator to meet emerging needs and by integrating a RACI-like model to clarify roles.
...
The ACO concurs with what is noted and has no further comment.
Opportunities to Improve
Recommendation 2.1: Although the ACO team has made great progress in building an organizational structure for ACCESS governance, the panel feels that they should take more ownership for the overall health of ACCESS and for making progress towards overall ACCESS goals.
...
The panel recognizes that ACO is the facilitator of ACCESS-wide governance functions (e.g. the EC, EAB and quarterly meetings) and may not be solely responsible for the strengths and weaknesses of those functions. Nevertheless, whether it is via facilitation or influence amongst its peers, the ACO is crucial to successful ACCESS-wide governance.
Strengths
There are a number of positive developments in ACCESS governance during the past year.
...
The ACO concurs with what is noted. The ACO PI will consider how to distill useful feedback to NSF from the informal co-advisory group. The value of this is evident, but the need to maintain confidentiality is key to the success of those interactions.
Opportunities to Improve
Recommendation 3.1: To strengthen the overall ACCESS community, set priorities, and build a sense of common purpose, it would be beneficial to consider moving to a rotating leadership model, where perhaps the ACCESS PIs rotate a chair position once a year. This builds a better understanding of the overall health and operations of ACCESS and challenges/opportunities of the multi-functioning collaboration.
...
4. Communication and Outreach - Rating: Achieved Results
Strengths
Strength 4.1: The communication team has skills sufficient to their tasks, a broad knowledge of the tools at their disposal, and a clear understanding of industry standards. The team has been active in generating a brand and furthering its identity. The initial survey forays were encouraging, and the team is poised to take on new challenges. We have confidence in their current efforts and future endeavors.
...
The ACO concurs with what is noted and has no further comment.
Opportunities to Improve
Recommendation 4.1: The committee is excited about the opportunities that lay ahead of the communication team. We strongly encourage a pivot to a more proactive orientation, where the communication team does more than disseminating information on various web pages, networking, visiting campuses, and attending conferences. There is a pressing need to engage targeted groups, with the aim of receiving feedback, creating awareness, and developing relationships.
...
The panel recognizes that a lot of work was completed this year, and that the ACO had to significantly pivot this year in terms of leadership and direction. The panel was impressed with the amount of work that had been completed by the evaluation team, but felt that the necessary work to complete goals and decide on measures should have been further along at this point in time. In general, the panel was hoping to see, by the end of the second year of funding, a clearer presentation of overall goals for ACCESS and progress made, as well as overall goals for ACO and progress made.
Strengths
Strength 5.1: The reboot of the evaluation plan has been a positive step and there has been substantial progress in the development of program-wide goals and associated metrics, as well as ACO-specific goals and metrics.
...
The ACO concurs with what is noted and has no further comment.
Opportunities to Improve
Recommendation 5.1: The selection process of measures to support the goals of the ACO, and longitudinal tracking of those measures, should be transparent and available for review. The ones presented during the site review created a good picture, but it is unclear how they were selected and the totality of all measures that were available to be selected is not well understood.
...
6. ACO Project Office - Rating: Achieved Results
Strengths
Strength 6.1: The role that the ACO plays in the organization of meetings is very good, with clear delineation of tasks, decisions, and parking lot items.
...
The ACO concurs with what is noted and has no further comment.
Opportunities to Improve
Recommendation 6.1: The panel recommends that ACCESS team members focus on risks, either service-specific or program-wide, as a technique to not only manage those risks, but to develop goals, plans and metrics.
...