Responses to Observations and Opportunities for Improvement Indicated by the Review Panel
Broader Impact - Opportunities to Improve
DEI
Recommendation 1.1: In terms of DEI in the broader community, ACO can do more to take advantage of the unique and influential position ACCESS has by:
Advocating within the EC to clearly define goals for DEI as they relate to the broader community that ACCESS engages with.
Refining Personas and user journeys to actively invite, and perhaps even prioritize, engagement with researcher communities that have not historically engaged HPC resources.
The ACO concurs with this recommendation but emphasizes for both of the bullets above that while the ACO can and does advocate for these things via the EC, Working Groups, and Standing Committees, it is the EC that must make decisions regarding prioritization and allocation of staff time to support these program-wide efforts.
Recommendation 1.2: Work within the EC to develop a coherent program-wide strategy that involves all services to bring under-represented groups into the computational science research community. The ACO concurs with this recommendation though sees it as a recommendation primarily to the EC and not specifically to the ACO. The ACO is already advocating and expending effort to support this.
The EC concurs with this priority and thus chartered the DEI Working Group to address part of this. As we noted during our presentation the Community Building & Engagement (CB&E) efforts have been largely suspended in anticipation of further guidance from NSF on how the program should proceed in this space. The ACO took it upon itself to facilitate discussion during the February Quarterly Meeting to identify the priority events via which ACCESS should engage with the community for PY3. This would normally have been a function of the program-wide CB&E team.
Recommendation 1.3: ACCESS (and ACO) are uniquely placed to make a real impact on DEI in the broader community - if this is taken advantage of. This unique position to shape the composition of the community is a privilege that should be seen as such.
We take the panel’s recommendation very seriously and agree that ACCESS can and should do more to impact DEI in the community. As the panel notes, this is a recommendation to both the EC and the ACO. We will continue to use our role as the EC facilitator and peer member of the team to ensure that DEI remains a top priority of the program. We look forward to the DEI Working Group – in which the ACO participates – bringing recommendations on this to the EC as soon as possible.
Goals and Metrics
Recommendation 1.4: Whereas measures supporting goals within the overall health of ACCESS and how it is enabling greater science to occur were presented and good, it was not clear how these measures were selected and how they will be tracked over a longer period. It would be helpful to explain the methodology for selecting the measures, and show the full picture of all available measures, to better understand how ACCESS is moving science forward.
The ACO appreciates this observation by the review panel, and the opportunity to provide some context. Fundamentally, the metrics are intended to provide indicators of progress toward realizing the program-wide goals that were developed through a highly collaborative process involving all staff across the program. The metrics presented in the shared team slides were put forth by the Evaluation Working Group following an initial brainstorming session at the September 2023 Quarterly meeting facilitated by ACO PI Towns. We view these as v1 of the program-wide metrics and will refine these over time with broad program input. The Evaluation Standing Committee is now reviewing the metrics in light of the results from the 3 ACCESS surveys and other program developments and will bring forth recommendations for consideration. Time is of the essence as this work will impact PY3 surveys.
Recommendation 1.5: While communications that work towards achieving broader impact goals are noted and ongoing, the ACO should pursue more concerted efforts at engaging with traditionally non-HPC communities.
The ACO recognizes the need to focus on communications with traditionally non-HPC communities. In collaboration with the other awardee teams, the ACO continues to seek means by which we can be more effective in this area. Some of those efforts were shared during the review, but others have not been pursued due to resource and staff time constraints. Nonetheless, the ACO will work with the Executive Council to identify additional means to engage the broader community within our resource constraints.
Recommendation 1.6: Showcasing how the science being enabled by ACCESS is contributing in practical ways to society would allow for talking points when approaching non-traditional HPC communities. Perhaps this could be done through metrics or communications stories.
The ACO does highlight the impact of the science supported through science stories, events support, annual publications, and social media. This recommendation suggests the ACCESS Communications Standing Committee consider spending some time distilling talking points from these stories to support those directly engaging with non-traditional HPC communities as well as others. Perhaps these talking points can be maintained on the ACCESS Communications Standing Committee wiki pages as a resource to any ACCESS staff member engaging members of non-traditional HPC communities.
Intellectual Merit - Rating: Achieved Results
Strengths
Strength 2.1: The ACO team leadership includes appropriate expertise and a structure for performing important functions such as project management, communications, and events. These groups seem to be working well together. ACO leadership has shown agility and responsiveness by, for example, shifting the role of the external evaluator to meet emerging needs and by integrating a RACI-like model to clarify roles.
Strength 2.2: In addition, the ACO PI has been innovative in creating regular meetings with leadership of other Federally funded projects for co-advising and sharing common management experiences and lessons learned. This is an important mechanism for knowledge sharing and the development of best practices.
The ACO concurs with what is noted and has no further comment.
Opportunities to Improve
Recommendation 2.1: Although the ACO team has made great progress in building an organizational structure for ACCESS governance, the panel feels that they should take more ownership for the overall health of ACCESS and for making progress towards overall ACCESS goals.
We concur with the panel that there are opportunities for the ACO to take more ownership for the overall health of ACCESS, but with the caveat that that ACO’s influence is limited. While we do take every effort to identify and focus the EC’s collective efforts on matters of importance, forward progress is determined by the contributions of individual team members and the leadership of the PIs. It is our sense that there is now, more than ever before in the program, an understanding by ACCESS leadership that progress beyond single award teams will only come through better collaboration and teamwork. We are seeing positive signs across the program and will do our part to support these efforts.
Recommendation 2.2: The ACO is the only ACCESS entity that can take this overarching view within the scope of their work, and the panel would like to see more leadership by example and efforts to influence working groups/committees to achieve results in a timely manner. An example of this is being able to deliver on unified goals (including DEI) and evaluation measures within the first half of the funded award.
We appreciate the Panel’s perspective on this and agree to a point that the ACO can do more to influence Working Group/Standing Committee output. Again though, there are limitations to our influence. While the role of the ACO is certainly more oriented toward a global view, it is incumbent on all of the PIs, as part of the EC, to take this overarching view of the program. The ACO feels that it has led by example and shared a number of examples of this during the review (facilitating development of program-wide goals and metrics, facilitating the reboot of evaluation efforts, advocating for the formation of the DEI Working Group, etc.). We appreciate the panel’s recognition of the ACOs efforts here. However, the ACO simply is not staffed at a level necessary to take a leading role in all program-wide efforts. To the extent the ACO can, we will continue to lead by example with opportunities to do so on the horizon.
Recommendation 2.3: The ACO leadership team has experienced significant staff turnover including in its evaluation team. It is currently engaging the challenges of quickly filling open positions and bringing new people up to speed. It is not clear that ACO has considered how it can systematically develop resilience through methods such as succession planning or reassignment of personnel from other projects.
The ACO has considered how we can improve resilience in staff turn-over, and considers this an important issue to manage. With a total staff level of 2.58 FTE with individuals possessing disparate skill sets, there are minimal areas of redundancy.. Still, we have made efforts to address this, particularly with respect to the PIs but also for all staff members. In some cases, it is reasonable to consider re-assigning staff from other projects, but this clearly runs the risk of creating the same problem for another project. Nonetheless, we have done exactly this in some cases (e.g., Sr. Project Manager Shannon Bradley being reassigned to support the role of Sr. Project Manager Ron Payne following his departure from the Center). It is more challenging to address this with respect to the PI and co-PI, however, the PI and co-PI stand ready to support each other’s current roles should one of them become unavailable to the project until a replacement can be identified.
Program Governance, including EC and EAB
Rating: Partially Achieved
The panel recognizes that ACO is the facilitator of ACCESS-wide governance functions (e.g. the EC, EAB and quarterly meetings) and may not be solely responsible for the strengths and weaknesses of those functions. Nevertheless, whether it is via facilitation or influence amongst its peers, the ACO is crucial to successful ACCESS-wide governance.
Strengths
There are a number of positive developments in ACCESS governance during the past year:
Strength 3.1: Establishing program-wide quarterly meetings is very beneficial to the program. The Executive Council (EC) meets frequently, with good participation by stakeholders, and has developed a set of collaborative procedures and decision-making processes. The ACO team reports that there is an increasing level of trust and collaboration amongst EC members. The EC members (service PIs) have demonstrated that they can effectively work together to develop important common documents, such as a common response to PY1 review feedback, developing common PY2 review slides, and establishing program-wide goals.
Strength 3.2: The External Advisory Board (EAB) has strong members (including representatives of the Researcher Advisory Council and Resource Provider Forum) and the level of EAB engagement is strong. It is encouraging that participants are willing to contribute beyond just attending the meetings. We hope this continues as members rotate on/off the EAB.
Strength 3.3: Finally, initiating interactions with other NSF (and NIH) programs that have similar management structures is a positive step. While we understand the benefits of confidentiality, it would be valuable to define take-aways from these discussions, including commonly-shared feedback for NSF regarding management/coordination of future programs. The ACO concurs with what is noted. The ACO PI will consider how to distill useful feedback to NSF from the informal co-advisory group. The value of this is evident, but the need to maintain confidentiality is key to the success of those interactions.
Opportunities to Improve
Recommendation 3.1: To strengthen the overall ACCESS community, set priorities, and build a sense of common purpose, it would be beneficial to consider moving to a rotating leadership model, where perhaps the ACCESS PIs rotate a chair position once a year. This builds a better understanding of the overall health and operations of ACCESS and challenges/opportunities of the multi-functioning collaboration.
The ACO concurs with this recommendation and will advocate for this with the rest of the EC. The ACO did make this recommendation during the development of the EC charter. At that time the Service Team PIs did not feel this was warranted, rejecting also the suggestion that there be a Chair of the EC at all.
Recommendation 3.2: We suggest that the EC make every effort to “get on the same page” regarding the overall ACCESS goals within PY3. This will require hard conversations, and no doubt compromise, but should then also end up with defined procedures and processes that benefit ACCESS governance.
The ACO concurs that the EC members should be “on the same page” regarding overall ACCESS goals, but believes that there is in fact already significant alignment. The issue raised in the review discussions was with respect to the prioritization of efforts across the program and lack of complete alignment there, an observation with which the ACO fully agrees. The ACO has initiated a process to work toward a shared view of priorities.
Recommendation 3.3: With this unified vision and set of goals at the EC level, the staff within the separate projects as well as the wider community as a whole can then more effectively and efficiently work towards those goals. The ACO should build upon current efforts underway within the EC towards harmony to achieve this wider goal.
The ACO Concurs with this recommendation and has already initiated efforts to build upon this.
Recommendation 3.4: While the EC generally functions well, there appear to be some areas of conflict. Dissension within the EC represents a significant risk for ACCESS overall, and the ACO along with other members of the EC should make every effort to resolve any internal conflicts. For example, the External Evaluator report suggests some internal conflicts within the EC (and their respective teams) about the appropriate role of the ACO within ACCESS.
The ACO concurs with this recommendation. As the annual reviews of the program awardees complete and follow-up activities are done, the ACO will be bringing these issues to the EC for discussion. The ACO sees it as critical for there to be a shared understanding of its scope in order for it to have the opportunity for success. To illustrate the point, we call the panel’s attention to a comment made by the ACO evaluator in her
PY2 report (which was provided to the panel): “Without breaching confidentiality agreements, it is safe to say that the Service Track PIs have very different, in some cases irreconcilable views about the role of the ACO. In particular, when a conflict arises among members of the Service Tracks, some of the Service Track representatives look to the ACO to address the conflict. Other Service Track representatives believe the ACO should not take the lead in this way. Acting as a Track representative with an opinion on a contentious issue can easily be at odds with acting as a meeting facilitator.”
Recommendation 3.5: Also, the ACO team discussed a recent, specific conflict related to an ACCESS-wide DEI discussion. We are pleased that the ACO team has taken constructive steps to address this issue, including using an external conflict resolution expert. Continued efforts to ensure that issue is resolved are important, including perhaps a broader discussion regarding DEI priorities within the ACCESS team.
The ACO concurs with this recommendation and sees it also as a recommendation to the EC. The ACO appreciates the panel’s recognition of the importance of resolving conflict within the EC. The facilitated discussion is seen as a first step in a process to first gain alignment among the five PIs, followed by expanding this to the EC and then further to all of the ACCESS awardee teams. While this has a particular focus on DEI issues and priority of DEI efforts across the program, it is also intended to address alignment more generally.
Recommendation 3.6: There are indications that interactions between ACCESS and Resource Providers need to be improved - e.g. the low response rate from RPs to the RP survey and the modest RP
satisfaction rating.
The ACO concurs with this recommendation and noted in the review the collective effort now underway by the program to better engage with RPs. These efforts are outlined in the document we provided to the panel, “Improving the ACCESS Program RP Pathways (March 20, 2024)”. This includes, among other recommendations, having the RP Forum Chair join the EC (a step already taken).
Recommendation 3.7: There should be strong coupling with substantive feedback in both directions
with the RP community. The RP-related slides in the “ACCESS Annual Surveys” presentation and the
document provided to the panel “Improving the ACCESS Program RP Pathways (March 20, 2024)”
outline a number of constructive steps that the EC plans to take to address this issue. We look forward
to improvements in the coming year.
The ACO concurs with this recommendation and will be advocating these changes and their
implementation in the EC.
Recommendation 3.8: There is concern that the ACO is not fully attuned to all of the work occurring
between working groups and standing committees. Many important functions are being discussed
within these populations, and having a member of the ACO in the group is likely not strong enough
positioning to be able to understand the work and how it fits into the overall health and structure of
ACCESS - which is a primary concern of the ACO.
While it would be ideal if the ACO (and more importantly the EC) to be fully attuned to all of the
work occurring between Working Groups and Standing Committee, it is simply beyond the scope of
the ACO to do so. The ACO has established structure for working groups and standing committees,
and strives to be cognizant of the efforts across the program, but it is challenging for the ACO to
deliver on its many other obligations to allocate the level of staff time necessary for the oversight
activity the panel suggests. Further, not all work that has a program-wide impact is being done
within Working Groups and Standing Committees, but within specific teams. The situation is
improving as the need for greater cross-team communication and transparency becomes more
evident. The ACO is responsible for scheduling regular updates of the working group and standing
committees to the EC and those have not happened as frequently as they should due to other
pressing priorities. The ACO is already moving to rectify this.
8
Recommendation 3.9: We encourage a more robust system of bringing the ACO up to date on all
activities and work within the committee/group structures, and inviting leaders to attend your
meetings on a regular basis, so that the communication and group work is better supported by your
team to help achieve all desired outcomes.
The ACO generally concurs with this, but sees it as an issue for the EC and the ACO as part of the EC.
See response to Recommendation 3.8 above.
Communication and Outreach - Rating: Achieved Results
Strengths
Strength 4.1: The communication team has skills sufficient to their tasks, a broad knowledge of the
tools at their disposal, and a clear understanding of industry standards. The team has been active in
generating a brand and furthering its identity. The initial survey forays were encouraging, and the
team is poised to take on new challenges. We have confidence in their current efforts and future
endeavors.
Strength 4.2: The communications team are using excellent metrics to understand their impact and
look for ways to increase ACCESS visibility and usage. We acknowledge their rapid development and
efforts to survey the larger ACCESS community. We acknowledge the innovative approaches to build
communities by scanning badges at conferences, etc.
Strength 4.3: It is a positive development that RPs and standing committees will be included in the
calls for inputs to the newsletters.
Strength 4.4: Developing ‘personas’ is effective for targeting audiences and helping to structure
communications/outreach efforts.
Strength 4.5: Coordination of the ongoing work to change how the website(s) is/are being changed
shows [trying to say: move from “pieces make up the website” to “website is made up of pieces” is very
positive since it shows the projects starting to pull together towards the common good of ACCESS. This
willingness is a strength.
The ACO concurs with what is noted and has no further comment.
Opportunities to Improve
Recommendation 4.1: The committee is excited about the opportunities that lay ahead of the
communication team. We strongly encourage a pivot to a more proactive orientation, where the
communication team does more than disseminating information on various web pages, networking,
9
visiting campuses, and attending conferences. There is a pressing need to engage targeted groups, with
the aim of receiving feedback, creating awareness, and developing relationships.
In our Plan Year 3 Communications Plan, we provide for targeted campaigns and will work with the
Executive Council to identify priorities according to our personas as well as resources available,
including budget and time, to do more targeted work.
Recommendation 4.2: The potential shift that’s being considered from service-based structure of
website to personas-based structure has good potential to improve website useability. We look forward
to seeing progress on this in PY3, since it is understood that there are challenges with separate awards
having responsibility for their portions of the website. However, the approach to date is mostly passive,
in that information is created and the community must choose to engage. The panel encourages the
communications team to consider more active ways to engage communities, build diverse readership,
and ultimately lead to a more diverse and engaged user community for ACCESS.
Our web stories form the foundation of our communications platform, creating content we believe
those representing our personas will see themselves in. We actively promote those stories on our
social media, and our newsletters reach well beyond those already on our site. And we ask the
subjects of our stories to share on their own social media pages to try to introduce ACCESS to new
targets. So we are consistently reaching out with other tools to draw people to the ACCESS web
presence.
Recommendation 4.3: To develop a more proactive orientation, the panel encourages the
communication team to engage in separate, purposeful focus groups of graduate students,
under-represented communities, users known to be high-level influencers, and essential users.
Thankfully, we have had an opportunity to get feedback from focus groups conducted by our
ACCESS Support colleagues as well as getting feedback through surveys shared with 22,641
recipients engaged with ACCESS as our current resources do not allow for specific focus group
activities in addition to what we are already doing. We can work with our Evaluation Standing
Committee group to try to identify users from under-represented communities as well as the others
the panel has identified.
Recommendation 4.4: With acknowledgement of the value of sharing information through LinkedIn,
we encourage the communications team to broaden its outreach to scientific communities within
LinkedIn, engage in specific exploration of users with appropriate job titles and research interests, and
build its brand within LinkedIn in addition to posting information updates.
We appreciate this suggestion and want to add that we do use strategies such as reposting and
hashtags to engage beyond our page. We know there are private groups we could join, but
understanding each may have different rules for posting, this may require more bandwidth than our
team can afford at this time. However, it would be good to look into paid ads on LinkedIn to target
specific groups. If budget allows, that may be the most achievable way to broaden outreach.
10
Recommendation 4.5: Even without an official community building designation, your team members
will still be attending conferences and looking to build a stronger user base. We encourage you to
consider venues that host large communities of underrepresented populations, such as HACU, NSBE,
BDPA, BIT, NACME, etc.
The ACCESS staff has discussed which conferences to attend to reach underrepresented populations
and has done some reconnaissance, such as attending the Grace Hopper Celebration last year. The
direction has been to continue to attend both PEARC and SC to which Communications personnel
travel; given our capacity, our team is best positioned to provide tools for the rest of our ACCESS
colleagues who may also be attending conferences (such as TAPIA) to spread the word about
ACCESS. We also note here that this is a function of the now-suspended Community Building &
Engagement team. While the ACO and the broader program leadership are working to advance
efforts in this area, progress in this area is hampered by the absence of a CB&E effort.
Recommendation 4.6: The development of ‘personas’ should be continued and refined. One
suggestion is to leverage DEI goals in the development of some personas.
The Communications team will coordinate with the DEI working group on ways to position the
personas outputs (web work) to maximum impact. This is the first plan year we will leverage the
personas so we look forward to determining how their incorporation into our plans impact
communications successes.
Recommendation 4.7: The communications team should partner with Resource Providers so that at
least 2 survey items are included in RPs own sampling of its user community.
We have already begun greater engagement with our Resource Providers in developing
communications so this is something we will take under advisement and will work with the
Evaluation Standing Committee to see if we can get more focused feedback from them.
Recommendation 4.8: The communication team should continue to develop news pieces for select
Resource Providers and share opportunities with other Resource Providers.
We have begun the process of gathering news items from our Resource Providers to develop
bimonthly spotlights in our ACCESS Advance newsletter as part of our Plan Year 3 Communications
Plan.
Recommendation 4.9: The development of materials for use on mobile devices is a critical
opportunity to reach a vast set of users. If materials are not already aligned with mobile device use, the
committee encourages this advancement.
The ACO website is a responsive design and we encourage those ACCESS awardees whose sites are
not working well on mobile to make those adjustments. That said, the vast majority of our users
visit the site on a desktop.
11Evaluation - Rating: Achieved Results
The panel recognizes that a lot of work was completed this year, and that the ACO had to significantly
pivot this year in terms of leadership and direction. The panel was impressed with the amount of work
that had been completed by the evaluation team, but felt that the necessary work to complete goals
and decide on measures should have been further along at this point in time. In general, the panel was
hoping to see, by the end of the second year of funding, a clearer presentation of overall goals for
ACCESS and progress made, as well as overall goals for ACO and progress made.
Strengths
Strength 5.1: The reboot of the evaluation plan has been a positive step and there has been
substantial progress in the development of program-wide goals and associated metrics, as well as
ACO-specific goals and metrics.
Strength 5.2: The former is especially important as it reflects close collaboration with other services,
and helps to unify the EC/service awards in prioritizing collaborative efforts going forward.
Strength 5.3: In addition, the inclusion of an external evaluator is very positive, and indicates a strong
commitment to continuous improvement within the ACO team, as well as throughout ACCESS.
The ACO concurs with what is noted and has no further comment.
Opportunities to Improve
Recommendation 5.1: The selection process of measures to support the goals of the ACO, and
longitudinal tracking of those measures, should be transparent and available for review. The ones
presented during the site review created a good picture, but it is unclear how they were selected and
the totality of all measures that were available to be selected is not well understood.
Program-wide metrics, which were the metrics presented in the review, were developed through
the Evaluation Work Group. Each member of the working group (at least one representative per
Service Team) suggested metrics for each of the program-wide goals. From there the group
discussed and winnowed down the list to a few metrics for each goal. These were presented to the
full ACCESS team at the February in-person meeting for comment. As noted, the metrics are
considered initial versions and are now going through a review and revision via the Evaluation
Standing Committee in preparation for the PY3 survey. In summary, there was an extensive process
of development that led to these.
Recommendation 5.2: In addition, for the selected measures, the ACO team should include a measure
of confidence with these results.
12
We appreciate this suggestion and will confer with the Evaluation Standing Committee on its
potential implementation.
Recommendation 5.3: There is concern that ACO and ACCESS goals and measures are not fully
developed at this point in the life of the award.
While we share the panel’s concern in this area, we have been working diligently on both ACO and
ACCESS-wide goals. As noted in the review, we have prioritized ACCESS-wide goals given the need
to reformulate the ACCESS evaluation plan. It is worth noting that there is an opportunity to use the
ACCESS-wide goals to revise the ACO-specific goals and ensure that the ACO’s goals align with the
ACCESS-wide goals.
Recommendation 5.4: As the community facilitator/influencer, the ACO needs to take a stronger lead
in creating urgency to complete defining program-wide and ACO-specific metrics that will be
supported by the work of the entire community.
We appreciate the panel’s sense of urgency in defining metrics. The ACO has taken a significant
leadership role in the development of program-wide metrics. A current focus of the Evaluation
Standing Committee (which includes 2 ACO staff and the ACO External Evaluator) is on refining the
initial set of metrics that were presented at the review.
Recommendation 5.5: In addition, ACO should establish a baseline of ‘fixed’ metrics that can be
tracked over time to establish trends.
This is an excellent suggestion that we will take back to the Evaluation Standing Committee.
Recommendation 5.6: The ACO is urged to Identify mechanisms to improve the response rate for the
evaluation surveys.
We appreciate the panel highlighting the importance of improving response rates. This is a current
focus of the Evaluation Working Group.
Recommendation 5.7: The ACO should work with the RP Forum to understand the best methods of
approach to ensure higher RP response rates, and evaluate techniques to improve the ‘community’
response rate, perhaps with a focus on high-priority subsets of the community.
We agree with the panel that there is a need to improve RP survey response rates. There is an
ACCESS-wide effort to better engage with the RP Forum (and RPs more directly) on a wide-range of
issues, including improving the survey methodology and response rates.
Recommendation 5.8: The ACO should consider tapping into subject matter expertise on survey
methodology to improve questions posed and response rates.
The ACO External Evaluator has some expertise in survey methodology and she has directly
contributed to the development of the initial surveys. We will explore additional sources of
expertise.
13ACO Project Office - Rating: Achieved Results
Strengths
Strength 6.1: The role that the ACO plays in the organization of meetings is very good, with clear
delineation of tasks, decisions, and parking lot items.
Strength 6.2: This undoubtedly positively contributes to smoothly operating groups. In addition, the
developed tools appear to be well-used and provide transparency into the communication and sharing
of documents.
The ACO concurs with what is noted and has no further comment.
Opportunities to Improve
Recommendation 6.1: The panel recommends that ACCESS team members focus on risks, either
service-specific or program-wide, as a technique to not only manage those risks, but to develop goals,
plans and metrics.
We agree with the panel’s observations to utilize risks as opportunities to develop goals, plans and
metrics. The ACO currently notes risks internal to the ACO and program-wide risks which are
reviewed by the EC and contingencies executed as needed. The notion of using risk to drive
development of goals, plans and metrics is interesting and we will certainly investigate this
approach.
Recommendation 6.2: In this aspect, the ACO should consider how to more fully capture, track and
share the various risks that should become visible to the full ACCESS staff.
The ACO appreciates the panel's recommendation and will continue to search for ways to improve
the overall risk capturing and tracking process. The Risk Register which is created and maintained
by the ACO is fully accessible to all ACCESS staff, including but not limited to the ACO risks, Service
Team risks and program wide risks. As we further develop the goals and metrics, it will be natural to
develop the associated risks. This can be used in the manner suggested in the previous item, but
also provides an opportunity to remind the Service Teams of the risk register capability that the
ACO is providing and improve adoption. This can be in the context of improved transparency in
sharing of award-level risks as well.
Recommendation 6.3: The ACO should consider using some of the tools, perhaps SLACK, to augment
survey capabilities and capture risk/opportunities identified throughout the community.
The ACO appreciates the panel's recommendation to utilize SLACK as a platform to augment the
existing survey capabilities and plans to identify risks and opportunities for improvement within
the program and the broader community.
14
Appendix: Original PY2 Review Panel Report
We include here the original report received from the review panel for completeness.
15
Panel ID V242294
Summary Statement - Rating: Achieved Results
Overall the panel felt that a lot of progress had been made this year, in particular rebooting
evaluation efforts, continuing to lead and manage the governance structure, and aiming towards
a coalesced ACCESS structure as a whole. There are a number of elements in place that will be
helpful in achieving the goals (once finalized), although not all of the results are ready to be
reviewed. In particular, the panel would like to see urgency in developing ACCESS goals and
measures, to include DEI. The panel is eager to see how future engagements, including RP on
the EC and other management integrative steps, help build cohesion among all the ACCESS
elements. The panel would like to recognize the willingness, commitment, and capability of the
ACO to help ACCESS achieve all of its goals, even when challenging issues must be
addressed.
Broader Impact - Rating: Partially Achieved
While the potential scope of Broader Impacts can be intimidating, we see the ACCESS program
to be in a unique position to have the ability to shape the computational research communities
in important and very impactful ways. We suggest the EC, the EAB, the projects and other
stakeholders identify what broader impacts ACCESS could achieve that only ACCESS could
achieve, and then to focus on one or two of those. The panel had difficulty identifying what
broader impacts the ACO project achieved vs ACCESS Program achievements.
● Strengths:
○ Willingness to have the conversation about DEI and to leverage external
expertise when needed - this is the first step and incredibly important.
○ Efforts to increase diversity (including those by communications showcasing the
use of ACCESS resources by researchers at non-R1 and minority-serving
institutions) seems to be increasing allocations to non-R1 institution researchers.
○ ACO (and ACCESS) have in the last program year put indicators and tools in
place that allow for the evaluation, and when needed redirection, of efforts
related to Broader Impact. These measures are important and should continue to
receive necessary resources and priority.
● Opportunities to Improve:
○ In terms of DEI in the broader community, ACO can do more to take advantage
of the unique and influential position ACCESS has by:
■ Advocating within the EC to clearly define goals for DEI as they relate to
the broader community that ACCESS engages with.
■ Refining Personas and user journeys to actively invite, and perhaps even
prioritize, engagement with researcher communities that have not
historically engaged HPC resources.
■ Work within the EC to develop a coherent program-wide strategy that
involves all services to bring under-represented groups into the
computational science research community.
○ ACCESS (and ACO) are uniquely placed to make a real impact on DEI in the
broader community - if this is taken advantage of. This unique position to shape
the composition of the community is a privilege that should be seen as such.
○ Whereas measures supporting goals within the overall health of ACCESS and
how it is enabling greater science to occur were presented and good, it was not
clear how these measures were selected and how they will be tracked over a
longer period. It would be helpful to explain the methodology for selecting the
measures, and show the full picture of all available measures, to better
understand how ACCESS is moving science forward.
○ While communications that work towards achieving broader impact goals are
noted and ongoing, the ACO should pursue more concerted efforts at engaging
with traditionally non-HPC communities.
○ Showcasing how the science being enabled by ACCESS is contributing in
practical ways to society would allow for talking points when approaching
non-traditional HPC communities. Perhaps this could be done through metrics or
communications stories.
Intellectual Merit - Rating: Achieved Results
● Strengths:
The ACO team leadership includes appropriate expertise and a structure for performing
important functions such as project management, communications, and events. These groups
seem to be working well together. ACO leadership has shown agility and responsiveness by, for
example, shifting the role of the external evaluator to meet emerging needs and by integrating a
RACI-like model to clarify roles. In addition, the ACO PI has been innovative in creating regular
meetings with leadership of other Federally funded projects for co-advising and sharing
common management experiences and lessons learned. This is an important mechanism for
knowledge sharing and the development of best practices.
● Opportunities to Improve:
Although the ACO team has made great progress in building an organizational structure for
ACCESS governance, the panel feels that they should take more ownership for the overall
health of ACCESS and for making progress towards overall ACCESS goals. The ACO is the
only ACCESS entity that can take this overarching view within the scope of their work, and the
panel would like to see more leadership by example and efforts to influence working
groups/committees to achieve results in a timely manner. An example of this is being able to
deliver on unified goals (including DEI) and evaluation measures within the first half of the
funded award.
The ACO leadership team has experienced significant staff turnover including in its evaluation
team. It is currently engaging the challenges of quickly filling open positions and bringing new
people up to speed. It is not clear that ACO has considered how it can systematically develop
resilience through methods such as succession planning or reassignment of personnel from
other projects.
Program Governance, including EC and EAB - Rating: Partially Achieved
The panel recognizes that ACO is the facilitator of ACCESS-wide governance functions (e.g. the
EC, EAB and quarterly meetings) and may not be solely responsible for the strengths and
weaknesses of those functions. Nevertheless, whether it is via facilitation or influence amongst
its peers, the ACO is crucial to successful ACCESS-wide governance.
● Strengths:
There are a number of positive developments in ACCESS governance during the past year.
Establishing program-wide quarterly meetings is very beneficial to the program. The Executive
Council (EC) meets frequently, with good participation by stakeholders, and has developed a set
of collaborative procedures and decision-making processes. The ACO team reports that there is
an increasing level of trust and collaboration amongst EC members. The EC members (service
PIs) have demonstrated that they can effectively work together to develop important common
documents, such as a common response to PY1 review feedback, developing common PY2
review slides, and establishing program-wide goals.
The External Advisory Board (EAB) has strong members (including representatives of the
Researcher Advisory Council and Resource Provider Forum) and the level of EAB engagement
is strong. It is encouraging that participants are willing to contribute beyond just attending the
meetings. We hope this continues as members rotate on/off the EAB.
Finally, initiating interactions with other NSF (and NIH) programs that have similar management
structures is a positive step. While we understand the benefits of confidentiality, it would be
valuable to define take-aways from these discussions, including commonly-shared feedback for
NSF regarding management/coordination of future programs.
● Opportunities to Improve:
To strengthen the overall ACCESS community, set priorities, and build a sense of common
purpose, it would be beneficial to consider moving to a rotating leadership model, where
perhaps the ACCESS PIs rotate a chair position once a year. This builds a better
understanding of the overall health and operations of ACCESS and challenges/opportunities of
the multi-functioning collaboration.
We suggest that the EC make every effort to “get on the same page” regarding the overall
ACCESS goals within PY3. This will require hard conversations, and no doubt compromise, but
should then also end up with defined procedures and processes that benefit ACCESS
governance. With this unified vision and set of goals at the EC level, the staff within the separate
projects as well as the wider community as a whole can then more effectively and efficiently
work towards those goals. The ACO should build upon current efforts underway within the EC
towards harmony to achieve this wider goal.
While the EC generally functions well, there appear to be some areas of conflict. Dissension
within the EC represents a significant risk for ACCESS overall, and the ACO along with other
members of the EC should make every effort to resolve any internal conflicts. For example, the
External Evaluator report suggests some internal conflicts within the EC (and their respective
teams) about the appropriate role of the ACO within ACCESS. Also, the ACO team discussed a
recent, specific conflict related to an ACCESS-wide DEI discussion. We are pleased that the
ACO team has taken constructive steps to address this issue, including using an external
conflict resolution expert. Continued efforts to ensure that issue is resolved are important,
including perhaps a broader discussion regarding DEI priorities within the ACCESS team.
There are indications that interactions between ACCESS and Resource Providers need to be
improved - e.g. the low response rate from RPs to the RP survey and the modest RP
satisfaction rating. There should be strong coupling with substantive feedback in both directions
with the RP community. The RP-related slides in the “ACCESS Annual Surveys” presentation
and the document provided to the panel “Improving the ACCESS Program RP Pathways
(March 20, 2024)” outline a number of constructive steps that the EC plans to take to address
this issue. We look forward to improvements in the coming year.
There is concern that the ACO is not fully attuned to all of the work occurring between working
groups and standing committees. Many important functions are being discussed within these
populations, and having a member of the ACO in the group is likely not strong enough
positioning to be able to understand the work and how it fits into the overall health and structure
of ACCESS - which is a primary concern of the ACO. We encourage a more robust system of
bringing the ACO up to date on all activities and work within the committee/group structures,
and inviting leaders to attend your meetings on a regular basis, so that the communication and
group work is better supported by your team to help achieve all desired outcomes.
Communication and Outreach - Rating: Achieved Results
● Strengths:
The communication team has skills sufficient to their tasks, a broad knowledge of the tools at
their disposal, and a clear understanding of industry standards. The team has been active in
generating a brand and furthering its identity. The initial survey forays were encouraging, and
the team is poised to take on new challenges. We have confidence in their current efforts and
future endeavors.
The communications team are using excellent metrics to understand their impact and look for
ways to increase ACCESS visibility and usage. We acknowledge their rapid development and
efforts to survey the larger ACCESS community. We acknowledge the innovative approaches to
build communities by scanning badges at conferences, etc. h
○ It is a positive development that RPs and standing committees will be included in
the calls for inputs to the newsletters.
○ Developing ‘personas’ is effective for targeting audiences and helping to structure
communications/outreach efforts.
○ Coordination of the ongoing work to change how the website(s) is/are being
changed shows [trying to say: move from “pieces make up the website” to
“website is made up of pieces” is very positive since it shows the projects starting
to pull together towards the common good of ACCESS. This willingness is a
strength.
● Opportunities to Improve:
The committee is excited about the opportunities that lay ahead of the communication team. We
strongly encourage a pivot to a more proactive orientation, where the communication team does
more than disseminating information on various webpages, networking, visiting campuses, and
attending conferences. There is a pressing need to engage targeted groups, with the aim of
receiving feedback, creating awareness, and developing relationships.
The potential shift that’s being considered from service-based structure of website to
personas-based structure has good potential to improve website useability. We look forward to
seeing progress on this in PY3, since it is understood that there are challenges with separate
awards having responsibility for their portions of the website. However, the approach to date is
mostly passive, in that information is created and the community must choose to engage. The
panel encourages the communications team to consider more active ways to engage
communities, build diverse readership, and ultimately lead to a more diverse and engaged user
community for ACCESS.
○ To develop a more proactive orientation, the panel encourages the
communication team to engage in separate, purposeful focus groups of graduate
students, under-represented communities, users known to be high-level
influencers, and essential users.
○ With acknowledgement of the value of sharing information through LinkedIn, we
encourage the communications team to broaden its outreach to scientific
communities within LinkedIn, engage in specific exploration of users with
appropriate job titles and research interests, and build its brand within LinkedIn in
addition to posting information updates.
○ Even without an official community building designation, your team members will
still be attending conferences and looking to build a stronger user base. We
encourage you to consider venues that host large communities of
underrepresented populations, such as HACU, NSBE, BDPA, BIT, NACME, etc.
○ The development of ‘personas’ should be continued and refined. One suggestion
is to leverage DEI goals in the development of some personas.
○ The communications team should partner with Resource Providers so that at
least 2 survey items are included in RPs own sampling of its user community.
○ The communication team should continue to develop news pieces for select
Resource Providers and share opportunities with other Resource Providers.
○ The development of materials for use on mobile devices is a critical opportunity
to reach a vast set of users. If materials are not already aligned with mobile
device use, the committee encourages this advancement.
Evaluation - Rating: Achieved Results
The panel recognizes that a lot of work was completed this year, and that the ACO had to
significantly pivot this year in terms of leadership and direction. The panel was impressed with
the amount of work that had been completed by the evaluation team, but felt that the necessary
work to complete goals and decide on measures should have been further along at this point in
time. In general, the panel was hoping to see, by the end of the second year of funding, a
clearer presentation of overall goals for ACCESS and progress made, as well as overall goals
for ACO and progress made.
● Strengths:
The reboot of the evaluation plan has been a positive step and there has been substantial
progress in the development of program-wide goals and associated metrics, as well as
ACO-specific goals and metrics. The former is especially important as it reflects close
collaboration with other services, and helps to unify the EC/service awards in prioritizing
collaborative efforts going forward. In addition, the inclusion of an external evaluator is very
positive, and indicates a strong commitment to continuous improvement within the ACO team,
as well as throughout ACCESS.
● Opportunities to Improve:
The selection process of measures to support the goals of the ACO, and longitudinal tracking of
those measures, should be transparent and available for review. The ones presented during the
site review created a good picture, but it is unclear how they were selected and the totality of all
measures that were available to be selected is not well understood. In addition, for the selected
measures, the ACO team should include a measure of confidence with these results.
There is concern that ACO and ACCESS goals and measures are not fully developed at this
point in the life of the award. As the community facilitator/influencer, the ACO needs to take a
stronger lead in creating urgency to complete defining program-wide and ACO-specific metrics
that will be supported by the work of the entire community. In addition, ACO should establish a
baseline of ‘fixed’ metrics that can be tracked over time to establish trends.
The ACO is urged to Identify mechanisms to improve the response rate for the evaluation
surveys. The ACO should work with the RP Forum to understand the best methods of approach
to ensure higher RP response rates, and evaluate techniques to improve the ‘community’
response rate, perhaps with a focus on high-priority subsets of the community. The ACO should
consider tapping into subject matter expertise on survey methodology to improve questions
posed and response rates.
ACO Project Office - Rating: Achieved Results
● Strengths:
The role that the ACO plays in the organization of meetings is very good, with clear delineation
of tasks, decisions, and parking lot items. This undoubtedly positively contributes to smoothly
operating groups. In addition, the developed tools appear to be well-used and provide
transparency into the communication and sharing of documents.
● Opportunities to Improve:
The panel recommends that ACCESS team members focus on risks, either service-specific or
program-wide, as a technique to not only manage those risks, but to develop goals, plans and
metrics. In this aspect, the ACO should consider how to more fully capture, track and share the
various risks that should become visible to the full ACCESS staff.
The ACO should consider using some of the tools, perhaps SLACK, to augment survey
capabilities and capture risk/opportunities identified throughout the community.
The summary was read to/by the panel, and the panel concurred that the summary accurately
reflects the panel discussion.