ACCESS Coordination Office Response to the PY2 Review Panel Summary

[Submitted July 24, 2024]

Summary Statement - Rating: Achieved Results

Overall the panel felt that a lot of progress had been made this year, in particular rebooting evaluation efforts, continuing to lead and manage the governance structure, and aiming towards a coalesced ACCESS structure as a whole. There are a number of elements in place that will be helpful in achieving the goals (once finalized), although not all of the results are ready to be reviewed. In particular, the panel would like to see urgency in developing ACCESS goals and measures, to include DEI. The panel is eager to see how future engagements, including RP on the EC and other management integrative steps, help build cohesion among all the ACCESS elements. The panel would like to recognize the willingness, commitment, and capability of the ACO to help ACCESS achieve all of its goals, even when challenging issues must be addressed.

We appreciate the Panel's time, attention, and thoughtful comments provided to this review. The ACO is committed to providing the community with the highest level of service possible and strives to be responsible stewards of the support provided by NSF to the ACO on behalf of ACCESS.

We agree with many of the Panel's comments below. The ACO continues to do more than was scoped for it and has augmented the project with significant contributed resources to compensate for what we believe is an underfunded award for the expected scope. Moving into PY3 we believe it is essential for the EC to be more actively involved in prioritizing work across the program so that available resources can be put to use where they matter most. We will continue to put these important topics before the EC and provide the tools and mechanisms to make progress.

We appreciate the Panel's acknowledgment of the progress made since the last review. The ACO team has worked hard to address issues raised in the prior review as well as those that have emerged since that time. Progress on program-wide goals and metrics, and other program-wide initiatives is paced by the contributions and engagement from the other teams.

We appreciate the Panel's acknowledgment of the human and material resource issues. As presented during the review, it is important to note that the ACO has a total headcount of approximately 2.6 FTE. Additional contributed effort by the NCSA and SDSC partners has been necessary to make progress in key areas such as evaluation workgroup process, general program governance, program-wide strategic planning, and other activities.

The remainder of this response document focuses on the “Opportunities to Improve” indicated by the Panel. This is not to discount the recognition of the Strengths noted in each area and we do appreciate the Panel’s noting of those. In addition, there are some noted Strengths that we also provide comments regarding. Still, it is the case that improvement of the project, and by extension, the ACCESS program as a whole, must focus on the areas of potential improvement. To provide clarity, below the contents of the panel report are italicized and our comments and responses to that are in normal text.

PY2 Review Response Tasks

Responses to Observations and Opportunities for Improvement Indicated by the Review Panel

1.      Broader Impact - Rating: Partially Achieved

While the potential scope of Broader Impacts can be intimidating, we see the ACCESS program to be in a unique position to have the ability to shape the computational research communities in important and very impactful ways. We suggest the EC, the EAB, the projects and other stakeholders identify what broader impacts ACCESS could achieve that only ACCESS could achieve, and then to focus on one or two of those. The panel had difficulty identifying what broader impacts the ACO project achieved vs ACCESS Program achievements.

Strengths

Strength 1.1: Willingness to have the conversation about DEI and to leverage external expertise when needed - this is the first step and incredibly important.

Strength 1.2: Efforts to increase diversity (including those by communications showcasing the use of ACCESS resources by researchers at non-R1 and minority-serving institutions) seems to be increasing allocations to non-R1 institution researchers.

Strength 1.3: ACO (and ACCESS) have in the last program year put indicators and tools in place that allow for the evaluation, and when needed redirection, of efforts related to Broader Impact. These measures are important and should continue to receive necessary resources and priority.

The ACO appreciates these observations and has no further comment.

Opportunities to Improve

 Recommendation 1.1: In terms of DEI in the broader community, ACO can do more to take advantage of the unique and influential position ACCESS has by:

• Advocating within the EC to clearly define goals for DEI as they relate to the broader community that ACCESS engages with.

• Refining Personas and user journeys to actively invite, and perhaps even prioritize, engagement with researcher communities that have not historically engaged HPC resources.

The ACO concurs with this recommendation but emphasizes for both of the bullets above that while the ACO can and does advocate for these things via the EC, Working Groups, and Standing Committees, it is the the EC that must make decisions regarding prioritization and allocation of staff time to support these program-wide efforts.

Recommendation 1.2: Work within the EC to develop a coherent program-wide strategy that involves all services to bring under-represented groups into the computational science research community.

The ACO concurs with this recommendation though sees it as a recommendation primarily to the EC and not specifically to the ACO. The ACO is already advocating and expending effort to support this. The EC concurs with this priority and thus chartered the DEI Working Group to address part of this. As we noted during our presentation the Community Building & Engagement (CB&E) efforts have been largely suspended in anticipation of further guidance from NSF on how the program should proceed in this space. The ACO took it upon itself to facilitate discussion during the February Quarterly Meeting to identify the priority events via which ACCESS should engage with the community for PY3. This would normally have been a function of the program-wide CB&E team.

Recommendation 1.3: ACCESS (and ACO) are uniquely placed to make a real impact on DEI in the broader community - if this is taken advantage of. This unique position to shape the composition of the community is a privilege that should be seen as such.

We take the panel’s recommendation very seriously and agree that ACCESS can and should do more to impact DEI in the community. As the panel notes, this is a recommendation to both the EC and the ACO. We will continue to use our role as the EC facilitator and peer member of the team to ensure that DEI remains a top priority of the program. We look forward to the DEI Working Group – in which the ACO participates – bringing recommendations on this to the EC as soon as possible.

 Recommendation 1.4: Whereas measures supporting goals within the overall health of ACCESS and how it is enabling greater science to occur were presented and good, it was not clear how these measures were selected and how they will be tracked over a longer period. It would be helpful to explain the methodology for selecting the measures, and show the full picture of all available measures, to better understand how ACCESS is moving science forward.

The ACO appreciates this observation by the review panel, and the opportunity to provide some context. Fundamentally, the metrics are intended to provide indicators of progress toward realizing the program-wide goals that were developed through a highly collaborative process involving all staff across the program. The metrics presented in the shared team slides were put forth by the Evaluation Working Group following an initial brainstorming session at the September 2023 Quarterly meeting facilitated by ACO PI Towns. We view these as v1 of the program-wide metrics and will refine these over time with broad program input. The Evaluation Standing Committee is now reviewing the metrics in light of the results from the 3 ACCESS surveys and other program developments and will bring forth recommendations for consideration. Time is of the essence as this work will impact PY3 surveys.

Recommendation 1.5: While communications that work towards achieving broader impact goals are noted and ongoing, the ACO should pursue more concerted efforts at engaging with traditionally non-HPC communities.

The ACO recognizes the need to focus on communications with traditionally non-HPC communities. In collaboration with the other awardee teams, the ACO continues to seek means by which we can be more effective in this area. Some of those efforts were shared during the review, but others have not been pursued due to resource and staff time constraints. Nonetheless, the ACO will work with the Executive Council to identify additional means to engage the broader community within our resource constraints.

Recommendation 1.6: Showcasing how the science being enabled by ACCESS is contributing in practical ways to society would allow for talking points when approaching non-traditional HPC communities. Perhaps this could be done through metrics or communications stories.

The ACO does highlight the impact of the science supported through science stories, events support, annual publications, and social media. This recommendation suggests the ACCESS Communications Standing Committee consider spending some time distilling talking points from these stories to support those directly engaging with non-traditional HPC communities as well as others. Perhaps these talking points can be maintained on the ACCESS Communications Standing Committee wiki pages as a resource to any ACCESS staff member engaging members of non-traditional HPC communities.

2.      Intellectual Merit - Rating: Achieved Results

Strengths

Strength 2.1: The ACO team leadership includes appropriate expertise and a structure for performing important functions such as project management, communications, and events. These groups seem to be working well together. ACO leadership has shown agility and responsiveness by, for example, shifting the role of the external evaluator to meet emerging needs and by integrating a RACI-like model to clarify roles.

Strength 2.2: In addition, the ACO PI has been innovative in creating regular meetings with leadership of other Federally funded projects for co-advising and sharing common management experiences and lessons learned. This is an important mechanism for knowledge sharing and the development of best practices.

The ACO concurs with what is noted and has no further comment. 

Opportunities to Improve

Recommendation 2.1: Although the ACO team has made great progress in building an organizational structure for ACCESS governance, the panel feels that they should take more ownership for the overall health of ACCESS and for making progress towards overall ACCESS goals.

We concur with the panel that there are opportunities for the ACO to take more ownership for the overall health of ACCESS, but with the caveat that that ACO’s influence is limited. While we do take every effort to identify and focus the EC’s collective efforts on matters of importance, forward progress is determined by the contributions of individual team members and the leadership of the PIs. It is our sense that there is now, more than ever before in the program, an understanding by ACCESS leadership that progress beyond single award teams will only come through better collaboration and teamwork. We are seeing positive signs across the program and will do our part to support these efforts.

 Recommendation 2.2: The ACO is the only ACCESS entity that can take this overarching view within the scope of their work, and the panel would like to see more leadership by example and efforts to influence working groups/committees to achieve results in a timely manner. An example of this is being able to deliver on unified goals (including DEI) and evaluation measures within the first half of the funded award.

We appreciate the Panel’s perspective on this and agree to a point that the ACO can do more to influence Working Group/Standing Committee output. Again though, there are limitations to our influence. While the role of the ACO is certainly more oriented toward a global view, it is incumbent on all of the PIs, as part of the EC, to take this overarching view of the program. The ACO feels that it has led by example and shared a number of examples of this during the review (facilitating development of program-wide goals and metrics, facilitating the reboot of evaluation efforts, advocating for the formation of the DEI Working Group, etc.). We appreciate the panel’s recognition of the ACOs efforts here. However, the ACO simply is not staffed at a level necessary to take a leading role in all program-wide efforts. To the extent the ACO can, we will continue to lead by example with opportunities to do so on the horizon.

Recommendation 2.3: The ACO leadership team has experienced significant staff turnover including in its evaluation team. It is currently engaging the challenges of quickly filling open positions and bringing new people up to speed. It is not clear that ACO has considered how it can systematically develop resilience through methods such as succession planning or reassignment of personnel from other projects.

The ACO has considered how we can improve resilience in staff turn-over, and considers this an important issue to manage. With a total staff level of 2.58 FTE with individuals possessing disparate skill sets, there are minimal areas of redundancy.. Still, we have made efforts to address this, particularly with respect to the PIs but also for all staff members. In some cases, it is reasonable to consider re-assigning staff from other projects, but this clearly runs the risk of creating the same problem for another project. Nonetheless, we have done exactly this in some cases (e.g., Sr. Project Manager Shannon Bradley being reassigned to support the role of Sr. Project Manager Ron Payne following his departure from the Center). It is more challenging to address this with respect to the PI and co-PI, however, the PI and co-PI stand ready to support each other’s current roles should one of them become unavailable to the project until a replacement can be identified.

3.      Program Governance, including EC and EAB - Rating: Partially Achieved

The panel recognizes that ACO is the facilitator of ACCESS-wide governance functions (e.g. the EC, EAB and quarterly meetings) and may not be solely responsible for the strengths and weaknesses of those functions. Nevertheless, whether it is via facilitation or influence amongst its peers, the ACO is crucial to successful ACCESS-wide governance.

Strengths

There are a number of positive developments in ACCESS governance during the past year.

Strength 3.1: Establishing program-wide quarterly meetings is very beneficial to the program. The Executive Council (EC) meets frequently, with good participation by stakeholders, and has developed a set of collaborative procedures and decision-making processes. The ACO team reports that there is an increasing level of trust and collaboration amongst EC members. The EC members (service PIs) have demonstrated that they can effectively work together to develop important common documents, such as a common response to PY1 review feedback, developing common PY2 review slides, and establishing program-wide goals.

Strength 3.2: The External Advisory Board (EAB) has strong members (including representatives of the Researcher Advisory Council and Resource Provider Forum) and the level of EAB engagement is strong. It is encouraging that participants are willing to contribute beyond just attending the meetings. We hope this continues as members rotate on/off the EAB.

Strength 3.3: Finally, initiating interactions with other NSF (and NIH) programs that have similar management structures is a positive step. While we understand the benefits of confidentiality, it would be valuable to define take-aways from these discussions, including commonly-shared feedback for NSF regarding management/coordination of future programs.

The ACO concurs with what is noted. The ACO PI will consider how to distill useful feedback to NSF from the informal co-advisory group. The value of this is evident, but the need to maintain confidentiality is key to the success of those interactions.

Opportunities to Improve

Recommendation 3.1: To strengthen the overall ACCESS community, set priorities, and build a sense of common purpose, it would be beneficial to consider moving to a rotating leadership model, where perhaps the ACCESS PIs rotate a chair position once a year. This builds a better understanding of the overall health and operations of ACCESS and challenges/opportunities of the multi-functioning collaboration.

The ACO concurs with this recommendation and will advocate for this with the rest of the EC. The ACO did make this recommendation during the development of the EC charter. At that time the Service Team PIs did not feel this was warranted, rejecting also the suggestion that there be a Chair of the EC at all.

Recommendation 3.2: We suggest that the EC make every effort to “get on the same page” regarding the overall ACCESS goals within PY3. This will require hard conversations, and no doubt compromise, but should then also end up with defined procedures and processes that benefit ACCESS governance.

The ACO concurs that the EC members should be “on the same page” regarding overall ACCESS goals, but believes that there is in fact already significant alignment. The issue raised in the review discussions was with respect to the prioritization of efforts across the program and lack of complete alignment there, an observation with which the ACO fully agrees. The ACO has initiated a process to work toward a shared view of priorities.

Recommendation 3.3: With this unified vision and set of goals at the EC level, the staff within the separate projects as well as the wider community as a whole can then more effectively and efficiently work towards those goals. The ACO should build upon current efforts underway within the EC towards harmony to achieve this wider goal.

The ACO Concurs with this recommendation and has already initiated efforts to build upon this.

Recommendation 3.4: While the EC generally functions well, there appear to be some areas of conflict. Dissension within the EC represents a significant risk for ACCESS overall, and the ACO along with other members of the EC should make every effort to resolve any internal conflicts. For example, the External Evaluator report suggests some internal conflicts within the EC (and their respective teams) about the appropriate role of the ACO within ACCESS.

The ACO concurs with this recommendation. As the annual reviews of the program awardees complete and follow-up activities are done, the ACO will be bringing these issues to the EC for discussion. The ACO sees it as critical for there to be a shared understanding of its scope in order for it to have the opportunity for success.

To illustrate the point, we call the panel’s attention to a comment made by the ACO evaluator in her PY2 report (which was provided to the panel): “Without breaching confidentiality agreements, it is safe to say that the Service Track PIs have very different, in some cases irreconcilable views about the role of the ACO. In particular, when a conflict arises among members of the Service Tracks, some of the Service Track representatives look to the ACO to address the conflict. Other Service Track representatives believe the ACO should not take the lead in this way. Acting as a Track representative with an opinion on a contentious issue can easily be at odds with acting as a meeting facilitator.”

Recommendation 3.5: Also, the ACO team discussed a recent, specific conflict related to an

ACCESS-wide DEI discussion. We are pleased that the ACO team has taken constructive steps to address this issue, including using an external conflict resolution expert. Continued efforts to ensure that issue is resolved are important, including perhaps a broader discussion regarding DEI priorities within the ACCESS team.

The ACO concurs with this recommendation and sees it also as a recommendation to the EC. The ACO appreciates the panel’s recognition of the importance of resolving conflict within the EC. The facilitated discussion is seen as a first step in a process to first gain alignment among the five PIs, followed by expanding this to the EC and then further to all of the ACCESS awardee teams. While this has a particular focus on DEI issues and priority of DEI efforts across the program, it is also intended to address alignment more generally.

Recommendation 3.6: There are indications that interactions between ACCESS and Resource Providers need to be improved - e.g. the low response rate from RPs to the RP survey and the modest RP satisfaction rating.

The ACO concurs with this recommendation and noted in the review the collective effort now underway by the program to better engage with RPs. These efforts are outlined in the document we provided to the panel, “Improving the ACCESS Program RP Pathways (March 20, 2024)”. This includes, among other recommendations, having the RP Forum Chair join the EC (a step already taken).

Recommendation 3.7: There should be strong coupling with substantive feedback in both directions with the RP community. The RP-related slides in the “ACCESS Annual Surveys” presentation and the document provided to the panel “Improving the ACCESS Program RP Pathways (March 20, 2024)” outline a number of constructive steps that the EC plans to take to address this issue. We look forward to improvements in the coming year.

The ACO concurs with this recommendation and will be advocating these changes and their implementation in the EC.

 Recommendation 3.8: There is concern that the ACO is not fully attuned to all of the work occurring between working groups and standing committees. Many important functions are being discussed within these populations, and having a member of the ACO in the group is likely not strong enough positioning to be able to understand the work and how it fits into the overall health and structure of ACCESS - which is a primary concern of the ACO.

While it would be ideal if the ACO (and more importantly the EC) to be fully attuned to all of the work occurring between Working Groups and Standing Committee, it is simply beyond the scope of the ACO to do so. The ACO has established structure for working groups and standing committees, and strives to be cognizant of the efforts across the program, but it is challenging for the ACO to deliver on its many other obligations to allocate the level of staff time necessary for the oversight activity the panel suggests. Further, not all work that has a program-wide impact is being done within Working Groups and Standing Committees, but within specific teams. The situation is improving as the need for greater cross-team communication and transparency becomes more evident. The ACO is responsible for scheduling regular updates of the working group and standing committees to the EC and those have not happened as frequently as they should due to other pressing priorities. The ACO is already moving to rectify this.

Recommendation 3.9: We encourage a more robust system of bringing the ACO up to date on all activities and work within the committee/group structures, and inviting leaders to attend your meetings on a regular basis, so that the communication and group work is better supported by your team to help achieve all desired outcomes.

The ACO generally concurs with this, but sees it as an issue for the EC and the ACO as part of the EC. See response to Recommendation 3.8 above.

4.      Communication and Outreach - Rating: Achieved Results

Strengths

Strength 4.1: The communication team has skills sufficient to their tasks, a broad knowledge of the tools at their disposal, and a clear understanding of industry standards. The team has been active in generating a brand and furthering its identity. The initial survey forays were encouraging, and the team is poised to take on new challenges. We have confidence in their current efforts and future endeavors. 

Strength 4.2: The communications team are using excellent metrics to understand their impact and look for ways to increase ACCESS visibility and usage. We acknowledge their rapid development and efforts to survey the larger ACCESS community. We acknowledge the innovative approaches to build communities by scanning badges at conferences, etc.

Strength 4.3: It is a positive development that RPs and standing committees will be included in the calls for inputs to the newsletters.

Strength 4.4: Developing ‘personas’ is effective for targeting audiences and helping to structure communications/outreach efforts.

Strength 4.5: Coordination of the ongoing work to change how the website(s) is/are being changed shows [trying to say: move from “pieces make up the website” to “website is made up of pieces” is very positive since it shows the projects starting to pull together towards the common good of ACCESS. This willingness is a strength.

The ACO concurs with what is noted and has no further comment.

Opportunities to Improve

Recommendation 4.1: The committee is excited about the opportunities that lay ahead of the communication team. We strongly encourage a pivot to a more proactive orientation, where the communication team does more than disseminating information on various web pages, networking, visiting campuses, and attending conferences. There is a pressing need to engage targeted groups, with the aim of receiving feedback, creating awareness, and developing relationships.

In our Plan Year 3 Communications Plan, we provide for targeted campaigns and will work with the Executive Council to identify priorities according to our personas as well as resources available, including budget and time, to do more targeted work.

Recommendation 4.2: The potential shift that’s being considered from service-based structure of website to personas-based structure has good potential to improve website useability. We look forward to seeing progress on this in PY3, since it is understood that there are challenges with separate awards having responsibility for their portions of the website. However, the approach to date is mostly passive, in that information is created and the community must choose to engage. The panel encourages the communications team to consider more active ways to engage communities, build diverse readership, and ultimately lead to a more diverse and engaged user community for ACCESS.

Our web stories form the foundation of our communications platform, creating content we believe those representing our personas will see themselves in. We actively promote those stories on our social media, and our newsletters reach well beyond those already on our site. And we ask the subjects of our stories to share on their own social media pages to try to introduce ACCESS to new targets. So we are consistently reaching out with other tools to draw people to the ACCESS web presence.

 Recommendation 4.3: To develop a more proactive orientation, the panel encourages the communication team to engage in separate, purposeful focus groups of graduate students, under-represented communities, users known to be high-level influencers, and essential users.

Thankfully, we have had an opportunity to get feedback from focus groups conducted by our ACCESS Support colleagues as well as getting feedback through surveys shared with 22,641 recipients engaged with ACCESS as our current resources do not allow for specific focus group activities in addition to what we are already doing. We can work with our Evaluation Standing Committee group to try to identify users from under-represented communities as well as the others the panel has identified.

 Recommendation 4.4: With acknowledgement of the value of sharing information through LinkedIn, we encourage the communications team to broaden its outreach to scientific communities within LinkedIn, engage in specific exploration of users with appropriate job titles and research interests, and build its brand within LinkedIn in addition to posting information updates.

 We appreciate this suggestion and want to add that we do use strategies such as reposting and hashtags to engage beyond our page. We know there are private groups we could join, but understanding each may have different rules for posting, this may require more bandwidth than our team can afford at this time. However, it would be good to look into paid ads on LinkedIn to target specific groups. If budget allows, that may be the most achievable way to broaden outreach.

Recommendation 4.5: Even without an official community building designation, your team members will still be attending conferences and looking to build a stronger user base. We encourage you to consider venues that host large communities of underrepresented populations, such as HACU, NSBE, BDPA, BIT, NACME, etc.

The ACCESS staff has discussed which conferences to attend to reach underrepresented populations and has done some reconnaissance, such as attending the Grace Hopper Celebration last year. The direction has been to continue to attend both PEARC and SC to which Communications personnel travel; given our capacity, our team is best positioned to provide tools for the rest of our ACCESS colleagues who may also be attending conferences (such as TAPIA) to spread the word about ACCESS. We also note here that this is a function of the now-suspended Community Building & Engagement team. While the ACO and the broader program leadership are working to advance efforts in this area, progress in this area is hampered by the absence of a CB&E effort.

 Recommendation 4.6: The development of ‘personas’ should be continued and refined. One suggestion is to leverage DEI goals in the development of some personas.

The Communications team will coordinate with the DEI working group on ways to position the personas outputs (web work) to maximum impact. This is the first plan year we will leverage the personas so we look forward to determining how their incorporation into our plans impact communications successes.

Recommendation 4.7: The communications team should partner with Resource Providers so that at least 2 survey items are included in RPs own sampling of its user community.

We have already begun greater engagement with our Resource Providers in developing communications so this is something we will take under advisement and will work with the Evaluation Standing Committee to see if we can get more focused feedback from them.

 Recommendation 4.8: The communication team should continue to develop news pieces for select Resource Providers and share opportunities with other Resource Providers.

We have begun the process of gathering news items from our Resource Providers to develop bimonthly spotlights in our ACCESS Advance newsletter as part of our Plan Year 3 Communications Plan.

Recommendation 4.9: The development of materials for use on mobile devices is a critical opportunity to reach a vast set of users. If materials are not already aligned with mobile device use, the committee encourages this advancement.

The ACO website is a responsive design and we encourage those ACCESS awardees whose sites are not working well on mobile to make those adjustments. That said, the vast majority of our users visit the site on a desktop.

5.      Evaluation - Rating: Achieved Results

The panel recognizes that a lot of work was completed this year, and that the ACO had to significantly pivot this year in terms of leadership and direction. The panel was impressed with the amount of work that had been completed by the evaluation team, but felt that the necessary work to complete goals and decide on measures should have been further along at this point in time. In general, the panel was hoping to see, by the end of the second year of funding, a clearer presentation of overall goals for ACCESS and progress made, as well as overall goals for ACO and progress made.

Strengths

Strength 5.1: The reboot of the evaluation plan has been a positive step and there has been substantial progress in the development of program-wide goals and associated metrics, as well as ACO-specific goals and metrics.

Strength 5.2: The former is especially important as it reflects close collaboration with other services, and helps to unify the EC/service awards in prioritizing collaborative efforts going forward.

Strength 5.3: In addition, the inclusion of an external evaluator is very positive, and indicates a strong commitment to continuous improvement within the ACO team, as well as throughout ACCESS.

The ACO concurs with what is noted and has no further comment.

Opportunities to Improve

Recommendation 5.1: The selection process of measures to support the goals of the ACO, and longitudinal tracking of those measures, should be transparent and available for review. The ones presented during the site review created a good picture, but it is unclear how they were selected and the totality of all measures that were available to be selected is not well understood.

Program-wide metrics, which were the metrics presented in the review, were developed through the Evaluation Work Group. Each member of the working group (at least one representative per Service Team) suggested metrics for each of the program-wide goals. From there the group discussed and winnowed down the list to a few metrics for each goal. These were presented to the full ACCESS team at the February in-person meeting for comment. As noted, the metrics are considered initial versions and are now going through a review and revision via the Evaluation Standing Committee in preparation for the PY3 survey. In summary, there was an extensive process of development that led to these.

 Recommendation 5.2: In addition, for the selected measures, the ACO team should include a measure of confidence with these results.

We appreciate this suggestion and will confer with the Evaluation Standing Committee on its potential implementation.

 Recommendation 5.3: There is concern that ACO and ACCESS goals and measures are not fully developed at this point in the life of the award.

While we share the panel’s concern in this area, we have been working diligently on both ACO and ACCESS-wide goals. As noted in the review, we have prioritized ACCESS-wide goals given the need to reformulate the ACCESS evaluation plan. It is worth noting that there is an opportunity to use the ACCESS-wide goals to revise the ACO-specific goals and ensure that the ACO’s goals align with the ACCESS-wide goals.

Recommendation 5.4: As the community facilitator/influencer, the ACO needs to take a stronger lead in creating urgency to complete defining program-wide and ACO-specific metrics that will be supported by the work of the entire community.

 We appreciate the panel’s sense of urgency in defining metrics. The ACO has taken a significant leadership role in the development of program-wide metrics. A current focus of the Evaluation Standing Committee (which includes 2 ACO staff and the ACO External Evaluator) is on refining the initial set of metrics that were presented at the review.

Recommendation 5.5: In addition, ACO should establish a baseline of ‘fixed’ metrics that can be tracked over time to establish trends.

This is an excellent suggestion that we will take back to the Evaluation Standing Committee.

Recommendation 5.6: The ACO is urged to Identify mechanisms to improve the response rate for the evaluation surveys.

We appreciate the panel highlighting the importance of improving response rates. This is a current focus of the Evaluation Working Group.

Recommendation 5.7: The ACO should work with the RP Forum to understand the best methods of approach to ensure higher RP response rates, and evaluate techniques to improve the ‘community’ response rate, perhaps with a focus on high-priority subsets of the community.

We agree with the panel that there is a need to improve RP survey response rates. There is an ACCESS-wide effort to better engage with the RP Forum (and RPs more directly) on a wide-range of issues, including improving the survey methodology and response rates.

Recommendation 5.8: The ACO should consider tapping into subject matter expertise on survey methodology to improve questions posed and response rates.

The ACO External Evaluator has some expertise in survey methodology and she has directly contributed to the development of the initial surveys. We will explore additional sources of expertise.

6.      ACO Project Office - Rating: Achieved Results

Strengths

Strength 6.1: The role that the ACO plays in the organization of meetings is very good, with clear delineation of tasks, decisions, and parking lot items.

Strength 6.2: This undoubtedly positively contributes to smoothly operating groups. In addition, the developed tools appear to be well-used and provide transparency into the communication and sharing of documents.

The ACO concurs with what is noted and has no further comment.

Opportunities to Improve

Recommendation 6.1: The panel recommends that ACCESS team members focus on risks, either service-specific or program-wide, as a technique to not only manage those risks, but to develop goals, plans and metrics.

We agree with the panel’s observations to utilize risks as opportunities to develop goals, plans and metrics. The ACO currently notes risks internal to the ACO and program-wide risks which are reviewed by the EC and contingencies executed as needed. The notion of using risk to drive development of goals, plans and metrics is interesting and we will certainly investigate this approach.

Recommendation 6.2: In this aspect, the ACO should consider how to more fully capture, track and share the various risks that should become visible to the full ACCESS staff.

The ACO appreciates the panel's recommendation and will continue to search for ways to improve the overall risk capturing and tracking process. The Risk Register which is created and maintained by the ACO is fully accessible to all ACCESS staff, including but not limited to the ACO risks, Service Team risks and program wide risks. As we further develop the goals and metrics, it will be natural to develop the associated risks. This can be used in the manner suggested in the previous item, but also provides an opportunity to remind the Service Teams of the risk register capability that the ACO is providing and improve adoption. This can be in the context of improved transparency in sharing of award-level risks as well.

Recommendation 6.3: The ACO should consider using some of the tools, perhaps SLACK, to augment survey capabilities and capture risk/opportunities identified throughout the community.

The ACO appreciates the panel's recommendation to utilize SLACK as a platform to augment the existing survey capabilities and plans to identify risks and opportunities for improvement within the program and the broader community.