Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 2 Next »

ACCESS Coordination Office Response to the PY2 Review
Panel Summary

Responses to Observations and Opportunities for Improvement Indicated by the Review Panel

  1. Broader Impact -
    Opportunities to Improve
    Recommendation 1.1: In terms of DEI in the broader community, ACO can do more to take advantage
    of the unique and influential position ACCESS has by:
    ■ Advocating within the EC to clearly define goals for DEI as they relate to the broader
    community that ACCESS engages with.
    ■ Refining Personas and user journeys to actively invite, and perhaps even prioritize,
    engagement with researcher communities that have not historically engaged HPC resources.

    The ACO concurs with this recommendation but emphasizes for both of the bullets above that while
    the ACO can and does advocate for these things via the EC, Working Groups, and Standing
    Committees, it is the the EC that must make decisions regarding prioritization and allocation of staff
    time to support these program-wide efforts.

    Recommendation 1.2: Work within the EC to develop a coherent program-wide strategy that involves
    all services to bring under-represented groups into the computational science research community.
    The ACO concurs with this recommendation though sees it as a recommendation primarily to the EC
    and not specifically to the ACO. The ACO is already advocating and expending effort to support this.
    The EC concurs with this priority and thus chartered the DEI Working Group to address part of this.
    As we noted during our presentation the Community Building & Engagement (CB&E) efforts have
    been largely suspended in anticipation of further guidance from NSF on how the program should
    proceed in this space. The ACO took it upon itself to facilitate discussion during the February
    Quarterly Meeting to identify the priority events via which ACCESS should engage with the
    community for PY3. This would normally have been a function of the program-wide CB&E team.
    Recommendation 1.3: ACCESS (and ACO) are uniquely placed to make a real impact on DEI in the
    broader community - if this is taken advantage of. This unique position to shape the composition of the
    community is a privilege that should be seen as such.
    We take the panel’s recommendation very seriously and agree that ACCESS can and should do more
    to impact DEI in the community. As the panel notes, this is a recommendation to both the EC and
    the ACO. We will continue to use our role as the EC facilitator and peer member of the team to
    ensure that DEI remains a top priority of the program. We look forward to the DEI Working Group –
    in which the ACO participates – bringing recommendations on this to the EC as soon as possible.
    Recommendation 1.4: Whereas measures supporting goals within the overall health of ACCESS and
    how it is enabling greater science to occur were presented and good, it was not clear how these
    measures were selected and how they will be tracked over a longer period. It would be helpful to
    explain the methodology for selecting the measures, and show the full picture of all available measures,
    to better understand how ACCESS is moving science forward.
    The ACO appreciates this observation by the review panel, and the opportunity to provide some
    context. Fundamentally, the metrics are intended to provide indicators of progress toward realizing
    the program-wide goals that were developed through a highly collaborative process involving all
    staff across the program. The metrics presented in the shared team slides were put forth by the
    Evaluation Working Group following an initial brainstorming session at the September 2023
    Quarterly meeting facilitated by ACO PI Towns. We view these as v1 of the program-wide metrics
    and will refine these over time with broad program input. The Evaluation Standing Committee is
    now reviewing the metrics in light of the results from the 3 ACCESS surveys and other program
    developments and will bring forth recommendations for consideration. Time is of the essence as
    this work will impact PY3 surveys.
    Recommendation 1.5: While communications that work towards achieving broader impact goals are
    noted and ongoing, the ACO should pursue more concerted efforts at engaging with traditionally
    non-HPC communities.
    The ACO recognizes the need to focus on communications with traditionally non-HPC communities.
    In collaboration with the other awardee teams, the ACO continues to seek means by which we can
    3
    be more effective in this area. Some of those efforts were shared during the review, but others have
    not been pursued due to resource and staff time constraints. Nonetheless, the ACO will work with
    the Executive Council to identify additional means to engage the broader community within our
    resource constraints.
    Recommendation 1.6: Showcasing how the science being enabled by ACCESS is contributing in
    practical ways to society would allow for talking points when approaching non-traditional HPC
    communities. Perhaps this could be done through metrics or communications stories.
    The ACO does highlight the impact of the science supported through science stories, events support,
    annual publications, and social media. This recommendation suggests the ACCESS Communications
    Standing Committee consider spending some time distilling talking points from these stories to
    support those directly engaging with non-traditional HPC communities as well as others. Perhaps
    these talking points can be maintained on the ACCESS Communications Standing Committee wiki
    pages as a resource to any ACCESS staff member engaging members of non-traditional HPC
    communities.

  2. Intellectual Merit - Rating: Achieved Results
    Strengths
    Strength 2.1: The ACO team leadership includes appropriate expertise and a structure for performing
    important functions such as project management, communications, and events. These groups seem to
    be working well together. ACO leadership has shown agility and responsiveness by, for example,
    shifting the role of the external evaluator to meet emerging needs and by integrating a RACI-like model
    to clarify roles.
    Strength 2.2: In addition, the ACO PI has been innovative in creating regular meetings with leadership
    of other Federally funded projects for co-advising and sharing common management experiences and
    lessons learned. This is an important mechanism for knowledge sharing and the development of best
    practices.
    The ACO concurs with what is noted and has no further comment.
    Opportunities to Improve
    Recommendation 2.1: Although the ACO team has made great progress in building an organizational
    structure for ACCESS governance, the panel feels that they should take more ownership for the overall
    health of ACCESS and for making progress towards overall ACCESS goals.
    We concur with the panel that there are opportunities for the ACO to take more ownership for the
    overall health of ACCESS, but with the caveat that that ACO’s influence is limited. While we do take
    4
    every effort to identify and focus the EC’s collective efforts on matters of importance, forward
    progress is determined by the contributions of individual team members and the leadership of the
    PIs. It is our sense that there is now, more than ever before in the program, an understanding by
    ACCESS leadership that progress beyond single award teams will only come through better
    collaboration and teamwork. We are seeing positive signs across the program and will do our part
    to support these efforts.
    Recommendation 2.2: The ACO is the only ACCESS entity that can take this overarching view within
    the scope of their work, and the panel would like to see more leadership by example and efforts to
    influence working groups/committees to achieve results in a timely manner. An example of this is being
    able to deliver on unified goals (including DEI) and evaluation measures within the first half of the
    funded award.
    We appreciate the Panel’s perspective on this and agree to a point that the ACO can do more to
    influence Working Group/Standing Committee output. Again though, there are limitations to our
    influence. While the role of the ACO is certainly more oriented toward a global view, it is incumbent
    on all of the PIs, as part of the EC, to take this overarching view of the program. The ACO feels that it
    has led by example and shared a number of examples of this during the review (facilitating
    development of program-wide goals and metrics, facilitating the reboot of evaluation efforts,
    advocating for the formation of the DEI Working Group, etc.). We appreciate the panel’s recognition
    of the ACOs efforts here. However, the ACO simply is not staffed at a level necessary to take a
    leading role in all program-wide efforts. To the extent the ACO can, we will continue to lead by
    example with opportunities to do so on the horizon.
    Recommendation 2.3: The ACO leadership team has experienced significant staff turnover including
    in its evaluation team. It is currently engaging the challenges of quickly filling open positions and
    bringing new people up to speed. It is not clear that ACO has considered how it can systematically
    develop resilience through methods such as succession planning or reassignment of personnel from
    other projects.
    The ACO has considered how we can improve resilience in staff turn-over, and considers this an
    important issue to manage. With a total staff level of 2.58 FTE with individuals possessing disparate
    skill sets, there are minimal areas of redundancy.. Still, we have made efforts to address this,
    particularly with respect to the PIs but also for all staff members. In some cases, it is reasonable to
    consider re-assigning staff from other projects, but this clearly runs the risk of creating the same
    problem for another project. Nonetheless, we have done exactly this in some cases (e.g., Sr. Project
    Manager Shannon Bradley being reassigned to support the role of Sr. Project Manager Ron Payne
    following his departure from the Center). It is more challenging to address this with respect to the
    PI and co-PI, however, the PI and co-PI stand ready to support each other’s current roles should one
    of them become unavailable to the project until a replacement can be identified.
    5

  3. Program Governance, including EC and EAB - Rating: Partially Achieved
    The panel recognizes that ACO is the facilitator of ACCESS-wide governance functions (e.g. the EC, EAB
    and quarterly meetings) and may not be solely responsible for the strengths and weaknesses of those
    functions. Nevertheless, whether it is via facilitation or influence amongst its peers, the ACO is crucial
    to successful ACCESS-wide governance.
    Strengths
    There are a number of positive developments in ACCESS governance during the past year.
    Strength 3.1: Establishing program-wide quarterly meetings is very beneficial to the program. The
    Executive Council (EC) meets frequently, with good participation by stakeholders, and has developed a
    set of collaborative procedures and decision-making processes. The ACO team reports that there is an
    increasing level of trust and collaboration amongst EC members. The EC members (service PIs) have
    demonstrated that they can effectively work together to develop important common documents, such
    as a common response to PY1 review feedback, developing common PY2 review slides, and establishing
    program-wide goals.
    Strength 3.2: The External Advisory Board (EAB) has strong members (including representatives of
    the Researcher Advisory Council and Resource Provider Forum) and the level of EAB engagement is
    strong. It is encouraging that participants are willing to contribute beyond just attending the
    meetings. We hope this continues as members rotate on/off the EAB.
    Strength 3.3: Finally, initiating interactions with other NSF (and NIH) programs that have similar
    management structures is a positive step. While we understand the benefits of confidentiality, it would
    be valuable to define take-aways from these discussions, including commonly-shared feedback for NSF
    regarding management/coordination of future programs.
    The ACO concurs with what is noted. The ACO PI will consider how to distill useful feedback to NSF
    from the informal co-advisory group. The value of this is evident, but the need to maintain
    confidentiality is key to the success of those interactions.
    Opportunities to Improve
    Recommendation 3.1: To strengthen the overall ACCESS community, set priorities, and build a sense
    of common purpose, it would be beneficial to consider moving to a rotating leadership model, where
    perhaps the ACCESS PIs rotate a chair position once a year. This builds a better understanding of the
    overall health and operations of ACCESS and challenges/opportunities of the multi-functioning
    collaboration.
    The ACO concurs with this recommendation and will advocate for this with the rest of the EC. The
    ACO did make this recommendation during the development of the EC charter. At that time the
    6
    Service Team PIs did not feel this was warranted, rejecting also the suggestion that there be a Chair
    of the EC at all.
    Recommendation 3.2: We suggest that the EC make every effort to “get on the same page” regarding
    the overall ACCESS goals within PY3. This will require hard conversations, and no doubt compromise,
    but should then also end up with defined procedures and processes that benefit ACCESS governance.
    The ACO concurs that the EC members should be “on the same page” regarding overall ACCESS
    goals, but believes that there is in fact already significant alignment. The issue raised in the review
    discussions was with respect to the prioritization of efforts across the program and lack of complete
    alignment there, an observation with which the ACO fully agrees. The ACO has initiated a process to
    work toward a shared view of priorities.
    Recommendation 3.3: With this unified vision and set of goals at the EC level, the staff within the
    separate projects as well as the wider community as a whole can then more effectively and efficiently
    work towards those goals. The ACO should build upon current efforts underway within the EC towards
    harmony to achieve this wider goal.
    The ACO Concurs with this recommendation and has already initiated efforts to build upon this.
    Recommendation 3.4: While the EC generally functions well, there appear to be some areas of
    conflict. Dissension within the EC represents a significant risk for ACCESS overall, and the ACO along
    with other members of the EC should make every effort to resolve any internal conflicts. For example,
    the External Evaluator report suggests some internal conflicts within the EC (and their respective
    teams) about the appropriate role of the ACO within ACCESS.
    The ACO concurs with this recommendation. As the annual reviews of the program awardees
    complete and follow-up activities are done, the ACO will be bringing these issues to the EC for
    discussion. The ACO sees it as critical for there to be a shared understanding of its scope in order
    for it to have the opportunity for success.
    To illustrate the point, we call the panel’s attention to a comment made by the ACO evaluator in her
    PY2 report (which was provided to the panel): “Without breaching confidentiality agreements, it is
    safe to say that the Service Track PIs have very different, in some cases irreconcilable views about
    the role of the ACO. In particular, when a conflict arises among members of the Service Tracks, some
    of the Service Track representatives look to the ACO to address the conflict. Other Service Track
    representatives believe the ACO should not take the lead in this way. Acting as a Track
    representative with an opinion on a contentious issue can easily be at odds with acting as a meeting
    facilitator.”
    Recommendation 3.5: Also, the ACO team discussed a recent, specific conflict related to an
    ACCESS-wide DEI discussion. We are pleased that the ACO team has taken constructive steps to address
    this issue, including using an external conflict resolution expert. Continued efforts to ensure that issue
    is resolved are important, including perhaps a broader discussion regarding DEI priorities within the
    ACCESS team.
    7
    The ACO concurs with this recommendation and sees it also as a recommendation to the EC. The
    ACO appreciates the panel’s recognition of the importance of resolving conflict within the EC. The
    facilitated discussion is seen as a first step in a process to first gain alignment among the five PIs,
    followed by expanding this to the EC and then further to all of the ACCESS awardee teams. While
    this has a particular focus on DEI issues and priority of DEI efforts across the program, it is also
    intended to address alignment more generally.
    Recommendation 3.6: There are indications that interactions between ACCESS and Resource
    Providers need to be improved - e.g. the low response rate from RPs to the RP survey and the modest RP
    satisfaction rating.
    The ACO concurs with this recommendation and noted in the review the collective effort now
    underway by the program to better engage with RPs. These efforts are outlined in the document we
    provided to the panel, “Improving the ACCESS Program RP Pathways (March 20, 2024)”. This
    includes, among other recommendations, having the RP Forum Chair join the EC (a step already
    taken).
    Recommendation 3.7: There should be strong coupling with substantive feedback in both directions
    with the RP community. The RP-related slides in the “ACCESS Annual Surveys” presentation and the
    document provided to the panel “Improving the ACCESS Program RP Pathways (March 20, 2024)”
    outline a number of constructive steps that the EC plans to take to address this issue. We look forward
    to improvements in the coming year.
    The ACO concurs with this recommendation and will be advocating these changes and their
    implementation in the EC.
    Recommendation 3.8: There is concern that the ACO is not fully attuned to all of the work occurring
    between working groups and standing committees. Many important functions are being discussed
    within these populations, and having a member of the ACO in the group is likely not strong enough
    positioning to be able to understand the work and how it fits into the overall health and structure of
    ACCESS - which is a primary concern of the ACO.
    While it would be ideal if the ACO (and more importantly the EC) to be fully attuned to all of the
    work occurring between Working Groups and Standing Committee, it is simply beyond the scope of
    the ACO to do so. The ACO has established structure for working groups and standing committees,
    and strives to be cognizant of the efforts across the program, but it is challenging for the ACO to
    deliver on its many other obligations to allocate the level of staff time necessary for the oversight
    activity the panel suggests. Further, not all work that has a program-wide impact is being done
    within Working Groups and Standing Committees, but within specific teams. The situation is
    improving as the need for greater cross-team communication and transparency becomes more
    evident. The ACO is responsible for scheduling regular updates of the working group and standing
    committees to the EC and those have not happened as frequently as they should due to other
    pressing priorities. The ACO is already moving to rectify this.
    8
    Recommendation 3.9: We encourage a more robust system of bringing the ACO up to date on all
    activities and work within the committee/group structures, and inviting leaders to attend your
    meetings on a regular basis, so that the communication and group work is better supported by your
    team to help achieve all desired outcomes.
    The ACO generally concurs with this, but sees it as an issue for the EC and the ACO as part of the EC.
    See response to Recommendation 3.8 above.

  4. Communication and Outreach - Rating: Achieved Results
    Strengths
    Strength 4.1: The communication team has skills sufficient to their tasks, a broad knowledge of the
    tools at their disposal, and a clear understanding of industry standards. The team has been active in
    generating a brand and furthering its identity. The initial survey forays were encouraging, and the
    team is poised to take on new challenges. We have confidence in their current efforts and future
    endeavors.
    Strength 4.2: The communications team are using excellent metrics to understand their impact and
    look for ways to increase ACCESS visibility and usage. We acknowledge their rapid development and
    efforts to survey the larger ACCESS community. We acknowledge the innovative approaches to build
    communities by scanning badges at conferences, etc.
    Strength 4.3: It is a positive development that RPs and standing committees will be included in the
    calls for inputs to the newsletters.
    Strength 4.4: Developing ‘personas’ is effective for targeting audiences and helping to structure
    communications/outreach efforts.
    Strength 4.5: Coordination of the ongoing work to change how the website(s) is/are being changed
    shows [trying to say: move from “pieces make up the website” to “website is made up of pieces” is very
    positive since it shows the projects starting to pull together towards the common good of ACCESS. This
    willingness is a strength.
    The ACO concurs with what is noted and has no further comment.
    Opportunities to Improve
    Recommendation 4.1: The committee is excited about the opportunities that lay ahead of the
    communication team. We strongly encourage a pivot to a more proactive orientation, where the
    communication team does more than disseminating information on various web pages, networking,
    9
    visiting campuses, and attending conferences. There is a pressing need to engage targeted groups, with
    the aim of receiving feedback, creating awareness, and developing relationships.
    In our Plan Year 3 Communications Plan, we provide for targeted campaigns and will work with the
    Executive Council to identify priorities according to our personas as well as resources available,
    including budget and time, to do more targeted work.
    Recommendation 4.2: The potential shift that’s being considered from service-based structure of
    website to personas-based structure has good potential to improve website useability. We look forward
    to seeing progress on this in PY3, since it is understood that there are challenges with separate awards
    having responsibility for their portions of the website. However, the approach to date is mostly passive,
    in that information is created and the community must choose to engage. The panel encourages the
    communications team to consider more active ways to engage communities, build diverse readership,
    and ultimately lead to a more diverse and engaged user community for ACCESS.
    Our web stories form the foundation of our communications platform, creating content we believe
    those representing our personas will see themselves in. We actively promote those stories on our
    social media, and our newsletters reach well beyond those already on our site. And we ask the
    subjects of our stories to share on their own social media pages to try to introduce ACCESS to new
    targets. So we are consistently reaching out with other tools to draw people to the ACCESS web
    presence.
    Recommendation 4.3: To develop a more proactive orientation, the panel encourages the
    communication team to engage in separate, purposeful focus groups of graduate students,
    under-represented communities, users known to be high-level influencers, and essential users.
    Thankfully, we have had an opportunity to get feedback from focus groups conducted by our
    ACCESS Support colleagues as well as getting feedback through surveys shared with 22,641
    recipients engaged with ACCESS as our current resources do not allow for specific focus group
    activities in addition to what we are already doing. We can work with our Evaluation Standing
    Committee group to try to identify users from under-represented communities as well as the others
    the panel has identified.
    Recommendation 4.4: With acknowledgement of the value of sharing information through LinkedIn,
    we encourage the communications team to broaden its outreach to scientific communities within
    LinkedIn, engage in specific exploration of users with appropriate job titles and research interests, and
    build its brand within LinkedIn in addition to posting information updates.
    We appreciate this suggestion and want to add that we do use strategies such as reposting and
    hashtags to engage beyond our page. We know there are private groups we could join, but
    understanding each may have different rules for posting, this may require more bandwidth than our
    team can afford at this time. However, it would be good to look into paid ads on LinkedIn to target
    specific groups. If budget allows, that may be the most achievable way to broaden outreach.
    10
    Recommendation 4.5: Even without an official community building designation, your team members
    will still be attending conferences and looking to build a stronger user base. We encourage you to
    consider venues that host large communities of underrepresented populations, such as HACU, NSBE,
    BDPA, BIT, NACME, etc.
    The ACCESS staff has discussed which conferences to attend to reach underrepresented populations
    and has done some reconnaissance, such as attending the Grace Hopper Celebration last year. The
    direction has been to continue to attend both PEARC and SC to which Communications personnel
    travel; given our capacity, our team is best positioned to provide tools for the rest of our ACCESS
    colleagues who may also be attending conferences (such as TAPIA) to spread the word about
    ACCESS. We also note here that this is a function of the now-suspended Community Building &
    Engagement team. While the ACO and the broader program leadership are working to advance
    efforts in this area, progress in this area is hampered by the absence of a CB&E effort.
    Recommendation 4.6: The development of ‘personas’ should be continued and refined. One
    suggestion is to leverage DEI goals in the development of some personas.
    The Communications team will coordinate with the DEI working group on ways to position the
    personas outputs (web work) to maximum impact. This is the first plan year we will leverage the
    personas so we look forward to determining how their incorporation into our plans impact
    communications successes.
    Recommendation 4.7: The communications team should partner with Resource Providers so that at
    least 2 survey items are included in RPs own sampling of its user community.
    We have already begun greater engagement with our Resource Providers in developing
    communications so this is something we will take under advisement and will work with the
    Evaluation Standing Committee to see if we can get more focused feedback from them.
    Recommendation 4.8: The communication team should continue to develop news pieces for select
    Resource Providers and share opportunities with other Resource Providers.
    We have begun the process of gathering news items from our Resource Providers to develop
    bimonthly spotlights in our ACCESS Advance newsletter as part of our Plan Year 3 Communications
    Plan.
    Recommendation 4.9: The development of materials for use on mobile devices is a critical
    opportunity to reach a vast set of users. If materials are not already aligned with mobile device use, the
    committee encourages this advancement.
    The ACO website is a responsive design and we encourage those ACCESS awardees whose sites are
    not working well on mobile to make those adjustments. That said, the vast majority of our users
    visit the site on a desktop.
    11

  5. Evaluation - Rating: Achieved Results
    The panel recognizes that a lot of work was completed this year, and that the ACO had to significantly
    pivot this year in terms of leadership and direction. The panel was impressed with the amount of work
    that had been completed by the evaluation team, but felt that the necessary work to complete goals
    and decide on measures should have been further along at this point in time. In general, the panel was
    hoping to see, by the end of the second year of funding, a clearer presentation of overall goals for
    ACCESS and progress made, as well as overall goals for ACO and progress made.
    Strengths
    Strength 5.1: The reboot of the evaluation plan has been a positive step and there has been
    substantial progress in the development of program-wide goals and associated metrics, as well as
    ACO-specific goals and metrics.
    Strength 5.2: The former is especially important as it reflects close collaboration with other services,
    and helps to unify the EC/service awards in prioritizing collaborative efforts going forward.
    Strength 5.3: In addition, the inclusion of an external evaluator is very positive, and indicates a strong
    commitment to continuous improvement within the ACO team, as well as throughout ACCESS.
    The ACO concurs with what is noted and has no further comment.
    Opportunities to Improve
    Recommendation 5.1: The selection process of measures to support the goals of the ACO, and
    longitudinal tracking of those measures, should be transparent and available for review. The ones
    presented during the site review created a good picture, but it is unclear how they were selected and
    the totality of all measures that were available to be selected is not well understood.
    Program-wide metrics, which were the metrics presented in the review, were developed through
    the Evaluation Work Group. Each member of the working group (at least one representative per
    Service Team) suggested metrics for each of the program-wide goals. From there the group
    discussed and winnowed down the list to a few metrics for each goal. These were presented to the
    full ACCESS team at the February in-person meeting for comment. As noted, the metrics are
    considered initial versions and are now going through a review and revision via the Evaluation
    Standing Committee in preparation for the PY3 survey. In summary, there was an extensive process
    of development that led to these.
    Recommendation 5.2: In addition, for the selected measures, the ACO team should include a measure
    of confidence with these results.
    12
    We appreciate this suggestion and will confer with the Evaluation Standing Committee on its
    potential implementation.
    Recommendation 5.3: There is concern that ACO and ACCESS goals and measures are not fully
    developed at this point in the life of the award.
    While we share the panel’s concern in this area, we have been working diligently on both ACO and
    ACCESS-wide goals. As noted in the review, we have prioritized ACCESS-wide goals given the need
    to reformulate the ACCESS evaluation plan. It is worth noting that there is an opportunity to use the
    ACCESS-wide goals to revise the ACO-specific goals and ensure that the ACO’s goals align with the
    ACCESS-wide goals.
    Recommendation 5.4: As the community facilitator/influencer, the ACO needs to take a stronger lead
    in creating urgency to complete defining program-wide and ACO-specific metrics that will be
    supported by the work of the entire community.
    We appreciate the panel’s sense of urgency in defining metrics. The ACO has taken a significant
    leadership role in the development of program-wide metrics. A current focus of the Evaluation
    Standing Committee (which includes 2 ACO staff and the ACO External Evaluator) is on refining the
    initial set of metrics that were presented at the review.
    Recommendation 5.5: In addition, ACO should establish a baseline of ‘fixed’ metrics that can be
    tracked over time to establish trends.
    This is an excellent suggestion that we will take back to the Evaluation Standing Committee.
    Recommendation 5.6: The ACO is urged to Identify mechanisms to improve the response rate for the
    evaluation surveys.
    We appreciate the panel highlighting the importance of improving response rates. This is a current
    focus of the Evaluation Working Group.
    Recommendation 5.7: The ACO should work with the RP Forum to understand the best methods of
    approach to ensure higher RP response rates, and evaluate techniques to improve the ‘community’
    response rate, perhaps with a focus on high-priority subsets of the community.
    We agree with the panel that there is a need to improve RP survey response rates. There is an
    ACCESS-wide effort to better engage with the RP Forum (and RPs more directly) on a wide-range of
    issues, including improving the survey methodology and response rates.
    Recommendation 5.8: The ACO should consider tapping into subject matter expertise on survey
    methodology to improve questions posed and response rates.
    The ACO External Evaluator has some expertise in survey methodology and she has directly
    contributed to the development of the initial surveys. We will explore additional sources of
    expertise.
    13

  6. ACO Project Office - Rating: Achieved Results
    Strengths
    Strength 6.1: The role that the ACO plays in the organization of meetings is very good, with clear
    delineation of tasks, decisions, and parking lot items.
    Strength 6.2: This undoubtedly positively contributes to smoothly operating groups. In addition, the
    developed tools appear to be well-used and provide transparency into the communication and sharing
    of documents.
    The ACO concurs with what is noted and has no further comment.
    Opportunities to Improve
    Recommendation 6.1: The panel recommends that ACCESS team members focus on risks, either
    service-specific or program-wide, as a technique to not only manage those risks, but to develop goals,
    plans and metrics.
    We agree with the panel’s observations to utilize risks as opportunities to develop goals, plans and
    metrics. The ACO currently notes risks internal to the ACO and program-wide risks which are
    reviewed by the EC and contingencies executed as needed. The notion of using risk to drive
    development of goals, plans and metrics is interesting and we will certainly investigate this
    approach.
    Recommendation 6.2: In this aspect, the ACO should consider how to more fully capture, track and
    share the various risks that should become visible to the full ACCESS staff.
    The ACO appreciates the panel's recommendation and will continue to search for ways to improve
    the overall risk capturing and tracking process. The Risk Register which is created and maintained
    by the ACO is fully accessible to all ACCESS staff, including but not limited to the ACO risks, Service
    Team risks and program wide risks. As we further develop the goals and metrics, it will be natural to
    develop the associated risks. This can be used in the manner suggested in the previous item, but
    also provides an opportunity to remind the Service Teams of the risk register capability that the
    ACO is providing and improve adoption. This can be in the context of improved transparency in
    sharing of award-level risks as well.
    Recommendation 6.3: The ACO should consider using some of the tools, perhaps SLACK, to augment
    survey capabilities and capture risk/opportunities identified throughout the community.
    The ACO appreciates the panel's recommendation to utilize SLACK as a platform to augment the
    existing survey capabilities and plans to identify risks and opportunities for improvement within
    the program and the broader community.
    14
    Appendix: Original PY2 Review Panel Report
    We include here the original report received from the review panel for completeness.
    15
    Panel ID V242294
    Summary Statement - Rating: Achieved Results
    Overall the panel felt that a lot of progress had been made this year, in particular rebooting
    evaluation efforts, continuing to lead and manage the governance structure, and aiming towards
    a coalesced ACCESS structure as a whole. There are a number of elements in place that will be
    helpful in achieving the goals (once finalized), although not all of the results are ready to be
    reviewed. In particular, the panel would like to see urgency in developing ACCESS goals and
    measures, to include DEI. The panel is eager to see how future engagements, including RP on
    the EC and other management integrative steps, help build cohesion among all the ACCESS
    elements. The panel would like to recognize the willingness, commitment, and capability of the
    ACO to help ACCESS achieve all of its goals, even when challenging issues must be
    addressed.
    Broader Impact - Rating: Partially Achieved
    While the potential scope of Broader Impacts can be intimidating, we see the ACCESS program
    to be in a unique position to have the ability to shape the computational research communities
    in important and very impactful ways. We suggest the EC, the EAB, the projects and other
    stakeholders identify what broader impacts ACCESS could achieve that only ACCESS could
    achieve, and then to focus on one or two of those. The panel had difficulty identifying what
    broader impacts the ACO project achieved vs ACCESS Program achievements.
    ● Strengths:
    ○ Willingness to have the conversation about DEI and to leverage external
    expertise when needed - this is the first step and incredibly important.
    ○ Efforts to increase diversity (including those by communications showcasing the
    use of ACCESS resources by researchers at non-R1 and minority-serving
    institutions) seems to be increasing allocations to non-R1 institution researchers.
    ○ ACO (and ACCESS) have in the last program year put indicators and tools in
    place that allow for the evaluation, and when needed redirection, of efforts
    related to Broader Impact. These measures are important and should continue to
    receive necessary resources and priority.
    ● Opportunities to Improve:
    ○ In terms of DEI in the broader community, ACO can do more to take advantage
    of the unique and influential position ACCESS has by:
    ■ Advocating within the EC to clearly define goals for DEI as they relate to
    the broader community that ACCESS engages with.
    ■ Refining Personas and user journeys to actively invite, and perhaps even
    prioritize, engagement with researcher communities that have not
    historically engaged HPC resources.
    ■ Work within the EC to develop a coherent program-wide strategy that
    involves all services to bring under-represented groups into the
    computational science research community.
    ○ ACCESS (and ACO) are uniquely placed to make a real impact on DEI in the
    broader community - if this is taken advantage of. This unique position to shape
    the composition of the community is a privilege that should be seen as such.
    ○ Whereas measures supporting goals within the overall health of ACCESS and
    how it is enabling greater science to occur were presented and good, it was not
    clear how these measures were selected and how they will be tracked over a
    longer period. It would be helpful to explain the methodology for selecting the
    measures, and show the full picture of all available measures, to better
    understand how ACCESS is moving science forward.
    ○ While communications that work towards achieving broader impact goals are
    noted and ongoing, the ACO should pursue more concerted efforts at engaging
    with traditionally non-HPC communities.
    ○ Showcasing how the science being enabled by ACCESS is contributing in
    practical ways to society would allow for talking points when approaching
    non-traditional HPC communities. Perhaps this could be done through metrics or
    communications stories.
    Intellectual Merit - Rating: Achieved Results
    ● Strengths:
    The ACO team leadership includes appropriate expertise and a structure for performing
    important functions such as project management, communications, and events. These groups
    seem to be working well together. ACO leadership has shown agility and responsiveness by, for
    example, shifting the role of the external evaluator to meet emerging needs and by integrating a
    RACI-like model to clarify roles. In addition, the ACO PI has been innovative in creating regular
    meetings with leadership of other Federally funded projects for co-advising and sharing
    common management experiences and lessons learned. This is an important mechanism for
    knowledge sharing and the development of best practices.
    ● Opportunities to Improve:
    Although the ACO team has made great progress in building an organizational structure for
    ACCESS governance, the panel feels that they should take more ownership for the overall
    health of ACCESS and for making progress towards overall ACCESS goals. The ACO is the
    only ACCESS entity that can take this overarching view within the scope of their work, and the
    panel would like to see more leadership by example and efforts to influence working
    groups/committees to achieve results in a timely manner. An example of this is being able to
    deliver on unified goals (including DEI) and evaluation measures within the first half of the
    funded award.
    The ACO leadership team has experienced significant staff turnover including in its evaluation
    team. It is currently engaging the challenges of quickly filling open positions and bringing new
    people up to speed. It is not clear that ACO has considered how it can systematically develop
    resilience through methods such as succession planning or reassignment of personnel from
    other projects.
    Program Governance, including EC and EAB - Rating: Partially Achieved
    The panel recognizes that ACO is the facilitator of ACCESS-wide governance functions (e.g. the
    EC, EAB and quarterly meetings) and may not be solely responsible for the strengths and
    weaknesses of those functions. Nevertheless, whether it is via facilitation or influence amongst
    its peers, the ACO is crucial to successful ACCESS-wide governance.
    ● Strengths:
    There are a number of positive developments in ACCESS governance during the past year.
    Establishing program-wide quarterly meetings is very beneficial to the program. The Executive
    Council (EC) meets frequently, with good participation by stakeholders, and has developed a set
    of collaborative procedures and decision-making processes. The ACO team reports that there is
    an increasing level of trust and collaboration amongst EC members. The EC members (service
    PIs) have demonstrated that they can effectively work together to develop important common
    documents, such as a common response to PY1 review feedback, developing common PY2
    review slides, and establishing program-wide goals.
    The External Advisory Board (EAB) has strong members (including representatives of the
    Researcher Advisory Council and Resource Provider Forum) and the level of EAB engagement
    is strong. It is encouraging that participants are willing to contribute beyond just attending the
    meetings. We hope this continues as members rotate on/off the EAB.
    Finally, initiating interactions with other NSF (and NIH) programs that have similar management
    structures is a positive step. While we understand the benefits of confidentiality, it would be
    valuable to define take-aways from these discussions, including commonly-shared feedback for
    NSF regarding management/coordination of future programs.
    ● Opportunities to Improve:
    To strengthen the overall ACCESS community, set priorities, and build a sense of common
    purpose, it would be beneficial to consider moving to a rotating leadership model, where
    perhaps the ACCESS PIs rotate a chair position once a year. This builds a better
    understanding of the overall health and operations of ACCESS and challenges/opportunities of
    the multi-functioning collaboration.
    We suggest that the EC make every effort to “get on the same page” regarding the overall
    ACCESS goals within PY3. This will require hard conversations, and no doubt compromise, but
    should then also end up with defined procedures and processes that benefit ACCESS
    governance. With this unified vision and set of goals at the EC level, the staff within the separate
    projects as well as the wider community as a whole can then more effectively and efficiently
    work towards those goals. The ACO should build upon current efforts underway within the EC
    towards harmony to achieve this wider goal.
    While the EC generally functions well, there appear to be some areas of conflict. Dissension
    within the EC represents a significant risk for ACCESS overall, and the ACO along with other
    members of the EC should make every effort to resolve any internal conflicts. For example, the
    External Evaluator report suggests some internal conflicts within the EC (and their respective
    teams) about the appropriate role of the ACO within ACCESS. Also, the ACO team discussed a
    recent, specific conflict related to an ACCESS-wide DEI discussion. We are pleased that the
    ACO team has taken constructive steps to address this issue, including using an external
    conflict resolution expert. Continued efforts to ensure that issue is resolved are important,
    including perhaps a broader discussion regarding DEI priorities within the ACCESS team.
    There are indications that interactions between ACCESS and Resource Providers need to be
    improved - e.g. the low response rate from RPs to the RP survey and the modest RP
    satisfaction rating. There should be strong coupling with substantive feedback in both directions
    with the RP community. The RP-related slides in the “ACCESS Annual Surveys” presentation
    and the document provided to the panel “Improving the ACCESS Program RP Pathways
    (March 20, 2024)” outline a number of constructive steps that the EC plans to take to address
    this issue. We look forward to improvements in the coming year.
    There is concern that the ACO is not fully attuned to all of the work occurring between working
    groups and standing committees. Many important functions are being discussed within these
    populations, and having a member of the ACO in the group is likely not strong enough
    positioning to be able to understand the work and how it fits into the overall health and structure
    of ACCESS - which is a primary concern of the ACO. We encourage a more robust system of
    bringing the ACO up to date on all activities and work within the committee/group structures,
    and inviting leaders to attend your meetings on a regular basis, so that the communication and
    group work is better supported by your team to help achieve all desired outcomes.
    Communication and Outreach - Rating: Achieved Results
    ● Strengths:
    The communication team has skills sufficient to their tasks, a broad knowledge of the tools at
    their disposal, and a clear understanding of industry standards. The team has been active in
    generating a brand and furthering its identity. The initial survey forays were encouraging, and
    the team is poised to take on new challenges. We have confidence in their current efforts and
    future endeavors.
    The communications team are using excellent metrics to understand their impact and look for
    ways to increase ACCESS visibility and usage. We acknowledge their rapid development and
    efforts to survey the larger ACCESS community. We acknowledge the innovative approaches to
    build communities by scanning badges at conferences, etc. h
    ○ It is a positive development that RPs and standing committees will be included in
    the calls for inputs to the newsletters.
    ○ Developing ‘personas’ is effective for targeting audiences and helping to structure
    communications/outreach efforts.
    ○ Coordination of the ongoing work to change how the website(s) is/are being
    changed shows [trying to say: move from “pieces make up the website” to
    “website is made up of pieces” is very positive since it shows the projects starting
    to pull together towards the common good of ACCESS. This willingness is a
    strength.
    ● Opportunities to Improve:
    The committee is excited about the opportunities that lay ahead of the communication team. We
    strongly encourage a pivot to a more proactive orientation, where the communication team does
    more than disseminating information on various webpages, networking, visiting campuses, and
    attending conferences. There is a pressing need to engage targeted groups, with the aim of
    receiving feedback, creating awareness, and developing relationships.
    The potential shift that’s being considered from service-based structure of website to
    personas-based structure has good potential to improve website useability. We look forward to
    seeing progress on this in PY3, since it is understood that there are challenges with separate
    awards having responsibility for their portions of the website. However, the approach to date is
    mostly passive, in that information is created and the community must choose to engage. The
    panel encourages the communications team to consider more active ways to engage
    communities, build diverse readership, and ultimately lead to a more diverse and engaged user
    community for ACCESS.
    ○ To develop a more proactive orientation, the panel encourages the
    communication team to engage in separate, purposeful focus groups of graduate
    students, under-represented communities, users known to be high-level
    influencers, and essential users.
    ○ With acknowledgement of the value of sharing information through LinkedIn, we
    encourage the communications team to broaden its outreach to scientific
    communities within LinkedIn, engage in specific exploration of users with
    appropriate job titles and research interests, and build its brand within LinkedIn in
    addition to posting information updates.
    ○ Even without an official community building designation, your team members will
    still be attending conferences and looking to build a stronger user base. We
    encourage you to consider venues that host large communities of
    underrepresented populations, such as HACU, NSBE, BDPA, BIT, NACME, etc.
    ○ The development of ‘personas’ should be continued and refined. One suggestion
    is to leverage DEI goals in the development of some personas.
    ○ The communications team should partner with Resource Providers so that at
    least 2 survey items are included in RPs own sampling of its user community.
    ○ The communication team should continue to develop news pieces for select
    Resource Providers and share opportunities with other Resource Providers.
    ○ The development of materials for use on mobile devices is a critical opportunity
    to reach a vast set of users. If materials are not already aligned with mobile
    device use, the committee encourages this advancement.
    Evaluation - Rating: Achieved Results
    The panel recognizes that a lot of work was completed this year, and that the ACO had to
    significantly pivot this year in terms of leadership and direction. The panel was impressed with
    the amount of work that had been completed by the evaluation team, but felt that the necessary
    work to complete goals and decide on measures should have been further along at this point in
    time. In general, the panel was hoping to see, by the end of the second year of funding, a
    clearer presentation of overall goals for ACCESS and progress made, as well as overall goals
    for ACO and progress made.
    ● Strengths:
    The reboot of the evaluation plan has been a positive step and there has been substantial
    progress in the development of program-wide goals and associated metrics, as well as
    ACO-specific goals and metrics. The former is especially important as it reflects close
    collaboration with other services, and helps to unify the EC/service awards in prioritizing
    collaborative efforts going forward. In addition, the inclusion of an external evaluator is very
    positive, and indicates a strong commitment to continuous improvement within the ACO team,
    as well as throughout ACCESS.
    ● Opportunities to Improve:
    The selection process of measures to support the goals of the ACO, and longitudinal tracking of
    those measures, should be transparent and available for review. The ones presented during the
    site review created a good picture, but it is unclear how they were selected and the totality of all
    measures that were available to be selected is not well understood. In addition, for the selected
    measures, the ACO team should include a measure of confidence with these results.
    There is concern that ACO and ACCESS goals and measures are not fully developed at this
    point in the life of the award. As the community facilitator/influencer, the ACO needs to take a
    stronger lead in creating urgency to complete defining program-wide and ACO-specific metrics
    that will be supported by the work of the entire community. In addition, ACO should establish a
    baseline of ‘fixed’ metrics that can be tracked over time to establish trends.
    The ACO is urged to Identify mechanisms to improve the response rate for the evaluation
    surveys. The ACO should work with the RP Forum to understand the best methods of approach
    to ensure higher RP response rates, and evaluate techniques to improve the ‘community’
    response rate, perhaps with a focus on high-priority subsets of the community. The ACO should
    consider tapping into subject matter expertise on survey methodology to improve questions
    posed and response rates.
    ACO Project Office - Rating: Achieved Results
    ● Strengths:
    The role that the ACO plays in the organization of meetings is very good, with clear delineation
    of tasks, decisions, and parking lot items. This undoubtedly positively contributes to smoothly
    operating groups. In addition, the developed tools appear to be well-used and provide
    transparency into the communication and sharing of documents.
    ● Opportunities to Improve:
    The panel recommends that ACCESS team members focus on risks, either service-specific or
    program-wide, as a technique to not only manage those risks, but to develop goals, plans and
    metrics. In this aspect, the ACO should consider how to more fully capture, track and share the
    various risks that should become visible to the full ACCESS staff.
    The ACO should consider using some of the tools, perhaps SLACK, to augment survey
    capabilities and capture risk/opportunities identified throughout the community.


The summary was read to/by the panel, and the panel concurred that the summary accurately
reflects the panel discussion.

  • No labels