5.0 8 November 2022: Evaluation Reactions and Planning

Agenda

Goal of today’s meeting: discuss what we learned from last meeting regarding Measurement and Evaluation and what we should try to do.

• Some initial thoughts (harvested from the discussion):

  • What would help us with managing our program?

  • Evaluation instruments and IRB: how best to proceed?

  • Coordination with evaluation leads for each track

  • What do we need to track to measure diversification and expansion of our community?

  • Evaluation framework

    • Awareness

    • Satisfaction

    • Impact

  • Formative data

    • How are things going?

    • Was this a good idea?

    • Is this working?

  • Baseline data: one possibility is the community survey data (at the bottom of this document)

  • How far should we track each effort?

  • What would we like to know but is out of our grasp (as best as we can tell, given budgetary constraints)?

Notes

• Allocations shared DEI plan and plans for initial AARC meeting (held December 5, 2022)

• Measurement suggested that the XDMoD database could be useful to show growth in number of users as well as user diversity (based on the data stored which includes number of users, NSF role of each user and institutions each user represents).

• Discussion points:

  • Short term: try to collect data now, even without capacity and framework for evaluation

  • Longer term: think about what we should prioritize for a possible supplement focused on evaluation

    • What should come from databases?

    • What should come from survey instruments?

    • How to handle Institutional Review Boards and their requirements

      • Really important for human subject research and publishing results of human subject research.