Want to make creations as awesome as this one?

Transcript

Start

Program Evaluation

What is Program Evaluation?

Routine, systematic, deliberate collection of information gathered.

To determine the factors that contribute to success of a program and what actions need to be taken to address the findings of the evaluation.

Program evaluation should have three outcomes:

  • Assess program implementation
  • Assess program results
  • Highlight methods of program improvement

Program evaluation is a process that consists of collecting, analyzing, and using information to assess the relevance of a public program, its effectiveness, and its efficiency The aim of program evaluation is to answer questions about a program’s performance and value.

Program Evaluation

Program evaluation is a valuable tool for program managers who are seeking to strengthen the quality of their programs and improve outcomes.

Program evaluation is an important part of continuous improvement and accountability

Evaluation is necessary for any program regardless of size, nature, and duration.

Program evaluation answers basic questions about a program’s effectiveness, and evaluation data can be used to improve program services.

Importance of Program Evaluation

Sampling: An Exaple

  • Solicit input from stakeholders and identify program goals
  • Design a plan for examining progress toward program goals
  • Create a consolidated data collection plan
  • Plan the data analyses to examine program goals
  • Estimate the financial & time costs of the evaluation needed
  • Come to a final agreement about services

Steps to Guide a Focused Program Evaluation

Monitoring (Process Evaluation): How well is the program working?

Cost/Benefits Studies: A systematic approach used to evaluate the strengths and weaknesses of available options through a critical comparison of benefits and costs.

Formative Evaluation: Studies that aim to determine the existence and extent of problems, typically among a segment of the population.

Needs Assessment: A systematic process for determining and addressing needs, or 'gaps' between current conditions and desired conditions or 'wants.'

Topics for Program Evaluation

The term program generally refers to any group of related, complementary activities intended to achieve specfific outcomes or results.

What is a "Program"?

Program evaluation also applies to projects and initiatives.

  • To determine the achievement of objectives related to improved status
  • To improve program implementation
  • To provide accountability to funders, the community, and other stakeholders
  • To increase community support for initiatives
  • To contribute to the scientific base for interventions
  • To inform policy decisions

Purpose of a Program Evaluation

Click each category to learn more.

4.Improve staff practice with participants

3.Showcase effectivesness of the community

2. Enables program managers to answer basic questions

1. Find out "what works" and "what does not work"

Benefits of Program Evaluations

Knowing “what works” helps program managers to focus resources on the essential components of the program model that benefit participants.

Knowing “what does not work” allows program managers to improve and strengthen their service delivery models.

Not knowing what is working may waste valuable time and resources.

1. Find out “what works” and “what does not work.”

Questions about a program’s effectiveness, including:

  • Are participants satisfied with the program?
  • How much and what kind of a difference did the intervention make for the participants?
  • Do staff have the necessary skills and training?
  • Are some sub-groups benefiting, but not others?

2. Enables program managers to answer basic questions.

Sharing findings within the community can serve as a good outreach tool for:

  • Attracting collaborative partners
  • Recruiting participants and volunteers
  • Building trust with families and community members.

Evaluation findings can demonstrate to funders that a program is worthwhile. Funders often require:

  • That a program evaluation be conducted when they agree to fund a program,
  • Some funders will not fund, or refund, a program until an evaluation has been conducted and outcomes have been demonstrated

3. Showcase effectiveness to the community.

Evaluation questions may include:

  • Do staff have the necessary skills to work effectively with program participants?
  • What types of additional training would benefit staff?
  • Are staff receiving the ongoing coaching and mentoring necessary to do their work?
  • Do staff have the necessary support to function effectively?

  • Improving how staff members deliver services will increase the likelihood of achieving positive outcomes.
  • This allows a manager to systematically assess staff performance, and figure out where staff members are succeeding and where they need support or training.
  • Further, it can provide staff with opportunities to discuss the challenges they face and offer potential solutions.

4. Improve staff practice with participants.

  • Two purposes of program evaluation are:
1. Assessing and improving quality2. Determinig program effectiveness
  • Two main types of program evulations:
1. Process Evaluation2. Outcome Evaluation

Program evaluation is a systematic method of collecting, analyzing, and using information to answer basic questions about a program.

Program Evaluation

Evaluation Types

Click each category to learn more.

Other types of Program Evaluation

Outcome Evaluation

Process Evaluation

Image source: JT. (2020, Oct. 7). What’s the difference between evaluation and analytics? https://ccnyinc.org/whats-the-difference-between-evaluation-and-analytics/

  • Whether an intervention or program was implemented as planned (fidelity of treatment implementation).
  • Whether the intended target population was reached.
  • What were the major challenges and successful strategies associated with the program implementation?

Assess:

Process Evaluation

Click the table to enlarge the view.

Outcome Evaluations:

The table below provides an example of the format and the information that will be needed to complete an outcome evaluation.

Basic Components of Outcome Evaluation

Mohajeri, M. (n.d.). Outcomes Evaluation for Community Organization - PPT Download. SlidePlayer. https:slideplayer.com/slide/17989731/ (Adapted)

The observable and measurable "milestones" toward an outcome target

The targeted number and percent of participants to be achieved

The real impacts/ benefits/ changes for program participatnts

The units of service regarding to the program

  • Determine whether, and to what extent:
    • The expected changes in outcomes occur and
    • Whether these changes can be attributed to the program or program activities
  • Can focus on short-term outcomes like behavior change or changes in attitudes, knowledge, and awareness or long-term outcomes like the degree to which diseases or injuries actually decreased.

Activities, or processes to fulfil client's needs

Materials and resources

Outputs
Outcomes
Outcome Targets
Outcome Indicators
Activities
Inputs

Outcome Evaluation

Other Types Program Evaluation

  • Relies on rigorous methods to determine the changes in outcomes which can be attributed to a specific intervention based on cause-and-effect analysis.
  • Serve an accountability purpose to determine if and how well a program worked.

Impact

Intended to provide a package of results used to assess whether a program works (or not) at the end of the program.

Summative

Typically conducted during the development or improvement of a program and is conducted more than once with the intent to improve.

Formative

Internal Evaluator

  • Advantages:
    • Familiar with organization and program history
    • Able to communicate results frequently and clearly
    • Less expensive
  • Disadvantages:
    • Possibility of evaluator bias or conflict of interest

An individual trained in evaluation and personally involved with the program conducts the evaluation.

External Evaluator

  • Advantages:
    • More objective review and fresh perspective
    • Can ensure unbiased evaluation outcome
    • Typically brings more breadth and depth of technical expertise
  • Disadvantages
    • More expensive
    • Can be somewhat isolate, often lacking knowledge of and experience with the program.

Conducted by someone who is not connected with the program (i.e., evaluation consultant)

Who Conducts the Evaluation?

An evaluation plan consists of two components:

Think research proposal but for a program evaluation.

  • Program Profile
  • Logic Model

An evaluation plan is a written document that describes how one will monitor and evaluate a program as well as how one intends to use evaluation results for program improvement and decision-making.

Writing an Evaluation Plan

A program profile describes:

  • Background and context
  • Current status of the program
  • Rationale for the program
  • Primary intended users (target population)
  • Main goals
  • Evaluation stakeholders

Program Profile is a one-page summary that highlights the components of the program. Similar to a logic model, program profiles make it easy to understand what components define the program; the direct results of the program component(s); and what the program is collecting, documenting, or measuring to discern if what is being done is working.

Program Profile

Sampling: An Exaple

  • Program Goals
    • What is the program striving to achieve?
  • Background and Context
    • Why was the program established?
    • What does existing research say about this issue?
    • Is the program just getting started?
    • How long has the program been in existence?
    • What factors and trends in the larger environment may influence program success or failure?
  • Rationale
    • Any related studies to provide a rationale for the program
    • Any programs that offer similar services
  • Target Population
    • Who is the program intended to serve?
    • How will the program benefit the target population?
  • Evaluation Stakeholders
    • Who is the target audience of evaluation results?
    • Who will be interested in or use these results?
    • Who do these results impact?

Program Profile: Example

Communication: helps to communicate about the program with staff, funders, or other stakeholders

Evaluation: helps select evaluation questions and performance indicators

Program management: displays connections between resources, activities and outcomes

Planning: enable planners to plan activities for specific outcomes

Displays the program components (inputs, activities, outputs, outcomes)

A graphical or table description of how a program is intended to work (i.e., the program theory)

Click the image to enlarge the view.

A logic model describes:

  • How the program works and what is involved (i.e., program components)
  • Program Components include:
    • Inputs/Resources
    • Activities
    • Outputs/what is generated by the program
    • Outcomes/intended results or goals

A Logic Model is a graphic depiction (road map) that presents the shared relationships among the resources, activities, outputs, outcomes, and impact for a program. It depicts the relationship between the program’s activities and its intended effects.

Logic Model

Sampling: An Exaple

  • Services or products generated by program activities
  • Usually are within the control of the organization and can be controlled by modifying program activities
  • Outputs may include:
    • Procedures completed
    • The care/treatment plan completed
    • Creation of documents (e.g. manuals)
    • Health promotion pamphlets delivered

Outputs

  • The intended results or goals of a program
  • Can be immediate, intermediate, or final (i.e., short-term or long-term)
  • Outcomes are written as change statements (e.g., increase, decrease)

Outcomes

  • The inputs/resources required to implement program activities necessary to accomplish intended outcomes
  • Activities may include:
    • Assessment – Counseling – Treatment
    • Consultation – Education – Management

Acitivities

  • The inputs/resources required to implement program activities necessary to accomplish intended outcomes
  • Financial and non-financial resources
  • Inputs may include:
    • Money - Personnel - Equipment
    • Facilities – Supplies – Partners
    • Clinical Guidelines – Policies

Inputs/Resources

Logic Model Components

  • Immediate Outcome
    • An increase in awareness among a client
      • Increased awareness of the consequences of drug use
  • Intermediate Outcome
    • A behavior change in a client
      • Decreased personal use of drugs
  • Final Outcome
    • A change of state/condition among the target population
      • Decreased rates of drug use among urban youth in Tampa

Logic Model: Example Outcomes

Outcomes
Outputs
Activities
Inputs

Here is an example of a logic model template. This template shows the basic information that an organization or agency needs to complete. Then, you will find a more detailed accounting of how the basic information links to providing a picture of how an agency or organization is supposed to work, followed by Logic Model examples from plain to elaborate.

Logic Model Development

A Series of Casual Links

Final Outcomes
If the intended immediate and intermediate outcomes occur, then certain system changes (in organizations or communities) might occur.
If the outputs are accomplished, then the program is expected to have this influence (these outcomes) on the target population.
If the activities are undertaken, then these products/services (outputs) are expected to be delivered.
If the resources are used then these planned activities will occur.
These inputs/resources are necessary to accomplish the activities.
Immediate/Intermediate Outcomes
Outputs
Activities
Inputs

Logic Model Example Development: A Series of Casual Links

Funding Personnel Political Support

Personnel Supplies (computer, paper, printer, etc.)

Funding Personnel

Plan public advocacy program

  • Every three months the programs will hold a statewide public awareness program
  • Every six weeks counseling coordinators will meet to review plan
  • Improved quality of life for youth
  • Develop Organized community support for policies increasing drug intervention programs
  • The state will have adequate and coordinated treatment programs
  • By the end of the year, three new peer counseling programs will have been established

Develop training material

Develop peer counseling curriculum

Hire Coordinators

Outcomes
Outputs
Activities
Inputs

Logic Model Example Development

Final Outcomes
Immediate/Intermediate Outcomes
Outputs
Activities
Inputs
Program: Educational Parenting Program Target Population: Parents of children 2 to 4 years of age.

Logic Model Example Development:An Example in Healthcare

  • Increased number of children able to attain their optimal level of development
  • Increased number of parents able to adopt healthy parenting behaviors
  • Increases knowledge about caring for a young child
  • Increased ongoing peer support
  • Increased knowledge of available services
  • 50 parents of children 2-4 years of age attend the sessions
  • Organize sessions
  • Distribute reading material

Instructor Note: The page numbers in this example refer to the documents from the agency I worked with to create this logic model.

In this document, we identify a number of common short-, medium-, and long-term outcomes for school leadership interventions (see page 11). How to use it: Understand how leadership interventions work to achieve improved outcomes for students, identify the principal competencies the current or future intervention aims to affect, and align the design of the intervention with outcomes.

In the logic models, we identify outputs—or the immediate things that should happen if the intervention is implemented effectively—for each of the six intervention categories, and indicate possible indicators that can be used to measure implementation success (see pages 12 to 35). How to use: it Identify the key outputs that you might want to examine and measure to determine whether the intervention is being implemented properly.

In the logic models, we identify activities associated with each of the six categories of leadership interventions and report additional detail on how specific interventions we reviewed undertake these activities (see pages 12 to 35). How to use it: Determine whether your current or future intervention has the activities commonly found in evidence-based programs.

In this document, we identify a list of resource types and some questions about needed resources for states and districts to consider asking (see pages 36 and 37).How to use it: When identifying an intervention and the primary activities, use our guiding questions to identify resources. If sufficient resources are not available, consider other intervention types.

Outcomes"What are my goals?"

Outputs"What happens immediately?"

Activities"What do I do?"

Resources"What do I need?"

In this document, we describe each of our six intervention types according to the problem they aim to address (see pages 9 and 10). How to use it: Use these problem statements to help you focus on logic models related to the types of interventions that most directly apply to your needs and priorities.

Problem Statement"What issue am I addressing?"

Logic Model Example Development:"Road Map" to Logic Model Components

Click to view or download a PDF version of this logic model.

CYD Logic Model. (n.d.). Create Your Dreams. Retrieved from createyourdreams.org.

Logic Model Example Development:CYD Logic Model

You have reached the end of the presentation.

Please return to the course Canvas page.