The case study, “ECS PROGRAMMING FOR CHILDREN WITH SEVERE DISABILITIES” aims to provide funding for students with severe disabilities, in order to meet their educational needs.
It is difficult to choose just one approach because there may be more than one aspect of the program evaluated.
For instance, if one were interested in evaluating the impacts and benefits during or after the program implementation on clients, an Outcomes-Based approach would be appropriate. In ECS PROGRAMMING FOR CHILDREN WITH SEVERE DISABILITIES, an outcomes based evaluation would investigate if the program is actually making a difference in the form of enhanced learning, behaviour, or skill development for children with severe disabilities. In addition, an outcomes-based evaluation uses a system that investigates the inputs, outputs, outcomes, outcome targets, and outcomes indicators of the program. The use of this system gives the evaluator a holistic look at the program and helps in evaluating the effectiveness of the program.
Another possible approach to evaluating this program is a Goal-Based approach. McNamara states that a goal-based evaluation aims to evaluate if predetermined goals are being met. The evaluator would consider the predetermined timelines, resources, and goals in order to come to a conclusion. An example using our case-study is examining the amount of money and time allocated for each student to judge the program's ability to meet each child’s needs.
The last approach that could be used to evaluate this program is Process-Based. This approach would be useful to examine how the program works, especially if an outside group was interested in learning how the program received its results. This approach is known to be used when, as McNamara states; there are inefficiencies or a large number of complaints about the program.
All three of these approaches could be used to evaluate this program. It is entirely dependent on what aspect of the program is being evaluated. It is also entirely possible that any of these three programs be combined.
When starting this assignment I was inclined to choose between a goal-based approach and an outcomes-based approach. Now, after studying the two, I see that there isn’t a patently obvious line between the two and that an overlap of the two is a likely circumstance.
Wednesday, September 16, 2009
Thursday, September 10, 2009
Assignment 1
The program evaluation that I evaluated is titled "Youth Financial Education" and was conducted by the University of Wisconsin.
Citation: University of Wisconsin-Extension, Cooperative Extension (2008). Program evaluation report: Youth financial education. Madison, WI: UW-Extension, Program Development and Evaluation. http://www.uwex.edu/ces/pdande/evaluation/evalstudies.html
This evaluation is a summative, outcomes-based evaluation that aims to answer the following questions:
1) To what extent are desired outcomes occuring for youth participants related to changes in knowledge, skills, confidence, motivation, and behaviour in their financial practices?
2) What do youth view as the value of this type of reality/experiential program?
3) What suggestions and lessons can be learned to improve these reality programs?
The strengths of this evaluation include its transparency in regards to information and the authors diligence in following format protocol as proposed by Carter McNamara ( http://www.managementhelp.org/evaluatn/fnl_eval.htm).
Its logic model also clearly outline the input, output, and outcomes.
This evaluation has many strengths.
The evaluation was conducted by a third party, rather than by the organizatons (schools, program creators), thus giving the evaluation more credibility.
Furthermore, the authors of this evaluation have ensured that all 4 levels of evaluation (reactions and feelings, learning, change in skills, effectiveness) were explored and reported.
The selection of an outcomes based evaluation is appropriate because it evaluates whether the desired outcome ( if youth financial education does indeed make youth more knowledgeable about finances). The Y.F.E. evaluation is based on the increased knowledge, perceptions/attitudes, or skills of the participants.
The Y.F.E. evaluation uses questionnaire surveys with open and close ended items to elicit this information, which is an instrument that is used in many studies wishing to explore participant knowledge, perceptions, and known skills.
Additionally, the authors have included participants in the evaluation which some suggest is essential to the evaluation process ( Michael Scriven ).
The evaluators have used a variety of data collection methods, such as questionnaires with open and closed form questions. This strengthen the results. The evaluators also did a questionnaire-survey nine months after the initial questionnaire-surveys were given.
As there are strengths there are also weaknesses.First, the instrument used for data collection ( questionnaire survey) is not the most reliable instrument. There is a possibility that participants did not give careful feedback. This is more likely since participants were high school and middle school aged and the questions dealt with complex financial concepts (credit).
Second, the questionnaires were quite lengthy which may have caused participants to rush through it.
Third, the use of a questionnaire/survey may not allow participants to express their full opinions. Lastly and most importantly this evaluation does not include evidence of actual behaviour, rather it focuses on participants' perspectives. This leaves us with a crucial question: How are we to know if the program is actually effective?
While this was a very interesting evaluation to read through because I learned so much about how an evaluation should look and be conducted, I am left with the feeling that an additional evaluation must be undertaken to truly understand the outcomes of Youth Financial Education. Perhaps an evaluation where participants' finances are charted over the course of five years would offer hard evidence of Youth Financial Education outcomes.
Citation: University of Wisconsin-Extension, Cooperative Extension (2008). Program evaluation report: Youth financial education. Madison, WI: UW-Extension, Program Development and Evaluation. http://www.uwex.edu/ces/pdande/evaluation/evalstudies.html
This evaluation is a summative, outcomes-based evaluation that aims to answer the following questions:
1) To what extent are desired outcomes occuring for youth participants related to changes in knowledge, skills, confidence, motivation, and behaviour in their financial practices?
2) What do youth view as the value of this type of reality/experiential program?
3) What suggestions and lessons can be learned to improve these reality programs?
The strengths of this evaluation include its transparency in regards to information and the authors diligence in following format protocol as proposed by Carter McNamara ( http://www.managementhelp.org/evaluatn/fnl_eval.htm).
Its logic model also clearly outline the input, output, and outcomes.
This evaluation has many strengths.
The evaluation was conducted by a third party, rather than by the organizatons (schools, program creators), thus giving the evaluation more credibility.
Furthermore, the authors of this evaluation have ensured that all 4 levels of evaluation (reactions and feelings, learning, change in skills, effectiveness) were explored and reported.
The selection of an outcomes based evaluation is appropriate because it evaluates whether the desired outcome ( if youth financial education does indeed make youth more knowledgeable about finances). The Y.F.E. evaluation is based on the increased knowledge, perceptions/attitudes, or skills of the participants.
The Y.F.E. evaluation uses questionnaire surveys with open and close ended items to elicit this information, which is an instrument that is used in many studies wishing to explore participant knowledge, perceptions, and known skills.
Additionally, the authors have included participants in the evaluation which some suggest is essential to the evaluation process ( Michael Scriven ).
The evaluators have used a variety of data collection methods, such as questionnaires with open and closed form questions. This strengthen the results. The evaluators also did a questionnaire-survey nine months after the initial questionnaire-surveys were given.
As there are strengths there are also weaknesses.First, the instrument used for data collection ( questionnaire survey) is not the most reliable instrument. There is a possibility that participants did not give careful feedback. This is more likely since participants were high school and middle school aged and the questions dealt with complex financial concepts (credit).
Second, the questionnaires were quite lengthy which may have caused participants to rush through it.
Third, the use of a questionnaire/survey may not allow participants to express their full opinions. Lastly and most importantly this evaluation does not include evidence of actual behaviour, rather it focuses on participants' perspectives. This leaves us with a crucial question: How are we to know if the program is actually effective?
While this was a very interesting evaluation to read through because I learned so much about how an evaluation should look and be conducted, I am left with the feeling that an additional evaluation must be undertaken to truly understand the outcomes of Youth Financial Education. Perhaps an evaluation where participants' finances are charted over the course of five years would offer hard evidence of Youth Financial Education outcomes.
Subscribe to:
Posts (Atom)