Engage Stakeholders
Who should be involved?
Teachers, administrators, superintendents, division coordinators,
How might they be engaged?
Staff meetings, email correspondence, volunteering, questionnaires
Focus the Evaluation
What are you going to evaluate? Describe program (logic model).
Teacher Professional Growth Plans- see logic model
What is the purpose of the evaluation?
The purpose of this evaluation is to evaluate the extent to which P.G.P.s are meeting the goals of the program, namely the enhancement of teacher instruction and student achievement.
Who will use the evaluation? How will they use it?
How will they use the information?
Administrators, superintendents, division coordinators, and teachers
To assess the effectiveness of the program and make changes ( if necessary) to help teachers meet the goals.
What questions will the evaluation seek to answer?
Do Professional Growth Plans promote professional growth, enhance instruction, and student achievement? (Are P.G.P.s meeting the goals set out by the division?)
Sub questions:
Do teachers have adequate resources to implement P.G.P.s?
Do teachers and administrators see growth in their students as a result of their P.G.P.s?
Do P.G.P.s encourage teachers to develop professionally?
How are P.G.P.s perceived by teachers and administrators?
Are P.G.P.s being used in the way that they are intended?
What are the benefits to teacher created PGP?
What information do you need to answer the questions?
Indicators – How will I know it?
I need to obtain teacher/administrator testimonials of their experience with Professional Growth Plans in order to answer the questions.
I will use the questionnaire and interview data to inform me of the programme's effectiveness.
When is the evaluation needed?
This is to be determined
What evaluation design will you use?
A goal-based evaluation design will be used.
Collect the information
What sources of information will you use?
Existing information:
Professional Growth Plan Tool Kit, Division PowerPoint,
People:
Teachers at Bishop Klein School
What data collection method(s) will you use?
Questionnaire with a larger sample. Interview with a small sample.
Monday, October 12, 2009
Assignment 3- logic model
GOAL
Statement of the overall purpose of the project
Professional Growth Plans are a yearly requirement for all teachers working for Greater Saskatoon Catholic School Division. Their rationale states that G.S.C.S.D. believes that a) the professional growth of the teachers enhances instruction and student achievement. B) the teacher has the primary responsibility for his or her professional growth.
OBJECTIVES
Specific statements of what the project sets out to accomplish
Statement of Policy:On an annual basis, each teacher shall develop a professional growth plan.
Rationale:
The Professional Growth Plan, designed to focus the goal(s) of the teacher in the upcoming year, will include at least one of the following strategies: peer coaching, mentorship, portfolio, reflective journal, action research, learning group, cognitive coaching, or other initiatives that enhance instruction and student achievement.
ACTIVITIES
Specific tasks to complete through implementation of the project
-creation of a committee comprised of teachers, administrators, division coordinators, superintendents, and teachers
-research
-Creation and publishing of Professional Growth Plan Toolkit for teachers
-workshops and meetings to discuss PGP with staff
-Collaboration between staff and administration
OUTPUTS
Immediate results (direct products of project activities)
1.The Professional growth Plan will be reviewed by the teacher and principal, or designate, and reported to the designated Superintendent of Education no later than October 31.
2.The principal, or designate, and teacher will revisit the Professional Growth Plan by May 3
Strategies for goal
achievement. These are
the actions that can be taken to achieve a P.G.P.
-taking a course
-attending conferences, seminars and workshops
-visiting another worksite
-reading or doing research
-peer coaching
- cognitive coaching
-creation of professional portfolios
-participating in learning communities
-mentorship
OUTCOMES
Intermediate results (1 to 3 years after project starts)
-Teacher participation in P.G.P.
-Enrollment in programs that allow professional growth
-enhanced student achievement
-growth in teachers/administrators and professional partnerships
a change of focus for supervision of instruction away from the individual, toward program effectiveness and improvement of instruction
-a more productive use of administrative time
-increased professional interactions and discussions among staff members
-greater sharing of responsibility for instructional leadership
-increased professional knowledge, involvement, and ownership.
-continuous learning
IMPACTS
Long-term results (3 to 10 years after project starts)
-increased professional knowledge, involvement, and ownership.
-continuous learning
-Enrollment in programs that allow professional growth
-enhanced student achievement
-growth in teachers/administrators and professional partnerships
a change of focus for supervision of instruction away from the individual, toward program effectiveness and improvement of instruction
-a more productive use of administrative time
-increased professional interactions and discussions among staff members
-greater sharing of responsibility for instructional leadership
-increased professional knowledge, involvement, and ownership.
-continuous learning
IMPACTS
Long-term results (3 to 10 years after project starts)
-increased professional knowledge, involvement, and ownership.
-continuous learning
Wednesday, September 16, 2009
Assignment 2
The case study, “ECS PROGRAMMING FOR CHILDREN WITH SEVERE DISABILITIES” aims to provide funding for students with severe disabilities, in order to meet their educational needs.
It is difficult to choose just one approach because there may be more than one aspect of the program evaluated.
For instance, if one were interested in evaluating the impacts and benefits during or after the program implementation on clients, an Outcomes-Based approach would be appropriate. In ECS PROGRAMMING FOR CHILDREN WITH SEVERE DISABILITIES, an outcomes based evaluation would investigate if the program is actually making a difference in the form of enhanced learning, behaviour, or skill development for children with severe disabilities. In addition, an outcomes-based evaluation uses a system that investigates the inputs, outputs, outcomes, outcome targets, and outcomes indicators of the program. The use of this system gives the evaluator a holistic look at the program and helps in evaluating the effectiveness of the program.
Another possible approach to evaluating this program is a Goal-Based approach. McNamara states that a goal-based evaluation aims to evaluate if predetermined goals are being met. The evaluator would consider the predetermined timelines, resources, and goals in order to come to a conclusion. An example using our case-study is examining the amount of money and time allocated for each student to judge the program's ability to meet each child’s needs.
The last approach that could be used to evaluate this program is Process-Based. This approach would be useful to examine how the program works, especially if an outside group was interested in learning how the program received its results. This approach is known to be used when, as McNamara states; there are inefficiencies or a large number of complaints about the program.
All three of these approaches could be used to evaluate this program. It is entirely dependent on what aspect of the program is being evaluated. It is also entirely possible that any of these three programs be combined.
When starting this assignment I was inclined to choose between a goal-based approach and an outcomes-based approach. Now, after studying the two, I see that there isn’t a patently obvious line between the two and that an overlap of the two is a likely circumstance.
It is difficult to choose just one approach because there may be more than one aspect of the program evaluated.
For instance, if one were interested in evaluating the impacts and benefits during or after the program implementation on clients, an Outcomes-Based approach would be appropriate. In ECS PROGRAMMING FOR CHILDREN WITH SEVERE DISABILITIES, an outcomes based evaluation would investigate if the program is actually making a difference in the form of enhanced learning, behaviour, or skill development for children with severe disabilities. In addition, an outcomes-based evaluation uses a system that investigates the inputs, outputs, outcomes, outcome targets, and outcomes indicators of the program. The use of this system gives the evaluator a holistic look at the program and helps in evaluating the effectiveness of the program.
Another possible approach to evaluating this program is a Goal-Based approach. McNamara states that a goal-based evaluation aims to evaluate if predetermined goals are being met. The evaluator would consider the predetermined timelines, resources, and goals in order to come to a conclusion. An example using our case-study is examining the amount of money and time allocated for each student to judge the program's ability to meet each child’s needs.
The last approach that could be used to evaluate this program is Process-Based. This approach would be useful to examine how the program works, especially if an outside group was interested in learning how the program received its results. This approach is known to be used when, as McNamara states; there are inefficiencies or a large number of complaints about the program.
All three of these approaches could be used to evaluate this program. It is entirely dependent on what aspect of the program is being evaluated. It is also entirely possible that any of these three programs be combined.
When starting this assignment I was inclined to choose between a goal-based approach and an outcomes-based approach. Now, after studying the two, I see that there isn’t a patently obvious line between the two and that an overlap of the two is a likely circumstance.
Thursday, September 10, 2009
Assignment 1
The program evaluation that I evaluated is titled "Youth Financial Education" and was conducted by the University of Wisconsin.
Citation: University of Wisconsin-Extension, Cooperative Extension (2008). Program evaluation report: Youth financial education. Madison, WI: UW-Extension, Program Development and Evaluation. http://www.uwex.edu/ces/pdande/evaluation/evalstudies.html
This evaluation is a summative, outcomes-based evaluation that aims to answer the following questions:
1) To what extent are desired outcomes occuring for youth participants related to changes in knowledge, skills, confidence, motivation, and behaviour in their financial practices?
2) What do youth view as the value of this type of reality/experiential program?
3) What suggestions and lessons can be learned to improve these reality programs?
The strengths of this evaluation include its transparency in regards to information and the authors diligence in following format protocol as proposed by Carter McNamara ( http://www.managementhelp.org/evaluatn/fnl_eval.htm).
Its logic model also clearly outline the input, output, and outcomes.
This evaluation has many strengths.
The evaluation was conducted by a third party, rather than by the organizatons (schools, program creators), thus giving the evaluation more credibility.
Furthermore, the authors of this evaluation have ensured that all 4 levels of evaluation (reactions and feelings, learning, change in skills, effectiveness) were explored and reported.
The selection of an outcomes based evaluation is appropriate because it evaluates whether the desired outcome ( if youth financial education does indeed make youth more knowledgeable about finances). The Y.F.E. evaluation is based on the increased knowledge, perceptions/attitudes, or skills of the participants.
The Y.F.E. evaluation uses questionnaire surveys with open and close ended items to elicit this information, which is an instrument that is used in many studies wishing to explore participant knowledge, perceptions, and known skills.
Additionally, the authors have included participants in the evaluation which some suggest is essential to the evaluation process ( Michael Scriven ).
The evaluators have used a variety of data collection methods, such as questionnaires with open and closed form questions. This strengthen the results. The evaluators also did a questionnaire-survey nine months after the initial questionnaire-surveys were given.
As there are strengths there are also weaknesses.First, the instrument used for data collection ( questionnaire survey) is not the most reliable instrument. There is a possibility that participants did not give careful feedback. This is more likely since participants were high school and middle school aged and the questions dealt with complex financial concepts (credit).
Second, the questionnaires were quite lengthy which may have caused participants to rush through it.
Third, the use of a questionnaire/survey may not allow participants to express their full opinions. Lastly and most importantly this evaluation does not include evidence of actual behaviour, rather it focuses on participants' perspectives. This leaves us with a crucial question: How are we to know if the program is actually effective?
While this was a very interesting evaluation to read through because I learned so much about how an evaluation should look and be conducted, I am left with the feeling that an additional evaluation must be undertaken to truly understand the outcomes of Youth Financial Education. Perhaps an evaluation where participants' finances are charted over the course of five years would offer hard evidence of Youth Financial Education outcomes.
Citation: University of Wisconsin-Extension, Cooperative Extension (2008). Program evaluation report: Youth financial education. Madison, WI: UW-Extension, Program Development and Evaluation. http://www.uwex.edu/ces/pdande/evaluation/evalstudies.html
This evaluation is a summative, outcomes-based evaluation that aims to answer the following questions:
1) To what extent are desired outcomes occuring for youth participants related to changes in knowledge, skills, confidence, motivation, and behaviour in their financial practices?
2) What do youth view as the value of this type of reality/experiential program?
3) What suggestions and lessons can be learned to improve these reality programs?
The strengths of this evaluation include its transparency in regards to information and the authors diligence in following format protocol as proposed by Carter McNamara ( http://www.managementhelp.org/evaluatn/fnl_eval.htm).
Its logic model also clearly outline the input, output, and outcomes.
This evaluation has many strengths.
The evaluation was conducted by a third party, rather than by the organizatons (schools, program creators), thus giving the evaluation more credibility.
Furthermore, the authors of this evaluation have ensured that all 4 levels of evaluation (reactions and feelings, learning, change in skills, effectiveness) were explored and reported.
The selection of an outcomes based evaluation is appropriate because it evaluates whether the desired outcome ( if youth financial education does indeed make youth more knowledgeable about finances). The Y.F.E. evaluation is based on the increased knowledge, perceptions/attitudes, or skills of the participants.
The Y.F.E. evaluation uses questionnaire surveys with open and close ended items to elicit this information, which is an instrument that is used in many studies wishing to explore participant knowledge, perceptions, and known skills.
Additionally, the authors have included participants in the evaluation which some suggest is essential to the evaluation process ( Michael Scriven ).
The evaluators have used a variety of data collection methods, such as questionnaires with open and closed form questions. This strengthen the results. The evaluators also did a questionnaire-survey nine months after the initial questionnaire-surveys were given.
As there are strengths there are also weaknesses.First, the instrument used for data collection ( questionnaire survey) is not the most reliable instrument. There is a possibility that participants did not give careful feedback. This is more likely since participants were high school and middle school aged and the questions dealt with complex financial concepts (credit).
Second, the questionnaires were quite lengthy which may have caused participants to rush through it.
Third, the use of a questionnaire/survey may not allow participants to express their full opinions. Lastly and most importantly this evaluation does not include evidence of actual behaviour, rather it focuses on participants' perspectives. This leaves us with a crucial question: How are we to know if the program is actually effective?
While this was a very interesting evaluation to read through because I learned so much about how an evaluation should look and be conducted, I am left with the feeling that an additional evaluation must be undertaken to truly understand the outcomes of Youth Financial Education. Perhaps an evaluation where participants' finances are charted over the course of five years would offer hard evidence of Youth Financial Education outcomes.
Subscribe to:
Posts (Atom)