Defining "Evaluation"

Contents

1. Foreword
   • Note to class
   • Ana-Paula Quote
   • Fitzpatrick Book Reference
2. Definitions of 'Evaluation' by Various Resources
   • Google Definition of Evaluation
   • Wikipedia Definition of Evaluation
   • University of North Carolina Greensboro Evaluation Resources
   • Handbook of Human Performance Technology
3. Discussion Questions and Response
   • Question 1. How does Fitzpatrick, Sanders & Worthen (2011) define “evaluation”?
   • Question 1. How does Kirkpatrick (1998) define “evaluation”?
   • Question 1. How does Newby (1992) define “evaluation”?
   • Question 2. What are the differences and similarities in the meanings, contexts and scopes of their definitions?
4. Resources

Foreward

I'm looking forward to reading everyone's thoughts on the definition of evaluation, thoughts on the required reading, and how they might be using these in real life.

"This week’s discussion is around other broad concepts in Evaluation. For example, what does evaluation mean? How to define evaluation? We are also looking at how to make distinctions about different terminology: Needs Assessment, Monitoring, Outcome Studies, Testing, Measurement, and Assessment."

-- Ana-Paula, Welcome to Week 3 Online Discussion

I highlighted pages 7 and 9 of Fitzpatrick, J., Sanders, J. & Worthen, B. (2011). Program Evaluation: Alternative Approaches and Practical Guidelines (4th ed.). They seem to define the word evaluation for the book. The book includes the dictionary definition, a research definition, and a methods-based definition for  evaluators with external stakeholders.

Definitions of 'Evaluation' by Various Resources

1. Google Search: 'define evaluation':

"e·val·u·a·tion əˌvalyəˈwāSH(ə)n/ noun: evaluation; plural noun: evaluations

the making of a judgment about the amount, number, or value of something; assessment. "the evaluation of each method" synonyms: assessment, appraisal, judgment, gauging, rating, estimation,consideration"

• Google (2016), Define Evaluation. https://www.google.com/webhp?sourceid=chrome-instant&ion=1&espv=2&es_th=1&ie=UTF-8#q=define%20evaluation

---

2. Wikipedia: 'Evaluation'

For other uses, see Evaluation (disambiguation).

"Evaluation is a systematic determination of a subject's merit, worth and significance, using criteria governed by a set of standards. It can assist an organization, program, project or any other intervention or initiative to assess any aim, realizable concept/proposal, or any alternative, to help in decision-making; or to ascertain the degree of achievement or value in regard to the aim and objectives and results of any such action that has been completed. The primary purpose of evaluation, in addition to gaining insight into prior or existing initiatives, is to enable reflection and assist in the identification of future change.

Evaluation is often used to characterize and appraise subjects of interest in a wide range of human enterprises, including the arts, criminal justice, foundations, non-profit organizations, government, health care, and other human services."

• Wikipedia (2016), Evaluation. https://en.Wikipedia.org/wiki/Evaluation

---

3. University of North Carolina Greensboro

"Program evaluation is a systematic process of gathering evidence to inform judgments of whether a program, product, policy, or system is meeting is goals and how it can be improved to better do so. This definition of program evaluation reflects the evolution in the conceptualization of program evaluation that began with the early work of Scriven (1967) and extends to the later works of Michael Scriven (1991), Carol Weiss (1998), Melvin Mark, Gary Henry, and George Julnes (2000), Robert Stake (2000), Daniel Stufflebeam (2001), and Thomas Schwandt (2008).

Several key terms are commonly used in describing the evaluation process. These key terms are:

Evaluand: The evaluand is the program, product, policy, or system that is being evaluated.

Evaluator: The evaluator is an individual involved in conducting the program evaluation. Evaluators who are internal to the client’s organization or group are referred to as internal evaluators. Evaluators who are hired from outside of the client’s organization or group are referred to as external evaluators.

Stakeholders: The stakeholders associated with a program evaluation are the individuals who participate in, or are affected by, the program, product, policy, or system being evaluated.

Formative Evaluation: The primary purpose of a formative evaluation is to provide ongoing information for evaluand improvement.

Summative Evaluation: The primary purpose of a summative evaluation is to provide information to make programmatic decisions or to judge whether a program should be adopted, continued, or expanded."

• The University of North Carolina Greensboro (2016). Program Evaluation Resource Center. Overview of Program Evaluation. http://erm.uncg.edu/oaers/methodology-resources/program-evaluation/

---

7. Handbook of Human Performance Technology

According to the Handbook of Human Performance Technology (2006), "Evaluation is the means to ascertain the worth or value of a performance-improvement initiative. It can be used to improve a performance-improvement process or to decide to discontinue the effort. It is also useful in judging the relative worth of performance-improvement alternatives. Two types of evaluation are formative and summative."

This definition seems to correspond well with pages 7-9 of Program Evaluation: Alternative Approaches and Practical Guidelines (4th ed.). The HHPT continues on page 25 to explain the definitions of formative and summative evaluation methods. Formative being a method for improvements while a products is being built or designed and summative being a method used to judge merits after a project is complete.

• Harold Stolovitch, Erica Keeps, James Pershing (2006). Handbook of Human Performance Technology. (3rd Edition). Pfeiffer.

---

8. Dictionary Definition of Evaluation

"...one typical dictionary definition of evaluation is "to determine or fix the value of: to examine and judge."

• Fitzpatrick, J., Sanders, J. & Worthen, B. (2011). Program Evaluation: Alternative Approaches and Practical Guidelines (4th ed.), page 5. New York: Pearson.

Discussion Questions and Response

Question 1. How do Fitzpatrick, Sanders & Worthen (2011), Kirkpatrick (1998) and Newby (1992) each define “evaluation”?

Question 1. How do Fitzpatrick, Sanders & Worthen (2011), Kirkpatrick (1998) and Newby (1992) each define “evaluation”?

According to page 5 of Program Evaluation: Alternative Approaches and Practical Guidelines (PEAAPG), the authors, Fitzpatrick, J., Sanders, J. & Worthen (who I will refer to as the PEAAPG authors for brevity) state that, "..among professional evaluators there is no uniformly agreed-upon definition of precisely what the term evaluation means" they go on to imply that the role is also not defined.

Still on page 5 of PEAAPG the authors define the uses for inquiry and judgement methods in three guiding principles including, " 1. determining standards for judging quality and deciding whether those standards should be relative or absolute, 2. collecting relevant information, and 3. applying the standards to determine value, quality, utility, effectiveness, or significance."

On pages 6-7 the authors differentiate the evaluation and research based on their purpose, definition, implementation, stakeholders, criteria, and methodologies.

On pages 8-9, a section about informal versus formal evaluation the authors show examples of how old the concept of evaluation is based on the description, "examining and judging to determine value." These examples include the Welsh longbows in the Hundred Year War, Neanderthals, students, clients, and managers. Other sections like Uses and Objects Evaluation (page 14) also layout examples and use cases for evaluators. These examples set a foundation for the authors to show how 'faulty' judgments which are, "characterized by an absence of breadth and depth...(or) systematic procedures and formally collected evidence...(or) based on past experience" can cause the evaluators to make inaccurate decisions based on "biases" that may lead to unintended consequences or outcomes. The authors further identify the flaws in informal evaluations and evaluators as less cognizant of limitations, biased, lacking information, unrealistic, but sometimes the only realistic approach due to time/budget constraints. They do note that sometimes, if the evaluator is, "knowledgeable, experienced, and fair" an informal approach may be the best option for situations where formal evaluations can not be conducted.

On page 10 the authors reiterate, "that the basic purpose of evaluation is to render judgments about the value of whatever is being evaluated. Many different uses may be made of those value judgments, as we shall discuss shortly but in every instance the central purpose of the evaluative act is the same: to determine the merit or worth of some thing (in program evaluation, of the program or some part of it)."

Page 11 offers differing views and methods on evaluation, but they are not relevant to this question.

On page 13 the authors seem to agree with Lipsey (2000) that evaluators may play a secondary role as a scientific expert whose, "expertise is to track things down, systematically observe and measure..compare, analyze, and interpret with a good faith attempt at objectivity." They also state that the role of an evaluator is to educate users to an evaluators purpose including operational roles, managerial, program management, business development, and requirements gathering while advocating for users who may be disenfranchised.

On page 15, they define the evaluation object, evaluand, or evaluee is defined as anything/anyone who is being evaluated. The authors however have decided to avoid technical jargon ("precise language") except in direct quotes.

Pages 23-25 define the roles of internal vs. external evaluators.

Based on the consistent references to Scriven and the positive references to his quotes, I would also assert that the authors of PEAAPG agree with most of his methods and definitions related to evaluation.

Page 27 defines the limitations of evaluations. In short, the authors state that it is not a magic wand and won't effect a solution. It is a model to give context and bring about improvement.

Page 27-28, section, "Major Concepts and Theories" specifically defines and recaps the six key definitions relating to evaluation in this chapter:

1. Evaluation is the identification, clarification, and application of defensible criteria to determine an evaluation object's value, its merit or worth, in regard to those criteria. The specification and use of explicit criteria distinguish formal evaluation from the informal evaluations most of us make daily.
2. Evaluation differs from research in its purpose, its concern with generalizability, its involvement of stakeholders, and the breadth of training those practicing it require.
3. The basic purpose of evaluation is to render judgments about the value of the object under evaluation. Other purposes include providing information for program improvement, working to better society, encouraging meaningful dialogue among many diverse stakeholders, and providing oversight and compliance for programs.
4. Evaluators play many roles including scientific expert, facilitator, planner, collaborator, aid to decision makers and critical friend.
5. Evaluations can be formative or summative. Formative evaluations are designed for program improvement and the audience is, most typically, stakeholders close to the program. Summative evaluations serve decisions about program adoption, continuation, or expansion. Audiences for these evaluations must have the ability to make such "go-no go" decisions.
6. Evaluators may be internal or external to the organization. Internal evaluators know the organizational environment and can facilitate communication and use of results. External evaluators can provide more credibility in high-profile evaluations and bring a fresh perspective and different skills to the evaluation.

• Fitzpatrick, J., Sanders, J. & Worthen, B. (2011). Program Evaluation: Alternative Approaches and Practical Guidelines (4th ed.). New York: Pearson.

---

Question 1. How does Kirkpatrick define “evaluation”?

Question 1. How do Fitzpatrick, Sanders & Worthen (2011), Kirkpatrick (1998) and Newby (1992) each define “evaluation”?

Kirpatrick is introduced on page 15 of the book Program Evaluation: Alternative Approaches and Practical Guidelines (PEAAPG). In a section that defines the object of a formal evaluation study, Kirkpatrick's (1983) model for evaluating training efforts with broad and varying categories. This however, is not directly related to the question regarding the 1998 definition of evaluation by Kirkpatrick.

• Fitzpatrick, J., Sanders, J. & Worthen, B. (2011). Program Evaluation: Alternative Approaches and Practical Guidelines (4th ed.). New York: Pearson.

---

Question 1. How does Newby (1992) define “evaluation”?

Question 1. How do Fitzpatrick, Sanders & Worthen (2011), Kirkpatrick (1998) and Newby (1992) each define “evaluation”?

Newby offers a model for evaluating training, called context evaluation.

Will fill in.

• Fitzpatrick, J., Sanders, J. & Worthen, B. (2011). Program Evaluation: Alternative Approaches and Practical Guidelines (4th ed.). New York: Pearson.

Question 2. What are the differences and similarities in the meanings, contexts and scopes of their definitions?

Each definition of evaluation offers limitations to inform evaluators so that they can be cognizant of risk and influences to decisions.

• Fitzpatrick, J., Sanders, J. & Worthen, B. (2011). Program Evaluation: Alternative Approaches and Practical Guidelines (4th ed.). New York: Pearson.

Additional discussion points

This book was last published in 2011. It is five years old, and was published four years after the first iPhone. All of the information related to the terms evaluation are still very relevant and valid, except perhaps the perception that evaluation is undefined, it may be disagreed upon, but based on my brief research for this discussion board it appears to be thoroughly discussed, referenced, and defined in multiple formats for various types of work and methodologies. I wonder what the authors would say about modern evaluation methodology and its influence on stakeholders.

Resources: 

• Wikipedia (2016), Evaluation. https://en.wikipedia.org/wiki/Evaluation
• Rutgers, New Jersey Agricultural Experiment Station (2016). Program Evaluation Resources: Conducting Needs Assessments. http://njaes.rutgers.edu/evaluation/resources/needs-assessment.asp
• Google (2016), Define Evaluation. https://www.google.com/webhp?sourceid=chrome-instant&ion=1&espv=2&es_th=1&ie=UTF-8#q=define%20evaluation
• Fitzpatrick, J., Sanders, J. & Worthen, B. (2011). Program Evaluation: Alternative Approaches and Practical Guidelines (4th ed.), Chapter 1. New York: Pearson.
• The University of North Carolina Greensboro (2016). Program Evaluation Resource Center. Overview of Program Evaluation. http://erm.uncg.edu/oaers/methodology-resources/program-evaluation/
• Newby, A.C. (1992). Training Evaluation Handbook, London: Gower.
• Mark Spilsbury (1995). Measuring the Effectiveness of Training.
• Pulley (1994). Continuous Professional Development in Social Work, Responsive Evaluation (page 191). Jessica Kingsley Publishers.
• Dale Shunk, Barry J. Zimmerman (1998). Self Regulated Learning From Teaching to Self-Reflective Practive. The Guilford Press.
• Harold Stolovitch, Erica Keeps, James Pershing (2006). Handbook of Human Performance Technology. (3rd Edition). Pfeiffer.
• Peggy A. Ertmer, Timothy J. Newby (1996). Purdue University. The expert learner: Strategic, self-regulated, and reflective. Kluwer Academic Publishers. http://www.brown.edu/about/administration/sheridan-center/sites/brown.edu.about.administration.sheridan-center/files/uploads/ErtmerNewby1996_0.pdf