Thursday, February 4, 2016

How to develop skills for Evaluation including: Search for Side Effects, Determine Value, Account for Viewpoints, Controversial Issues, Synthesize

How is it possible to develop the skills necessary to search for unintended side effects, to determine values, accounting for different view points, logically and diplomatically addressing controversial issues, as well as synthesizing facts and values?

These are a varied list of skills sets that all compliment one another but may require different methods to address or become expert in addressing them professionally. I will break each skill down by method, include references, and include examples from my own experience. If anyone reading my response disagrees, has additional information they would like to add, or can think of useful related resources please include them in this discussion.

Search for Unintended Side Effects

The ability to search for unintended side effects of a project or evaluation, in my experience, have often required actively engaging in cognitive evaluation. According to Google's dictionary definition, "Cogntivite thinking is the mental action or process of acquiring knowledge and understanding through thought, experience, and the senses." There is an excellent senior undergraduate psychology course at ISU that delves deeper into analyzing cognitive thinking. 
There is also a theory called Cognitive Evaluation Theory which explains the effects of consequences/events on motivation. These events may be relevant to the regulation of behavior (I really like using behavior analyses in my research/evaluations) and external factors. Some of the factors this may effect include perceived competence (Deci and Ryan, 1985),  informational value, perceptions, influence, motivation, perception of successs/failure, comparisons, perceived incompetence, disinterest, significance, and the theory can help evaluators better understand why individuals may undermine themselves or others based on perceived success in relation to past experiences. 
To summarize, having experienced failure or success in the past may effect an evaluators ability to recognize future side effects.
Brainstorming sessions and methodology that facilitate group ideation may also help evaluators recognize potential side effects. It is important to note, that the stakeholders in a brainstorming session need to include field experts related to the project in order for side effects to be properly addressed. A room full of technology neophytes may be able to bring up user experience issues a technology enthusiast may overlook. Both groups are unlikely to address specific technical issues of a large scale software development application the way a room full of experts would.
Brainstorming sessions can be facilitated in innumerable ways but a few useful examples may include:
  • A large round table meant for open discussion where ground rules are set to prevent ideas being shot down
  • Card sorting
  • Diary notes
  • Ask users for feedback
  • Quality assurance testing
  • Ask experts who may have worked on similar projects
  • Create a planning document that lays out a time line for the project. Break the time line down into manageable pieces. Evaluate the manageable pieces on a macro scale to see if there are any unexpected items to address or take note of.
  • Create a pros / con list
Other ways to learn to account for the unexpected:
  • Take a class on risk management.
  • Take a bootcamp workshop on project management and get a PMP certification (more HCI professionals should do this, in my opinion)
  • Create a budget for expected risks and unexpected risks (known as deviations, see also the Three Modes of Deviation Handling: Coping with Unexpected Events in Project Management)
  • Maintain good interpersonal team communication - this is key to any project or team and will go along way in facilitating successful risk mitigation
  • Hire competent team members, with proven ability to deal with known levels of stress related to your project - basically make sure you hire a team who is experienced (ESPECIALLY THE TEAM LEADER)
  • Know how to research things on your own (take a class on search engine optimizations and how to use Google/Libraries)
  • Know when to ask for help within your team or from external resources (don't waste your budget looking for answers you know are there when you could collaborate)

Determine Values

Determine values is vague. I would also like to point out, again, that the sentence Mathison wrote on page 186 of Fundamental issues of evaluation is her interpretation of Professor Scriven's work. The terminology she used and exact sentence written on page 186 do not appear in any resources I found referenced in her book.
While searching for resources that evaluate Mathison's writing I discovered, Fundamental Issues of Evaluation (2008). This is sited in my references. 
Page 5 (the preface) sites a resource that specifically addresses determining values. That paragraph is quoted here:
"Part III: Issues of Practice begins with “Complexities in Setting Program Standards in Collaborative Evaluation” (Chapter 7) by J. Bradley Cousins and Lyn M. Shulha. Cousins and Shulha address the fundamental issue, What level of performance must an evaluand achieve for its performance to be considered adequate or satisfactory? They discuss aspects of selecting bases for comparison, determining values, setting standards, and ensuring cultural sensitivity when stakeholders and evaluators work together to conduct formative collaborative evaluations. In such evaluations, power over evaluation decision making is shared equally among the evaluators, who have technical skills, and program personnel, whose understanding of the substance, organization, and daily operations of the program is far greater than that of the program’s evaluators. Such evaluations should be ideal for explicitly and ..."
I was unable to find a copy of the book site above (J. Bradley Cousins and Lyn M. Shulha, 1997), however I've sited it and a similar resource by the author in my references.
Additional resources for Determining Value:
  • Karl Stark and Bill Stewart (12/2011) take a simplistic economic approach and break complicated business theory and micro-economics into 4 steps that can be summed up to: determining value into yearly customer ROI, retention, cost of retention, overhead, business costs, and project
  • Qualitative analysis (community outreach, environmental impact, employee relations)
  • Market Value
  • Competitor Analysis
  • Stock or shareholder value
  • Nick L. Smith, Paul R. Brandon (2008) starting on page 29 has a case study of the U.S. Department of Education (2003a) proposed priority titled “Scientifically Based Evaluation Methods” - on page 30 the book goes on to note the AEA's response to this proposal including side effects. It's a worthwhile case study relevant to our discussion.
    • • Randomized control group trials are not the only means of establishing causality. • Randomized control group trials are not always the best way to establish causality. • Randomized control group trials sometimes should not be used for ethical reasons. • Randomized control group trials sometimes should not be used because data sources are insufficient. • Evaluators should employ a repertoire of methods. • Conditionality of effectiveness, not just causality, should be considered. 

Account for Different View Points

Quote from page 243 of Nick L. Smith, Paul R. Brandon (2008):
One definition of indigenous is “having originated . . . in a particular region” (Merriam-Webster’s Collegiate Dictionary, 1993, p. 591). Some key ideas regarding indigenousness include “We were here first. Newcomer, you do not have the right to impose your values on us, even if you have more destructive weapons than we do.” 
By decolonizing evaluation methodologies, we aim to recenter ourselves within our own lands. From here we challenge the viewpoints of those outside of our communities who see us as less than a “norm” that is based within their worldview rather than within ours. We are therefore advocating evaluation practices that are “of, for, by, and with us”— that is, K naka Maoli and Kaupapa M ori methodologies (Pihama, Cram, & Walker, 2002; Porima, 2005).

Logically and Diplomatically Address Controversial Issues

Addressing controversial issues may depend on context. For example, some issues do not have to be addressed. Granted, if someone directly confronts another individual on an issues it may further provoke the aggressor to ignore the issue and confrontation altogether, but it's still a viable option. At work, this may be a poor way to collaborate but online- especially on social media it may be best to simply block, ignore, or not participate at all in controversial discussion. I personally choose to read the comments of issues I would like to expand or better understand external viewpoints on, I do not however engage in many public online social media discussions. In fact, I've made a habit of blocking hundreds of people that I've never met or conversed with based on their online commentary and perceived repose (or lack of).
Back to context, rarely does it benefit a team to altogether ignore or leave controversial issues readdressed. Keep in mind though, if personal issues like financial, dating, politics, or controversial personal past times are irrelevant to the team or work place it may be acceptable, or moreover recommended to agree not to discuss them. Agreeing or making it a workplace policy to ignore these controversial topics, may even be considered a way to address that particular issue but in the case of harassment it is not acceptable to simply ignore an issue. It may also be important to offer private services like therapy for controversial past times like drinking, especially if those past times are harmful to team members. That is to say, drinking in itself may not be a controversial issue (unless discussed in a religious context) but drinking in excess or during necessary team related hours may be harmful and therefor may need to be addressed. 
Again, these are hypothetical examples with varied factors that would need to be addressed in further detail to properly articulate.
Controversial issues may be related to individuals who dislike or disagree. Disagreeing is not a bad thing, it may in fact be an opportunity to expand ones perspective and knowledge of the outside world when evaluated constructively. However, some disagreements may be articulated with words or actions that may have negative connotations such as words or gestures that may be considered offensive. Two colleagues may appreciate and respect one another without disliking each other, or they may dislike the other person on a fundamental level but really value their place on a team. In the same sense, a perspective or feedback may be appreciated, valued and respected regardless of personal cognitive reactions.
Ways to address controversial issues:
  • Discuss the issue openly in a group
  • Discuss the issue privately
  • Privately confront an individual or ask a leader to do so
  • Brainstorm on ways to mitigate issues
  • Brainstorm constructive or positive side-effects of an issue
  • Ignore the issue and choose to accept the risks/issue(s) in whole or in part
  • Determine ways to mitigate risk(s) by first acknowledging the risk then listing potential outcomes. Once a preferred outcome is determined implement policies, and assign team members to specifically address and insure tasks are taken to lead the project toward the preferred outcome.
  • Talk it out with a mentor - Discernment

Synthesize Facts and Values

  • Temple University Writing Center. What is Synthesis?
"A library presents an imposing vision: books neatly arranged according to reference numbers on endless rows of shelves. Initially, the wall-to-wall books make you feel that any fact you want to know must be in one of them and that the ideas in these books should fit together as neatly as the books fit together on the shelves. You have a comforting feeling that all knowledge in books interlocks to provide a smooth carpet of learning—everywhere even and firm under foot, no matter where you tread. 
When you actually start to look for specific information or try to find agreement between the books on a particular topic, you are more likely to feel that you have stepped into the Bad Lands of the Dakotas or the swamps of Florida. You cannot always find what you are looking for; what you do find may be contradictory or confusing. On the positive side, you may uncover some wonderful surprises-ideas and information that you had no idea existed. 
If you stop to consider why and how books are written, the unevenness of ground may not be so surprising. Each writer makes a particular statement, based on personal thinking and perceptions, to address a specific problem. Although authors may share common knowledge and familiarity with statements made by others, each individual uses these background materials and ideas in a unique way. 
... examine how to identify when texts truly disagree, how to locate their exact points of disagreement, and how to evaluate their disagreements to judge which side states a better case. This is the task of evaluative comparison."
According to Norman Herr, Bloom's Taxonomy (2016):
Synthesis- Synthesis refers to the ability to put parts together to form a new whole. This may involve the production of a unique communication (theme or speech), a plan of operations (research proposal), or a set of abstract relations (scheme for classifying information). Learning outcomes in this area stress creative behaviours, with major emphasis on the formulation of new patterns and structures.
Illustrative General Instructional Objectives Writes a well-organised theme. Gives a well-organised speech. Writes a creative short story (or poem). Proposes a plan for an experiment. Integrates learning from different areas into a plan for solving a problem. Formulates a new scheme for classifying objects (or events, or ideas).
Illustrative Verbs for Stating Specific Learning Outcomes Categorises, combines, complies, composes, creates, devises, designs, explains, generates, modifies, organises, plans, rearranges, reconstructs, relates, reorganises, revises, rewrites, summarises, tells, writes.



  • Google (2016). Define Cognition.
  • Deci, E.L., & Ryan, R.M. (1985). Intrinsic Motivation and Self-Determination in Human Behavior. New York: Plenum.
  • Technology Management for the Global Future, 2006. PICMET 2006 (2006, Volume:5 ) Three Modes of Deviation Handling: Coping with Unexpected Events in Project Management. Case Studies. IEEE.
  • Mathison, Sandra (2007). What is the difference between evaluation and research? And why do we care? In N.L. Smith & P. Brandon (Eds.).Fundamental issues in evaluation. New York: Guilford Publishers.
  • Coffman, J. (2003-2004), Winter). Michael Scriven on the differences between evaluation and social science research. Volume IX. Issue Topic: Reflecting on the Past and Future of Evaluation. Ask the Expert. The evaluation Exchange, 9(4).
  • Scriven, M. (1991). Evaluation thesaurus (4th ed.). Newbury Park, CA: Sage. or
  • Edited by Nick L. Smith, Paul R. Brandon (2008). Fundamental Issues in Evaluation. Guilford Publishers. or
  • Cousins, J. Bradley and Lyn M. Shulha (2008) Complexities in Setting Program Standards in Collaborative Evaluation. In Fundamental Issues in Evaluation,
  • J. Bradley Cousins and Lyn M. Shulha (1997). Evaluation Use: Theory, Research, and Practice Since 1986.
  • Karl Stark and Bill Stewart (12/2011). 4 Steps for Calculating Customer Value. INC Magazine Online. Accessed 2/2016.
  • Sharon Brisolara, Denise SeigartSaumitra SenGupta (2014). Feminist Evaluation and Research: Theory and Practice. Intersections and Divergence page 43. Guilford Publications.
  • Norman Herr (2007). Bloom's Taxonomy. Accessed 2016.
  • Colorado State University. COMPARING AND SYNTHESIZING SOURCES. Knowlege is Messy. Page 2. Accessed 2016.
    • Charles Bazerman (1992). The Informed Writer.
  • Temple University Writing Center. What is Synthesis?
My Photo

Senior UI/UX web designer at a large-scale IT contractor for defense, intelligence, and civilian government solutions. Adventurist and certified Yoga / Barre Instructor. Love aviation, books, and travel.Prefer long light hearted series in mystery, comedy, fantasy, and romance.

Share This Post

Share to Facebook Share to Twitter Email This Pin This Share on Google Plus Share on Tumblr

No comments:

Post a Comment

Backlinks will be hidden. Crude comments may be removed.