Theorybased evaluation and objectivebased evaluation. While both performance measurement and program evaluation are considered measurement activities, the two activities serve different purposes. Baron and hers hey 1 explored outcome bias in evaluation of decision making on either medical matters or cor respondi ng au thor. The construction of an individual item on an evaluation scale which adversely affects some subset of the target audience. Develop a logic model for a fictitious case example. Such biases can be interpreted as a tendency to decide in certain simpli ed ways which can often lead to suboptimal. Overview of evaluation design corporation for national and. Outcome bias in clinical decision making and the assessment.
Obe is useful both as a planning tool and as an evaluation tool. After one week, about half of the monkeys will have made a profit and the other half a loss. Never judge a decision by its outcome meaning ring. As with other fields, medical sciences are subject to different sources of bias. How to successfully complete an evaluation and utilize. Sep 07, 2017 outcome based evaluation obe is defined by imls as a systemic way of assessing the extent to which a program has achieved its intended result. However, i would admit that such an investment decision was one where we deviated from all our norms and broke all our. It answers questions such as what difference did the program make. Outcomes evaluation report washtenaw coordinated funders. We introduced the evaluation of a decision without an outcome. Outcome bias is a cognitive bias that enables us to judge our decision making based on the results of the process rather than the quality of the process itself. Data were provided by 176 adults, ages 18 to 93 years, who completed computerized decision making vignettes and a battery of demographic and cognitive measures. Outcomes knowledge may bias radiological decision making.
Subjects rated the quality of thinking of the decisions, the competence of the decision maker, or their willingness to let the decision maker decide on their behalf. Decisions concerned either medical matters or monetary gambles. Say one million monkeys speculate on the stock market. Outcomes knowledge may bias radiological decisionmaking. Effect of intention on outcome bias in decision making. Overview of impact evaluation page 2 an impact evaluation should only be undertaken when its intended use can be clearly identified and when it is likely to be able to produce useful findings, taking into account the availability of resources and the timing of decisions about the programme or policy under investigation. Evaluation looks at the results of your investment of time, expertise, resources and energy, and compares those results with what you said you wanted to achieve in your programs logic model. Outcomes evaluation report prepared for washtenaw coordinated funders may 2017 two penn center philadelphia, pa 19102 philadelphia new york san francisco 31 west 27th street 225 bush street suite 1910 4th floor suite 1600 new york, ny 1 san francisco, ca 94104 2155680399 2129490990 4154398368. Outcome bias describes the phenomenon by which evaluators tend to take information about the outcome into account when evaluating the quality of a decision baron and hershey, 1988. Develop an action plan for using a logic model approach to planning and evaluation for hisher program. In the following we provide an overview of example decision biases 100s of these exist and their role in recommender systems. We document outcome bias in situations where an agent makes risky financial decisions for a principal. It is almost always better to make an informed decision than to make a decision based on gut feelings, in haste, etc.
What we miss when we judge a decision by the outcome. The evidence of outcome bias was explored in a twoplayer player 1. Outcome bias in the evaluation of financial agents christian konigkersting 1, monique pollmann2, jan potters2, and stefan t. While understanding sources of bias is a key element for drawing valid. The primary aim of this study was to examine the contributions of individual characteristics and strategic processing to the prediction of decision quality. Hershey department of decision sciences university of pennsylvania in 5 studies, undergraduate subjects were given descriptions and outcomes of decisions made by others under conditions of uncertainty. These findings that joint evaluation exacerbates the outcome bias seemingly contradict advice based on prior literature that prescribes joint evaluation as a tool for mitigating decision making biases bazerman et al. Pdf outcome bias in decision making on punishment or reward. Outcome evaluation definition of outcome evaluation by. United nations evaluation group standards for evaluation in. Colorado probation research in brief outcome bias in. Biases in decision making alexander felfernig institute for software technology, in eldgasse 16b, 8010 graz, austria alexander. A decisiontheoretic evaluation of early stopping rules.
Whereas research seeks to definitively answer a preformed hypothesis, triangulation. Apr 25, 2008 rather unsurprising study results, titled no harm, no foul, demonstrate that we judge the morality of choices by outcome we call the same decision immoral when it leads to a bad outcome, but moral when it leads to a good outcome and that we have a penchant both for punishing choices or choicermakers. This is at least as true in investing as it is in sports. Some of the challenges you face in doing an evaluation. Simply put, outcome bias in medicine occurs when the assessment of the quality of a clinical decision, such as the ordering of a particular test or treatment, is affected by knowledge. They buy and sell stocks like crazy and, of course, completely at random. Scope and method of the evaluation the evaluation of the effort sharing decision was performed in 2015 by the commission and was guided by a steering group consisting of members from the concerned directorategenerals in the commission. The experimental factors were the intention of an allocator player 1, the type of chosen dice selfish, fair, and generous, and the outcome selfish, fair, and generous. The systematic tendency to evaluate decisions based on their outcomes is called outcome bias ob baron.
Hershey department of decision sciences university of pennsylvania in s studies. One cognitive bias yet to be specifically addressed on the pages of sciencebased medicine, and it is one that i encounter regularly in practice, is outcome bias. Littlefair s1, brennan p2, mellothoms c2, dung p2, pietryzk m3, talanow r4, reed w2. In two experiments, the current paper documents evidence on outcome bias in financial decision making, specifically in the evaluation of. Interestingly even when individuals stated that the outcome should not have factored into the quality of the decision, they still showed a bias toward the decisions with positive outcomes. Refinement of outcome bias measurement in the parental decision. When people evaluate decisions after the outcomes of those decisions have been observed, they succumb to outcome bias. Pdf in 5 studies, undergraduate subjects were given descriptions and outcomes of decisions made by others under conditions of uncertainty. When the more risky option is not taken, they differ in whether the foregone outcome was greater 3, 8, 12, and 16 or less 4, 7, 11, 15 than the outcome obtained. The link between theorybased evaluation and objectivebased evaluation is logic modeling, which has been widely used, especially among practitioners, as a tool for evaluability assessment, management and evaluation. Outcome bias in decision evaluation jonathan baron university of pennsylvania john c. These cases can be used to look for a foregone outcome bias on the evaluation of decisions. Abstract subjects were given descriptions of decisions made by others under conditions of uncertainty, together with outcomes of those decisions.
Both individuals and groups exhibit bias in decisionmaking, and we should be aware of. Outcome bias does not involve analysis of factors that lead to a. A second reason for outcome bias is that the outcome calls attention to those arguments that would make the decision good or bad. How to avoid bias in your daytoday management decisions.
For this reason, it may often be adaptive for individuals to learn which options to avoid in the future. An external file that holds a picture, illustration, etc. We consider the decision theoretic evaluation of early stopping rules for clinical trials. The quality of the process that led to the investment decision was definitely not one that was rushed or haphazardly done. Evaluation is the use of research methods to assess a programs design, implementation, andor outcomes. Although the entire chapter is based on the example of grading students, it should be stressed that \grades are often used in contexts unrelated to the evaluation of the performance of students.
A decision theoretic evaluation of early stopping rules. Article pdf available in journal of personality and social psychology 544. Overview of evaluation design corporation for national. Triangulation can be used by public health officials to assess the impact of widely implemented interventions at the population level. Every evaluation is essentially a research or discovery project. Upon completion of this training, participants will be able to. Before we discuss evaluation design, we first want to provide you with a brief overview of performance measurement and program evaluation and, in particular, how these activities differ from one another. The outcome bias is an error made in evaluating the quality of a decision when the outcome of that decision is already known. A broad view of evaluation in health care includes three approaches, directed toward structure, process, and outcome, depending on the focus of evaluation and the criteria or standards being used. A decision is taken depending on the context in which alternatives are presented. Decisions are typically taken on the basis of heuristics that are a door opener for di erent types of decision biases. Good decisionmaking appears to be an expected outcome of education and. Outcome bias in decision evaluation school of arts and sciences. The head of evaluation should ensure that the evaluation function is.
Some decisions were medical decisions made by a physician or a patient, and others were decisions about monetary gambles. Outcome bias in decision evaluation article pdf available in journal of personality and social psychology 544. For example, a southwest native american child who has. Jul 19, 2019 outcome bias arises when a decision is based on the outcome of previous events, without regard to how the past events developed. One of the great est pains to hu man na ture is the pain of. The european environment agency also participated in this group. Outcome bias arises when a decision is based on the outcome of previous events, without regard to how the past events developed.
Looking back at the thesis, i do believe it was logical and wellthoughtout. The fact that something worked doesnt mean it was the result of a correct decision, and the fact that something failed doesnt mean the decision was wrong. How to use output measures to plan your effort or improve your programs operations. Fischhoff1975 demonstrated the existence of a hindsight bias, an effect of outcome information on the judged probabil. We examined the relations among age, domainspecific experience, working. In 5 studies, undergraduate subjects were given descriptions and outcomes of decisions made by others under conditions of uncertainty. Outcome bias in decision evaluation jonathan baron and john c. The supported decisionmaking new york or sdmny project goals are to. When a sample is inadvertently connected to desired outcomes, the evaluation will be affected. The issue is that people might infer from the outcome what the decision maker must have known at the time of the decision. In one study, we found that requiring evaluators to make judgments about decision makers choices before the result of those outcomes was known reduced the outcome bias under jointevaluation. Pdf outcome bias in decision evaluation researchgate.
1162 651 1109 1410 1431 833 354 626 983 347 747 1161 1458 269 369 374 552 160 657 1302 282 1184 1353 98 1061 339 743 461 222 320 812 1148 431 135 741 363 470 1059 127 771 886 429 535