4.1. Experimental Materials

Six scenarios were developed in which different types of stakes were manipulated (lives; physical injury; embarrassment; money; damage to objects of personal value). Four versions of each scenario were created in which the stakes were scaled in magnitude (see Supplementary Material for all scenarios and versions). For example, the ‘vaccine’ scenario involves changing the number of lives at stake should a vaccine be made incorrectly and administered to research participants:

Stakes 1: Low

Elaine is a medical researcher. Her task is to create a vaccine for a virus. Elaine has done this before, and she has a check list that specifies all of the steps she needs to take to make the vaccine. Elaine is following all of the steps correctly.

Elaine’s assistant has informed her that there is one human research participant who has volunteered to trial the vaccine before it is distributed more widely. If Elaine does not follow the steps correctly, it will produce an ineffective combination that when administered to the research participant will give them mild cold-like symptoms. 

In the above ‘low’ stakes version of the scenario, one individual will experience mild symptoms. These stakes are then incrementally raised in the remaining versions of the scenario:

Stakes 2

… one human research participant who has volunteered to trial the vaccine before it is distributed more widely. If Elaine does not follow the steps correctly, it will produce an ineffective combination that when administered to the research participant will kill him within days. 

Stakes 3

… 15 human research participants who have volunteered to trial the vaccine before it is distributed more widely. If Elaine does not follow the steps correctly, it will produce an ineffective combination that when administered to the research participants will kill them all within days. 

Stakes 4: High

… 100 human research participants who have volunteered to trial the vaccine before it is distributed more widely. If Elaine does not follow the steps correctly, it will produce an ineffective combination that when administered to the research participants will kill them all after several days of excruciating pain. 

After reading each scenario, participants were asked to respond to a knowledge prompt along a 7-point Likert-type scale (the endpoints of the Likert scale were labeled as follows: 1-Strongly disagree, 7-Strongly agree). Whether participants received a negative or positive knowledge prompt was determined by initial condition assignment (positive polarity condition; negative polarity condition). For example, having read the vaccine scenario (above) participants in the positive polarity condition were asked to rate their level of agreement with the statement:

Elaine knows that she is making the vaccine correctly

Participants in the negative polarity condition were asked to rate their level of agreement with the statement:

Elaine doesn’t know that she is making the vaccine correctly

To exclude any participants who failed to understand the experimental instructions, all participants had to respond correctly to a control question at the beginning of the experiment (Prompt 1) and a control question at the end of the experiment (Prompt 2) in order to be included in the data analysis. Participants who responded “agree” or “strongly agree” to Prompt 1 were removed and participants who responded “disagree” or “strongly disagree” to Prompt 2 were removed.⁴

Prompt 1

You have a fair coin, with heads on one side and tails on the other, that you flip into the air and catch on the back of your hand without looking at it.

Please indicate whether you agree or disagree with the following statement about the scenario:

You know the coin landed heads

1 (strongly disagree) – 7 (strongly agree)

Prompt 2

You have a fair coin, with heads on one side and tails on the other, that you flip into the air and catch on the back of your hand without looking at it.

Please indicate whether you agree or disagree with the following description of the scenario:

You don’t know that the coin landed heads

1 (strongly agree) – 7 (strongly disagree)⁵

4.2. Procedure

Participants in both the positive and negative polarity conditions were presented with six scenarios in four different degrees of stakes each (24 pairs of scenario + degree of stakes) in a randomized block design. Using this design ensures that all scenarios occur once in a sequence (or block) before any of them is repeated.

4.3. Participants

One hundred and twenty participants were recruited from MTurk and paid $1.75 each for participating. This research received ethical approval from the Department of Philosophy, University of Reading, UK and informed consent was obtained from all participants. Procedures were performed ex post to identify suspicious or low-quality responses in all datasets.⁶ Following screening procedures, eight responses were flagged as Virtual Private Server (VPS) responses and subsequently removed. As an additional screening procedure, participants who responded incorrectly to one of the above control prompts were removed from further data analysis. A total of 15 participants were removed having failed one or more of these checks, leaving a final sample of 97 participants (44 females, 52 males, 1 non-binary gender identity) between 20 and 71 years old (M = 39.64 years, SD = 12.00 years). Participants were randomly assigned to the positive polarity condition (N = 55) or the negative polarity condition (N = 42).

4.4. Hypothesis

If knowledge is sensitive to stakes, we should observe differences in levels of agreement with the knowledge prompts in at least some of the different degrees of stakes of a particular scenario. For example, we should find some difference between the “low stakes” and the “high stakes” versions of at least one of the six scenarios we tested, in either the positive or negative polarity. Independent of the stakes sensitivity of knowledge, there should also be an effect of switching the polarity of the prompt: If participants are willing to agree with the statement that the protagonist of a story knows that p in a particular scenario with a particular degree of stakes, participants should be less willing to agree with the statement that the protagonist doesn’t know that p in the same combination of scenario and degree of stakes.

4.5. Results

Overall and contrary to our hypothesis, there were consistent levels of agreement and disagreement across all stakes versions of all scenarios (see Figure 1). There was no main effect of stakes in any of the scenarios, meaning that we did not detect any influence of what or how much was at stake on levels of agreement with knowledge prompts (positive or negative). Our hypothesis regarding polarity was supported; across all scenarios, participants were more willing to agree that the protagonist knows that p and less willing to agree that the protagonist doesn’t know that p (for a full summary of individual statistics for each scenario, see Appendix I).

Figure 1.

Figure 1. Levels of agreement and disagreement with knowledge prompts across all degrees of stakes for all six scenarios. Levels of agreement and disagreement are consistent across all stakes scales (note the lines are level with little slope). Levels of agreement are significantly higher for positive polarity prompts than for negative polarity prompts. Error bars represent +/- 1 Standard Error (SE).

4.6. Discussion

Using the evidence-fixed approach, we found no evidence of a stakes effect on judgments about knowledge. This finding was consistent across all six of our scenarios when varying what was at stake—whether what was at stake was people’s lives, money, objects of personal value, or the degree of the protagonist’s embarrassment.

While these findings are in accordance with several previous evidence-fixed studies which failed to find an effect, they conflict with findings from Sripada and Stanley (2012), who did find evidence of a stakes effect on judgments about knowledge using an evidence-fixed design. Given the conflicting findings, we carried out a pre-registered replication of Sripada and Stanley’s (2012) experiment (see Appendix II for full method and analyses).

Using Sripada and Stanley’s between-groups experimental design, groups of participants were presented with one of six scenarios (N = 58 – 68 per group). Investigating three pairs of vignettes with a low and high stakes version, we asked all participants to respond to a question about the protagonist’s strength of evidence (what is the strength of S’s evidence that P), followed by a judgment about the protagonist’s knowledge (level of agreement/disagreement with: S knows that P). In the “basic” vignettes, we replicated Sripada and Stanley’s finding of an effect of stakes on quality of evidence: we found a medium-sized effect of stakes, with participants stating that the protagonist’s evidence was weaker in the high stakes case (see the left-hand side of Figure 2). But we did not replicate their finding of a stakes effect on quality of evidence in the “implicit/explicit” or “ignorant” vignette pairs. In judgments about knowledge, we found a small effect of stakes with participants being less inclined to agree with the knowledge claim in the “basic” vignette pair (see the right-hand side of Figure 2). We did not find evidence of a stakes effect on judgments about evidence or knowledge in the “implicit/explicit” or “ignorant” vignette pairs; these controlled for stakes being implicitly versus explicitly described (implicit low/explicit high) and the protagonist being ignorant of the stakes involved (ignorant low/ignorant high) (see Implicit/Explicit and Ignorant bars in figures). This is the opposite pattern of stakes effects that Sripada and Stanley found; they found effects in the “implicit/explicit” and “ignorant” vignette pairs, but not in the “basic” vignette pairs.

Figure 2:

Figure 2: Strength of evidence across the three vignette pairs (left hand panel) and Levels of agreement and disagreement with the knowledge claim across the three vignette pairs. (right hand panel). Error bars represent +/- 1SE. * = small effect size. ** = medium effect size (Cohen’s d). In the strength of evidence comparison (left panel) there was a medium effect of stakes in the basic vignette pair, (t(124) = 3.17, p =.002, d = 0.57) with strength of evidence higher in the low stakes scenario. The effect of stakes was not significant in the other vignettes [Implicit/Explicit: t(117) = 0.54, p = .588; Ignorant: t(120) = 0.84, p =.511]. For the levels of agreement comparison (right panel) there was a smaller effect of stakes in the basic vignette pair, (t(124) = -.2.57, p =.011, d = 0.46) with levels of agreement are higher in the low stakes scenario. Once again there was no significant effect of stakes in the other vignettes [Implicit/Explicit: t(117) = -1.48, p = .142; Ignorant: t(120) = -0.98, p =.329].

The results of our evidence-fixed experiment alongside the findings from our replication of Sripada and Stanley (2012) indicate that while the “evidence-fixed” experimental design is capable of uncovering stakes effects on judgments about knowledge, those effects are hard to find and, if found, are small. One additional worry about the effects found in the original Sripada and Stanley (2012) study—as well as our replication—is that the question asking participants whether they agree with the knowledge claim always follows the quality of evidence question—the order of those questions is not varied. This ordering could potentially be influencing how participants are responding to each prompt and contributing to the observed effect.⁷ In our larger, more diverse “evidence-fixed” experiment, we did not ask participants about the quality of the evidence available to the protagonist before asking them whether they agreed with a claim about knowledge. That difference could account for the divergent finding in the different evidence-fixed experiments we conducted. We return to this possibility in our General Discussion (§6).

Although the results of the “evidence-fixed” design have provided only mixed results, the “evidence-seeking” design used in Pinillos (2012) and Pinillos and Simpson (2014) has consistently found evidence of stakes effects. For that reason, we therefore conducted an “evidence-seeking” version of our experiment to further probe for the scalarity of stakes effects.

5.1. Materials and Procedure

The same set of scenarios were presented to participants in the same randomized block design as in Experiment 1. As before, one group of participants received positive prompts and the other received negative prompts.

In this experiment, the original knowledge prompts from Experiment 1 were replaced with evidence-seeking prompts. For example, in response to the vaccine scenario, participants in the positive polarity condition were asked:

and participants in the negative polarity were asked:

As in Pinillos and Simpson (2014), after reading the positive polarity prompt, participants were asked to respond as follows:

enter a whole number: 1, 2, 3... etc. If you think Elaine knows without having to check, write "0". If you think Elaine will never know no matter how many times she checks, write "never"

Participants were asked to respond as follows in the negative polarity condition:

enter a whole number: 1, 2, 3... etc. If you think Elaine will never know no matter how many times she checks, write "never"⁸

5.2. Participants

One hundred and twenty participants were recruited from MTurk and paid $1.75 each for participating. This research received ethical approval from the Department of Philosophy, University of Reading, UK and informed consent was obtained from all participants. As before, screening procedures were performed ex post to identify suspicious or low-quality responses in all datasets. Following screening procedures, three suspicious responses were flagged and removed. Additionally, participants who responded incorrectly to one of two control prompts (these were the same coin control prompts used in Experiment 1) were removed from further data analysis. A total of 8 participants were removed having failed these checks leaving a final sample of 109 participants (54 females, 55 males) between 21 and 74 years old (M = 38.98 years, SD = 11.76 years). Participants were randomly assigned to a positive polarity condition (N = 58) or a negative polarity condition (N = 51).

5.3. Hypothesis

If knowledge is sensitive to stakes, we expect to find some effect of changing stakes on responses to a given polarity in a given scenario. For example, we should find some significant difference in responses to the “low” and “high” stakes conditions for positive or negative polarity prompts in at least one of the six scenarios we considered.

Regarding the effect of polarity, we do not expect to find a significant difference between responses to the negative and positive polarity prompts. That’s because participants should be searching for the threshold of how much evidence is required for knowledge when presented with either the positive or the negative prompts. To explain why, consider a particular example of a positive prompt in the “high” stakes version of the vaccine scenario:

Positive: How many times does Elaine need to consult her check list before she knows that she is making the vaccine correctly?

If participants respond to the positive prompt with 3 times, they should give a similar number in response to the negative prompt in the same combination of scenario and degree of stakes:

Negative: How many times can Elaine consult her check list and still not know that she is making the vaccine correctly?

It would be unexpected to find that participants would say that Elaine could consult her check list a significantly different number of times and still not know that she’s making the vaccine correctly while saying that she needs to consult her check list only 3 times before she knows that she’s making the vaccine correctly. ⁹

5.4. Results

Overall and in line with our hypothesis, there were effects of changing stakes on responses across both the negative and positive polarities in all scenarios (in the possessions scenario, this change in responses was observed in the positive polarity only) (see Figure 3).¹⁰ Across scenarios, as the stakes increased, there was a general pattern of participants stating that more evidence would be needed. In terms of the scalarity of these stakes effects, there was a clear pattern of lower stakes scenarios requiring less evidence than higher stakes scenarios (for a full summary of scenario and stakes scales analyses, see Appendix III). We did not observe any significant difference in responses to positive and negative polarity prompts (for a full summary of individual statistics for each scenario, see Appendix III).

Figure 3.

Figure 3. The amount of evidence participants state is needed in order for the protagonist to know/or that can be had while the protagonist still doesn’t know across all stakes versions of all six scenarios. Amount of evidence required increased as the stakes were raised in each scenario (note positive gradient of lines). In the possessions scenario, this effect was observed in the positive polarity only. Error bars represent +/- 1SE.

In the evidence-seeking task, we also collected two additional types of response from participants:

enter a whole number: 1, 2, 3... etc. If you think Elaine knows without having to check, write "0" (in the positive polarity condition). If you think Elaine will never know no matter how many times she checks, write "never" (in both the positive and negative polarity conditions)

Previous papers simply discarded these “never” responses from further analysis (Pinillos & Simpson 2014: 21) and in terms of zero responses, our analyses did not include values less than or equal to 0.¹¹

Upon closer inspection, we noticed something unexpected about these responses: There was a much larger number of “never” answers given in response to the negative polarity prompt concerning how much evidence a protagonist could have and still not know that p than in response to the positive polarity prompt. “Never” answers made up 22% of the overall responses in the negative polarity group but only 2% of responses in the positive polarity group (see Figure 4). In effect, many more participants were responding to the prompt “skeptically”—that is, responding that they thought that no matter how many times she checks, S will never know that p.¹² We also observed a larger percentage of “0” responses to the positive (8.5%) than in the negative prompts (4%), but that isn’t surprising given that participants weren’t explicitly given the option to respond with “0” if the subject knows without having to check in the negative polarity prompts (see Figure 4). The meaning of a “0” response when given in response to a positive vs. a negative polarity prompt is therefore probably different: in response to a positive prompt such a response means S knows without having to check; in response to a negative prompt, it’s not clear what a “0” response means.

Figure 4.

Figure 4. The percentage of “never” responses (left) and zero responses (right) given in response to positive and negative prompts (across all scenarios). The proportion of “never” responses was significantly higher in the negative polarity (χ²(1) = 188.52, p < .001). The proportion of “0” responses was significantly higher in the positive polarity (χ²(1) = 28.51, p < .001). Error bars represent +/- 1SE_p (converted to %).

Once we noticed these large disparities in “never” and “0” responses between the positive and negative polarity prompts, we wondered whether there might be a stakes effect on the rate at which participants gave these responses. If knowledge is sensitive to stakes, it’s possible that the number of these declarations (S will never know no matter how many times she checks and S knows without having to check) will vary when the stakes vary.

In order to investigate this, we analysed the frequency of “never” and “0” responses in each stakes scale across all scenarios. We found no main effect of stakes in global analysis across all stakes scales for “never” responses (see left-hand side Figure 5).¹³ But we did find an overall pattern of participants being less likely to say that S knows P without having to check (“0”) as the stakes increased in the positive polarity (see right-hand side Figure 5).¹⁴

Figure 5.

Figure 5. The percentage of “never” responses (left-hand side) and zero responses (right-hand side) given in each stakes scale (across all scenarios) in each polarity condition. The frequency of “never” answers given in response to negative prompts increased as the stakes were raised (note positive gradient of line) although this was not significant in inferential analyses. The frequency of zero answers given in response to positive prompts significantly decreased as the stakes were raised (note negative gradient of line). Error bars represent +/- 1SE_p (converted to %).

5.5. Discussion

Using the evidence-seeking approach, we found evidence of a stakes effect: As the stakes were raised, individuals stated that the protagonist would need to gather more evidence in order to know something or that she could gather more evidence and still not know something. This finding was consistent across the majority of our scenarios (bar the scenario involving personal possessions in which a stakes effect was found for the positive polarity only). We further found evidence that these stakes effects were scalar, not binary. In the majority of scenarios, participants stated that less evidence would be required by the protagonist in stakes 1 [low] cases when compared to the stakes 4 [high] cases. However, the details of how degrees on the scale of stakes affected knowledge judgments varied across different scenarios. For both the paramedic and vaccine scenarios (the vignettes in which lives were at stake), the stakes 1 [low] case was significantly different to the stakes 3 case with the amount of evidence plateauing after this point. That could be due to participants reaching a saturation point of how much the number of lives at stake affect knowledge. In the mountaineering scenario, the stakes 1 [low] case was significantly different to all of the subsequent stakes cases (i.e., stakes 2, stakes 3, and stakes 4 [high]). In the possessions scenario, the reverse pattern was found with the stakes 4 [high] case being significantly different to all previous stakes cases (i.e., stakes 1 [low], stakes 2, and stakes 3). Finally, in the game show scenario, the stakes 1 [low] case was significantly different to the stakes 3 and stakes 4 [high] cases. But the stakes 2 case was also significantly different to the stakes 4 [high] case.

The variability we observed in scalar stakes effects across the different scenarios is unsurprising, given that (i) the scenarios differed in their details, including what the scale of stakes was measuring (lives, money, embarrassment, non-monetary possessions, personal injury), and (ii) we relied on our own intuitive, non-systematic sense of what would make for noticeable differences between different degrees of stakes on each scale. Future investigations of scalar stakes effects could systematically construct particular types of stakes scales in order to evaluate more precisely how the shape of particular stakes scales affect knowledge judgments. For example, according to Kahneman and Tversky’s (1979) prospect theory, people experience monetary losses and gains with diminishing sensitivity: moving from $5 to $10 is experienced as a bigger gain than going from $95 to $100. And people are “loss-averse”—losing $5 hurts more than gaining $5 feels good. With these factors in mind, it would be possible to more systematically construct monetary stakes scales with the aim of making the differences between degrees on those scales more regular. That would also serve to bring the discussion of stakes understood as differences in expected utility (as on the Anderson and Hawthorne proposal discussed in §2) into contact with what we know about human decision making under risk and uncertainty. That is a potentially rich area for future experimental work on stakes effects.

With regards to the “never” responses, we found a large framing effect in the different polarities: people were more likely to respond with “never” (S can never know that P) when presented with a negative prompt. That result is surprising. We predicted that there should be no significant difference in numerical responses to the positive and negative polarity prompts, so we assumed that there should be no significant difference between the “never” responses to the positive and negative polarity prompts as well. What could be driving this framing effect? One possibility is that participants are being encouraged by the negative prompt to consider possible situations in which someone could fail to know that P in spite of having a great deal of evidence that P is the case. When that possibility is suggested, it might appear reasonable to respond that S will never know, no matter how much evidence she has, because she can’t rule out the possibility that she is in such a situation. When confronted with the positive polarity prompt, in contrast, participants are not being encouraged to consider such a situation.¹⁵

The fact that we found a stakes effect on “0” responses in the positive polarity group is a previously unnoticed stakes effect: people are less likely to say that S knows P without having to check as the stakes increase. Although we did not find a significant stakes effect on the “never” responses, there was a pattern of people stating that S can never know that P in greater numbers as the stakes were raised. As we will discuss below, in a follow-up experiment we did observe a stakes effect on “never” responses.

In order to illuminate these unexpected findings, we ran two follow-up studies that addressed two worries that we had with the existing experimental design:

1. The existing negative prompt included the phrase (in bold): “how many times can S check and still not know that P” which may have triggered the presupposition that the protagonist does not know that P, which might have contributed to the framing effect on “never” and “0” responses.

2. The option to “write ‘0’ if you think S knows without having to check” was only included after positive prompts (not negative prompts) which may have contributed to the framing effect on “never” responses by giving the positive polarity group a different response anchor.

To address our first concern, we ran a follow-up experiment (Symmetrical Experiment) which replicated the existing paradigm but with symmetrical negative and positive prompts, removing the phrase “still not know” (see Appendix IV for full experimental details). And to address our second concern, we ran another follow-up experiment (Matched Experiment) which replicated the design of the Symmetrical Experiment but simply removed the option to “write ‘0’ if you think S knows without having to check” from the positive polarity prompts (see Appendix IV for full experimental details).

The framing effect on “never” responses was preserved across both experiments when controlling for the wording of the prompt and when removing the additional prompt option in the positive polarity (see Figure 6).

Figure 6.

Figure 6. The percentage of “never” answers in the Symmetrical Experiment (left-hand side) and the Matched Experiment (right-hand side) given in response to positive and negative prompts (across all scenarios). The proportion of “never” responses was significantly higher in the negative polarity in the Symmetrical Experiment (χ²(1) = 115.32, p < .001) and in the Matched Experiment (χ²(1) = 113.72, p < .001). Error bars represent +/- 1SE_p (converted to %).

In the second, Symmetrical, experiment, we found a stakes effect on “never” responses (see figure 7).¹⁶ (For full analyses, see Appendix IV.)

Figure 7.

Figure 7. The percentage of “never” responses given in each stakes scale (across all scenarios) in each polarity condition of the Symmetrical Experiment. The frequency of “never” answers given in response to prompts increased as the stakes were raised from stakes 1 [low] to stakes 3.

The finding of a stakes effect on “never” responses is important because it offers a response to a powerful theoretical objection that has been made to the use of evidence-seeking experiments as a way of investigating stakes effects on knowledge. Buckwalter and Schaffer (2015: 214) argue that the reason that changing stakes affect judgments about evidence-seeking prompts is that the evidence-seeking prompts contain a deontic modal (“need”, or “has” in Pinillos 2012), and it is uncontroversial that what is at stake affects judgments about what someone needs to do; it is the effect of changing stakes on the deontic modal that is driving participants’ responses, rather than any effect of stakes on knowledge. However, when participants respond with “never”, they are responding to a secondary prompt that does not contain a deontic modal, namely:

If you think Elaine will never know no matter how many times she checks, write "never".

Since we found evidence that the rate at which participants respond with “never” is itself sensitive to stakes, that looks like evidence of a genuine stakes effect on participants’ judgments about whether someone can ever know that p which cannot be explained as an effect of stakes interacting with an interpretation of a deontic modal.¹⁷

We offer this as a tentative response only, because we only found a stakes effect on skeptical “never” responses in one out of three of the evidence-seeking experiments we conducted. In the third and final Matched Experiment that we ran, as in the first experiment, we did not find a stakes effect on skeptical “never” responses. (See Appendix IV for detailed analyses.) The mixed findings across our three evidence-seeking experiments regarding the stakes effect on skeptical “never” responses might arise from the fact that the effect is small—this is something that will have to be addressed in future research designed specifically to investigate stakes effects on skeptical judgments.

A referee pointed out that our finding of a stakes effect on both positive and negative knowledge prompts (“S doesn’t know that P”) also presents a challenge to Buckwalter and Schaffer’s objection to the results of evidence-seeking experiments. The negative prompt, “How many times can S [check] and not know that P?” contains a modal expression (“can”), but it’s not a deontic modal—it’s either an ability modal or a modal expressing metaphysical possibility. While stakes clearly affect what one should do (for example, in how we interpret the deontic modal “have” in the positive prompt), stakes don’t obviously affect what one is able to do, or what is possible in a metaphysical sense. The stakes effect we found on responses to the negative prompts therefore isn’t easily explained in the same way that Buckwalter and Schaffer explain the stakes effect on positive prompts in evidence-seeking experiments.

A stakes effect on “0” responses was found in the Symmetrical Experiment (see analyses in Appendix IV) although this time, the effect was seen across both prompts. So, participants were more likely to respond with “0” in lower stakes scenarios when presented with both positive and negative prompts.

As well as allowing us to investigate the “never” responses further, these follow-up experiments also served as opportunities to replicate the stakes effects found in the first evidence-seeking experiment. Overall, we found that the stakes effects observed in our first evidence-seeking experiment replicated across one or both of the follow-up experiments (for a full summary of individual scenario analyses see Appendix IV), with the paramedic scenario (number of lives at stake) and the game show scenario (amount of money at stake) consistently producing stakes effects across all three of the evidence-seeking experiments.

6.1. Evidence-Fixed versus Evidence-Seeking Results

Using the classic evidence-fixed design employed in earlier experimental studies of stakes effects on knowledge, we did not find evidence of a stakes effect on judgments about knowledge across several epistemic scenarios.¹⁸ However, in a second series of experiments which employ the evidence-seeking approach developed in Pinillos (2012), we did find evidence of a stakes effect across multiple scenarios.

Based on our findings and the results of previous studies, there are two types of competing explanations for this pattern of negative and positive results in the two types of experiment. The first type of explanation, which is favorable to the existence of genuine stakes effects on knowledge, is that there is some feature of the evidence-fixed design that obscures such a stakes effect. Pinillos (2012: 198) and Sripada and Stanley (2012: 10) hypothesize that stakes effects on knowledge exist, but they can be difficult to observe in evidence-fixed experimental designs because participants assume that protagonists in higher stakes situations will have gathered more evidence than those in lower stakes scenarios, leading to a tendency to judge that subjects know that P at greater rates in higher stakes situations, which would suppress any effect of higher stakes lowering the tendency to judge that subjects know that P. The possibility that participants may be revising their sense of how much evidence the protagonist has upwards in the high stakes case could therefore potentially explain why participants are equally likely to agree with the statement that the protagonist knows in the high stakes case as in the low stakes case.

Another factor that might be obscuring an underlying stakes effect in the evidence-fixed design is that the quality of the subject’s evidence that P is so high in all conditions (from low to high) that stakes aren’t having an observable effect on judgments about whether the subject knows that P, because judgments that the subject knows will already be at or near ceiling. The data represented in Figure 1 is compatible with this possibility, since agreement with “S knows that P” is consistently high across scenarios and degrees of stakes, and agreement with “S doesn’t know that P” is consistently low across scenarios and degrees of stakes. It’s possible that scenarios in which the subject’s evidence is lower quality might leave room for the stakes effect to show up in participants’ responses.¹⁹

A second type of explanation of the divergent patterns of results is not favorable to the existence of genuine stakes effects on knowledge. One version of this type of explanation holds that the failure to detect a stakes effect in most evidence-fixed experiments is because there is no stakes effect on knowledge, while the finding of an effect in the evidence-seeking experiments arises not from a stakes effect on knowledge, but a stakes effect on the deontic modal (“have”) that appears in the evidence-seeking prompt (Buckwalter & Schaffer 2015).

Our finding of a stakes effect on sceptical “never” responses in our second “Symmetrical” experiment, and our finding of a stakes effect on responses to the negative polarity prompts provide a novel response to this explanation: since neither the prompt to which the “never” responses are directly offered, nor the negative polarity prompts, contains a deontic modal, the observed stakes effect cannot be explained away as the effect of stakes on a deontic modal.²⁰

Another version of the second type of explanation (which is not favorable to the existence of a stakes effect on knowledge) is proposed in Gerken (2017). Gerken explains the apparent stakes effect in Pinillos’s evidence-seeking experiments as resulting from what he calls an “Epistemic Actionability-Proxy” heuristic, which leads participants to interpret the prompts in the evidence-seeking design as “concerning how much evidence S should gather before it is reasonable to act on P, rather than concerning the nature of knowledge” (2017: 271).

While Gerken’s explanation accounts for the stakes effect we found in responses to positive polarity prompts, it doesn’t easily account for our finding of a stakes effect on “never” responses, and on responses to the negative polarity prompts (“How many times can S check and not know that P?”). Gerken might argue that “never” responses indicate that participants think that the subject in the scenario should never perform the relevant action (checking that the steps for creating the vaccine have been correctly carried out, e.g.), but that strikes us as implausible (see Footnote 17, above). Also problematic for Gerken’s explanation is the fact that we found a stakes effect in response to negative polarity prompts, which can’t be interpreted as asking how much evidence S should gather before it is reasonable to act on P. He could potentially argue that the negative prompts are proxies for a question about how much evidence S could gather and yet not act on P, but as Nagel (2011) discusses, sentential negation (such as occurs in our negative prompts) generally triggers effortful type-2 processing, which would interfere with Gerken’s heuristic-based (type-1) explanation of responses to the evidence-seeking prompts.

Like Gerken, Nagel (2008) provides an explanation of stakes effects in terms of a psychological effect that is not (directly) a stakes effect on knowledge, namely “need-for-closure”. “Closure” is the arrival at a settled belief; prior to closure, one’s mental state is “open” or non-committed (2008: 287). “Low need-for-closure” is a state in which a subject is “strongly averse to inaccurate or premature judgment, as in [high-stakes scenarios]”, while subjects in low-stakes scenarios are in a state of “neutral” need-for-closure (2008: 288). Subjects who have a lower need-for-closure seek more evidence before settling on a belief, and are characterized by lower degrees of confidence in the belief even once settled. Given that subjects in high-stakes scenarios are also in a state of low need-for-closure, Nagel argues that stakes effects might be driven by the fact that it is belief formation that is directly sensitive to stakes, while knowledge is only sensitive to stakes indirectly (assuming belief is a component of knowledge). Nagel’s competing explanation of stakes effects in terms of need-for-closure is not ruled out by the stakes effects we found in the evidence-seeking experiments.²¹

While our experiments do not rule out the possibility that stakes effects could be explained in terms of need-for-closure, Pinillos (2012: 202) ran an experiment in which participants were explicitly told that the subject in the scenario “forms the belief” that P before they are asked to judge how many times the subject has to check before she knows that P. Even with this modification to the evidence-seeking design, Pinillos still found a significant stakes effect on responses to the knowledge prompt, which provides reason to doubt that stakes are affecting knowledge indirectly through affecting belief formation.²²

As discussed above, Gerken (2017) and Nagel (2008) use mechanisms drawn from cognitive psychology to explain apparent stakes effects in a way that is consistent with “intellectualist invariantism”, the view that practical factors like stakes do not directly affect knowledge. Another approach to making intellectual invariantism compatible with the apparent stakes effects on knowledge is to invoke pragmatic linguistic mechanisms, like conversational implicature, to explain what participants are responding to when asked to judge whether a subject knows something.²³ To the best of our knowledge, no one has proposed a pragmatic explanation of the apparent stakes effect revealed in evidence-seeking experiments. But we did consider one possible pragmatic confound present in the first version of our evidence-seeking experiment, in the form of the presupposition trigger “still” that appeared in the negative prompts in our first evidence-seeking experiment (“How many times can S [check] and still not know that P?”). But we replicated our findings of a stakes effect and framing effect (in which there were far greater numbers of “never” responses in response to negative prompts than to positive prompts) even when “still” was removed from the negative prompts (see §5.5).

6.2. Framing Effects and Skepticism

We uncovered a large framing effect on participants’ willingness to say that a subject in a scenario never can know that something is the case. These skeptical responses appeared at a much greater rate when participants responded to negative polarity prompts.

6.3. Advantages of Our Methodology

In order to investigate the stakes sensitivity of knowledge, we have incorporated a diverse set of scenarios that vary both what is at stake and how much is at stake. Previous research has predominately incorporated a single pair of vignettes that involve more or less commonplace scenarios. By including a variety of stakes, from extreme cases involving dozens of lives at risk in spectacular circumstances, to less extreme cases involving degrees of embarrassment, as well as vignettes that scale these stakes in magnitude, we can begin to build a finer-grained picture of the stakes sensitivity of knowledge than is possible from previous studies. Aside from being statistically more powerful, the variety of scenarios we employed also speaks to the generality of the effect—where we have consistently found (in the evidence-seeking design) stakes, framing and scalar effects across scenarios we can be more confident that such results are not unforeseen artefacts of the particular vignettes employed.

Stakes effects were elusive in our original evidence-fixed study, although a registered replication of Sripada and Stanley (2012) confirmed that such effects could be found (though the overall pattern of stakes effects on knowledge that we observed in our replication did not match those observed by Sripada and Stanley). In both the original Sripada and Stanley study and our replication, questions about the quality of the evidence available to the protagonist always preceded questioning the participants whether they agreed with a claim about knowledge, potentially contributing to the finding of an effect. In contrast, in our first experiment, where we did not find evidence of a stakes effect, we did not ask participants to assess the quality of evidence available. A direct comparison of evidence-quality question-present versus question-absent conditions is needed to clarify whether this difference contributes to determining whether stakes effects are observed in an evidence-fixed design.²⁴

The contrast between the mixed results in evidence-fixed designs and the more consistent results in evidence-seeking designs serves to reinforce the notion that evidence-seeking designs are likely to be more informative experimental tools for further research on stakes effects.

Finally, because we included both negative and positive polarity prompts, we have been able to determine the role played by the polarity of a prompt is affecting judgments about knowledge. Most importantly, if we had not included this positive-negative prompt distinction, then we would not have been able to detect and interpret the large framing effects observed in our evidence-seeking experiments or uncover the stakes effects on “0” and “never” responses that we observed.

Though no single experimental investigation of a stakes effect on knowledge can definitively settle the existence of such an effect, the results of this study provide new reasons to think that such an effect exists.

1. Individual Scenario Analyses

For each individual scenario 2 x 4 mixed model Analysis of Variance (ANOVA) was performed on levels of agreement, with polarity (know, doesn’t know) as the between-subjects factor and stakes scale (one [low], two, three, four [high]) as the within-subjects factor. This analysis was replicated with a generalized estimating equation (GEE) using a linear model. Note that, where these were conducted, the non-significant results of multiple comparison follow-up tests in the non-parametric analyses are likely due to the weak global stakes effect (η² = .03).

1. Open Science Protocol

Adopting the experimental design of Sripada and Stanley (2012), we preregistered the experiment (background, methods, and power analysis) using the Open Science Framework repository (osf.io/sqeau). Our preregistration was submitted prior to data collection and is accessible to the public.

2. Participants

Four hundred and thirty participants (183 females, 246 males, 1 non-binary gender identity) between 20 and 67 years (M = 35.99 years, SD = 10.55 years) were recruited from MTurk and paid $0.50 for participating. This research received ethical approval from the Department of Philosophy, University of Reading, UK and informed consent was obtained from all participants. As before, screening procedures were performed ex post to identify suspicious or low-quality responses in all datasets. Following screening procedures, 63 VPS and further suspicious responses were flagged and removed. As in the original experiment, participants were randomly assigned to one of six conditions (N = 58 – 68 per condition).

3. Materials and Procedure

As in the original study (Sripada & Stanley 2012), we investigated three pairs of vignettes, each of which had a low and high stakes version. After reading one vignette, participants were asked to respond to two questions:

1. What is the strength of Hannah’s evidence that her noodles are not topped with pine nuts?

2. Suppose it turns out that her noodles are not topped with pine nuts. Please rate how strongly you agree or disagree with the following sentence:

   • “Hannah knows her noodles are not topped with pine nuts.”

Participants responded to the first question (evidence prompt) along a 7-point Likert-type scale (1-Very weak evidence, 7-Very strong evidence) and to the second prompt (knowledge prompt) along another 7-point Likert-type scale (1-Strongly agree, 7-Strongly disagree). Questions were presented in the same order (as above).

1. Individual Scenario Analyses

Across all scenarios, there was a general pattern of participants stating that more evidence would be required by the protagonist in order to know something (positive polarity)/still not know something (negative polarity) as the stakes increased and this pattern remained following outlier removal.

2. Never and Zero Responses

In order to interpret the full range of responses given in the evidence-seeking experiments, separate analyses were performed on both “never” and “0” responses which could be given in response to the following prompts:

Never responses: If you think S will never know no matter how many times she checks, write “never”

Zero responses: If you think S knows without having to check, write “0”

The number of “never” responses given in response to scenarios in the negative polarity were significantly higher across all three experiments when compared to the positive polarity conditions (see the table below).

The number of “0” responses given in response to scenarios in the positive polarity were higher in both the original prompts and symmetrical prompts experiments when compared to the negative polarity conditions (see the table below). Note that zero responses are not recorded for the matched prompts follow-up experiment as the response option was removed to create the matched design (i.e., If you think S knows without having to check, write “0”).

1. Symmetrical Experiment

1.4. Summary of Descriptive Statistics (prior to outlier removal and following removal)

Across all scenarios, there was a general pattern of participants stating that more evidence would be required by the protagonist in order to know something (positive polarity)/still not know something (negative polarity) as the stakes increased. This pattern of responses remained with extreme outliers removed There was a general pattern of participants stating that more evidence would be required by the protagonist in order to know something (positive polarity)/still not know something (negative polarity) as the stakes increased.

In terms of replicating the stakes effects observed in the first evidence-seeking experiment, we found a main effect of stakes in the paramedic and game show scenarios (see Figure 9). However, we did not observe a stakes effect in the vaccine, mountaineering, introductions, or possessions scenarios. Additionally, across four of the six scenarios, we did not observe significant differences in responses to positive and negative polarity prompts. An effect of polarity in the paramedic and mountaineering scenarios, in which evidence scores were higher for the positive polarity. A full breakdown of analysis by scenario follows this summary.

Figure 9.

Figure 9. The amount of evidence participants state is needed in order for the protagonist to know/still not know across all stakes versions of all six scenarios. These figures show the data after removal of extreme outliers (i.e., the data used in the second analyses reported below). Amount of evidence required increased as the stakes were raised across polarities in the paramedic and game show scenarios (note positive gradient of lines) however in this experiment, there was no main effect of stakes in the vaccine, mountaineering, introduction, or possessions scenarios. Error bars represent +/- 1 SE.

2. Matched Experiment

2.4. Summary of Descriptive Statistics (prior to outlier removal and following removal)

Across all scenarios, there was a general pattern of participants stating that more evidence would be required by the protagonist in order to know something (positive polarity)/still not know something (negative polarity) as the stakes increased. This pattern of responses remained with extreme outliers removed. There was a general pattern of participants stating that more evidence would be required by the protagonist in order to know something (positive polarity)/still not know something (negative polarity) as the stakes increased.

In terms of replicating the stakes effects observed in the first evidence-seeking experiment, we found a main effect of stakes across all scenarios excluding the introduction scenario (see Figure 10). As in the first evidence-seeking experiment, the main effect of stakes in the possessions scenario was observed in the positive polarity only and unique to this experiment, so was the main effect of stakes in the game show scenario. Across four of the six scenarios, we did not observe significant differences in responses to positive and negative polarity prompts. However, in this experiment, we did observe an effect of polarity in the vaccine and possessions scenarios; evidence scores were higher for the positive polarity in the vaccine case but lower for the positive polarity in the possessions case. A full breakdown of analysis by scenario follows this summary. As previously, an initial GEE analysis was performed using a gamma (log link) model with stakes (one [low]; two; three; four [high]) as within-subjects factor and polarity (positive; negative) as between-subjects factor and then repeated once extreme outliers had been identified and removed.

Figure 10.

Figure 10. The amount of evidence participants state is needed in order for the protagonist to know/still not know across all stakes versions of all six scenarios. Amount of evidence required increased as the stakes were raised across all scenarios (note positive gradient of lines) except the introduction scenario. In both the possessions and game show scenario, the main effect was observed in the positive polarity only. Error bars represent +/- 1 SE.

3. Never and Zero Responses

In order to interpret the full range of responses given in the evidence-seeking experiments, separate analyses were performed on both “never” and “0” responses which could be given in response to the following prompts:

Never responses: If you think S will never know no matter how many times she checks, write “never” (in the positive polarity condition and in the negative polarity condition)

Zero responses: If you think S knows without having to check, write “0” (in the positive polarity condition)

The number of “never” responses given in response to scenarios in the negative polarity were significantly higher across all three experiments when compared to the positive polarity conditions (see Table below).

The number of “0” responses given in response to scenarios in the positive polarity were higher in both the original prompts and symmetrical prompts experiments when compared to the negative polarity conditions (see Table below). Again, as in the first evidence-seeking experiment, this isn’t surprising given that participants weren’t explicitly given the option to respond with “0” if the subject knows without having to check in the negative polarity prompts. The meaning of a “0” response when given in response to a positive vs. a negative polarity prompt is therefore probably different. Note that zero responses are not recorded for the matched prompts follow-up experiment as the response option was removed to create the matched design (i.e., If you think S knows without having to check, write “0”).

In order to investigate stakes effects on these responses, a GEE analysis with poisson (loglinear) model was performed on the frequency of “never” responses in the follow-up experiment using symmetrical prompts. Analysis revealed a main effect of polarity, (Wald X ²[1] = 20.82, p <.001), a main effect of stakes, (Wald X ²[3] = 14.65, p =.002), and a significant interaction of polarity x stakes, (Wald X ²[3] = 7.88, p =.049).

When interpreting the interaction, comparisons using sequential Bonferroni did not find any significant differences. Given the findings of these follow-up tests, we are unable to interpret this interaction. When interpreting the main effect of stakes, comparisons using sequential Bonferroni indicated a significant difference in the frequency of “never” responses between the stakes 1 [low] scenarios and the stakes 3 scenarios (p =.016). This effect was present across polarities (see Figure 11).

Figure 11

Figure 11 . The percentage of “never” responses given in each stakes scale (across all scenarios) in each polarity condition of the Symmetrical Experiment. The frequency of “never” answers given in response to prompts increased as the stakes were raised from low stakes to stakes 3. Note: this figure is Figure 7 in the manuscript.

In order to investigate stakes effects on zero responses in the Symmetrical Experiment, a GEE analysis with poisson (loglinear) model was also performed on the frequency of zero responses. Analysis revealed a main effect of stakes, (Wald X ²[3] = 8.73, p =.033). There was no main effect of polarity, (p =.599) and no interaction of polarity x stakes, (p =.330).

When interpreting the main effect of stakes, comparisons using sequential Bonferroni indicated a significant difference between the zero counts in the stakes 1 [low] scenarios and the stakes 3 scenarios (p =.045). The difference between the stakes 2 scenarios and the stakes 3 scenarios fell short of significance (p =.050). This effect was present across polarities (see Figure 12).

Figure 12

Figure 12 . The percentage of zero responses given in each stakes scale (across all scenarios) in each polarity condition of the Symmetrical Experiment. The frequency of zero answers given in response to prompts decreased as the stakes were raised from low stakes to stakes 3.

In the Matched Experiment, A GEE analysis with poisson (loglinear) model was performed on the frequency of “never” responses, revealing a main effect of polarity, (Wald X ²[1] = 18.66, p <.001), but no main effect of stakes, (p =.477), and no interaction of polarity x stakes, (p =.202) (see Figure 13).²⁷

Figure 13.

Figure 13. The percentage of “never” responses given in each stakes scale (across all scenarios) in each polarity condition of the Symmetrical Experiment. Here we find no main effect of stakes on “never” responses to the negative prompts.

Experimental Materials

Prompt variations