<\/p>\n\n\n\n
But what criteria determine whether some knowledge claims are legitimately disputable and others are not? Can we simply rely on our intuitions about how much we favor the claims or their consequences? Clearly, that would contradict the principles of the scientific method<\/a> and would leave society collectively adrift without a shared understanding of reality.<\/p>\n\n\n\n <\/p>\n\n\n\n Luckily, Daniel Sarewitz<\/a> and Steve Rayner<\/a>, two scholars of the relationship between science and society, have discussed several dimensions<\/a> that help to identify the trustworthiness of claims to expertise and knowledge. Two aspects I find particularly useful are 1) how feedback-rich and falsifiable the knowledge is, and 2) how high the stakes are in terms of broader value-laden ramifications. <\/p>\n\n\n\n <\/p>\n\n\n\n First, trustworthiness and reliability increase when knowledge producers or experts receive rapid, repeated, real-world feedback and clear performance metrics. Sarewitz and Rayner give examples of pilots and surgeons as \u201cpractitioner\u201d experts who occupy this space: their reliability and credibility are derived directly from the large number of feedback-rich trials on which they have built their expertise. Sarewitz and Rayner contrast this with \u201cinappropriate expertise,\u201d where supposed experts may be credentialed, but they seek to advise on matters in which they have not been practically able to accrue a demonstrably reliable track record. An example may be a group of international relations PhDs advising the government on whether to go to war. <\/p>\n\n\n\n <\/p>\n\n\n\n Second, trustworthiness and reliability decrease when stakes are high and values are contested because, in those cases, knowledge generation is highly susceptible to cultural forces and thus motivated cognition. The broader incentive structure and\/or often unconscious preferences lead knowledge producers (researchers, scientific institutions, universities, etc.) to begin with a strongly preferred broader conclusion and then gather evidence to support it.<\/p>\n\n\n\n When assessing the general <\/em>trustworthiness of scientific or science-adjacent claims to knowledge, I have found it useful to arrange these two ideas (slightly reframed and rephrased) in a two-dimensional space<\/a>:<\/p>\n\n\n\n <\/p>\n\n\n\n Below, I put \u201cHow testable is the relevant real-world conclusion?<\/strong>\u201d on the horizontal axis. <\/p>\n\n\n\n <\/p>\n\n\n\n Diagram made in Microsoft PowerPoint.<\/p>\n\n\n\n I put \u201cHow strong is the pull toward a preferred conclusion?<\/strong>\u201d on the vertical axis. <\/p>\n\n\n\n <\/p>\n\n\n\n Diagram made in Microsoft PowerPoint.<\/p>\n\n\n\n <\/p>\n\n\n\n With these two dimensions, we can broadly characterize scientific claims to knowledge into four quadrants (I, II, III, IV). <\/strong><\/p>\n\n\n\n <\/p>\n\n\n\n Diagram made in Microsoft PowerPoint.<\/p>\n\n\n\n <\/p>\n\n\n\n Diagram made in Microsoft PowerPoint. Icons made by ChatGPT.<\/p>\n\n\n\n <\/p>\n\n\n\n Quadrant I: Claims are reliably trustworthy and uncontroversial but relatively inconsequential. <\/strong><\/p>\n\n\n\n <\/p>\n\n\n\n Quadrant I is the purest realm of the scientific method, where knowledge producers are motivated solely by the pursuit of truth. They engage in an iterative cycle of posing testable hypotheses, deriving predictions, confronting these with reproducible observations\/experiments, analyzing the results, and revising or discarding the hypotheses accordingly.<\/p>\n\n\n\n <\/p>\n\n\n\n Examples in Quadrant I include the science facts that students learn and practice demonstrating in grade school, such as how dissolving salt in water lowers the freezing point of the water. Such a claim can be demonstrated quickly and easily anywhere, and it has no broader social ramifications or bearing on one’s career, identity, or beliefs.<\/p>\n\n\n\n <\/p>\n\n\n\n Quadrant II: Claims are contestable, but controversy remains academic.<\/strong><\/p>\n\n\n\n <\/p>\n\n\n\n In Quadrant II, broader conclusions are not subject to definitive tests either in principle or due to practical limits related to, for example, spatial scale or timescale. Thus, trustworthiness remains provisional and is based on evaluating the internal consistency of claims, indirect evidence, or mathematical elegance. <\/p>\n\n\n\n <\/p>\n\n\n\n At the same time, there\u2019s little incentive for the knowledge producers to desire specific conclusions since the broader moral or sociopolitical implications are small. <\/p>\n\n\n\n <\/p>\n\n\n\n An example in Quadrant II might be the hypothesized extra, compactified spatial dimensions of string theory. It\u2019s a well-structured idea supported by formalism, but direct, definitive testing is challenging, leading the arguments to focus mainly on theoretical plausibility.<\/p>\n\n\n\n <\/p>\n\n\n\n Quadrant III: Foundational claims are trustworthy, but controversy arises from different frameworks implicitly emphasizing different values.<\/strong><\/p>\n\n\n\n <\/p>\n\n\n\n In Quadrant III<\/strong>, core foundational claims and, to a lesser extent, broader conclusions are<\/em> testable. However, controversy arises due to disagreements on which evidence deserves the most weight and which conclusions to emphasize. <\/p>\n\n\n\n <\/p>\n\n\n\n When you are near the bottom of the vertical axis, there is a great deal of susceptibility to motivated cognition. Human beings generate scientific knowledge, and human beings are far from being purely rational robots. Thus, scientific knowledge is influenced by ethical intuitions, culture, and peer pressure among researchers, as well as the incentive structures of research funding and publishing systems (see the Bias and Science section of Clark et al. 2023<\/a>). <\/p>\n\n\n\n This means that there will often be pulls towards certain themes and broader conclusions over others, independent of the evidence. Since constructing research papers involves numerous researcher degrees of freedom<\/a> (such as<\/a> the research question to ask, the methodological specifics for addressing the question, and the focus of the paper’s abstract and title), there is significant latitude for selection biases<\/a> to favor preferred broader conclusions over nonpreferred ones.<\/p>\n\n\n\n <\/p>\n\n\n\n An example from Quadrant III might be the claim that raising the minimum wage helps poor people. That claim may seem straightforward enough to be tractable, but a problem is that \u201chelps\u201d is a value-laden verb and is interpretable in many different ways. Empirical evidence might show that raising the minimum wage increases wages for those low-paid workers who are actually employed, but has conflicting effects on broader hours worked, employment levels, or small-business survival. One might argue (with data and models) that a minimum wage narrowly raises incomes in lower terciles in the short term but stifles economic growth in the long term, eventually harming society overall, including poor people. Whether this policy \u201chelps\u201d depends on which outcomes are prioritized, the specific populations considered, and the time frame of concern. These debates may seem to focus only on the technical aspects of the question, but they also incorporate ethical and value disputes that are inherently political. These political disputes are not so much about fundamental facts but rather are about \u201cwho gets what, when, and how\u201d [Lasswell (1958)<\/a> via Pielke Jr. (2004)<\/a>].<\/p>\n\n\n\n <\/p>\n\n\n\n Quadrant IV: From the perspective of desiring neat, straightforward answers, Quadrant IV is a mess. <\/strong><\/p>\n\n\n\n <\/p>\n\n\n\n Claims are reliably controversial, contestable, and difficult to adjudicate because they are framework and model-dependent, difficult to test, embed value-laden assumptions, and are strongly susceptible to personally and culturally-influenced motivated cognition by the experts producing the knowledge. The same suite of evidence or underlying facts can be legitimately assembled into coherent narratives that seem completely at odds with each other, and there is no straightforward way to adjudicate which emphasis is \u201ccorrect\u201d (See Sarewitz, 2004<\/a>).<\/p>\n\n\n\n An example could be the claim that society should respond to a deadly pandemic by shutting down schools for a year. The counterfactuals are highly uncertain, objectives are numerous and incommensurate (mortality, learning, equity, livelihoods). Rapid, controlled, clear tests don\u2019t exist, and many interests influence the desirability of different conclusions. <\/p>\n\n\n\n <\/p>\n\n\n\n As a climate scientist by training, I pay close attention to climate science, and I have noticed that much of the field’s most prominent research and messaging tends to be highly susceptible to preferred conclusions and has low testability. Often, this is because the claims are inherently political, making them relatively easy to identify as native to Quadrant IV. <\/p>\n\n\n\n Diagram made in Microsoft PowerPoint.<\/p>\n\n\n\n <\/p>\n\n\n\n However, there are also many instances of Quadrant IV climate science claims that focus on physical and biological systems, making them difficult to properly identify as Quadrant IV claims that are legitimately contestable. <\/p>\n\n\n\n I will illustrate this with two examples from the past year.<\/p>\n\n\n\n <\/p>\n\n\n\n Example 1: Are all hurricanes now an entire category stronger? <\/strong><\/p>\n\n\n\n I have previously written<\/a> about a paper titled \u201cHuman-caused warming has intensified recent hurricanes<\/a>,\u201d which made the very bold claim that essentially all Atlantic hurricanes are one category stronger than they would have been under preindustrial conditions and accordingly received considerable attention<\/a>. Hurricane categories are defined by wind speed, so a basic sanity test of this claim (or at least the spirit of this claim) would be to look at long-term historical trends in convectional measures of tropical cyclone wind speed like Accumulated Cyclone Energy (ACE). When you do this, you find no global trend over the past 40 years<\/a> (when we have good data), despite substantial warming over that period. Essentially, to get their dramatic result, they focused primarily on the influence of a causal variable known to make tropical cyclones stronger (warming sea surface temperatures) and downplayed causal variables known to make the environment more hostile<\/a> to tropical cyclone formation and maintenance. The core idea that hurricane activity is much worse now due to warming could have been tested against empirical observations (and it would not have passed that test), but the research methodology was so specific and contained so many conditions that testing the narrow claim of the paper amounts to just reproducing their analysis rather than testing their claim for reproducibility. Robustness of results does not imply robustness of conclusions, so this places the claim on the left side of the horizontal axis. <\/p>\n\n\n\n <\/p>\n\n\n\n So were the methodological choices to emphasize exacerbating causal variables coincidences, or was there a strong pull towards a desired conclusion? The paper itself spells out that the study was at least partially motivated to influence public opinion stating the the idea was to \u201cconnect the dots\u201d between climate change and hurricanes because \u201clandfalling hurricanes with high intensities\u2014can act as \u2018focusing events\u2019 that draw public attention\u201d and that \u201cIncreased attention during and in the wake of storms creates opportunities for public and private discourse around climate and disaster preparedness.\u201d<\/p>\n\n\n\n <\/p>\n\n\n\n Also, all three authors are employed by Climate Central, which is explicitly an advocacy organization<\/a> whose main aim is to promote climate action. Additionally, all three study funders (Bezos Earth Fund, The Schmidt Family Foundation, and the CO\u2082 Foundation) have mission statements aligned with the paper\u2019s findings. For example, the CO\u2082 Foundation states that its mission<\/a> is communication about the urgent societal risks from extreme weather (linked to CO\u2082-related). After the paper’s publication, Climate Central fundraising emails highlighted that their analysis influenced media coverage of Hurricanes Helene and Milton, encouraging more financial support from subscribers. <\/p>\n\n\n\n <\/p>\n\n\n\n All this amounts to substantial evidence of a very strong pull towards a desired conclusion, putting the paper towards the bottom of the vertical axis. <\/p>\n\n\n\n <\/p>\n\n\n\n Example 2: Will crop production decrease in the future? <\/strong><\/p>\n\n\n\n <\/p>\n\n\n\n Another recent high-profile<\/a> example from this year was the study \u201cImpacts of climate change on global agriculture, accounting for adaptation<\/a>,\u201d published in Nature<\/em>. <\/p>\n\n\n\n <\/p>\n\n\n\n A main claim originating from the paper, promulgated in the media, was that \u201cChildren born now may live in a world where the U.S. can only produce half as much of its key food crops<\/a>.\u201d Indeed, the paper’s language and its projection figure (Fig. 3) suggest substantial declines in agricultural production over the remainder of the century. As in Example 1 above, a basic sanity test for this projection might be to examine historical trends in agricultural productivity as temperatures have risen over the past. That test would show that yields of major crops have dramatically increased<\/a> over that time period despite warming. So, what explains this discrepancy? The paper’s results are relative to an unspecified counterfactual<\/a>, and its projections inherently exclude innovation, which amounts to imagining that agricultural research and development (which has historically driven the dramatic increase in crop yields) suddenly stops tomorrow. Therefore, their claims cannot be taken as actual predictions of the future; and therefore they are essentially untestable and unfalsifiable, placing them to the left on the horizontal axis.<\/p>\n\n\n\n <\/p>\n\n\n\n Was it an oversight on the authors’ part that they failed to communicate that their results were not absolute predictions of future crop yields? There is strong evidence that it was not<\/a>. The paper’s construction and its interactions with the media suggest the authors were comfortable with this ambiguity. Further, authors of the study expressed that they hoped<\/em><\/a> \u201cthe study will provide more evidence for the urgent need to transform the energy system and the high cost of failing to do so\u201d. Overall, the prestigious venue where the article was published, the widespread attention and fawning media coverage, and the author’s expressed activism indicate a strong pull towards a preferred conclusion, placing the claims to the left on the horizontal axis.<\/p>\n\n\n\n <\/p>\n\n\n\n Claims being native to Quadrant IV wouldn’t be such a serious problem if these claims represented an unbiased sample of all the claims that could legitimately be made. In other words, different entities may have different preferred conclusions (often based on different moral frameworks), but if these preferred conclusions were randomly sampled, then assessing them in aggregate would accurately<\/em> indicate how wide a range of views are legitimate on the matter. A bigger problem arises when those people producing the knowledge (researchers, scientific institutions, universities, etc.) have a tendency to favor certain preferred conclusions over others<\/a>. Since scientific knowledge is<\/em> \u201cthe intellectual and social consensus of affiliated experts\u2026\u201d (Oreskes, 2004<\/a>), this gives the illusion of a much narrower picture of reality than is actually epistemically justified. <\/p>\n\n\n\n <\/p>\n\n\n\n How prediction markets can help<\/strong><\/p>\n\n\n\n <\/p>\n\n\n\n To increase the reliability and trustworthiness of claims, we\u2019d like to shift the orientation up and to the right. To raise a claim on the vertical axis, you\u2019d want to reduce the pull toward a preferred conclusion (which may be preferred for reasons other than accuracy) and increase the pull toward a conclusion that more purely emphasizes correctness.<\/p>\n\n\n\n On the horizontal axis, you\u2019d like to make sure that the accuracy of the conclusion will actually be evaluated<\/em>. This shifts the focus away from making somewhat obscure, hard-to-test claims on, e.g., past data and instead emphasizes making real-world predictions about the future.<\/p>\n\n\n\n <\/p>\n\n\n\n Diagram made in Microsoft PowerPoint.<\/p>\n\n\n\n <\/p>\n\n\n\n I am in charge of climate prediction markets at Interactive Brokers, and I see prediction markets<\/a> as being essentially designed to accomplish both of the above goals.<\/p>\n\n\n\n <\/p>\n\n\n\n Prediction markets allow people to take financial positions on the outcomes of future events. Thus, similar to the famous wager between Julian Simon and Paul Ehrlich<\/a> on resource depletion, they encourage intellectual disputes to shift from vague proclamations to precise, well-defined claims about the future that will be tested<\/em>. Furthermore, the monetary aspect aligns incentives, as there is no better motivation to put forth your best understanding of reality than to have financial (and, in the case of public wagers, reputational) skin in the game. This also shifts the emphasis away from explaining the past and towards the most powerful (and useful) kind of evidence in science: accurate predictions made about the future.<\/p>\n\n\n\n <\/p>\n\n\n\n The negotiation of the terms between even two prominent representatives of different views illuminates a great deal of information. Prediction markets, however, take this same idea and expand it by aggregating and distilling the wisdom of the crowd<\/a> into a single probability or price. Being naturally probabilistic, they also center inherent uncertainty rather than artificially side-stepping it.<\/p>\n\n\n\n <\/p>\n\n\n\n Prediction markets are not a new concept<\/a> or something emerging from outside academia. Numerous studies have examined their accuracy<\/a>, and they even have their own dedicated academic journal<\/a>. This body of research demonstrates that prediction markets have consistently performed as well as or significantly better than alternatives<\/a>. Pilot programs in climate prediction markets have shown promise,<\/a> demonstrating accurate predictions on variables as diverse as<\/a> UK monthly rainfall, average daily maximum temperature, annual wheat yields, monthly El Ni\u00f1o indices, annual Atlantic hurricanes, and annual US hurricane landfalls.<\/p>\n\n\n\n <\/p>\n\n\n\n So what would an embrace of prediction markets mean in practice? Academic researchers in climate could further study the accuracy and reliability of prediction markets, or they could formulate research questions and models specifically to inform predictions in these markets. <\/p>\n\n\n\n <\/p>\n\n\n\n Returning to the two examples above, this would reorient research questions from forms that are not particularly useful, like<\/p>\n\n\n\n <\/p>\n\n\n\n![\"\"](\"https:\/\/www.interactivebrokers.com\/campus\/wp-content\/uploads\/sites\/2\/2025\/11\/image-1100x825.png\")
<\/figure>\n\n\n\n![\"\"](\"https:\/\/www.interactivebrokers.com\/campus\/wp-content\/uploads\/sites\/2\/2025\/11\/image-1-1100x825.png\")
<\/figure>\n\n\n\n![\"\"](\"https:\/\/www.interactivebrokers.com\/campus\/wp-content\/uploads\/sites\/2\/2025\/11\/image-4-1100x825.png\")
<\/figure>\n\n\n\n![\"\"](\"https:\/\/www.interactivebrokers.com\/campus\/wp-content\/uploads\/sites\/2\/2025\/11\/image-5-1100x825.png\")
<\/figure>\n\n\n\nMuch of the most prominent climate science is native to Quadrant IV.<\/h1>\n\n\n\n
![\"\"](\"https:\/\/www.interactivebrokers.com\/campus\/wp-content\/uploads\/sites\/2\/2025\/11\/image-2-1100x825.png\")
<\/figure>\n\n\n\n![\"\"](\"https:\/\/www.interactivebrokers.com\/campus\/wp-content\/uploads\/sites\/2\/2025\/11\/image-3-1100x825.png\")
<\/figure>\n\n\n\n