{"id":114500,"date":"2021-12-07T11:50:46","date_gmt":"2021-12-07T16:50:46","guid":{"rendered":"https:\/\/ibkrcampus.com\/?p=114500"},"modified":"2022-11-21T09:49:53","modified_gmt":"2022-11-21T14:49:53","slug":"you-thought-p-hacking-was-bad-lets-talk-about-non-standard-errors","status":"publish","type":"post","link":"https:\/\/www.interactivebrokers.com\/campus\/ibkr-quant-news\/you-thought-p-hacking-was-bad-lets-talk-about-non-standard-errors\/","title":{"rendered":"You Thought P-Hacking was Bad? Let&#8217;s talk about &#8220;Non-Standard Errors&#8221;"},"content":{"rendered":"\n<p><em>The article &#8220;You Thought P-Hacking was Bad? Let&#8217;s talk about &#8220;Non-Standard Errors&#8221;&#8221; first appeared on <a href=\"https:\/\/alphaarchitect.com\/2021\/12\/03\/you-thought-p-hacking-was-bad-lets-talk-about-non-standard-errors\/\">Alpha Architect Blog<\/a>.<\/em><\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"h-non-standard-errors\"><strong>Non-Standard Errors<\/strong><\/h2>\n\n\n\n<ul class=\"wp-block-list\"><li>164 authors<\/li><li>Working paper<\/li><li>A version of this paper can be found&nbsp;<a href=\"https:\/\/papers.ssrn.com\/sol3\/papers.cfm?abstract_id=3961574\" target=\"_blank\" rel=\"noreferrer noopener\">here<\/a><\/li><li>Want to read our summaries of academic finance papers? Check out our&nbsp;<a href=\"https:\/\/alphaarchitect.com\/category\/architect-academic-insights\/academic-research-insight\" target=\"_blank\" rel=\"noreferrer noopener\">Academic Research Insight&nbsp;<\/a>category<\/li><\/ul>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"h-what-are-the-research-questions\">What are the research questions?<\/h2>\n\n\n\n<p>Most readers are familiar with&nbsp;<a href=\"https:\/\/alphaarchitect.com\/2017\/09\/13\/what-happens-when-you-data-mine-2-million-fundamental-quant-strategies\/\">p-hacking<\/a>&nbsp;and the so-called replication crisis in financial research (see&nbsp;<a href=\"https:\/\/alphaarchitect.com\/2021\/03\/23\/is-there-a-replication-crisis-in-finance\/\">here<\/a>,&nbsp;<a href=\"https:\/\/alphaarchitect.com\/2021\/03\/26\/more-on-the-factor-investing-replication-debate\/\">here<\/a>, and&nbsp;<a href=\"https:\/\/alphaarchitect.com\/2017\/03\/03\/evidence-based-investing-take-alpha-shove\/\">here<\/a>&nbsp;for differing views). Some claim that these research challenges are driven by a desire to find \u2018positive\u2019 results in the data because these results get published, whereas negative results do not get published (the&nbsp;<a href=\"https:\/\/alphaarchitect.com\/2018\/07\/30\/finance-journals-rarely-publish-articles-with-low-t-stats\/\">evidence backs<\/a>&nbsp;these claims).<\/p>\n\n\n\n<p>But this research project identifies and quantifies another potential issue with research \u2014 the researchers themselves! This \u201cnoise\u201d created by differences in empirical techniques, programming language, data pre-processing, and so forth are deemed \u201cnon-standard-errors,\u201d which may contribute even more uncertainty in our quest to determine intellectual truth. Yikes!<\/p>\n\n\n\n<p>In this epic study, the&nbsp;<a href=\"https:\/\/fincap.academy\/\">#fincap<\/a>&nbsp;community delivers a fresh standardized dataset and a set of hypotheses to 164 research teams across the globe. The authors then try and identify the variation in the results due to differences in the researcher\u2019s approach to tackling the problem.<\/p>\n\n\n\n<p>The research questions the paper seeks to address are as follows:<\/p>\n\n\n\n<ol class=\"wp-block-list\"><li>How large are non-standard errors for financial research?<\/li><li>What explains the differences?<\/li><li>Does peer-review minimize these differences?<\/li><li>How aware are researchers of the differences?<\/li><\/ol>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"h-what-are-the-academic-insights\">What are the Academic Insights?<\/h2>\n\n\n\n<ol class=\"wp-block-list\" type=\"1\"><li>How large are non-standard errors for financial research?<ol><li>Very large. On basic questions such as \u201cHow has market efficiency changed over time?\u201d one gets a huge dispersion in \u2018evidence-based\u2019 insights.<\/li><\/ol><\/li><li>What explains the differences?<ol><li>Hard to say, but intuitive concepts such as team quality, process quality, and paper quality are examined. The evidence, collectively, seems to suggest it is hard to explain the research differences on the dimensions listed.<\/li><\/ol><\/li><li>Does peer-review minimize these differences?<ol><li>Yes. Which reinforces why it is important to run a peer-review process and have people review your work with a critical eye.<\/li><\/ol><\/li><li>How aware are researchers of the differences?<ol><li>Not aware at all. But this is not surprising \u2014 we already know humans are&nbsp;<a href=\"https:\/\/alphaarchitect.com\/2014\/05\/13\/behavioral-finance-and-investing-are-you-trying-too-hard\/\">generally confused<\/a>, on average.<\/li><\/ol><\/li><\/ol>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"h-why-does-it-matter\">Why does it matter?<\/h2>\n\n\n\n<p>We already knew that data-mining was a problem in academic research and researchers are working hard to fix this problem. However, this paper brings up a new source of variability \u2014 the researchers themselves! And the sad part is all of these variations and biases embedded in research may not be tied to nefarious motives \u2014 they are simply part of the landscape and should be considered when reviewing academic research.<\/p>\n\n\n\n<p>Of course, we should be clear that the takeaway is NOT to disregard academic research and the scientific approach to learning new things. Relying on intuition and gut feel is a process that will likely lead to even more bias and warped conclusions! So while academic research is not flawless, it&#8217;s the best we got. To me, the key data point from this paper is that we should reinforce peer-review processes and establish a research culture where criticism is applauded, not derided.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"h-the-most-important-chart-from-the-paper\">The most important chart from the paper<\/h2>\n\n\n\n<figure class=\"wp-block-image size-full\"><img decoding=\"async\" width=\"800\" height=\"971\" data-src=\"\/campus\/wp-content\/uploads\/sites\/2\/2021\/12\/p-hacking-alpha-architect.png\" alt=\"\" class=\"wp-image-114507 lazyload\" data-srcset=\"https:\/\/ibkrcampus.com\/campus\/wp-content\/uploads\/sites\/2\/2021\/12\/p-hacking-alpha-architect.png 800w, https:\/\/ibkrcampus.com\/campus\/wp-content\/uploads\/sites\/2\/2021\/12\/p-hacking-alpha-architect-700x850.png 700w, https:\/\/ibkrcampus.com\/campus\/wp-content\/uploads\/sites\/2\/2021\/12\/p-hacking-alpha-architect-300x364.png 300w, https:\/\/ibkrcampus.com\/campus\/wp-content\/uploads\/sites\/2\/2021\/12\/p-hacking-alpha-architect-768x932.png 768w\" data-sizes=\"(max-width: 800px) 100vw, 800px\" src=\"data:image\/svg+xml;base64,PHN2ZyB3aWR0aD0iMSIgaGVpZ2h0PSIxIiB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciPjwvc3ZnPg==\" style=\"--smush-placeholder-width: 800px; aspect-ratio: 800\/971;\" \/><\/figure>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"h-abstract\">Abstract<\/h2>\n\n\n\n<blockquote class=\"wp-block-quote is-layout-flow wp-block-quote-is-layout-flow\"><p>In statistics, samples are drawn from a population in a data-generating process (DGP). Standard errors measure the uncertainty in sample estimates of population parameters. In science, evidence is generated to test hypotheses in an evidence-generating process (EGP). We claim that EGP variation across researchers adds uncertainty: non-standard errors. To study them, we let 164 teams test six hypotheses on the same sample. We find that non-standard errors are sizeable, on par with standard errors. Their size (i) co-varies only weakly with team merits, reproducibility, or peer rating, (ii) declines significantly after peer-feedback, and (iii) is underestimated by participants.<\/p><\/blockquote>\n","protected":false},"excerpt":{"rendered":"<p>Wesley Gray explores a research project that identifies another potential issue with research &#8211; the researchers themselves.<\/p>\n","protected":false},"author":87,"featured_media":48516,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[339,338,341,352,344],"tags":[8485,10754,806,10751,4922,10753,10749,1006,10755,10750,10752,4939],"contributors-categories":[13651],"class_list":{"0":"post-114500","1":"post","2":"type-post","3":"status-publish","4":"format-standard","5":"has-post-thumbnail","7":"category-data-science","8":"category-ibkr-quant-news","9":"category-quant-development","10":"category-quant-north-america","11":"category-quant-regions","12":"tag-data-mining","13":"tag-data-pre-processing","14":"tag-data-science","15":"tag-data-generating-process-dgp","16":"tag-econometrics","17":"tag-empirical-techniques","18":"tag-finance-crowd-analysis-project-fincap","19":"tag-fintech","20":"tag-non-standard-errors","21":"tag-p-hacking","22":"tag-replication-crisis-in-financial-research","23":"tag-statistics","24":"contributors-categories-alpha-architect"},"pp_statuses_selecting_workflow":false,"pp_workflow_action":"current","pp_status_selection":"publish","acf":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO Premium plugin v26.9 (Yoast SEO v27.4) - https:\/\/yoast.com\/product\/yoast-seo-premium-wordpress\/ -->\n<title>You Thought P-Hacking was Bad? Let&#8217;s talk about &#8220;Non-Standard Errors&#8221;<\/title>\n<meta name=\"description\" content=\"Wesley Gray explores a research project that identifies another potential issue with research - the researchers themselves.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.interactivebrokers.com\/campus\/wp-json\/wp\/v2\/posts\/114500\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"You Thought P-Hacking was Bad? Let&#039;s talk about &quot;Non-Standard Errors&quot; | IBKR Quant Blog\" \/>\n<meta property=\"og:description\" content=\"Wesley Gray explores a research project that identifies another potential issue with research - the researchers themselves.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.interactivebrokers.com\/campus\/ibkr-quant-news\/you-thought-p-hacking-was-bad-lets-talk-about-non-standard-errors\/\" \/>\n<meta property=\"og:site_name\" content=\"IBKR Campus US\" \/>\n<meta property=\"article:published_time\" content=\"2021-12-07T16:50:46+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2022-11-21T14:49:53+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.interactivebrokers.com\/campus\/wp-content\/uploads\/sites\/2\/2020\/06\/compass-trend.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"900\" \/>\n\t<meta property=\"og:image:height\" content=\"550\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"Wesley Gray\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Wesley Gray\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"3 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\n\t    \"@context\": \"https:\\\/\\\/schema.org\",\n\t    \"@graph\": [\n\t        {\n\t            \"@type\": \"NewsArticle\",\n\t            \"@id\": \"https:\\\/\\\/www.interactivebrokers.com\\\/campus\\\/ibkr-quant-news\\\/you-thought-p-hacking-was-bad-lets-talk-about-non-standard-errors\\\/#article\",\n\t            \"isPartOf\": {\n\t                \"@id\": \"https:\\\/\\\/www.interactivebrokers.com\\\/campus\\\/ibkr-quant-news\\\/you-thought-p-hacking-was-bad-lets-talk-about-non-standard-errors\\\/\"\n\t            },\n\t            \"author\": {\n\t                \"name\": \"Wesley Gray\",\n\t                \"@id\": \"https:\\\/\\\/ibkrcampus.com\\\/campus\\\/#\\\/schema\\\/person\\\/79af876152fb4c33c997f57aafcbae0d\"\n\t            },\n\t            \"headline\": \"You Thought P-Hacking was Bad? Let&#8217;s talk about &#8220;Non-Standard Errors&#8221;\",\n\t            \"datePublished\": \"2021-12-07T16:50:46+00:00\",\n\t            \"dateModified\": \"2022-11-21T14:49:53+00:00\",\n\t            \"mainEntityOfPage\": {\n\t                \"@id\": \"https:\\\/\\\/www.interactivebrokers.com\\\/campus\\\/ibkr-quant-news\\\/you-thought-p-hacking-was-bad-lets-talk-about-non-standard-errors\\\/\"\n\t            },\n\t            \"wordCount\": 651,\n\t            \"publisher\": {\n\t                \"@id\": \"https:\\\/\\\/ibkrcampus.com\\\/campus\\\/#organization\"\n\t            },\n\t            \"image\": {\n\t                \"@id\": \"https:\\\/\\\/www.interactivebrokers.com\\\/campus\\\/ibkr-quant-news\\\/you-thought-p-hacking-was-bad-lets-talk-about-non-standard-errors\\\/#primaryimage\"\n\t            },\n\t            \"thumbnailUrl\": \"https:\\\/\\\/www.interactivebrokers.com\\\/campus\\\/wp-content\\\/uploads\\\/sites\\\/2\\\/2020\\\/06\\\/compass-trend.jpg\",\n\t            \"keywords\": [\n\t                \"Data Mining\",\n\t                \"Data Pre-processing\",\n\t                \"Data Science\",\n\t                \"Data-generating Process (DGP)\",\n\t                \"Econometrics\",\n\t                \"Empirical Techniques\",\n\t                \"Finance Crowd Analysis Project (FinCap)\",\n\t                \"fintech\",\n\t                \"Non-standard-errors\",\n\t                \"P-Hacking\",\n\t                \"Replication Crisis in Financial Research\",\n\t                \"statistics\"\n\t            ],\n\t            \"articleSection\": [\n\t                \"Data Science\",\n\t                \"Quant\",\n\t                \"Quant Development\",\n\t                \"Quant North America\",\n\t                \"Quant Regions\"\n\t            ],\n\t            \"inLanguage\": \"en-US\"\n\t        },\n\t        {\n\t            \"@type\": \"WebPage\",\n\t            \"@id\": \"https:\\\/\\\/www.interactivebrokers.com\\\/campus\\\/ibkr-quant-news\\\/you-thought-p-hacking-was-bad-lets-talk-about-non-standard-errors\\\/\",\n\t            \"url\": \"https:\\\/\\\/www.interactivebrokers.com\\\/campus\\\/ibkr-quant-news\\\/you-thought-p-hacking-was-bad-lets-talk-about-non-standard-errors\\\/\",\n\t            \"name\": \"You Thought P-Hacking was Bad? Let's talk about \\\"Non-Standard Errors\\\" | IBKR Quant Blog\",\n\t            \"isPartOf\": {\n\t                \"@id\": \"https:\\\/\\\/ibkrcampus.com\\\/campus\\\/#website\"\n\t            },\n\t            \"primaryImageOfPage\": {\n\t                \"@id\": \"https:\\\/\\\/www.interactivebrokers.com\\\/campus\\\/ibkr-quant-news\\\/you-thought-p-hacking-was-bad-lets-talk-about-non-standard-errors\\\/#primaryimage\"\n\t            },\n\t            \"image\": {\n\t                \"@id\": \"https:\\\/\\\/www.interactivebrokers.com\\\/campus\\\/ibkr-quant-news\\\/you-thought-p-hacking-was-bad-lets-talk-about-non-standard-errors\\\/#primaryimage\"\n\t            },\n\t            \"thumbnailUrl\": \"https:\\\/\\\/www.interactivebrokers.com\\\/campus\\\/wp-content\\\/uploads\\\/sites\\\/2\\\/2020\\\/06\\\/compass-trend.jpg\",\n\t            \"datePublished\": \"2021-12-07T16:50:46+00:00\",\n\t            \"dateModified\": \"2022-11-21T14:49:53+00:00\",\n\t            \"description\": \"Wesley Gray explores a research project that identifies another potential issue with research - the researchers themselves.\",\n\t            \"inLanguage\": \"en-US\",\n\t            \"potentialAction\": [\n\t                {\n\t                    \"@type\": \"ReadAction\",\n\t                    \"target\": [\n\t                        \"https:\\\/\\\/www.interactivebrokers.com\\\/campus\\\/ibkr-quant-news\\\/you-thought-p-hacking-was-bad-lets-talk-about-non-standard-errors\\\/\"\n\t                    ]\n\t                }\n\t            ]\n\t        },\n\t        {\n\t            \"@type\": \"ImageObject\",\n\t            \"inLanguage\": \"en-US\",\n\t            \"@id\": \"https:\\\/\\\/www.interactivebrokers.com\\\/campus\\\/ibkr-quant-news\\\/you-thought-p-hacking-was-bad-lets-talk-about-non-standard-errors\\\/#primaryimage\",\n\t            \"url\": \"https:\\\/\\\/www.interactivebrokers.com\\\/campus\\\/wp-content\\\/uploads\\\/sites\\\/2\\\/2020\\\/06\\\/compass-trend.jpg\",\n\t            \"contentUrl\": \"https:\\\/\\\/www.interactivebrokers.com\\\/campus\\\/wp-content\\\/uploads\\\/sites\\\/2\\\/2020\\\/06\\\/compass-trend.jpg\",\n\t            \"width\": 900,\n\t            \"height\": 550,\n\t            \"caption\": \"Quant Trend\"\n\t        },\n\t        {\n\t            \"@type\": \"WebSite\",\n\t            \"@id\": \"https:\\\/\\\/ibkrcampus.com\\\/campus\\\/#website\",\n\t            \"url\": \"https:\\\/\\\/ibkrcampus.com\\\/campus\\\/\",\n\t            \"name\": \"IBKR Campus US\",\n\t            \"description\": \"Financial Education from Interactive Brokers\",\n\t            \"publisher\": {\n\t                \"@id\": \"https:\\\/\\\/ibkrcampus.com\\\/campus\\\/#organization\"\n\t            },\n\t            \"potentialAction\": [\n\t                {\n\t                    \"@type\": \"SearchAction\",\n\t                    \"target\": {\n\t                        \"@type\": \"EntryPoint\",\n\t                        \"urlTemplate\": \"https:\\\/\\\/ibkrcampus.com\\\/campus\\\/?s={search_term_string}\"\n\t                    },\n\t                    \"query-input\": {\n\t                        \"@type\": \"PropertyValueSpecification\",\n\t                        \"valueRequired\": true,\n\t                        \"valueName\": \"search_term_string\"\n\t                    }\n\t                }\n\t            ],\n\t            \"inLanguage\": \"en-US\"\n\t        },\n\t        {\n\t            \"@type\": \"Organization\",\n\t            \"@id\": \"https:\\\/\\\/ibkrcampus.com\\\/campus\\\/#organization\",\n\t            \"name\": \"Interactive Brokers\",\n\t            \"alternateName\": \"IBKR\",\n\t            \"url\": \"https:\\\/\\\/ibkrcampus.com\\\/campus\\\/\",\n\t            \"logo\": {\n\t                \"@type\": \"ImageObject\",\n\t                \"inLanguage\": \"en-US\",\n\t                \"@id\": \"https:\\\/\\\/ibkrcampus.com\\\/campus\\\/#\\\/schema\\\/logo\\\/image\\\/\",\n\t                \"url\": \"https:\\\/\\\/www.interactivebrokers.com\\\/campus\\\/wp-content\\\/uploads\\\/sites\\\/2\\\/2024\\\/05\\\/ibkr-campus-logo.jpg\",\n\t                \"contentUrl\": \"https:\\\/\\\/www.interactivebrokers.com\\\/campus\\\/wp-content\\\/uploads\\\/sites\\\/2\\\/2024\\\/05\\\/ibkr-campus-logo.jpg\",\n\t                \"width\": 669,\n\t                \"height\": 669,\n\t                \"caption\": \"Interactive Brokers\"\n\t            },\n\t            \"image\": {\n\t                \"@id\": \"https:\\\/\\\/ibkrcampus.com\\\/campus\\\/#\\\/schema\\\/logo\\\/image\\\/\"\n\t            },\n\t            \"publishingPrinciples\": \"https:\\\/\\\/www.interactivebrokers.com\\\/campus\\\/about-ibkr-campus\\\/\",\n\t            \"ethicsPolicy\": \"https:\\\/\\\/www.interactivebrokers.com\\\/campus\\\/cyber-security-notice\\\/\"\n\t        },\n\t        {\n\t            \"@type\": \"Person\",\n\t            \"@id\": \"https:\\\/\\\/ibkrcampus.com\\\/campus\\\/#\\\/schema\\\/person\\\/79af876152fb4c33c997f57aafcbae0d\",\n\t            \"name\": \"Wesley Gray\",\n\t            \"url\": \"https:\\\/\\\/www.interactivebrokers.com\\\/campus\\\/author\\\/wesley-gray\\\/\"\n\t        }\n\t    ]\n\t}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"You Thought P-Hacking was Bad? Let&#8217;s talk about &#8220;Non-Standard Errors&#8221;","description":"Wesley Gray explores a research project that identifies another potential issue with research - the researchers themselves.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.interactivebrokers.com\/campus\/wp-json\/wp\/v2\/posts\/114500\/","og_locale":"en_US","og_type":"article","og_title":"You Thought P-Hacking was Bad? Let's talk about \"Non-Standard Errors\" | IBKR Quant Blog","og_description":"Wesley Gray explores a research project that identifies another potential issue with research - the researchers themselves.","og_url":"https:\/\/www.interactivebrokers.com\/campus\/ibkr-quant-news\/you-thought-p-hacking-was-bad-lets-talk-about-non-standard-errors\/","og_site_name":"IBKR Campus US","article_published_time":"2021-12-07T16:50:46+00:00","article_modified_time":"2022-11-21T14:49:53+00:00","og_image":[{"width":900,"height":550,"url":"https:\/\/www.interactivebrokers.com\/campus\/wp-content\/uploads\/sites\/2\/2020\/06\/compass-trend.jpg","type":"image\/jpeg"}],"author":"Wesley Gray","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Wesley Gray","Est. reading time":"3 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"NewsArticle","@id":"https:\/\/www.interactivebrokers.com\/campus\/ibkr-quant-news\/you-thought-p-hacking-was-bad-lets-talk-about-non-standard-errors\/#article","isPartOf":{"@id":"https:\/\/www.interactivebrokers.com\/campus\/ibkr-quant-news\/you-thought-p-hacking-was-bad-lets-talk-about-non-standard-errors\/"},"author":{"name":"Wesley Gray","@id":"https:\/\/ibkrcampus.com\/campus\/#\/schema\/person\/79af876152fb4c33c997f57aafcbae0d"},"headline":"You Thought P-Hacking was Bad? Let&#8217;s talk about &#8220;Non-Standard Errors&#8221;","datePublished":"2021-12-07T16:50:46+00:00","dateModified":"2022-11-21T14:49:53+00:00","mainEntityOfPage":{"@id":"https:\/\/www.interactivebrokers.com\/campus\/ibkr-quant-news\/you-thought-p-hacking-was-bad-lets-talk-about-non-standard-errors\/"},"wordCount":651,"publisher":{"@id":"https:\/\/ibkrcampus.com\/campus\/#organization"},"image":{"@id":"https:\/\/www.interactivebrokers.com\/campus\/ibkr-quant-news\/you-thought-p-hacking-was-bad-lets-talk-about-non-standard-errors\/#primaryimage"},"thumbnailUrl":"https:\/\/www.interactivebrokers.com\/campus\/wp-content\/uploads\/sites\/2\/2020\/06\/compass-trend.jpg","keywords":["Data Mining","Data Pre-processing","Data Science","Data-generating Process (DGP)","Econometrics","Empirical Techniques","Finance Crowd Analysis Project (FinCap)","fintech","Non-standard-errors","P-Hacking","Replication Crisis in Financial Research","statistics"],"articleSection":["Data Science","Quant","Quant Development","Quant North America","Quant Regions"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/www.interactivebrokers.com\/campus\/ibkr-quant-news\/you-thought-p-hacking-was-bad-lets-talk-about-non-standard-errors\/","url":"https:\/\/www.interactivebrokers.com\/campus\/ibkr-quant-news\/you-thought-p-hacking-was-bad-lets-talk-about-non-standard-errors\/","name":"You Thought P-Hacking was Bad? Let's talk about \"Non-Standard Errors\" | IBKR Quant Blog","isPartOf":{"@id":"https:\/\/ibkrcampus.com\/campus\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.interactivebrokers.com\/campus\/ibkr-quant-news\/you-thought-p-hacking-was-bad-lets-talk-about-non-standard-errors\/#primaryimage"},"image":{"@id":"https:\/\/www.interactivebrokers.com\/campus\/ibkr-quant-news\/you-thought-p-hacking-was-bad-lets-talk-about-non-standard-errors\/#primaryimage"},"thumbnailUrl":"https:\/\/www.interactivebrokers.com\/campus\/wp-content\/uploads\/sites\/2\/2020\/06\/compass-trend.jpg","datePublished":"2021-12-07T16:50:46+00:00","dateModified":"2022-11-21T14:49:53+00:00","description":"Wesley Gray explores a research project that identifies another potential issue with research - the researchers themselves.","inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.interactivebrokers.com\/campus\/ibkr-quant-news\/you-thought-p-hacking-was-bad-lets-talk-about-non-standard-errors\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.interactivebrokers.com\/campus\/ibkr-quant-news\/you-thought-p-hacking-was-bad-lets-talk-about-non-standard-errors\/#primaryimage","url":"https:\/\/www.interactivebrokers.com\/campus\/wp-content\/uploads\/sites\/2\/2020\/06\/compass-trend.jpg","contentUrl":"https:\/\/www.interactivebrokers.com\/campus\/wp-content\/uploads\/sites\/2\/2020\/06\/compass-trend.jpg","width":900,"height":550,"caption":"Quant Trend"},{"@type":"WebSite","@id":"https:\/\/ibkrcampus.com\/campus\/#website","url":"https:\/\/ibkrcampus.com\/campus\/","name":"IBKR Campus US","description":"Financial Education from Interactive Brokers","publisher":{"@id":"https:\/\/ibkrcampus.com\/campus\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/ibkrcampus.com\/campus\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/ibkrcampus.com\/campus\/#organization","name":"Interactive Brokers","alternateName":"IBKR","url":"https:\/\/ibkrcampus.com\/campus\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/ibkrcampus.com\/campus\/#\/schema\/logo\/image\/","url":"https:\/\/www.interactivebrokers.com\/campus\/wp-content\/uploads\/sites\/2\/2024\/05\/ibkr-campus-logo.jpg","contentUrl":"https:\/\/www.interactivebrokers.com\/campus\/wp-content\/uploads\/sites\/2\/2024\/05\/ibkr-campus-logo.jpg","width":669,"height":669,"caption":"Interactive Brokers"},"image":{"@id":"https:\/\/ibkrcampus.com\/campus\/#\/schema\/logo\/image\/"},"publishingPrinciples":"https:\/\/www.interactivebrokers.com\/campus\/about-ibkr-campus\/","ethicsPolicy":"https:\/\/www.interactivebrokers.com\/campus\/cyber-security-notice\/"},{"@type":"Person","@id":"https:\/\/ibkrcampus.com\/campus\/#\/schema\/person\/79af876152fb4c33c997f57aafcbae0d","name":"Wesley Gray","url":"https:\/\/www.interactivebrokers.com\/campus\/author\/wesley-gray\/"}]}},"jetpack_featured_media_url":"https:\/\/www.interactivebrokers.com\/campus\/wp-content\/uploads\/sites\/2\/2020\/06\/compass-trend.jpg","_links":{"self":[{"href":"https:\/\/ibkrcampus.com\/campus\/wp-json\/wp\/v2\/posts\/114500","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/ibkrcampus.com\/campus\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/ibkrcampus.com\/campus\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/ibkrcampus.com\/campus\/wp-json\/wp\/v2\/users\/87"}],"replies":[{"embeddable":true,"href":"https:\/\/ibkrcampus.com\/campus\/wp-json\/wp\/v2\/comments?post=114500"}],"version-history":[{"count":0,"href":"https:\/\/ibkrcampus.com\/campus\/wp-json\/wp\/v2\/posts\/114500\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/ibkrcampus.com\/campus\/wp-json\/wp\/v2\/media\/48516"}],"wp:attachment":[{"href":"https:\/\/ibkrcampus.com\/campus\/wp-json\/wp\/v2\/media?parent=114500"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/ibkrcampus.com\/campus\/wp-json\/wp\/v2\/categories?post=114500"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/ibkrcampus.com\/campus\/wp-json\/wp\/v2\/tags?post=114500"},{"taxonomy":"contributors-categories","embeddable":true,"href":"https:\/\/ibkrcampus.com\/campus\/wp-json\/wp\/v2\/contributors-categories?post=114500"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}