{"id":47952,"date":"2020-06-08T10:20:13","date_gmt":"2020-06-08T14:20:13","guid":{"rendered":"https:\/\/ibkrcampus.com\/?p=47952"},"modified":"2022-11-21T09:45:40","modified_gmt":"2022-11-21T14:45:40","slug":"robot-welath-apache-beam-systematic-trading-data-pipeline","status":"publish","type":"post","link":"https:\/\/www.interactivebrokers.com\/campus\/ibkr-quant-news\/robot-welath-apache-beam-systematic-trading-data-pipeline\/","title":{"rendered":"Why We Use Apache Beam For Our Systematic Trading Data Pipeline"},"content":{"rendered":"\n<p><strong><em>Excerpt<\/em><\/strong><\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img decoding=\"async\" width=\"300\" height=\"300\" data-src=\"\/campus\/wp-content\/uploads\/sites\/2\/2020\/06\/robot-wealth-1.png\" alt=\"Apache Beam\" class=\"wp-image-48089 lazyload\" src=\"data:image\/svg+xml;base64,PHN2ZyB3aWR0aD0iMSIgaGVpZ2h0PSIxIiB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciPjwvc3ZnPg==\" style=\"--smush-placeholder-width: 300px; aspect-ratio: 300\/300;\" \/><\/figure>\n\n\n\n<p>Apache Beam is a unified programming model for batch and streaming data processing jobs. It comes with support for many runners such as Spark, Flink, Google Dataflow and many more&nbsp;<a href=\"https:\/\/beam.apache.org\/documentation\/runners\/capability-matrix\/\">(see here for all runners<\/a>).<\/p>\n\n\n\n<p>You can define your pipelines in Java, Python or Go. At this time the Java SDK is more mature with support for more database connections, but Python is being rapidly developed and comes at a close second, Go is still in its early stages of development.<\/p>\n\n\n\n<p>In the Robot Wealth batch data pipeline, we rely on Beam (on the Google&nbsp;<a href=\"https:\/\/cloud.google.com\/dataflow\">Dataflow<\/a>&nbsp;runner) for:<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>Downloading data from various APIs<\/li><li>Loading it to Google Storage<\/li><li>Transforming and enriching the data<\/li><li>Calculating features<\/li><li>Loading it to Big Query<\/li><li>Data integrity checks<\/li><\/ul>\n\n\n\n<p>This gives us a scalable data pipeline which is also cost-efficient, because you only pay for Beam when you are using it.<\/p>\n\n\n\n<p>Here is how our batch data pipeline currently looks:<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img decoding=\"async\" data-src=\"\/campus\/wp-content\/uploads\/sites\/2\/2020\/06\/robot-wealth-2.png\" alt=\"Why We Use Apache Beam For Our Systematic Trading Data Pipeline\" class=\"wp-image-48091 lazyload\" src=\"data:image\/svg+xml;base64,PHN2ZyB3aWR0aD0iMSIgaGVpZ2h0PSIxIiB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciPjwvc3ZnPg==\" \/><\/figure>\n\n\n\n<p>The great thing about running Beam on Google Cloud is how seamlessly everything works together. In fact, every connection between the technologies used in the pipeline has native support in Beam.<\/p>\n\n\n\n<p>The most appealing features that make Beam the right choice for our data pipeline are<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>Autoscaling<\/li><li>GCP Integration<\/li><li>Easy to maintain codebase.<\/li><\/ul>\n\n\n\n<p><strong>Apache Beam in Action in a Trading Workflow<\/strong><\/p>\n\n\n\n<p>Let\u2019s take a look at Beam in action.<\/p>\n\n\n\n<p>In the following code block, we will be defining a data integrity check pipeline that logs an error if OHLC data is faulty.<\/p>\n\n\n\n<p style=\"background-color:#fcfcdb;font-size:11px\" class=\"has-background\">\nimport apache_beam as beam<br>\nfrom apache_beam.options.pipeline_options import PipelineOptions<br>\nimport logging<br>\ndef price_integrity(row):<br>\n    &#8220;&#8221;&#8221;A function that checks the integrity of a OHLC row, <br>\n    High must be the largest or equal data point in the row and Low must be the smallest or equal<br>\n    Arguments:<br>\n        row {dict} &#8212; A dict with Keys being the column name and values being the column value<br>\n    Returns:<br>\n        bool &#8212; True if row has errors False if row is good<br>\n    &#8220;&#8221;&#8221;<br><br>\n    \n    #Checks that the low(high) price is the lowest(highest) or equall <br>\n    low_error = not(round(float(row[&#8216;low&#8217;]),3) <= round(float(row['high']),3) and round(float(row['low']),3)<= round(float(row['open']),3) and round(float(row['low']),3) <= round(float(row['close']),3))<br>\n    high_error = not(round(float(row[&#8216;high&#8217;]),3) >= round(float(row[&#8216;low&#8217;]),3) and round(float(row[&#8216;high&#8217;]),3) >= round(float(row[&#8216;open&#8217;]),3) and round(float(row[&#8216;high&#8217;]),3) >= round(float(row[&#8216;close&#8217;]),3))<br>\n    if low_error or high_error:<br>\n        row.update({&#8216;error&#8217;:True})<br>\n        logging.error(&#8216;Integrity Check error uploading to datastore &#8230;.&#8217;)<br>\n        return row<br>\n    else:<br>\n        row.update({&#8216;error&#8217;:False})<br>\n        return row<br><br>\n        \n        \ndef run():<br>\n    options = PipelineOptions()<br>\n    p = beam.Pipeline(options=options)<br>\n    PriceCheck = (<br>\n            p <br>\n            | &#8216;Mimic BigQuery Read&#8217; >> beam.Create([{&#8216;ticker&#8217;:&#8217;RW&#8217;,&#8217;open&#8217;:10,&#8217;high&#8217;:11,&#8217;low&#8217;:8,&#8217;close&#8217;:10},<br><\n                                              {'ticker':'RW','open':10,'high':11,'low':8,'close':11},<br>\n                                              {&#8216;ticker&#8217;:&#8217;RW&#8217;,&#8217;open&#8217;:11,&#8217;high&#8217;:11,&#8217;low&#8217;:8,&#8217;close&#8217;:9},<br>\n                                              {&#8216;ticker&#8217;:&#8217;RW&#8217;,&#8217;open&#8217;:10,&#8217;high&#8217;:12,&#8217;low&#8217;:8,&#8217;close&#8217;:10},<br>\n                                              {&#8216;ticker&#8217;:&#8217;RW&#8217;,&#8217;open&#8217;:10,&#8217;high&#8217;:11,&#8217;low&#8217;:8,&#8217;close&#8217;:17}])<br>\n            | &#8216;Row Integrity Check&#8217; >> beam.Map(price_integrity)<br>\n            | &#8216;Print Results&#8217; >> beam.Map(print)<br>\n        )<br>\n    p.run()<br>\nrun()<br>\n<\/p>\n\n\n\n<p>Visit Robot Wealth to read the full article and download additional programming code: <a href=\"https:\/\/robotwealth.com\/why-we-use-apache-beam-for-our-trading-data-pipeline\/?pa=CEFCE36499\">https:\/\/robotwealth.com\/why-we-use-apache-beam-for-our-trading-data-pipeline\/?pa=CEFCE36499<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Robot Wealth dives into Apache Beam programming model, and shows how to code in it, and use it for batch and streaming data processing jobs.<\/p>\n","protected":false},"author":439,"featured_media":47953,"comment_status":"closed","ping_status":"open","sticky":true,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[339,338,350,341,344],"tags":[7753,7755,7757,7752,1381,7756,596,7751,595,7754],"contributors-categories":[13676],"class_list":{"0":"post-47952","1":"post","2":"type-post","3":"status-publish","4":"format-standard","5":"has-post-thumbnail","7":"category-data-science","8":"category-ibkr-quant-news","9":"category-quant-asia-pacific","10":"category-quant-development","11":"category-quant-regions","12":"tag-apache-beam","13":"tag-flink","14":"tag-gcp-integration","15":"tag-go-programming","16":"tag-google-cloud","17":"tag-google-dataflow","18":"tag-java","19":"tag-java-sdk","20":"tag-python","21":"tag-spark","22":"contributors-categories-robot-wealth"},"pp_statuses_selecting_workflow":false,"pp_workflow_action":"current","pp_status_selection":"publish","acf":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO Premium plugin v26.9 (Yoast SEO v27.3) - https:\/\/yoast.com\/product\/yoast-seo-premium-wordpress\/ -->\n<title>Why We Use Apache Beam For Our Systematic Trading Data Pipeline<\/title>\n<meta name=\"description\" content=\"Robot Wealth dives into Apache Beam programming model, and shows how to code in it, and use it for batch and streaming data processing jobs.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.interactivebrokers.com\/campus\/wp-json\/wp\/v2\/posts\/47952\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Why We Use Apache Beam For Our Systematic Trading Data Pipeline | IBKR Quant Blog\" \/>\n<meta property=\"og:description\" content=\"Robot Wealth dives into Apache Beam programming model, and shows how to code in it, and use it for batch and streaming data processing jobs.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.interactivebrokers.com\/campus\/ibkr-quant-news\/robot-welath-apache-beam-systematic-trading-data-pipeline\/\" \/>\n<meta property=\"og:site_name\" content=\"IBKR Campus US\" \/>\n<meta property=\"article:published_time\" content=\"2020-06-08T14:20:13+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2022-11-21T14:45:40+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.interactivebrokers.com\/campus\/wp-content\/uploads\/sites\/2\/2020\/06\/tech-glow-impact.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"900\" \/>\n\t<meta property=\"og:image:height\" content=\"550\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"Ajet Luka\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Ajet Luka\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"2 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\n\t    \"@context\": \"https:\\\/\\\/schema.org\",\n\t    \"@graph\": [\n\t        {\n\t            \"@type\": \"NewsArticle\",\n\t            \"@id\": \"https:\\\/\\\/www.interactivebrokers.com\\\/campus\\\/ibkr-quant-news\\\/robot-welath-apache-beam-systematic-trading-data-pipeline\\\/#article\",\n\t            \"isPartOf\": {\n\t                \"@id\": \"https:\\\/\\\/www.interactivebrokers.com\\\/campus\\\/ibkr-quant-news\\\/robot-welath-apache-beam-systematic-trading-data-pipeline\\\/\"\n\t            },\n\t            \"author\": {\n\t                \"name\": \"Ajet Luka\",\n\t                \"@id\": \"https:\\\/\\\/ibkrcampus.com\\\/campus\\\/#\\\/schema\\\/person\\\/92acd149cd24c6217771ee2c4b274c26\"\n\t            },\n\t            \"headline\": \"Why We Use Apache Beam For Our Systematic Trading Data Pipeline\",\n\t            \"datePublished\": \"2020-06-08T14:20:13+00:00\",\n\t            \"dateModified\": \"2022-11-21T14:45:40+00:00\",\n\t            \"mainEntityOfPage\": {\n\t                \"@id\": \"https:\\\/\\\/www.interactivebrokers.com\\\/campus\\\/ibkr-quant-news\\\/robot-welath-apache-beam-systematic-trading-data-pipeline\\\/\"\n\t            },\n\t            \"wordCount\": 472,\n\t            \"publisher\": {\n\t                \"@id\": \"https:\\\/\\\/ibkrcampus.com\\\/campus\\\/#organization\"\n\t            },\n\t            \"image\": {\n\t                \"@id\": \"https:\\\/\\\/www.interactivebrokers.com\\\/campus\\\/ibkr-quant-news\\\/robot-welath-apache-beam-systematic-trading-data-pipeline\\\/#primaryimage\"\n\t            },\n\t            \"thumbnailUrl\": \"https:\\\/\\\/www.interactivebrokers.com\\\/campus\\\/wp-content\\\/uploads\\\/sites\\\/2\\\/2020\\\/06\\\/tech-glow-impact.jpg\",\n\t            \"keywords\": [\n\t                \"Apache Beam\",\n\t                \"Flink\",\n\t                \"GCP Integration\",\n\t                \"Go programming\",\n\t                \"Google Cloud\",\n\t                \"Google Dataflow\",\n\t                \"Java\",\n\t                \"Java SDK\",\n\t                \"Python\",\n\t                \"Spark\"\n\t            ],\n\t            \"articleSection\": [\n\t                \"Data Science\",\n\t                \"Quant\",\n\t                \"Quant Asia Pacific\",\n\t                \"Quant Development\",\n\t                \"Quant Regions\"\n\t            ],\n\t            \"inLanguage\": \"en-US\"\n\t        },\n\t        {\n\t            \"@type\": \"WebPage\",\n\t            \"@id\": \"https:\\\/\\\/www.interactivebrokers.com\\\/campus\\\/ibkr-quant-news\\\/robot-welath-apache-beam-systematic-trading-data-pipeline\\\/\",\n\t            \"url\": \"https:\\\/\\\/www.interactivebrokers.com\\\/campus\\\/ibkr-quant-news\\\/robot-welath-apache-beam-systematic-trading-data-pipeline\\\/\",\n\t            \"name\": \"Why We Use Apache Beam For Our Systematic Trading Data Pipeline | IBKR Quant Blog\",\n\t            \"isPartOf\": {\n\t                \"@id\": \"https:\\\/\\\/ibkrcampus.com\\\/campus\\\/#website\"\n\t            },\n\t            \"primaryImageOfPage\": {\n\t                \"@id\": \"https:\\\/\\\/www.interactivebrokers.com\\\/campus\\\/ibkr-quant-news\\\/robot-welath-apache-beam-systematic-trading-data-pipeline\\\/#primaryimage\"\n\t            },\n\t            \"image\": {\n\t                \"@id\": \"https:\\\/\\\/www.interactivebrokers.com\\\/campus\\\/ibkr-quant-news\\\/robot-welath-apache-beam-systematic-trading-data-pipeline\\\/#primaryimage\"\n\t            },\n\t            \"thumbnailUrl\": \"https:\\\/\\\/www.interactivebrokers.com\\\/campus\\\/wp-content\\\/uploads\\\/sites\\\/2\\\/2020\\\/06\\\/tech-glow-impact.jpg\",\n\t            \"datePublished\": \"2020-06-08T14:20:13+00:00\",\n\t            \"dateModified\": \"2022-11-21T14:45:40+00:00\",\n\t            \"description\": \"Robot Wealth dives into Apache Beam programming model, and shows how to code in it, and use it for batch and streaming data processing jobs.\",\n\t            \"inLanguage\": \"en-US\",\n\t            \"potentialAction\": [\n\t                {\n\t                    \"@type\": \"ReadAction\",\n\t                    \"target\": [\n\t                        \"https:\\\/\\\/www.interactivebrokers.com\\\/campus\\\/ibkr-quant-news\\\/robot-welath-apache-beam-systematic-trading-data-pipeline\\\/\"\n\t                    ]\n\t                }\n\t            ]\n\t        },\n\t        {\n\t            \"@type\": \"ImageObject\",\n\t            \"inLanguage\": \"en-US\",\n\t            \"@id\": \"https:\\\/\\\/www.interactivebrokers.com\\\/campus\\\/ibkr-quant-news\\\/robot-welath-apache-beam-systematic-trading-data-pipeline\\\/#primaryimage\",\n\t            \"url\": \"https:\\\/\\\/www.interactivebrokers.com\\\/campus\\\/wp-content\\\/uploads\\\/sites\\\/2\\\/2020\\\/06\\\/tech-glow-impact.jpg\",\n\t            \"contentUrl\": \"https:\\\/\\\/www.interactivebrokers.com\\\/campus\\\/wp-content\\\/uploads\\\/sites\\\/2\\\/2020\\\/06\\\/tech-glow-impact.jpg\",\n\t            \"width\": 900,\n\t            \"height\": 550,\n\t            \"caption\": \"Why We Use Apache Beam For Our Systematic Trading Data Pipeline\"\n\t        },\n\t        {\n\t            \"@type\": \"WebSite\",\n\t            \"@id\": \"https:\\\/\\\/ibkrcampus.com\\\/campus\\\/#website\",\n\t            \"url\": \"https:\\\/\\\/ibkrcampus.com\\\/campus\\\/\",\n\t            \"name\": \"IBKR Campus US\",\n\t            \"description\": \"Financial Education from Interactive Brokers\",\n\t            \"publisher\": {\n\t                \"@id\": \"https:\\\/\\\/ibkrcampus.com\\\/campus\\\/#organization\"\n\t            },\n\t            \"potentialAction\": [\n\t                {\n\t                    \"@type\": \"SearchAction\",\n\t                    \"target\": {\n\t                        \"@type\": \"EntryPoint\",\n\t                        \"urlTemplate\": \"https:\\\/\\\/ibkrcampus.com\\\/campus\\\/?s={search_term_string}\"\n\t                    },\n\t                    \"query-input\": {\n\t                        \"@type\": \"PropertyValueSpecification\",\n\t                        \"valueRequired\": true,\n\t                        \"valueName\": \"search_term_string\"\n\t                    }\n\t                }\n\t            ],\n\t            \"inLanguage\": \"en-US\"\n\t        },\n\t        {\n\t            \"@type\": \"Organization\",\n\t            \"@id\": \"https:\\\/\\\/ibkrcampus.com\\\/campus\\\/#organization\",\n\t            \"name\": \"Interactive Brokers\",\n\t            \"alternateName\": \"IBKR\",\n\t            \"url\": \"https:\\\/\\\/ibkrcampus.com\\\/campus\\\/\",\n\t            \"logo\": {\n\t                \"@type\": \"ImageObject\",\n\t                \"inLanguage\": \"en-US\",\n\t                \"@id\": \"https:\\\/\\\/ibkrcampus.com\\\/campus\\\/#\\\/schema\\\/logo\\\/image\\\/\",\n\t                \"url\": \"https:\\\/\\\/www.interactivebrokers.com\\\/campus\\\/wp-content\\\/uploads\\\/sites\\\/2\\\/2024\\\/05\\\/ibkr-campus-logo.jpg\",\n\t                \"contentUrl\": \"https:\\\/\\\/www.interactivebrokers.com\\\/campus\\\/wp-content\\\/uploads\\\/sites\\\/2\\\/2024\\\/05\\\/ibkr-campus-logo.jpg\",\n\t                \"width\": 669,\n\t                \"height\": 669,\n\t                \"caption\": \"Interactive Brokers\"\n\t            },\n\t            \"image\": {\n\t                \"@id\": \"https:\\\/\\\/ibkrcampus.com\\\/campus\\\/#\\\/schema\\\/logo\\\/image\\\/\"\n\t            },\n\t            \"publishingPrinciples\": \"https:\\\/\\\/www.interactivebrokers.com\\\/campus\\\/about-ibkr-campus\\\/\",\n\t            \"ethicsPolicy\": \"https:\\\/\\\/www.interactivebrokers.com\\\/campus\\\/cyber-security-notice\\\/\"\n\t        },\n\t        {\n\t            \"@type\": \"Person\",\n\t            \"@id\": \"https:\\\/\\\/ibkrcampus.com\\\/campus\\\/#\\\/schema\\\/person\\\/92acd149cd24c6217771ee2c4b274c26\",\n\t            \"name\": \"Ajet Luka\",\n\t            \"url\": \"https:\\\/\\\/www.interactivebrokers.com\\\/campus\\\/author\\\/ajetluka\\\/\"\n\t        }\n\t    ]\n\t}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"Why We Use Apache Beam For Our Systematic Trading Data Pipeline","description":"Robot Wealth dives into Apache Beam programming model, and shows how to code in it, and use it for batch and streaming data processing jobs.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.interactivebrokers.com\/campus\/wp-json\/wp\/v2\/posts\/47952\/","og_locale":"en_US","og_type":"article","og_title":"Why We Use Apache Beam For Our Systematic Trading Data Pipeline | IBKR Quant Blog","og_description":"Robot Wealth dives into Apache Beam programming model, and shows how to code in it, and use it for batch and streaming data processing jobs.","og_url":"https:\/\/www.interactivebrokers.com\/campus\/ibkr-quant-news\/robot-welath-apache-beam-systematic-trading-data-pipeline\/","og_site_name":"IBKR Campus US","article_published_time":"2020-06-08T14:20:13+00:00","article_modified_time":"2022-11-21T14:45:40+00:00","og_image":[{"width":900,"height":550,"url":"https:\/\/www.interactivebrokers.com\/campus\/wp-content\/uploads\/sites\/2\/2020\/06\/tech-glow-impact.jpg","type":"image\/jpeg"}],"author":"Ajet Luka","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Ajet Luka","Est. reading time":"2 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"NewsArticle","@id":"https:\/\/www.interactivebrokers.com\/campus\/ibkr-quant-news\/robot-welath-apache-beam-systematic-trading-data-pipeline\/#article","isPartOf":{"@id":"https:\/\/www.interactivebrokers.com\/campus\/ibkr-quant-news\/robot-welath-apache-beam-systematic-trading-data-pipeline\/"},"author":{"name":"Ajet Luka","@id":"https:\/\/ibkrcampus.com\/campus\/#\/schema\/person\/92acd149cd24c6217771ee2c4b274c26"},"headline":"Why We Use Apache Beam For Our Systematic Trading Data Pipeline","datePublished":"2020-06-08T14:20:13+00:00","dateModified":"2022-11-21T14:45:40+00:00","mainEntityOfPage":{"@id":"https:\/\/www.interactivebrokers.com\/campus\/ibkr-quant-news\/robot-welath-apache-beam-systematic-trading-data-pipeline\/"},"wordCount":472,"publisher":{"@id":"https:\/\/ibkrcampus.com\/campus\/#organization"},"image":{"@id":"https:\/\/www.interactivebrokers.com\/campus\/ibkr-quant-news\/robot-welath-apache-beam-systematic-trading-data-pipeline\/#primaryimage"},"thumbnailUrl":"https:\/\/www.interactivebrokers.com\/campus\/wp-content\/uploads\/sites\/2\/2020\/06\/tech-glow-impact.jpg","keywords":["Apache Beam","Flink","GCP Integration","Go programming","Google Cloud","Google Dataflow","Java","Java SDK","Python","Spark"],"articleSection":["Data Science","Quant","Quant Asia Pacific","Quant Development","Quant Regions"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/www.interactivebrokers.com\/campus\/ibkr-quant-news\/robot-welath-apache-beam-systematic-trading-data-pipeline\/","url":"https:\/\/www.interactivebrokers.com\/campus\/ibkr-quant-news\/robot-welath-apache-beam-systematic-trading-data-pipeline\/","name":"Why We Use Apache Beam For Our Systematic Trading Data Pipeline | IBKR Quant Blog","isPartOf":{"@id":"https:\/\/ibkrcampus.com\/campus\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.interactivebrokers.com\/campus\/ibkr-quant-news\/robot-welath-apache-beam-systematic-trading-data-pipeline\/#primaryimage"},"image":{"@id":"https:\/\/www.interactivebrokers.com\/campus\/ibkr-quant-news\/robot-welath-apache-beam-systematic-trading-data-pipeline\/#primaryimage"},"thumbnailUrl":"https:\/\/www.interactivebrokers.com\/campus\/wp-content\/uploads\/sites\/2\/2020\/06\/tech-glow-impact.jpg","datePublished":"2020-06-08T14:20:13+00:00","dateModified":"2022-11-21T14:45:40+00:00","description":"Robot Wealth dives into Apache Beam programming model, and shows how to code in it, and use it for batch and streaming data processing jobs.","inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.interactivebrokers.com\/campus\/ibkr-quant-news\/robot-welath-apache-beam-systematic-trading-data-pipeline\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.interactivebrokers.com\/campus\/ibkr-quant-news\/robot-welath-apache-beam-systematic-trading-data-pipeline\/#primaryimage","url":"https:\/\/www.interactivebrokers.com\/campus\/wp-content\/uploads\/sites\/2\/2020\/06\/tech-glow-impact.jpg","contentUrl":"https:\/\/www.interactivebrokers.com\/campus\/wp-content\/uploads\/sites\/2\/2020\/06\/tech-glow-impact.jpg","width":900,"height":550,"caption":"Why We Use Apache Beam For Our Systematic Trading Data Pipeline"},{"@type":"WebSite","@id":"https:\/\/ibkrcampus.com\/campus\/#website","url":"https:\/\/ibkrcampus.com\/campus\/","name":"IBKR Campus US","description":"Financial Education from Interactive Brokers","publisher":{"@id":"https:\/\/ibkrcampus.com\/campus\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/ibkrcampus.com\/campus\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/ibkrcampus.com\/campus\/#organization","name":"Interactive Brokers","alternateName":"IBKR","url":"https:\/\/ibkrcampus.com\/campus\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/ibkrcampus.com\/campus\/#\/schema\/logo\/image\/","url":"https:\/\/www.interactivebrokers.com\/campus\/wp-content\/uploads\/sites\/2\/2024\/05\/ibkr-campus-logo.jpg","contentUrl":"https:\/\/www.interactivebrokers.com\/campus\/wp-content\/uploads\/sites\/2\/2024\/05\/ibkr-campus-logo.jpg","width":669,"height":669,"caption":"Interactive Brokers"},"image":{"@id":"https:\/\/ibkrcampus.com\/campus\/#\/schema\/logo\/image\/"},"publishingPrinciples":"https:\/\/www.interactivebrokers.com\/campus\/about-ibkr-campus\/","ethicsPolicy":"https:\/\/www.interactivebrokers.com\/campus\/cyber-security-notice\/"},{"@type":"Person","@id":"https:\/\/ibkrcampus.com\/campus\/#\/schema\/person\/92acd149cd24c6217771ee2c4b274c26","name":"Ajet Luka","url":"https:\/\/www.interactivebrokers.com\/campus\/author\/ajetluka\/"}]}},"jetpack_featured_media_url":"https:\/\/www.interactivebrokers.com\/campus\/wp-content\/uploads\/sites\/2\/2020\/06\/tech-glow-impact.jpg","_links":{"self":[{"href":"https:\/\/ibkrcampus.com\/campus\/wp-json\/wp\/v2\/posts\/47952","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/ibkrcampus.com\/campus\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/ibkrcampus.com\/campus\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/ibkrcampus.com\/campus\/wp-json\/wp\/v2\/users\/439"}],"replies":[{"embeddable":true,"href":"https:\/\/ibkrcampus.com\/campus\/wp-json\/wp\/v2\/comments?post=47952"}],"version-history":[{"count":0,"href":"https:\/\/ibkrcampus.com\/campus\/wp-json\/wp\/v2\/posts\/47952\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/ibkrcampus.com\/campus\/wp-json\/wp\/v2\/media\/47953"}],"wp:attachment":[{"href":"https:\/\/ibkrcampus.com\/campus\/wp-json\/wp\/v2\/media?parent=47952"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/ibkrcampus.com\/campus\/wp-json\/wp\/v2\/categories?post=47952"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/ibkrcampus.com\/campus\/wp-json\/wp\/v2\/tags?post=47952"},{"taxonomy":"contributors-categories","embeddable":true,"href":"https:\/\/ibkrcampus.com\/campus\/wp-json\/wp\/v2\/contributors-categories?post=47952"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}