{"id":202845,"date":"2024-02-28T11:27:47","date_gmt":"2024-02-28T16:27:47","guid":{"rendered":"https:\/\/ibkrcampus.com\/?p=202845"},"modified":"2024-02-28T11:27:55","modified_gmt":"2024-02-28T16:27:55","slug":"robobrowser-automating-online-forms","status":"publish","type":"post","link":"https:\/\/www.interactivebrokers.com\/campus\/ibkr-quant-news\/robobrowser-automating-online-forms\/","title":{"rendered":"RoboBrowser: Automating Online Forms"},"content":{"rendered":"\n<h2 class=\"wp-block-heading\" id=\"h-background\"><strong>Background<\/strong><\/h2>\n\n\n\n<p>RoboBrowser is a Python 3.x package for crawling through the web and submitting online forms. It works similarly to the older Python 2.x package, mechanize. This post is going to give a simple introduction using RoboBrowser to submit a form on&nbsp;<a href=\"https:\/\/www.wunderground.com\/history\"><u>Wunderground<\/u><\/a>&nbsp;for scraping historical weather data.<br><br><\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"h-initial-setup\"><strong>Initial setup<\/strong><\/h2>\n\n\n\n<p>RoboBrowser can be installed via pip:<\/p>\n\n\n\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"generic\" data-enlighter-theme=\"\" data-enlighter-highlight=\"\" data-enlighter-linenumbers=\"\" data-enlighter-lineoffset=\"\" data-enlighter-title=\"\" data-enlighter-group=\"\">pip install robobrowser<\/pre>\n\n\n\n<p>Let\u2019s do the initial setup of the script by loading the RoboBrowser package. We\u2019ll also load pandas, as we\u2019ll be using that a little bit later.<\/p>\n\n\n\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"generic\" data-enlighter-theme=\"\" data-enlighter-highlight=\"\" data-enlighter-linenumbers=\"\" data-enlighter-lineoffset=\"\" data-enlighter-title=\"\" data-enlighter-group=\"\">from robobrowser import RoboBrowser\nimport pandas as pd<\/pre>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Create RoboBrowser Object<\/strong><\/h2>\n\n\n\n<p>Next, we create a RoboBrowser object. This object functions similarly to an actual web browser. It allows you to navigate to different websites, fill in forms, and get HTML from webpages. All of a RoboBrowser object\u2019s actions are completely invisible, so you won\u2019t actually see a physical browser while any of this is happening.<\/p>\n\n\n\n<p>Using our object,&nbsp;<em>browser<\/em>, we navigate to Wunderground\u2019s historical weather main site (see&nbsp;<a href=\"https:\/\/www.wunderground.com\/history\">https:\/\/www.wunderground.com\/history<\/a>). This is done using the&nbsp;<em>open<\/em>&nbsp;method, as below.<\/p>\n\n\n\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"generic\" data-enlighter-theme=\"\" data-enlighter-highlight=\"\" data-enlighter-linenumbers=\"\" data-enlighter-lineoffset=\"\" data-enlighter-title=\"\" data-enlighter-group=\"\">'''Create RoboBrowser object\n   This will act similarly to a typical web browser'''\nbrowser = RoboBrowser(history=True)\n \n'''Navigate browser to Wunderground's historical weather page'''\nbrowser.open('https:\/\/www.wunderground.com\/history')<\/pre>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Filling out the form<\/strong><\/h2>\n\n\n\n<p>Next, let\u2019s get the list of forms on the webpage,&nbsp;<a href=\"https:\/\/www.wunderground.com\/history\">https:\/\/www.wunderground.com\/history<\/a>.<\/p>\n\n\n\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"generic\" data-enlighter-theme=\"\" data-enlighter-highlight=\"\" data-enlighter-linenumbers=\"\" data-enlighter-lineoffset=\"\" data-enlighter-title=\"\" data-enlighter-group=\"\">forms = browser.get_forms()<\/pre>\n\n\n\n<figure class=\"wp-block-image size-large\"><img decoding=\"async\" width=\"1100\" height=\"148\" data-src=\"\/campus\/wp-content\/uploads\/sites\/2\/2024\/02\/robobrowser-the-automatic-net-1100x148.png\" alt=\"\" class=\"wp-image-202853 lazyload\" data-srcset=\"https:\/\/ibkrcampus.com\/campus\/wp-content\/uploads\/sites\/2\/2024\/02\/robobrowser-the-automatic-net-1100x148.png 1100w, https:\/\/ibkrcampus.com\/campus\/wp-content\/uploads\/sites\/2\/2024\/02\/robobrowser-the-automatic-net-700x94.png 700w, https:\/\/ibkrcampus.com\/campus\/wp-content\/uploads\/sites\/2\/2024\/02\/robobrowser-the-automatic-net-300x40.png 300w, https:\/\/ibkrcampus.com\/campus\/wp-content\/uploads\/sites\/2\/2024\/02\/robobrowser-the-automatic-net-768x103.png 768w, https:\/\/ibkrcampus.com\/campus\/wp-content\/uploads\/sites\/2\/2024\/02\/robobrowser-the-automatic-net.png 1115w\" data-sizes=\"(max-width: 1100px) 100vw, 1100px\" src=\"data:image\/svg+xml;base64,PHN2ZyB3aWR0aD0iMSIgaGVpZ2h0PSIxIiB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciPjwvc3ZnPg==\" style=\"--smush-placeholder-width: 1100px; aspect-ratio: 1100\/148;\" \/><\/figure>\n\n\n\n<p>Above, we can see there are two forms on the webpage. The last of these is the one we need, as this is what we\u2019ll use to submit a zip code with a historical date to get weather information. You can tell this by looking at the inputs in the print-out above of forms i.e. code, month, day etc.<\/p>\n\n\n\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"generic\" data-enlighter-theme=\"\" data-enlighter-highlight=\"\" data-enlighter-linenumbers=\"\" data-enlighter-lineoffset=\"\" data-enlighter-title=\"\" data-enlighter-group=\"\">form = forms[1]<\/pre>\n\n\n\n<p>Inputting data into our form is similar to adding values to a dictionary in Python. We do it like this:<\/p>\n\n\n\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"generic\" data-enlighter-theme=\"\" data-enlighter-highlight=\"\" data-enlighter-linenumbers=\"\" data-enlighter-lineoffset=\"\" data-enlighter-title=\"\" data-enlighter-group=\"\">'''Enter inputs into form'''\nform['code'] = '12345'\nform['month'] = 'September'\nform['day'] = '18'\nform['year'] = '2017'<\/pre>\n\n\n\n<p>As you can see, our inputs require a zip code (just called \u2018code\u2019 in the html), along with values for month, day, and year.<\/p>\n\n\n\n<p>Now, we can submit the web form in one line:<\/p>\n\n\n\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"generic\" data-enlighter-theme=\"\" data-enlighter-highlight=\"\" data-enlighter-linenumbers=\"\" data-enlighter-lineoffset=\"\" data-enlighter-title=\"\" data-enlighter-group=\"\">'''Submit form'''\nbrowser.submit_form(form)<\/pre>\n\n\n\n<p>You can get the HTML of the page following submission using the&nbsp;<em>parsed<\/em>&nbsp;method of&nbsp;<em>browser<\/em>.<\/p>\n\n\n\n<p>With the pandas&nbsp;<em>read_html<\/em>&nbsp;method, we can get the HTML tables on the page. In our example, we pull just the 0th (Python indexed 0) table on the page.<\/p>\n\n\n\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"generic\" data-enlighter-theme=\"\" data-enlighter-highlight=\"\" data-enlighter-linenumbers=\"\" data-enlighter-lineoffset=\"\" data-enlighter-title=\"\" data-enlighter-group=\"\">'''Scrape html of page following form submission'''\nhtml = str(browser.parsed)\n \n'''Use pandas to get primary table on page'''\ndata = pd.read_html(html)[0]<\/pre>\n\n\n\n<figure class=\"wp-block-image size-large\"><img decoding=\"async\" width=\"1100\" height=\"628\" data-src=\"\/campus\/wp-content\/uploads\/sites\/2\/2024\/02\/robobrowser-the-automatic-net-2-1100x628.png\" alt=\"\" class=\"wp-image-202854 lazyload\" data-srcset=\"https:\/\/ibkrcampus.com\/campus\/wp-content\/uploads\/sites\/2\/2024\/02\/robobrowser-the-automatic-net-2-1100x628.png 1100w, https:\/\/ibkrcampus.com\/campus\/wp-content\/uploads\/sites\/2\/2024\/02\/robobrowser-the-automatic-net-2-700x399.png 700w, https:\/\/ibkrcampus.com\/campus\/wp-content\/uploads\/sites\/2\/2024\/02\/robobrowser-the-automatic-net-2-300x171.png 300w, https:\/\/ibkrcampus.com\/campus\/wp-content\/uploads\/sites\/2\/2024\/02\/robobrowser-the-automatic-net-2-768x438.png 768w, https:\/\/ibkrcampus.com\/campus\/wp-content\/uploads\/sites\/2\/2024\/02\/robobrowser-the-automatic-net-2.png 1132w\" data-sizes=\"(max-width: 1100px) 100vw, 1100px\" src=\"data:image\/svg+xml;base64,PHN2ZyB3aWR0aD0iMSIgaGVpZ2h0PSIxIiB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciPjwvc3ZnPg==\" style=\"--smush-placeholder-width: 1100px; aspect-ratio: 1100\/628;\" \/><\/figure>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Putting it all together\u2026<\/strong><\/h2>\n\n\n\n<p>Let\u2019s wrap what we\u2019ve done into a function, like this:<\/p>\n\n\n\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"generic\" data-enlighter-theme=\"\" data-enlighter-highlight=\"\" data-enlighter-linenumbers=\"\" data-enlighter-lineoffset=\"\" data-enlighter-title=\"\" data-enlighter-group=\"\">def get_historical_data(code, month, day, year):\n     \n    '''Create RoboBrowser object\n       This will act similar to a typical web browser'''\n    browser = RoboBrowser(history=True)\n     \n    '''Navigate browser to Wunderground's historical weather page'''\n    browser.open('https:\/\/www.wunderground.com\/history')\n     \n    '''Find the form to input zip code and date'''\n    forms = browser.get_forms()\n     \n    form = forms[1]\n     \n    '''Enter inputs into form'''\n    form['code'] = code\n    form['month'] = month\n    form['day'] = day\n    form['year'] = year\n     \n    '''Submit form'''\n    browser.submit_form(form)\n     \n    '''Scrape html of page following form submission'''\n    html = str(browser.parsed)\n     \n    '''Use pandas to get primary table on page'''\n    data = pd.read_html(html)[0]\n \n    return data<\/pre>\n\n\n\n<p>We now have a function we can call by passing a zip code with a particular month, day, and year.<\/p>\n\n\n\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"generic\" data-enlighter-theme=\"\" data-enlighter-highlight=\"\" data-enlighter-linenumbers=\"\" data-enlighter-lineoffset=\"\" data-enlighter-title=\"\" data-enlighter-group=\"\">get_historical_data('23456','September','18','2000')    \n \nget_historical_data('23456','March','18','2000')<\/pre>\n\n\n\n<p>We can simplify the parameters we\u2019re passing so that we just need to input a zip code with a date, rather than three separate arguments for month, day, and year, respectively. We\u2019ll do this by converting a date input from the user to a pandas timestamp, which allows us to parse out these needed attributes about the date.<\/p>\n\n\n\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"generic\" data-enlighter-theme=\"\" data-enlighter-highlight=\"\" data-enlighter-linenumbers=\"\" data-enlighter-lineoffset=\"\" data-enlighter-title=\"\" data-enlighter-group=\"\">def get_historical_data(code, date):\n     \n    '''Create RoboBrowser object\n       This will act similar to a typical web browser'''\n    browser = RoboBrowser(history=True)\n     \n    '''Navigate browser to Wunderground's historical weather page'''\n    browser.open('https:\/\/www.wunderground.com\/history')\n     \n    '''Find the form to input zip code and date'''\n    forms = browser.get_forms()\n     \n    form = forms[1]\n     \n    '''Convert date to pandas timestamp'''\n    date = pd.Timestamp(date)\n     \n    '''Enter inputs into form'''\n    form['code'] = code\n    form['month'] = date.month\n    form['day'] = date.day\n    form['year'] = date.year\n     \n    '''Submit form'''\n    browser.submit_form(form)\n     \n    '''Scrape html of page following form submission'''\n    html = str(browser.parsed)\n     \n    '''Use pandas to get primary table on page'''\n    data = pd.read_html(html)[0]\n \n    return data<\/pre>\n\n\n\n<p>So calling our function now looks like this:<\/p>\n\n\n\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"generic\" data-enlighter-theme=\"\" data-enlighter-highlight=\"\" data-enlighter-linenumbers=\"\" data-enlighter-lineoffset=\"\" data-enlighter-title=\"\" data-enlighter-group=\"\">get_historical_data(\"23456\", \"9\/18\/2000\")\n \nget_historical_data(\"23456\", \"3\/18\/2000\")<\/pre>\n\n\n\n<p><em>Originally posted on <a href=\"https:\/\/theautomatic.net\/2017\/09\/19\/robobrowser-automating-online-forms\/\">TheAutomatic.net<\/a> blog.<\/em><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Next, we create a RoboBrowser object. This object functions similarly to an actual web browser. <\/p>\n","protected":false},"author":388,"featured_media":66906,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[339,343,349,338,341],"tags":[806,1224,595,14988],"contributors-categories":[13695],"class_list":{"0":"post-202845","1":"post","2":"type-post","3":"status-publish","4":"format-standard","5":"has-post-thumbnail","7":"category-data-science","8":"category-programing-languages","9":"category-python-development","10":"category-ibkr-quant-news","11":"category-quant-development","12":"tag-data-science","13":"tag-pandas","14":"tag-python","15":"tag-robobrowser","16":"contributors-categories-theautomatic-net"},"pp_statuses_selecting_workflow":false,"pp_workflow_action":"current","pp_status_selection":"publish","acf":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO Premium plugin v26.9 (Yoast SEO v27.4) - https:\/\/yoast.com\/product\/yoast-seo-premium-wordpress\/ -->\n<title>RoboBrowser: Automating Online Forms | IBKR Quant<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.interactivebrokers.com\/campus\/wp-json\/wp\/v2\/posts\/202845\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"RoboBrowser: Automating Online Forms | IBKR Campus US\" \/>\n<meta property=\"og:description\" content=\"Next, we create a RoboBrowser object. This object functions similarly to an actual web browser.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.interactivebrokers.com\/campus\/ibkr-quant-news\/robobrowser-automating-online-forms\/\" \/>\n<meta property=\"og:site_name\" content=\"IBKR Campus US\" \/>\n<meta property=\"article:published_time\" content=\"2024-02-28T16:27:47+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2024-02-28T16:27:55+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.interactivebrokers.com\/campus\/wp-content\/uploads\/sites\/2\/2020\/11\/python.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"900\" \/>\n\t<meta property=\"og:image:height\" content=\"550\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"Andrew Treadway\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Andrew Treadway\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"3 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\n\t    \"@context\": \"https:\\\/\\\/schema.org\",\n\t    \"@graph\": [\n\t        {\n\t            \"@type\": \"NewsArticle\",\n\t            \"@id\": \"https:\\\/\\\/www.interactivebrokers.com\\\/campus\\\/ibkr-quant-news\\\/robobrowser-automating-online-forms\\\/#article\",\n\t            \"isPartOf\": {\n\t                \"@id\": \"https:\\\/\\\/www.interactivebrokers.com\\\/campus\\\/ibkr-quant-news\\\/robobrowser-automating-online-forms\\\/\"\n\t            },\n\t            \"author\": {\n\t                \"name\": \"Andrew Treadway\",\n\t                \"@id\": \"https:\\\/\\\/ibkrcampus.com\\\/campus\\\/#\\\/schema\\\/person\\\/d4018570a16fb867f1c08412fc9c64bc\"\n\t            },\n\t            \"headline\": \"RoboBrowser: Automating Online Forms\",\n\t            \"datePublished\": \"2024-02-28T16:27:47+00:00\",\n\t            \"dateModified\": \"2024-02-28T16:27:55+00:00\",\n\t            \"mainEntityOfPage\": {\n\t                \"@id\": \"https:\\\/\\\/www.interactivebrokers.com\\\/campus\\\/ibkr-quant-news\\\/robobrowser-automating-online-forms\\\/\"\n\t            },\n\t            \"wordCount\": 478,\n\t            \"commentCount\": 0,\n\t            \"publisher\": {\n\t                \"@id\": \"https:\\\/\\\/ibkrcampus.com\\\/campus\\\/#organization\"\n\t            },\n\t            \"image\": {\n\t                \"@id\": \"https:\\\/\\\/www.interactivebrokers.com\\\/campus\\\/ibkr-quant-news\\\/robobrowser-automating-online-forms\\\/#primaryimage\"\n\t            },\n\t            \"thumbnailUrl\": \"https:\\\/\\\/www.interactivebrokers.com\\\/campus\\\/wp-content\\\/uploads\\\/sites\\\/2\\\/2020\\\/11\\\/python.jpg\",\n\t            \"keywords\": [\n\t                \"Data Science\",\n\t                \"Pandas\",\n\t                \"Python\",\n\t                \"RoboBrowser\"\n\t            ],\n\t            \"articleSection\": [\n\t                \"Data Science\",\n\t                \"Programming Languages\",\n\t                \"Python Development\",\n\t                \"Quant\",\n\t                \"Quant Development\"\n\t            ],\n\t            \"inLanguage\": \"en-US\",\n\t            \"potentialAction\": [\n\t                {\n\t                    \"@type\": \"CommentAction\",\n\t                    \"name\": \"Comment\",\n\t                    \"target\": [\n\t                        \"https:\\\/\\\/www.interactivebrokers.com\\\/campus\\\/ibkr-quant-news\\\/robobrowser-automating-online-forms\\\/#respond\"\n\t                    ]\n\t                }\n\t            ]\n\t        },\n\t        {\n\t            \"@type\": \"WebPage\",\n\t            \"@id\": \"https:\\\/\\\/www.interactivebrokers.com\\\/campus\\\/ibkr-quant-news\\\/robobrowser-automating-online-forms\\\/\",\n\t            \"url\": \"https:\\\/\\\/www.interactivebrokers.com\\\/campus\\\/ibkr-quant-news\\\/robobrowser-automating-online-forms\\\/\",\n\t            \"name\": \"RoboBrowser: Automating Online Forms | IBKR Campus US\",\n\t            \"isPartOf\": {\n\t                \"@id\": \"https:\\\/\\\/ibkrcampus.com\\\/campus\\\/#website\"\n\t            },\n\t            \"primaryImageOfPage\": {\n\t                \"@id\": \"https:\\\/\\\/www.interactivebrokers.com\\\/campus\\\/ibkr-quant-news\\\/robobrowser-automating-online-forms\\\/#primaryimage\"\n\t            },\n\t            \"image\": {\n\t                \"@id\": \"https:\\\/\\\/www.interactivebrokers.com\\\/campus\\\/ibkr-quant-news\\\/robobrowser-automating-online-forms\\\/#primaryimage\"\n\t            },\n\t            \"thumbnailUrl\": \"https:\\\/\\\/www.interactivebrokers.com\\\/campus\\\/wp-content\\\/uploads\\\/sites\\\/2\\\/2020\\\/11\\\/python.jpg\",\n\t            \"datePublished\": \"2024-02-28T16:27:47+00:00\",\n\t            \"dateModified\": \"2024-02-28T16:27:55+00:00\",\n\t            \"inLanguage\": \"en-US\",\n\t            \"potentialAction\": [\n\t                {\n\t                    \"@type\": \"ReadAction\",\n\t                    \"target\": [\n\t                        \"https:\\\/\\\/www.interactivebrokers.com\\\/campus\\\/ibkr-quant-news\\\/robobrowser-automating-online-forms\\\/\"\n\t                    ]\n\t                }\n\t            ]\n\t        },\n\t        {\n\t            \"@type\": \"ImageObject\",\n\t            \"inLanguage\": \"en-US\",\n\t            \"@id\": \"https:\\\/\\\/www.interactivebrokers.com\\\/campus\\\/ibkr-quant-news\\\/robobrowser-automating-online-forms\\\/#primaryimage\",\n\t            \"url\": \"https:\\\/\\\/www.interactivebrokers.com\\\/campus\\\/wp-content\\\/uploads\\\/sites\\\/2\\\/2020\\\/11\\\/python.jpg\",\n\t            \"contentUrl\": \"https:\\\/\\\/www.interactivebrokers.com\\\/campus\\\/wp-content\\\/uploads\\\/sites\\\/2\\\/2020\\\/11\\\/python.jpg\",\n\t            \"width\": 900,\n\t            \"height\": 550,\n\t            \"caption\": \"Python\"\n\t        },\n\t        {\n\t            \"@type\": \"WebSite\",\n\t            \"@id\": \"https:\\\/\\\/ibkrcampus.com\\\/campus\\\/#website\",\n\t            \"url\": \"https:\\\/\\\/ibkrcampus.com\\\/campus\\\/\",\n\t            \"name\": \"IBKR Campus US\",\n\t            \"description\": \"Financial Education from Interactive Brokers\",\n\t            \"publisher\": {\n\t                \"@id\": \"https:\\\/\\\/ibkrcampus.com\\\/campus\\\/#organization\"\n\t            },\n\t            \"potentialAction\": [\n\t                {\n\t                    \"@type\": \"SearchAction\",\n\t                    \"target\": {\n\t                        \"@type\": \"EntryPoint\",\n\t                        \"urlTemplate\": \"https:\\\/\\\/ibkrcampus.com\\\/campus\\\/?s={search_term_string}\"\n\t                    },\n\t                    \"query-input\": {\n\t                        \"@type\": \"PropertyValueSpecification\",\n\t                        \"valueRequired\": true,\n\t                        \"valueName\": \"search_term_string\"\n\t                    }\n\t                }\n\t            ],\n\t            \"inLanguage\": \"en-US\"\n\t        },\n\t        {\n\t            \"@type\": \"Organization\",\n\t            \"@id\": \"https:\\\/\\\/ibkrcampus.com\\\/campus\\\/#organization\",\n\t            \"name\": \"Interactive Brokers\",\n\t            \"alternateName\": \"IBKR\",\n\t            \"url\": \"https:\\\/\\\/ibkrcampus.com\\\/campus\\\/\",\n\t            \"logo\": {\n\t                \"@type\": \"ImageObject\",\n\t                \"inLanguage\": \"en-US\",\n\t                \"@id\": \"https:\\\/\\\/ibkrcampus.com\\\/campus\\\/#\\\/schema\\\/logo\\\/image\\\/\",\n\t                \"url\": \"https:\\\/\\\/www.interactivebrokers.com\\\/campus\\\/wp-content\\\/uploads\\\/sites\\\/2\\\/2024\\\/05\\\/ibkr-campus-logo.jpg\",\n\t                \"contentUrl\": \"https:\\\/\\\/www.interactivebrokers.com\\\/campus\\\/wp-content\\\/uploads\\\/sites\\\/2\\\/2024\\\/05\\\/ibkr-campus-logo.jpg\",\n\t                \"width\": 669,\n\t                \"height\": 669,\n\t                \"caption\": \"Interactive Brokers\"\n\t            },\n\t            \"image\": {\n\t                \"@id\": \"https:\\\/\\\/ibkrcampus.com\\\/campus\\\/#\\\/schema\\\/logo\\\/image\\\/\"\n\t            },\n\t            \"publishingPrinciples\": \"https:\\\/\\\/www.interactivebrokers.com\\\/campus\\\/about-ibkr-campus\\\/\",\n\t            \"ethicsPolicy\": \"https:\\\/\\\/www.interactivebrokers.com\\\/campus\\\/cyber-security-notice\\\/\"\n\t        },\n\t        {\n\t            \"@type\": \"Person\",\n\t            \"@id\": \"https:\\\/\\\/ibkrcampus.com\\\/campus\\\/#\\\/schema\\\/person\\\/d4018570a16fb867f1c08412fc9c64bc\",\n\t            \"name\": \"Andrew Treadway\",\n\t            \"description\": \"Andrew Treadway currently works as a Senior Data Scientist, and has experience doing analytics, software automation, and ETL. He completed a master\u2019s degree in computer science \\\/ machine learning, and an undergraduate degree in pure mathematics. Connect with him on LinkedIn: https:\\\/\\\/www.linkedin.com\\\/in\\\/andrew-treadway-a3b19b103\\\/In addition to TheAutomatic.net blog, he also teaches in-person courses on Python and R through my NYC meetup: more details.\",\n\t            \"sameAs\": [\n\t                \"https:\\\/\\\/theautomatic.net\\\/about-me\\\/\"\n\t            ],\n\t            \"url\": \"https:\\\/\\\/www.interactivebrokers.com\\\/campus\\\/author\\\/andrewtreadway\\\/\"\n\t        }\n\t    ]\n\t}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"RoboBrowser: Automating Online Forms | IBKR Quant","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.interactivebrokers.com\/campus\/wp-json\/wp\/v2\/posts\/202845\/","og_locale":"en_US","og_type":"article","og_title":"RoboBrowser: Automating Online Forms | IBKR Campus US","og_description":"Next, we create a RoboBrowser object. This object functions similarly to an actual web browser.","og_url":"https:\/\/www.interactivebrokers.com\/campus\/ibkr-quant-news\/robobrowser-automating-online-forms\/","og_site_name":"IBKR Campus US","article_published_time":"2024-02-28T16:27:47+00:00","article_modified_time":"2024-02-28T16:27:55+00:00","og_image":[{"width":900,"height":550,"url":"https:\/\/www.interactivebrokers.com\/campus\/wp-content\/uploads\/sites\/2\/2020\/11\/python.jpg","type":"image\/jpeg"}],"author":"Andrew Treadway","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Andrew Treadway","Est. reading time":"3 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"NewsArticle","@id":"https:\/\/www.interactivebrokers.com\/campus\/ibkr-quant-news\/robobrowser-automating-online-forms\/#article","isPartOf":{"@id":"https:\/\/www.interactivebrokers.com\/campus\/ibkr-quant-news\/robobrowser-automating-online-forms\/"},"author":{"name":"Andrew Treadway","@id":"https:\/\/ibkrcampus.com\/campus\/#\/schema\/person\/d4018570a16fb867f1c08412fc9c64bc"},"headline":"RoboBrowser: Automating Online Forms","datePublished":"2024-02-28T16:27:47+00:00","dateModified":"2024-02-28T16:27:55+00:00","mainEntityOfPage":{"@id":"https:\/\/www.interactivebrokers.com\/campus\/ibkr-quant-news\/robobrowser-automating-online-forms\/"},"wordCount":478,"commentCount":0,"publisher":{"@id":"https:\/\/ibkrcampus.com\/campus\/#organization"},"image":{"@id":"https:\/\/www.interactivebrokers.com\/campus\/ibkr-quant-news\/robobrowser-automating-online-forms\/#primaryimage"},"thumbnailUrl":"https:\/\/www.interactivebrokers.com\/campus\/wp-content\/uploads\/sites\/2\/2020\/11\/python.jpg","keywords":["Data Science","Pandas","Python","RoboBrowser"],"articleSection":["Data Science","Programming Languages","Python Development","Quant","Quant Development"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/www.interactivebrokers.com\/campus\/ibkr-quant-news\/robobrowser-automating-online-forms\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/www.interactivebrokers.com\/campus\/ibkr-quant-news\/robobrowser-automating-online-forms\/","url":"https:\/\/www.interactivebrokers.com\/campus\/ibkr-quant-news\/robobrowser-automating-online-forms\/","name":"RoboBrowser: Automating Online Forms | IBKR Campus US","isPartOf":{"@id":"https:\/\/ibkrcampus.com\/campus\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.interactivebrokers.com\/campus\/ibkr-quant-news\/robobrowser-automating-online-forms\/#primaryimage"},"image":{"@id":"https:\/\/www.interactivebrokers.com\/campus\/ibkr-quant-news\/robobrowser-automating-online-forms\/#primaryimage"},"thumbnailUrl":"https:\/\/www.interactivebrokers.com\/campus\/wp-content\/uploads\/sites\/2\/2020\/11\/python.jpg","datePublished":"2024-02-28T16:27:47+00:00","dateModified":"2024-02-28T16:27:55+00:00","inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.interactivebrokers.com\/campus\/ibkr-quant-news\/robobrowser-automating-online-forms\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.interactivebrokers.com\/campus\/ibkr-quant-news\/robobrowser-automating-online-forms\/#primaryimage","url":"https:\/\/www.interactivebrokers.com\/campus\/wp-content\/uploads\/sites\/2\/2020\/11\/python.jpg","contentUrl":"https:\/\/www.interactivebrokers.com\/campus\/wp-content\/uploads\/sites\/2\/2020\/11\/python.jpg","width":900,"height":550,"caption":"Python"},{"@type":"WebSite","@id":"https:\/\/ibkrcampus.com\/campus\/#website","url":"https:\/\/ibkrcampus.com\/campus\/","name":"IBKR Campus US","description":"Financial Education from Interactive Brokers","publisher":{"@id":"https:\/\/ibkrcampus.com\/campus\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/ibkrcampus.com\/campus\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/ibkrcampus.com\/campus\/#organization","name":"Interactive Brokers","alternateName":"IBKR","url":"https:\/\/ibkrcampus.com\/campus\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/ibkrcampus.com\/campus\/#\/schema\/logo\/image\/","url":"https:\/\/www.interactivebrokers.com\/campus\/wp-content\/uploads\/sites\/2\/2024\/05\/ibkr-campus-logo.jpg","contentUrl":"https:\/\/www.interactivebrokers.com\/campus\/wp-content\/uploads\/sites\/2\/2024\/05\/ibkr-campus-logo.jpg","width":669,"height":669,"caption":"Interactive Brokers"},"image":{"@id":"https:\/\/ibkrcampus.com\/campus\/#\/schema\/logo\/image\/"},"publishingPrinciples":"https:\/\/www.interactivebrokers.com\/campus\/about-ibkr-campus\/","ethicsPolicy":"https:\/\/www.interactivebrokers.com\/campus\/cyber-security-notice\/"},{"@type":"Person","@id":"https:\/\/ibkrcampus.com\/campus\/#\/schema\/person\/d4018570a16fb867f1c08412fc9c64bc","name":"Andrew Treadway","description":"Andrew Treadway currently works as a Senior Data Scientist, and has experience doing analytics, software automation, and ETL. He completed a master\u2019s degree in computer science \/ machine learning, and an undergraduate degree in pure mathematics. Connect with him on LinkedIn: https:\/\/www.linkedin.com\/in\/andrew-treadway-a3b19b103\/In addition to TheAutomatic.net blog, he also teaches in-person courses on Python and R through my NYC meetup: more details.","sameAs":["https:\/\/theautomatic.net\/about-me\/"],"url":"https:\/\/www.interactivebrokers.com\/campus\/author\/andrewtreadway\/"}]}},"jetpack_featured_media_url":"https:\/\/www.interactivebrokers.com\/campus\/wp-content\/uploads\/sites\/2\/2020\/11\/python.jpg","_links":{"self":[{"href":"https:\/\/ibkrcampus.com\/campus\/wp-json\/wp\/v2\/posts\/202845","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/ibkrcampus.com\/campus\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/ibkrcampus.com\/campus\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/ibkrcampus.com\/campus\/wp-json\/wp\/v2\/users\/388"}],"replies":[{"embeddable":true,"href":"https:\/\/ibkrcampus.com\/campus\/wp-json\/wp\/v2\/comments?post=202845"}],"version-history":[{"count":0,"href":"https:\/\/ibkrcampus.com\/campus\/wp-json\/wp\/v2\/posts\/202845\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/ibkrcampus.com\/campus\/wp-json\/wp\/v2\/media\/66906"}],"wp:attachment":[{"href":"https:\/\/ibkrcampus.com\/campus\/wp-json\/wp\/v2\/media?parent=202845"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/ibkrcampus.com\/campus\/wp-json\/wp\/v2\/categories?post=202845"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/ibkrcampus.com\/campus\/wp-json\/wp\/v2\/tags?post=202845"},{"taxonomy":"contributors-categories","embeddable":true,"href":"https:\/\/ibkrcampus.com\/campus\/wp-json\/wp\/v2\/contributors-categories?post=202845"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}