Asset Classes

Free investment financial education

Language

Multilingual content from IBKR

Close Navigation
Learn more about IBKR accounts
RoboBrowser: Automating Online Forms

RoboBrowser: Automating Online Forms

Posted February 28, 2024 at 11:27 am
Andrew Treadway
TheAutomatic.net

Background

RoboBrowser is a Python 3.x package for crawling through the web and submitting online forms. It works similarly to the older Python 2.x package, mechanize. This post is going to give a simple introduction using RoboBrowser to submit a form on Wunderground for scraping historical weather data.

Initial setup

RoboBrowser can be installed via pip:

pip install robobrowser

Let’s do the initial setup of the script by loading the RoboBrowser package. We’ll also load pandas, as we’ll be using that a little bit later.

from robobrowser import RoboBrowser
import pandas as pd

Create RoboBrowser Object

Next, we create a RoboBrowser object. This object functions similarly to an actual web browser. It allows you to navigate to different websites, fill in forms, and get HTML from webpages. All of a RoboBrowser object’s actions are completely invisible, so you won’t actually see a physical browser while any of this is happening.

Using our object, browser, we navigate to Wunderground’s historical weather main site (see https://www.wunderground.com/history). This is done using the open method, as below.

'''Create RoboBrowser object
   This will act similarly to a typical web browser'''
browser = RoboBrowser(history=True)
 
'''Navigate browser to Wunderground's historical weather page'''
browser.open('https://www.wunderground.com/history')

Filling out the form

Next, let’s get the list of forms on the webpage, https://www.wunderground.com/history.

forms = browser.get_forms()

Above, we can see there are two forms on the webpage. The last of these is the one we need, as this is what we’ll use to submit a zip code with a historical date to get weather information. You can tell this by looking at the inputs in the print-out above of forms i.e. code, month, day etc.

form = forms[1]

Inputting data into our form is similar to adding values to a dictionary in Python. We do it like this:

'''Enter inputs into form'''
form['code'] = '12345'
form['month'] = 'September'
form['day'] = '18'
form['year'] = '2017'

As you can see, our inputs require a zip code (just called ‘code’ in the html), along with values for month, day, and year.

Now, we can submit the web form in one line:

'''Submit form'''
browser.submit_form(form)

You can get the HTML of the page following submission using the parsed method of browser.

With the pandas read_html method, we can get the HTML tables on the page. In our example, we pull just the 0th (Python indexed 0) table on the page.

'''Scrape html of page following form submission'''
html = str(browser.parsed)
 
'''Use pandas to get primary table on page'''
data = pd.read_html(html)[0]

Putting it all together…

Let’s wrap what we’ve done into a function, like this:

def get_historical_data(code, month, day, year):
     
    '''Create RoboBrowser object
       This will act similar to a typical web browser'''
    browser = RoboBrowser(history=True)
     
    '''Navigate browser to Wunderground's historical weather page'''
    browser.open('https://www.wunderground.com/history')
     
    '''Find the form to input zip code and date'''
    forms = browser.get_forms()
     
    form = forms[1]
     
    '''Enter inputs into form'''
    form['code'] = code
    form['month'] = month
    form['day'] = day
    form['year'] = year
     
    '''Submit form'''
    browser.submit_form(form)
     
    '''Scrape html of page following form submission'''
    html = str(browser.parsed)
     
    '''Use pandas to get primary table on page'''
    data = pd.read_html(html)[0]
 
    return data

We now have a function we can call by passing a zip code with a particular month, day, and year.

get_historical_data('23456','September','18','2000')    
 
get_historical_data('23456','March','18','2000')

We can simplify the parameters we’re passing so that we just need to input a zip code with a date, rather than three separate arguments for month, day, and year, respectively. We’ll do this by converting a date input from the user to a pandas timestamp, which allows us to parse out these needed attributes about the date.

def get_historical_data(code, date):
     
    '''Create RoboBrowser object
       This will act similar to a typical web browser'''
    browser = RoboBrowser(history=True)
     
    '''Navigate browser to Wunderground's historical weather page'''
    browser.open('https://www.wunderground.com/history')
     
    '''Find the form to input zip code and date'''
    forms = browser.get_forms()
     
    form = forms[1]
     
    '''Convert date to pandas timestamp'''
    date = pd.Timestamp(date)
     
    '''Enter inputs into form'''
    form['code'] = code
    form['month'] = date.month
    form['day'] = date.day
    form['year'] = date.year
     
    '''Submit form'''
    browser.submit_form(form)
     
    '''Scrape html of page following form submission'''
    html = str(browser.parsed)
     
    '''Use pandas to get primary table on page'''
    data = pd.read_html(html)[0]
 
    return data

So calling our function now looks like this:

get_historical_data("23456", "9/18/2000")
 
get_historical_data("23456", "3/18/2000")

Originally posted on TheAutomatic.net blog.

Join The Conversation

If you have a general question, it may already be covered in our FAQs. If you have an account-specific question or concern, please reach out to Client Services.

Leave a Reply

Disclosure: Interactive Brokers

Information posted on IBKR Campus that is provided by third-parties does NOT constitute a recommendation that you should contract for the services of that third party. Third-party participants who contribute to IBKR Campus are independent of Interactive Brokers and Interactive Brokers does not make any representations or warranties concerning the services offered, their past or future performance, or the accuracy of the information provided by the third party. Past performance is no guarantee of future results.

This material is from TheAutomatic.net and is being posted with its permission. The views expressed in this material are solely those of the author and/or TheAutomatic.net and Interactive Brokers is not endorsing or recommending any investment or trading discussed in the material. This material is not and should not be construed as an offer to buy or sell any security. It should not be construed as research or investment advice or a recommendation to buy, sell or hold any security or commodity. This material does not and is not intended to take into account the particular financial conditions, investment objectives or requirements of individual customers. Before acting on this material, you should consider whether it is suitable for your particular circumstances and, as necessary, seek professional advice.

IBKR Campus Newsletters

This website uses cookies to collect usage information in order to offer a better browsing experience. By browsing this site or by clicking on the "ACCEPT COOKIES" button you accept our Cookie Policy.