{"id":187212,"date":"2023-03-23T11:24:45","date_gmt":"2023-03-23T15:24:45","guid":{"rendered":"https:\/\/ibkrcampus.com\/?p=187212"},"modified":"2023-03-24T10:34:32","modified_gmt":"2023-03-24T14:34:32","slug":"beautifulsoup-vs-rvest","status":"publish","type":"post","link":"https:\/\/www.interactivebrokers.com\/campus\/ibkr-quant-news\/beautifulsoup-vs-rvest\/","title":{"rendered":"BeautifulSoup vs. Rvest"},"content":{"rendered":"\n

This post will compare Python\u2019s BeautifulSoup<\/a> package to R\u2019s rvest<\/a> package for web scraping. We\u2019ll also talk about additional functionality in rvest<\/strong> (that doesn\u2019t exist in BeautifulSoup<\/strong>) in comparison to a couple of other Python packages (including pandas<\/strong> and RoboBrowser<\/strong>).<\/p>\n\n\n\n

Getting started<\/strong><\/h2>\n\n\n\n

<\/strong><\/p>\n\n\n\n

BeautifulSoup<\/a> and rvest<\/a> both involve creating an object that we can use to parse the HTML from a webpage. However, one immediate difference is that BeautifulSoup is just a web parser, so it doesn\u2019t connect to webpages. rvest<\/strong>, on the other hand, can connect to a webpage and scrape \/ parse its HTML in a single package.<\/p>\n\n\n\n

In BeautifulSoup, our initial setup looks like this:<\/p>\n\n\n\n

# load packages\nfrom bs4 import BeautifulSoup\nimport requests\n \n# connect to webpage\nresp = requests.get(\"\"https:\/\/www.azlyrics.com\/b\/beatles.html\"\")\n \n# get BeautifulSoup object\nsoup = BeautifulSoup(resp.content)<\/pre>\n\n\n\n
In comparison, here\u2019s what using rvest<\/strong> is like:<\/p>\n\n\n\n
# load rvest package\nlibrary(rvest)\n \n# get HTML object\nhtml_data <- read_html(\"https:\/\/www.azlyrics.com\/b\/beatles.html\")<\/pre>\n\n\n\n
Searching for specific HTML tags<\/strong><\/h2>\n\n\n\n
Next, let\u2019s take our parser objects and find all the links on the page.<\/p>\n\n\n\n
If you\u2019re familiar with BeautifulSoup<\/strong>, that would look like:<\/p>\n\n\n\n
links = soup.find_all(\"a\")<\/pre>\n\n\n\n
In rvest, however, we use syntax similar to dplyr<\/strong> and other tidyverse packages by using %>%.<\/p>\n\n\n\n
links <- html_data %>% html_nodes(\"a\")\n \nurls <- links %>% html_attr(\"href\")<\/pre>\n\n\n\n
In BeautifulSoup<\/strong>, we use the find_all<\/em> method to extract a list of all of a specific tag\u2019s objects from a webpage. Thus, in the links example, we specify we want to get all of the anchor tags (or \u201ca\u201d tags), which create HTML links on the page. If we wanted to scrape other types of tags, such as div<\/em> tags or p<\/em> tags, we just need to switch out \u201ca\u201d with \u201cdiv\u201d, \u201cp\u201d, or whatever tag we want.<\/p>\n\n\n\n
# get all div tags\nsoup.find_all(\"div\")\n \n# get all h1 tags\nsoup.find_all(\"h1\")<\/pre>\n\n\n\n
With rvest<\/strong>, we can get specific tags from HTML using html_nodes<\/em>. Thus, if we wanted to scrape different tags, such as the div tags or h1 tags, we could do this:<\/p>\n\n\n\n
# scrape all div tags\nhtml_data %>% html_nodes(\"div\")\n \n# scrape header h1 tags\nhtml_data %>% html_nodes(\"h1\")<\/pre>\n\n\n\n
Getting attributes and text from tags<\/strong><\/h2>\n\n\n\n
<\/strong><\/p>\n\n\n\n
In BeautifulSoup<\/strong>, we get attributes from HTML tags using the get<\/em> method. We can use a list comprehension<\/a> to get the href<\/a><\/em> attribute of each link (the href<\/em> attribute of a link is its destination URL).<\/p>\n\n\n\n
urls = [link.get(\"href\") for link in links]<\/pre>\n\n\n\n
To get other attributes, we just need to change our input to the get<\/em> method.<\/p>\n\n\n\n
# get the target attribute from each link\n[link.get(\"target\") for link in links]\n \n# get the ID attribute of each div tag\n[div.get(\"id\") for div in soup.find_all(\"div\")]<\/pre>\n\n\n\n
Using rvest<\/strong>, the html_attr<\/em> function can be used to get attributes from tags. So to get the URL of each link object we scrape, we need to specify that we want to get the href<\/em> attribute from each link, similarly to BeautifulSoup<\/strong>:<\/p>\n\n\n\n
urls <- links %>% html_attr(\"href\")<\/pre>\n\n\n\n
Likewise, if we want to scrape the IDs from the div tags, we can do this:<\/p>\n\n\n\n
html_data %>% html_nodes(\"div\") %>% html_attr(\"id\")<\/pre>\n\n\n\n
Notice how, like other tidyverse packages, we can chain together multiple operations.<\/p>\n\n\n\n
If we want to scrape the text from each of the links, we can use html_text<\/em>:<\/p>\n\n\n\n
links %>% html_text()<\/pre>\n\n\n\n
BeautifulSoup\u2019s<\/strong> way of accomplishing this is by using the text<\/em> method of a tag object:<\/p>\n\n\n\n
[link.text for link in links]<\/pre>\n\n\n\n
Scraping HTML tables<\/strong><\/h2>\n\n\n\n
Let\u2019s look at another example for scraping HTML tables.<\/p>\n\n\n\n
We can scrape HTML tables using rvest\u2019s<\/strong> html_table<\/em> method. This method will extract all tables found on the input webpage. The fill = TRUE<\/em> parameter is specifying that we want to fill any rows that have less than the maximum number of columns in a table with NAs. The tables will be stored as a list of data frames.<\/p>\n\n\n\n
city_data <- read_html(\"https:\/\/www.city-data.com\/city\/Florida.html\")\n \ncity_data %>% html_table(fill = TRUE)<\/pre>\n\n\n\n
Scraping tables with BeautifulSoup<\/strong> into a data frame object is a bit different. One way we can scrape tables with Python is to loop through the tr (row) or td (data cell in table) tags. But the closest analogy of rvest\u2019s<\/strong> functionality here is to use pandas<\/strong>:<\/p>\n\n\n\n
import pandas as pd\n \npd.read_html(\"https:\/\/www.city-data.com\/city\/Florida.html\")<\/pre>\n\n\n\n
Like using html_table<\/em>, this will return a list of data frames corresponding to the tables found on the webpage.<\/p>\n\n\n\n
Browser simulation with rvest<\/strong><\/h2>\n\n\n\n
An additional feature of rvest<\/strong> is that it can perform browser simulation. BeautifulSoup<\/strong> cannot do this; however, Python offers several alternatives including requests_html<\/strong> and RoboBrowser<\/strong> (each discussed here<\/a>).<\/p>\n\n\n\n
With rvest<\/strong>, we can start a browser session using the html_session<\/em> function:<\/p>\n\n\n\n
site = \"https:\/\/www.azlyrics.com\/b\/beatles.html\"\n \nsession <- html_session(site)<\/pre>\n\n\n\n
With our session object, we can navigate to different links on the page, just like a real web browser. There\u2019s a couple ways of doing this. One is to input the index of the link we want to go to. For example, to navigate to the third link of the page, we would write the below code:<\/p>\n\n\n\n
session %>% follow_link(3)<\/pre>\n\n\n\n
Here\u2019s a couple more examples:<\/p>\n\n\n\n
# navigate to the 5th link on the page\nsession %>% follow_link(5)\n \n# navigate to the 10th link on the page\nsession %>% follow_link(10)<\/pre>\n\n\n\n
You can also simulate clicking on links based off text. For example, the below code will navigate to the first link containing the text \u201cSun\u201d. The input is case sensitive.<\/p>\n\n\n\n
session %>% follow_link(\"Sun\")<\/pre>\n\n\n\n
You can use the session object to navigate directly to other webpages using the jump_to<\/em> function.<\/p>\n\n\n\n
session %>% jump_to(\"https:\/\/www.azlyrics.com\/a.html\")<\/pre>\n\n\n\n
If we use the RoboBrowser<\/strong> package in Python to somewhat replicate the R code above, we could write this:<\/p>\n\n\n\n
from robobrowser import RoboBrowser\n \n# create a RoboBrowser object\nbrowser = RoboBrowser(history = True)\n  \n# navigate to webpage\nbrowser.open(\"https:\/\/www.azlyrics.com\/b\/beatles.html\")\n \n# get links\nlinks = browser.get_links()\n \n# follow link\nbrowser.follow_link(links[5])\n \n# follow different link\nbrowser.follow_link(links[10])<\/pre>\n\n\n\n
The open<\/em> method used above is analogous to rvest\u2019s<\/strong> jump_to<\/em> function. The follow_link<\/em> method in RoboBrowser<\/strong> serves a similar purpose to the function of the same name in rvest<\/strong>, but behaves a little differently. This method takes a link object<\/em> as input, rather than the index of a link, or text within a link that you\u2019re searching for. Thus, we can use the links<\/em> object above to specify a link that we want to \u201cfollow\u201d or click. If we want to click on a link based off text like we did in the rvest<\/strong> example above, we could write the below code.<\/p>\n\n\n\n
# filter the list of links to only links containing \"Sun\" in their text\nsun_links = filter(lambda link: \"Sun\" in link.text, links)\n \n# Click on the first link containing the word \"Sun\"\nbrowser.follow_link(next(sun_links))<\/pre>\n\n\n\n
To learn more about browser simulation in Python, click here<\/a>.<\/p>\n\n\n\n
Originally posted on TheAutomatic.net<\/a> blog.<\/em><\/p>\n","protected":false},"excerpt":{"rendered":"
This post will compare Python\u2019s BeautifulSoup package to R\u2019s rvest package for web scraping.<\/p>\n","protected":false},"author":388,"featured_media":182355,"comment_status":"closed","ping_status":"open","sticky":true,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":"","jetpack_post_was_ever_published":false},"categories":[339,343,349,338,350,341,351,352,344,2197,342],"tags":[14986,806,2535,1224,14988,14987,1045,14989],"contributors-categories":[13695],"class_list":["post-187212","post","type-post","status-publish","format-standard","has-post-thumbnail","category-data-science","category-programing-languages","category-python-development","category-ibkr-quant-news","category-quant-asia-pacific","category-quant-development","category-quant-europe","category-quant-north-america","category-quant-regions","category-quant-south-america","category-r-development","tag-beautifulsoup","tag-data-science","tag-dplyr","tag-pandas","tag-robobrowser","tag-rvest","tag-tidyverse","tag-web-scraping","contributors-categories-theautomatic-net"],"pp_statuses_selecting_workflow":false,"pp_workflow_action":"current","pp_status_selection":"publish","acf":[],"yoast_head":"\nBeautifulSoup vs. Rvest | IBKR Quant<\/title>\n<meta name=\"description\" content=\"This post will compare Python\u2019s BeautifulSoup package to R\u2019s rvest package for web scraping.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.interactivebrokers.com\/campus\/wp-json\/wp\/v2\/posts\/187212\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"BeautifulSoup vs. Rvest | IBKR Campus US\" \/>\n<meta property=\"og:description\" content=\"This post will compare Python\u2019s BeautifulSoup package to R\u2019s rvest package for web scraping.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.interactivebrokers.com\/campus\/ibkr-quant-news\/beautifulsoup-vs-rvest\/\" \/>\n<meta property=\"og:site_name\" content=\"IBKR Campus US\" \/>\n<meta property=\"article:published_time\" content=\"2023-03-23T15:24:45+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2023-03-24T14:34:32+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.interactivebrokers.com\/campus\/wp-content\/uploads\/sites\/2\/2023\/02\/blue-abstract-server.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"1000\" \/>\n\t<meta property=\"og:image:height\" content=\"563\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"Andrew Treadway\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Andrew Treadway\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"5 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\n\t    \"@context\": \"https:\\\/\\\/schema.org\",\n\t    \"@graph\": [\n\t        {\n\t            \"@type\": \"NewsArticle\",\n\t            \"@id\": \"https:\\\/\\\/www.interactivebrokers.com\\\/campus\\\/ibkr-quant-news\\\/beautifulsoup-vs-rvest\\\/#article\",\n\t            \"isPartOf\": {\n\t                \"@id\": \"https:\\\/\\\/www.interactivebrokers.com\\\/campus\\\/ibkr-quant-news\\\/beautifulsoup-vs-rvest\\\/\"\n\t            },\n\t            \"author\": {\n\t                \"name\": \"Andrew Treadway\",\n\t                \"@id\": \"https:\\\/\\\/ibkrcampus.com\\\/campus\\\/#\\\/schema\\\/person\\\/d4018570a16fb867f1c08412fc9c64bc\"\n\t            },\n\t            \"headline\": \"BeautifulSoup vs. Rvest\",\n\t            \"datePublished\": \"2023-03-23T15:24:45+00:00\",\n\t            \"dateModified\": \"2023-03-24T14:34:32+00:00\",\n\t            \"mainEntityOfPage\": {\n\t                \"@id\": \"https:\\\/\\\/www.interactivebrokers.com\\\/campus\\\/ibkr-quant-news\\\/beautifulsoup-vs-rvest\\\/\"\n\t            },\n\t            \"wordCount\": 974,\n\t            \"publisher\": {\n\t                \"@id\": \"https:\\\/\\\/ibkrcampus.com\\\/campus\\\/#organization\"\n\t            },\n\t            \"image\": {\n\t                \"@id\": \"https:\\\/\\\/www.interactivebrokers.com\\\/campus\\\/ibkr-quant-news\\\/beautifulsoup-vs-rvest\\\/#primaryimage\"\n\t            },\n\t            \"thumbnailUrl\": \"https:\\\/\\\/www.interactivebrokers.com\\\/campus\\\/wp-content\\\/uploads\\\/sites\\\/2\\\/2023\\\/02\\\/blue-abstract-server.jpg\",\n\t            \"keywords\": [\n\t                \"BeautifulSoup\",\n\t                \"Data Science\",\n\t                \"dplyr\",\n\t                \"Pandas\",\n\t                \"RoboBrowser\",\n\t                \"rvest\",\n\t                \"tidyverse\",\n\t                \"Web Scraping\"\n\t            ],\n\t            \"articleSection\": [\n\t                \"Data Science\",\n\t                \"Programming Languages\",\n\t                \"Python Development\",\n\t                \"Quant\",\n\t                \"Quant Asia Pacific\",\n\t                \"Quant Development\",\n\t                \"Quant Europe\",\n\t                \"Quant North America\",\n\t                \"Quant Regions\",\n\t                \"Quant South America\",\n\t                \"R Development\"\n\t            ],\n\t            \"inLanguage\": \"en-US\"\n\t        },\n\t        {\n\t            \"@type\": \"WebPage\",\n\t            \"@id\": \"https:\\\/\\\/www.interactivebrokers.com\\\/campus\\\/ibkr-quant-news\\\/beautifulsoup-vs-rvest\\\/\",\n\t            \"url\": \"https:\\\/\\\/www.interactivebrokers.com\\\/campus\\\/ibkr-quant-news\\\/beautifulsoup-vs-rvest\\\/\",\n\t            \"name\": \"BeautifulSoup vs. Rvest | IBKR Campus US\",\n\t            \"isPartOf\": {\n\t                \"@id\": \"https:\\\/\\\/ibkrcampus.com\\\/campus\\\/#website\"\n\t            },\n\t            \"primaryImageOfPage\": {\n\t                \"@id\": \"https:\\\/\\\/www.interactivebrokers.com\\\/campus\\\/ibkr-quant-news\\\/beautifulsoup-vs-rvest\\\/#primaryimage\"\n\t            },\n\t            \"image\": {\n\t                \"@id\": \"https:\\\/\\\/www.interactivebrokers.com\\\/campus\\\/ibkr-quant-news\\\/beautifulsoup-vs-rvest\\\/#primaryimage\"\n\t            },\n\t            \"thumbnailUrl\": \"https:\\\/\\\/www.interactivebrokers.com\\\/campus\\\/wp-content\\\/uploads\\\/sites\\\/2\\\/2023\\\/02\\\/blue-abstract-server.jpg\",\n\t            \"datePublished\": \"2023-03-23T15:24:45+00:00\",\n\t            \"dateModified\": \"2023-03-24T14:34:32+00:00\",\n\t            \"description\": \"This post will compare Python\u2019s BeautifulSoup package to R\u2019s rvest package for web scraping.\",\n\t            \"inLanguage\": \"en-US\",\n\t            \"potentialAction\": [\n\t                {\n\t                    \"@type\": \"ReadAction\",\n\t                    \"target\": [\n\t                        \"https:\\\/\\\/www.interactivebrokers.com\\\/campus\\\/ibkr-quant-news\\\/beautifulsoup-vs-rvest\\\/\"\n\t                    ]\n\t                }\n\t            ]\n\t        },\n\t        {\n\t            \"@type\": \"ImageObject\",\n\t            \"inLanguage\": \"en-US\",\n\t            \"@id\": \"https:\\\/\\\/www.interactivebrokers.com\\\/campus\\\/ibkr-quant-news\\\/beautifulsoup-vs-rvest\\\/#primaryimage\",\n\t            \"url\": \"https:\\\/\\\/www.interactivebrokers.com\\\/campus\\\/wp-content\\\/uploads\\\/sites\\\/2\\\/2023\\\/02\\\/blue-abstract-server.jpg\",\n\t            \"contentUrl\": \"https:\\\/\\\/www.interactivebrokers.com\\\/campus\\\/wp-content\\\/uploads\\\/sites\\\/2\\\/2023\\\/02\\\/blue-abstract-server.jpg\",\n\t            \"width\": 1000,\n\t            \"height\": 563,\n\t            \"caption\": \"Autocorrelation and Autocovariance: Calculation, Examples, and More \u2013 Part II\"\n\t        },\n\t        {\n\t            \"@type\": \"WebSite\",\n\t            \"@id\": \"https:\\\/\\\/ibkrcampus.com\\\/campus\\\/#website\",\n\t            \"url\": \"https:\\\/\\\/ibkrcampus.com\\\/campus\\\/\",\n\t            \"name\": \"IBKR Campus US\",\n\t            \"description\": \"Financial Education from Interactive Brokers\",\n\t            \"publisher\": {\n\t                \"@id\": \"https:\\\/\\\/ibkrcampus.com\\\/campus\\\/#organization\"\n\t            },\n\t            \"potentialAction\": [\n\t                {\n\t                    \"@type\": \"SearchAction\",\n\t                    \"target\": {\n\t                        \"@type\": \"EntryPoint\",\n\t                        \"urlTemplate\": \"https:\\\/\\\/ibkrcampus.com\\\/campus\\\/?s={search_term_string}\"\n\t                    },\n\t                    \"query-input\": {\n\t                        \"@type\": \"PropertyValueSpecification\",\n\t                        \"valueRequired\": true,\n\t                        \"valueName\": \"search_term_string\"\n\t                    }\n\t                }\n\t            ],\n\t            \"inLanguage\": \"en-US\"\n\t        },\n\t        {\n\t            \"@type\": \"Organization\",\n\t            \"@id\": \"https:\\\/\\\/ibkrcampus.com\\\/campus\\\/#organization\",\n\t            \"name\": \"Interactive Brokers\",\n\t            \"alternateName\": \"IBKR\",\n\t            \"url\": \"https:\\\/\\\/ibkrcampus.com\\\/campus\\\/\",\n\t            \"logo\": {\n\t                \"@type\": \"ImageObject\",\n\t                \"inLanguage\": \"en-US\",\n\t                \"@id\": \"https:\\\/\\\/ibkrcampus.com\\\/campus\\\/#\\\/schema\\\/logo\\\/image\\\/\",\n\t                \"url\": \"https:\\\/\\\/www.interactivebrokers.com\\\/campus\\\/wp-content\\\/uploads\\\/sites\\\/2\\\/2024\\\/05\\\/ibkr-campus-logo.jpg\",\n\t                \"contentUrl\": \"https:\\\/\\\/www.interactivebrokers.com\\\/campus\\\/wp-content\\\/uploads\\\/sites\\\/2\\\/2024\\\/05\\\/ibkr-campus-logo.jpg\",\n\t                \"width\": 669,\n\t                \"height\": 669,\n\t                \"caption\": \"Interactive Brokers\"\n\t            },\n\t            \"image\": {\n\t                \"@id\": \"https:\\\/\\\/ibkrcampus.com\\\/campus\\\/#\\\/schema\\\/logo\\\/image\\\/\"\n\t            },\n\t            \"publishingPrinciples\": \"https:\\\/\\\/www.interactivebrokers.com\\\/campus\\\/about-ibkr-campus\\\/\",\n\t            \"ethicsPolicy\": \"https:\\\/\\\/www.interactivebrokers.com\\\/campus\\\/cyber-security-notice\\\/\"\n\t        },\n\t        {\n\t            \"@type\": \"Person\",\n\t            \"@id\": \"https:\\\/\\\/ibkrcampus.com\\\/campus\\\/#\\\/schema\\\/person\\\/d4018570a16fb867f1c08412fc9c64bc\",\n\t            \"name\": \"Andrew Treadway\",\n\t            \"description\": \"Andrew Treadway currently works as a Senior Data Scientist, and has experience doing analytics, software automation, and ETL. He completed a master\u2019s degree in computer science \\\/ machine learning, and an undergraduate degree in pure mathematics. Connect with him on LinkedIn: https:\\\/\\\/www.linkedin.com\\\/in\\\/andrew-treadway-a3b19b103\\\/In addition to TheAutomatic.net blog, he also teaches in-person courses on Python and R through my NYC meetup: more details.\",\n\t            \"sameAs\": [\n\t                \"https:\\\/\\\/theautomatic.net\\\/about-me\\\/\"\n\t            ],\n\t            \"url\": \"https:\\\/\\\/www.interactivebrokers.com\\\/campus\\\/author\\\/andrewtreadway\\\/\"\n\t        }\n\t    ]\n\t}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"BeautifulSoup vs. Rvest | IBKR Quant","description":"This post will compare Python\u2019s BeautifulSoup package to R\u2019s rvest package for web scraping.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.interactivebrokers.com\/campus\/wp-json\/wp\/v2\/posts\/187212\/","og_locale":"en_US","og_type":"article","og_title":"BeautifulSoup vs. Rvest | IBKR Campus US","og_description":"This post will compare Python\u2019s BeautifulSoup package to R\u2019s rvest package for web scraping.","og_url":"https:\/\/www.interactivebrokers.com\/campus\/ibkr-quant-news\/beautifulsoup-vs-rvest\/","og_site_name":"IBKR Campus US","article_published_time":"2023-03-23T15:24:45+00:00","article_modified_time":"2023-03-24T14:34:32+00:00","og_image":[{"width":1000,"height":563,"url":"https:\/\/www.interactivebrokers.com\/campus\/wp-content\/uploads\/sites\/2\/2023\/02\/blue-abstract-server.jpg","type":"image\/jpeg"}],"author":"Andrew Treadway","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Andrew Treadway","Est. reading time":"5 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"NewsArticle","@id":"https:\/\/www.interactivebrokers.com\/campus\/ibkr-quant-news\/beautifulsoup-vs-rvest\/#article","isPartOf":{"@id":"https:\/\/www.interactivebrokers.com\/campus\/ibkr-quant-news\/beautifulsoup-vs-rvest\/"},"author":{"name":"Andrew Treadway","@id":"https:\/\/ibkrcampus.com\/campus\/#\/schema\/person\/d4018570a16fb867f1c08412fc9c64bc"},"headline":"BeautifulSoup vs. Rvest","datePublished":"2023-03-23T15:24:45+00:00","dateModified":"2023-03-24T14:34:32+00:00","mainEntityOfPage":{"@id":"https:\/\/www.interactivebrokers.com\/campus\/ibkr-quant-news\/beautifulsoup-vs-rvest\/"},"wordCount":974,"publisher":{"@id":"https:\/\/ibkrcampus.com\/campus\/#organization"},"image":{"@id":"https:\/\/www.interactivebrokers.com\/campus\/ibkr-quant-news\/beautifulsoup-vs-rvest\/#primaryimage"},"thumbnailUrl":"https:\/\/www.interactivebrokers.com\/campus\/wp-content\/uploads\/sites\/2\/2023\/02\/blue-abstract-server.jpg","keywords":["BeautifulSoup","Data Science","dplyr","Pandas","RoboBrowser","rvest","tidyverse","Web Scraping"],"articleSection":["Data Science","Programming Languages","Python Development","Quant","Quant Asia Pacific","Quant Development","Quant Europe","Quant North America","Quant Regions","Quant South America","R Development"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/www.interactivebrokers.com\/campus\/ibkr-quant-news\/beautifulsoup-vs-rvest\/","url":"https:\/\/www.interactivebrokers.com\/campus\/ibkr-quant-news\/beautifulsoup-vs-rvest\/","name":"BeautifulSoup vs. Rvest | IBKR Campus US","isPartOf":{"@id":"https:\/\/ibkrcampus.com\/campus\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.interactivebrokers.com\/campus\/ibkr-quant-news\/beautifulsoup-vs-rvest\/#primaryimage"},"image":{"@id":"https:\/\/www.interactivebrokers.com\/campus\/ibkr-quant-news\/beautifulsoup-vs-rvest\/#primaryimage"},"thumbnailUrl":"https:\/\/www.interactivebrokers.com\/campus\/wp-content\/uploads\/sites\/2\/2023\/02\/blue-abstract-server.jpg","datePublished":"2023-03-23T15:24:45+00:00","dateModified":"2023-03-24T14:34:32+00:00","description":"This post will compare Python\u2019s BeautifulSoup package to R\u2019s rvest package for web scraping.","inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.interactivebrokers.com\/campus\/ibkr-quant-news\/beautifulsoup-vs-rvest\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.interactivebrokers.com\/campus\/ibkr-quant-news\/beautifulsoup-vs-rvest\/#primaryimage","url":"https:\/\/www.interactivebrokers.com\/campus\/wp-content\/uploads\/sites\/2\/2023\/02\/blue-abstract-server.jpg","contentUrl":"https:\/\/www.interactivebrokers.com\/campus\/wp-content\/uploads\/sites\/2\/2023\/02\/blue-abstract-server.jpg","width":1000,"height":563,"caption":"Autocorrelation and Autocovariance: Calculation, Examples, and More \u2013 Part II"},{"@type":"WebSite","@id":"https:\/\/ibkrcampus.com\/campus\/#website","url":"https:\/\/ibkrcampus.com\/campus\/","name":"IBKR Campus US","description":"Financial Education from Interactive Brokers","publisher":{"@id":"https:\/\/ibkrcampus.com\/campus\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/ibkrcampus.com\/campus\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/ibkrcampus.com\/campus\/#organization","name":"Interactive Brokers","alternateName":"IBKR","url":"https:\/\/ibkrcampus.com\/campus\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/ibkrcampus.com\/campus\/#\/schema\/logo\/image\/","url":"https:\/\/www.interactivebrokers.com\/campus\/wp-content\/uploads\/sites\/2\/2024\/05\/ibkr-campus-logo.jpg","contentUrl":"https:\/\/www.interactivebrokers.com\/campus\/wp-content\/uploads\/sites\/2\/2024\/05\/ibkr-campus-logo.jpg","width":669,"height":669,"caption":"Interactive Brokers"},"image":{"@id":"https:\/\/ibkrcampus.com\/campus\/#\/schema\/logo\/image\/"},"publishingPrinciples":"https:\/\/www.interactivebrokers.com\/campus\/about-ibkr-campus\/","ethicsPolicy":"https:\/\/www.interactivebrokers.com\/campus\/cyber-security-notice\/"},{"@type":"Person","@id":"https:\/\/ibkrcampus.com\/campus\/#\/schema\/person\/d4018570a16fb867f1c08412fc9c64bc","name":"Andrew Treadway","description":"Andrew Treadway currently works as a Senior Data Scientist, and has experience doing analytics, software automation, and ETL. He completed a master\u2019s degree in computer science \/ machine learning, and an undergraduate degree in pure mathematics. Connect with him on LinkedIn: https:\/\/www.linkedin.com\/in\/andrew-treadway-a3b19b103\/In addition to TheAutomatic.net blog, he also teaches in-person courses on Python and R through my NYC meetup: more details.","sameAs":["https:\/\/theautomatic.net\/about-me\/"],"url":"https:\/\/www.interactivebrokers.com\/campus\/author\/andrewtreadway\/"}]}},"jetpack_featured_media_url":"https:\/\/www.interactivebrokers.com\/campus\/wp-content\/uploads\/sites\/2\/2023\/02\/blue-abstract-server.jpg","_links":{"self":[{"href":"https:\/\/ibkrcampus.com\/campus\/wp-json\/wp\/v2\/posts\/187212","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/ibkrcampus.com\/campus\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/ibkrcampus.com\/campus\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/ibkrcampus.com\/campus\/wp-json\/wp\/v2\/users\/388"}],"replies":[{"embeddable":true,"href":"https:\/\/ibkrcampus.com\/campus\/wp-json\/wp\/v2\/comments?post=187212"}],"version-history":[{"count":0,"href":"https:\/\/ibkrcampus.com\/campus\/wp-json\/wp\/v2\/posts\/187212\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/ibkrcampus.com\/campus\/wp-json\/wp\/v2\/media\/182355"}],"wp:attachment":[{"href":"https:\/\/ibkrcampus.com\/campus\/wp-json\/wp\/v2\/media?parent=187212"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/ibkrcampus.com\/campus\/wp-json\/wp\/v2\/categories?post=187212"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/ibkrcampus.com\/campus\/wp-json\/wp\/v2\/tags?post=187212"},{"taxonomy":"contributors-categories","embeddable":true,"href":"https:\/\/ibkrcampus.com\/campus\/wp-json\/wp\/v2\/contributors-categories?post=187212"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}