{"id":45021,"date":"2020-05-13T12:31:43","date_gmt":"2020-05-13T16:31:43","guid":{"rendered":"https:\/\/ibkrcampus.com\/?p=45021"},"modified":"2022-11-21T09:45:30","modified_gmt":"2022-11-21T14:45:30","slug":"how-to-get-historical-sp-500-constituents-data-for-free","status":"publish","type":"post","link":"https:\/\/www.interactivebrokers.com\/campus\/ibkr-quant-news\/how-to-get-historical-sp-500-constituents-data-for-free\/","title":{"rendered":"How To Get Historical S&P 500 Constituents Data For Free"},"content":{"rendered":"\n

Excerpt<\/em><\/strong><\/p>\n\n\n\n

Getting Current S&P 500 Constituents for Free<\/strong><\/p>\n\n\n\n

Wikipedia publishes current S&P 500 component stocks here.<\/a><\/p>\n\n\n\n

If we use the chrome inspector we can see that the S&P 500 stock constituents are in an HTML table with id #constituents<\/code><\/p>\n\n\n\n

So let\u2019s use the rvest<\/code> R package to scrape that data into a data frame.<\/p>\n\n\n\n

\n# Load dependencies
\nif (!require(“pacman”)) install.packages(“pacman”)
\npacman::p_load(tidyverse, rvest)
\nwikispx <- read_html('https:\/\/en.wikipedia.org\/wiki\/List_of_S%26P_500_companies')
\ncurrentconstituents <- wikispx %>%
\n html_node(‘#constituents’) %>%
\n html_table(header = TRUE)
\ncurrentconstituents\n<\/p>\n\n\n\n

\"How<\/figure>\n\n\n\n

Getting S&P 500 Changes for Free<\/strong><\/p>\n\n\n\n

Wikipedia also publishes \u201cSelected Changes to the list of S&P 500 components\u201d on the same page.<\/p>\n\n\n\n

This lists stocks that have been added or removed from the index as a result of acquisitions, or as the companies grow and shrink in market capitalisation.<\/p>\n\n\n\n

I\u2019ve checked this against our data set and it\u2019s relatively accurate and complete up to about the year 2000. It gets less complete and accurate before then.<\/p>\n\n\n\n

But we don\u2019t need perfection<\/em> here\u2026 so let\u2019s scrape these changes.<\/p>\n\n\n\n

The Chrome Inspector shows us they live in a table with id #changes<\/code>.<\/p>\n\n\n\n

\nspxchanges <- wikispx %>%
\n html_node(‘#changes’) %>%
\n html_table(header = FALSE, fill = TRUE) %>%
\n filter(row_number() > 2) %>% # First two rows are headers
\n `colnames<-`(c('Date','AddTicker','AddName','RemovedTicker','RemovedName','Reason')) %>%
\n mutate(Date = as.Date(Date, format = ‘%B %d, %Y’),
\n year = year(Date),
\n month = month(Date))
\nspxchanges\n<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

Create Monthly Snapshot of S&P 500 Index Constituents<\/strong><\/p>\n\n\n\n

Now we\u2019re going to use this data to create monthly snapshots of what the SPX index used to look like.<\/p>\n\n\n\n

To do this we:<\/p>\n\n\n\n