I recently needed market capitalisation data for a large sample of stocks and had no access to paid databases. I therefore resorted to scrape the data from https://finance.yahoo.com using a small webscraping function:
library(tidyverse)
library(rvest)
library(crayon)
scrape_MC <- function(ticker){
message(blue("Scraping "),
white(ticker))
ret <- try(paste0("https://finance.yahoo.com/quote/",
ticker) %>%
read_html()%>%
html_table() %>%
reduce(.f=bind_rows) %>%
as_tibble() %>%
filter(X1 == "Market Cap") %>%
pull(X2),
silent = T)
ifelse(class(ret)=="try-error",
ticker,
ret)
}
You can use this function then to scrape individual tickers:
scrape_MC("AAPL")
which returns:
Scraping AAPL
[1] "1.969T"
The data is quoted depending on the unit and sometimes errors are reported, I fixed this using this code snippet:
MC <- MC_data %>%
filter(str_detect(MC, "^\\d")) %>%
mutate(MC = str_replace(MC, ",", ""),
Unit = str_extract(MC, "[:alpha:]$"),
MC = as.numeric(str_remove(MC, "[:alpha:]$")),
MC = case_when(Unit == "B" ~ MC * 1000,
Unit == "M" ~ MC,
Unit == "T" ~ MC * 1000*1000,
is.na(Unit) ~ MC / (1000*1000)
)
) %>%
select(-Unit)
You could adapt this to scrape the industry or sector as well.