{"id":96139,"date":"2021-08-27T11:30:00","date_gmt":"2021-08-27T15:30:00","guid":{"rendered":"https:\/\/ibkrcampus.com\/?p=96139"},"modified":"2022-11-21T09:47:57","modified_gmt":"2022-11-21T14:47:57","slug":"automated-eda-with-python","status":"publish","type":"post","link":"https:\/\/www.interactivebrokers.com\/campus\/ibkr-quant-news\/automated-eda-with-python\/","title":{"rendered":"Automated EDA with Python"},"content":{"rendered":"\n

In this post, we will investigate the\u00a0pandas_profiling<\/strong>\u00a0and\u00a0sweetviz<\/strong>\u00a0packages, which can be used to speed up EDA (exploratory data analysis) with Python. In a previous article, we talked about an analogous package in R (see this link<\/a>).<\/p>\n\n\n\n

Getting started with pandas_profiling<\/strong><\/h2>\n\n\n\n

pandas_profiling<\/strong> can be installed using pip, like this:<\/p>\n\n\n\n

pip install pandas-profiling[notebook]<\/code><\/pre>\n\n\n\n
Next, let\u2019s read in our dataset. The data we\u2019ll be using is a heart attack-related dataset, which can be found here<\/a>.<\/p>\n\n\n\n
import pandas as pd\n \nheart_data = pd.read_csv(\"heart.csv\")\n \nheart_data.head()\n<\/code><\/pre>\n\n\n\n
\"\"<\/figure>\n\n\n\n
Now, let\u2019s import ProfileReport<\/em> from pandas_profiling<\/strong>.<\/p>\n\n\n\n
from pandas_profiling import ProfileReport\n \nreport = ProfileReport(heart_data, title = \"Sample Report\")\n \nreport<\/code><\/pre>\n\n\n\n
\"\"<\/figure>\n\n\n\n
If you\u2019re running this code in Jupyter Notebook, you should see the report generated within your notebook file.<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
The report shows several pieces of analysis. First, it gives a summary glimpse of the data, giving the number of variables, observations, missing values and percentages, data type information, and number of duplicate rows (if any).<\/p>\n\n\n\n
Additionally, there are several other tabs available in the report. For example, the package automatically generates correlation heatmaps, like below. Also, it\u2019s possible to see how many missing values the dataset has by column.<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
The variables tab shows the number of distinct values, missing values, and plots a histogram or barplot for the variable (for numeric or categorical variables, respectively).<\/p>\n\n\n\n
If you want to save the report to an external HTML file, you can do that by using the to_file<\/em> method.<\/p>\n\n\n\n
# save report to file\nreport.to_html(\"report_file.html\")<\/code><\/pre>\n\n\n\n
Handling larger datasets<\/strong><\/h2>\n\n\n\n
Handling larger datasets is more of a challenge using the default settings, but can be done by making some adjustments. The ProfileReport<\/em> class comes with a parameter called minimal<\/em>, which we can set to True to reduce the number of computations that will be made.<\/p>\n\n\n\n
This setting will generate a report showing the Overview and Variables tabs, so you can still see a visualization of each variable, missing value analysis, and a few summary stats.<\/p>\n\n\n\n
min_report = ProfileReport(heart_data, title = \"Minimal Report\", minimal = True)\n \nmin_report<\/code><\/pre>\n\n\n\n
If you want to run the full profiling on a large dataset, it may be better to start with a sample to avoid computational issues.<\/p>\n\n\n\n
The sweetviz library<\/strong><\/h2>\n\n\n\n
An alternative to pandas_profiling<\/strong> is the sweetviz<\/strong>, which can also generate an automated EDA report. We can install sweetviz<\/strong> using pip:<\/p>\n\n\n\n
pip install sweetviz<\/code><\/pre>\n\n\n\n
Similar to pandas_profiling<\/strong>, you can generate an EDA report using a short code snippet:<\/p>\n\n\n\n
import sweetviz as sv\n \n# generate report\nsweet_report = sv.analyze(heart_data)\n \n# show the output\nsv.show_notebook()<\/code><\/pre>\n\n\n\n
\"\"<\/figure>\n\n\n\n
You can click on a tab for any of the variables to expand the analysis done on any variable. This analysis shows descriptive statistics, a histogram (or barplot) of the variable\u2019s distribution, and information around the most frequently occurring values. Additionally, sweetviz<\/strong> generates a heatmap showing the associations between the variables in the dataset. Between continuous variables, this heatmap shows the Pearson correlation coefficient. For continuous and categorical features, the correlation ratio is shown. The uncertainty coefficient is shown for categorical-to-categorical associations.<\/p>\n\n\n\n
The show_notebook<\/em> method above generates the report within Jupyter Notebook. If you want to create an external HTML file, you can use show_html<\/em>, like this:<\/p>\n\n\n\n
sv.show_html()<\/code><\/pre>\n\n\n\n
Comparing datasets with sweetviz<\/strong><\/h2>\n\n\n\n
Sweetviz<\/strong> can also be used to compare two datasets. For example, if you want to compare training and validation datasets, you could do that with sweetviz<\/strong>.<\/p>\n\n\n\n
compare_report = sv.compare([train_data, \"Train\"], [val_data, \"Test\"], \"output\")\n \n# show output within notebook file\ncompare_report.show_notebook()<\/code><\/pre>\n\n\n\n
This report will be similar to the one above, except it will break out the analysis by each dataset.<\/p>\n\n\n\n
Conclusion<\/strong><\/h2>\n\n\n\n
That\u2019s all for now! If you enjoyed this post, please share it with your friends. To learn more about the packages discussed in this post, check out these links:<\/p>\n\n\n\n