{"id":50787,"date":"2020-07-01T10:21:00","date_gmt":"2020-07-01T14:21:00","guid":{"rendered":"https:\/\/ibkrcampus.com\/?p=50787"},"modified":"2023-02-28T09:43:05","modified_gmt":"2023-02-28T14:43:05","slug":"evaluate-your-model-with-mlmetrics","status":"publish","type":"post","link":"https:\/\/www.interactivebrokers.com\/campus\/ibkr-quant-news\/evaluate-your-model-with-mlmetrics\/","title":{"rendered":"Evaluate Your Model with MLmetrics"},"content":{"rendered":"\n

This post will explore using R\u2019s MLmetrics<\/strong> to evaluate machine learning models. MLmetrics<\/strong> provides several functions to calculate common metrics for ML models, including AUC, precision, recall, accuracy, etc.<\/p>\n\n\n\n

Building an example model<\/strong><\/p>\n\n\n\n

Firstly, we need to build a model to use as an example. For this post, we\u2019ll be using a dataset on\u00a0pulsar stars from Kaggle<\/a>. Let\u2019s save the file as \u201cpulsar_stars.csv\u201d. Each record in the file represents a pulsar star candidate. The goal will be to predict if a record is a pulsar star based upon the attributes available.<\/p>\n\n\n\n

To get started, let\u2019s load the packages we\u2019ll need and read in our dataset.<\/p>\n\n\n\n

\nlibrary(MLmetrics)
\nlibrary(dplyr)

\n \nstars = read.csv(“pulsar_stars.csv”)\n<\/p>\n\n\n\n

Next, let\u2019s split our data into train vs. test. We\u2019ll do a standard 70\/30 split here.<\/p>\n\n\n\n

\nset.seed(0)
\ntrain_indexes = sample(1:nrow(stars), .7 * nrow(stars))

\n \ntrain_set <- stars[train_indexes,]
\ntest_set <- stars[-train_indexes,]
\n<\/p>\n\n\n\n

Now, let\u2019s build a simple logistic regression model<\/a>.<\/p>\n\n\n\n

\ntrain_set <- data.frame(train_set %>% select(target_class), train_set %>% select(-target_class))

\n \n# build model
\nmodel <- glm(formula(train_set), train_set, family = \"binomial\")

\n \n# get predictions on train and test datasets
\ntrain_pred <- predict(model, train_set, type = \"response\")
\ntest_pred <- predict(model, test_set, type = \"response\")
\n<\/p>\n\n\n\n

AUC \/ precision \/ recall \/ accuracy<\/strong><\/p>\n\n\n\n

Let\u2019s calculate a few metrics. One of the most common metrics for classification is calculating AUC, which can be done using MLMetrics\u2019<\/strong> AUC function. Intuitively, AUC is a score between 0 and 1 that measures how well a model rank-orders predictions. See here<\/a> for a more detailed explanation.<\/p>\n\n\n\n

\n# get AUC on test and train set
\nAUC(test_pred, test_set$target_class) # 0.974172
\nAUC(train_pred, train_set$target_class) # 0.9773794\n<\/p>\n\n\n\n

As a refresher, here\u2019s a quick overview of precision, recall, and accuracy:<\/p>\n\n\n\n