{"id":194391,"date":"2023-08-10T14:34:00","date_gmt":"2023-08-10T18:34:00","guid":{"rendered":"https:\/\/ibkrcampus.com\/?p=194391"},"modified":"2023-08-10T14:34:36","modified_gmt":"2023-08-10T18:34:36","slug":"how-to-build-a-logistic-regression-model-from-scratch-in-r","status":"publish","type":"post","link":"https:\/\/www.interactivebrokers.com\/campus\/ibkr-quant-news\/how-to-build-a-logistic-regression-model-from-scratch-in-r\/","title":{"rendered":"How to Build a Logistic Regression Model from Scratch in R"},"content":{"rendered":"\n

Originally posted on TheAutomatic.net<\/a>.<\/em><\/p>\n\n\n\n

Excerpt<\/em><\/p>\n\n\n\n

Background<\/strong><\/h2>\n\n\n\n

In a previous post<\/a>, we showed how using vectorization in R can vastly speed up fuzzy matching. Here, we will show you how to use vectorization to efficiently build a logistic regression<\/a> model from scratch in R. Now we could just use the caret<\/a> or stats<\/a> packages to create a model, but building algorithms from scratch is a great way to develop a better understanding of how they work under the hood.<\/p>\n\n\n\n

Definitions & Assumptions<\/strong><\/h2>\n\n\n\n

In developing our code for the logistic regression algorithm, we will consider the following definitions and assumptions:<\/p>\n\n\n\n

x<\/strong> = A dxn matrix of d<\/em> predictor variables, where each column xi<\/sub> represents the vector of predictors corresponding to one data point (with n<\/em> such columns i.e. n<\/em> data points)<\/p>\n\n\n\n

d = The number of predictor variables (i.e. the number of dimensions)<\/p>\n\n\n\n

n = The number of data points<\/p>\n\n\n\n

y<\/strong> = A vector of labels i.e. yi<\/sub> equals the label associated with xi<\/sub>; in our case we\u2019ll assume y = 1 or -1<\/p>\n\n\n\n

\u0398<\/strong> = The vector of coefficients, \u03981<\/sub>, \u03982<\/sub>\u2026\u0398d<\/sub> trained via gradient descent. These correspond to x1<\/sub>, x2<\/sub>\u2026xd<\/sub><\/p>\n\n\n\n

\u03b1 = Step size, controls the rate of gradient descent<\/p>\n\n\n\n

Logistic regression, being a binary classification algorithm, outputs a probability between 0 and 1 of a given data point being associated with a positive label. This probability is given by the equation below:<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

Recall that <\u0398<\/strong>, x> refers to the dot product<\/a> of \u0398<\/strong> and x<\/strong>.<\/p>\n\n\n\n

In order to calculate the above formula, we need to know the value of \u0398<\/strong>. Logistic regression uses a method called gradient descent<\/a> to learn the value of \u0398<\/strong>. There are a few variations of gradient descent, but the variation we will use here is based upon the following update formula for \u0398j<\/sub>:<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

This formula updates the jth<\/sup> element of the \u0398<\/strong> vector. Logistic regression models run this gradient descent update of \u0398<\/strong> until either 1) a maximum number of iterations has been reached or 2) the difference between the current update of \u0398<\/strong> and the previous value is below a set threshold. To run this update of theta, we\u2019re going to write the following function, which we\u2019ll break down further along in the post. This function will update the entire \u0398<\/strong> vector in one function call i.e. all j elements<\/em> of \u0398<\/strong>.<\/p>\n\n\n\n

# function to update theta via gradient descent\nupdate_theta <- function(theta_arg,n,x,y,d,alpha)\n{\n   \n  # calculate numerator of the derivative\n  numerator <- t(replicate(d , y)) * x\n   \n  # perform element-wise multiplication between theta and x,\n  # prior to getting their dot product\n  theta_x_prod <- t(replicate(n,theta_arg)) * t(x)\n  dotprod <- rowSums(theta_x_prod)\n   \n  denominator <- 1 + exp(-y * dotprod)\n   \n  # cast the denominator as a matrix\n  denominator <- t(replicate(d,denominator)) \n   \n  # final step, get new theta result based off update formula\n  theta_arg <- theta_arg - alpha * rowSums(numerator \/ denominator) \n   \n   \n  return(theta_arg)\n   \n}<\/pre>\n\n\n\n
Simplifying the update formula<\/strong><\/h2>\n\n\n\n
To simplify the update formula for \u0398j<\/sub>, we need to calculate the derivative<\/a> in the formula above.<\/p>\n\n\n\n
Let\u2019s suppose z = (yi<\/sub>)(<\u0398<\/strong>, xi<\/sub>>). We\u2019ll abbreviate the summation of 1 to n by simply using \u03a3.<\/p>\n\n\n\n
Then, calculating the derivative gives us the following:<\/p>\n\n\n\n
\u03a3 (1 \/ exp(1 + z)) * exp(z) * xi<\/sub>yi<\/sub><\/p>\n\n\n\n
= \u03a3 (exp(z) \/ exp(1 + z)) * xi<\/sub>yi<\/sub><\/p>\n\n\n\n
Since exp(z) \/ (1 + exp(z)) is a known identity for 1 \/ (1 + exp(-z)), we can substitute above to get:<\/p>\n\n\n\n
\u03a3 1 \/ (1 + exp(-z)) * xi<\/sub>yi<\/sub><\/p>\n\n\n\n
= \u03a3 xi<\/sub>yi<\/sub> \/ (1 + exp(-z))<\/p>\n\n\n\n
Now, substituting (yi<\/sub>)(<\u0398<\/strong>, xi<\/sub>>) back for z:<\/p>\n\n\n\n
= \u03a3 xi<\/sub>yi<\/sub> \/ (1 + exp(-(yi<\/sub>)(<\u0398<\/strong>, xi<\/sub>>)))<\/p>\n\n\n\n
Plugging this derivative result into the rest of the update formula, the below expression tells us how to update \u0398j<\/sub>:<\/p>\n\n\n\n
\u0398j<\/sub> \u2190 \u0398j<\/sub> \u2013 \u03b1\u03a3 xi<\/sub>yi<\/sub> \/ (1 + exp(-(yi<\/sub>)(<\u0398<\/strong>, xi<\/sub>>)))<\/p>\n\n\n\n
To convert this math into R code, we\u2019ll split up the formula above into three main steps:<\/p>\n\n\n\n