PMF for the binomial distribution<\/h3>\n\n\n\n
Given success probability p<\/em> and the number of trials n<\/em>, binomial distribution produces the probability of occurring x(\u2264n)<\/em> success as follows.<\/p>\n\n\n\n From the above equation for P(X=x;n,p), It is worth noticing that since P(X=x;n,p)<\/em> have one p<\/em>, the symbol of combination is used. This logic is similar to that of calculating one weighted average.<\/p>\n\n\n\n Standard linear regression is used for the calculation of the conditional mean, which is varying with the realization of covariates (X<\/em>). This means that in addition to the unconditional mean, additional information from covariates are useful for the improvement of forecast performance. This story is also applied to logistic regression. Assuming p<\/em> is neither a fixed nor an unconditional probability, we can estimate the conditional probability using explanatory variables (X<\/em>).<\/p>\n\n\n\n Since explanatory variables X<\/em> have useful information for explaining variation of a binary variable Y<\/em>, we can set up a linear regression model for this binary response variable. But output from a standard linear regression can be of less than zero or more than one. This is, therefore, not appropriate because Y<\/em> is the binary variable which have a range from 0 and 1. From this reason, we need the following logit transformation which converts 0~1 binary variable to \u2212\u221e~\u221e real number.<\/p>\n\n\n\n Using this transformation, we can obtain the logistic regression model in the following way.<\/p>\n\n\n\n Since each We can find that there is no term like It is typical to estimate parameters using MLE by calling numerical optimization. But for ready-made models like linear regression, logistic regression, etc, it is convenient to use glm() R package which performs optimization automatically.<\/p>\n\n\n\n The following R code shows how to implement logistic regression using R. We provide two-method: the one uses glm() conveniently and the other maximize log likelihood function by numerical MLE directly. To check for the accuracy of estimation, we assume population parameters as ![\"binomial](\"\/campus\/wp-content\/uploads\/sites\/2\/2022\/09\/logistic-regression-1.png\")
<\/figure>\n\n\n\n![](\"\/campus\/wp-content\/uploads\/sites\/2\/2022\/09\/logistic-regression-3.png\")
denotes the case when the number of success is x<\/em> from n<\/em> Bernoulli trials. Let’s take an example: suppose n = 3, x = 1.<\/p>\n\n\n\n![\"Let's](\"\/campus\/wp-content\/uploads\/sites\/2\/2022\/09\/logistic-regression-2.png\")
<\/figure>\n\n\n\n![](\"\/campus\/wp-content\/uploads\/sites\/2\/2022\/09\/logistic-regression-11.png\")
denotes the probability of . This probability of the above example (n = 3, x = 1) is as follows.<\/p>\n\n\n\n![\"this](\"\/campus\/wp-content\/uploads\/sites\/2\/2022\/09\/logistic-regression-5.png\")
<\/figure>\n\n\n\nLogit transformation<\/h3>\n\n\n\n
![\"we](\"\/campus\/wp-content\/uploads\/sites\/2\/2022\/09\/logistic-regression-12.png\")
<\/figure>\n\n\n\n![\"we](\"\/campus\/wp-content\/uploads\/sites\/2\/2022\/09\/logistic-regression-6.png\")
<\/figure>\n\n\n\nLog-likelihood function<\/h3>\n\n\n\n
![](\"\/campus\/wp-content\/uploads\/sites\/2\/2022\/09\/logistic-regression-7.png\")
is different with covariates, likelihood function of logistic regression as successive Bernoulli trials, has the following form.<\/p>\n\n\n\n![\"likelihood](\"\/campus\/wp-content\/uploads\/sites\/2\/2022\/09\/logistic-regression-8.png\")
<\/figure>\n\n\n\n in the likelihood function. The reason is that is different with covariates as mentioned previously. Combination is used only when occurrence probability are fixed and all the same. In particular, to facilitate a numerical optimization, we use the following log likelihood function.<\/p>\n\n\n\n![\"facilitate](\"\/campus\/wp-content\/uploads\/sites\/2\/2022\/09\/logistic-regression-9.png\")
<\/figure>\n\n\n\nR code for Logistic Regression<\/h3>\n\n\n\n
![](\"\/campus\/wp-content\/uploads\/sites\/2\/2022\/09\/logistic-regression-10.png\")
. It is, therefore, expected that two methods produce the same results.<\/p>\n\n\n\n#=========================================================================#\n# Financial Econometrics & Derivatives, ML\/DL using R, Python, Tensorflow \n# by Sang-Heon Lee \n#-------------------------------------------------------------------------#\n# Logistic Regression\n#=========================================================================#\n \ngraphics.off() # clear all graphs\nrm(list = ls()) # remove all files from your workspace\n \n#-----------------------------------------------------\n# Simulated data\n#-----------------------------------------------------\n \n # number of observation\n nobs <- 10000\n \n # parameters assumed\n beta <- c(0.3, 0.5, -0.2)\n \n # simulated x\n x1 <- rnorm(n = nobs)\n x2 <- rnorm(n = nobs)\n \n bx = beta[1] + beta[2]*x1 + beta[3]*x2\n p = 1 \/ (1 + exp(-bx))\n \n # simulated y\n y <- rbinom(n = nobs, size = 1, prob = p)\n \n # simulated y and x\n df.yx <- data.frame(y, x1, x2) \n \n#-----------------------------------------------------\n# Estimation by glm\n#-----------------------------------------------------\n \n fit_glm = glm(y ~ ., data = df.yx, family = binomial)\n summary(fit_glm)\n \n#-----------------------------------------------------\n# Estimation by optimization\n#-----------------------------------------------------\n \n # objective function = negative log-likelihood\n obj.func <- function(b0, df.yx) {\n \n beta <- b0\n y <- df.yx[,1]\n x <- as.matrix(cbind(1,df.yx[,-1]))\n \n bx = x%*%beta\n p = 1 \/ (1 + exp(-bx)) \n \n # we can calculate log-likelihood\n # using one of the two below\n \n # 1. vector operation\n #\n loglike <- sum(y*log(p) + (1-y)*log(1-p))\n \n # 2. loop operation\n #\n # loglike <- 0\n # for(i in 1:nrow(df.yx)) {\n # loglike <- loglike + y[i]*log(p[i]) + \n # (1-y[i])*log(1-p[i])\n # }\n \n return(-1*loglike)\n }\n \n # run optimization\n m<-optim(c(0.1,0.1,0.1), obj.func,\n control = list(maxit=5000, trace=2),\n method=c(\"Nelder-Mead\"),df.yx=df.yx)\n \n#-----------------------------------------------------\n# Comparison of results\n#-----------------------------------------------------\n beta\n m$par\n coef(fit_glm)<\/code><\/pre>\n\n\n\n
![\"R](\"\/campus\/wp-content\/uploads\/sites\/2\/2022\/09\/logistic_results-sh-fintech.png\")
<\/figure>\n\n\n\n