{"id":223003,"date":"2025-04-30T12:38:30","date_gmt":"2025-04-30T16:38:30","guid":{"rendered":"https:\/\/ibkrcampus.com\/campus\/?p=223003"},"modified":"2025-05-01T07:11:19","modified_gmt":"2025-05-01T11:11:19","slug":"bayesian-inference-methods-and-formula-explained","status":"publish","type":"post","link":"https:\/\/www.interactivebrokers.com\/campus\/ibkr-quant-news\/bayesian-inference-methods-and-formula-explained\/","title":{"rendered":"Bayesian Inference Methods and Formula Explained"},"content":{"rendered":"\n

The article “Bayesian Inference Methods and Formula Explained” was originally published on QuantInsti<\/a> blog.<\/em><\/p>\n\n\n\n

This post on Bayesian inference is the second of a multi-part series on Bayesian statistics and methods used in quantitative finance.<\/p>\n\n\n\n

In my previous post<\/a>, I gave a leisurely introduction to Bayesian statistics and while doing so distinguished between the frequentist and the Bayesian outlook of the world. I dwelt on how each of their underlying philosophies influenced their analysis of various probabilistic phenomena. I then discussed the Bayes’ Theorem along with some illustrations to help lay the building blocks of Bayesian statistics.<\/p>\n\n\n\n

Intent of this Post<\/strong><\/h2>\n\n\n\n

My objective here is to help develop a deeper understanding of statistical analysis by focusing on the methodologies adopted by frequentist statistics and Bayesian statistics. I consciously choose to tackle the programming and simulation aspects using Python in my next post.<\/p>\n\n\n\n

I now instantiate the previously discussed ideas with a simple coin-tossing example adapted from “Introduction to Bayesian Econometrics (2nd Edition)”.<\/p>\n\n\n\n

Example: A Repeated Coin-Tossing Experiment<\/h3>\n\n\n\n

Suppose we are interested in estimating the bias of a coin whose fairness is unknown. We define \u03b8 (the Greek letter ‘theta’) as the probability of getting a head after a coin is tossed.<\/strong> \u03b8 is the unknown parameter we want to estimate. We intend to do so by inspecting the results of tossing the coin multiple times. Let us denote y as a realization of the random variable Y (representing the outcome of a coin toss). Let Y=1<\/strong> if a coin toss results in heads and Y=0<\/strong> if a coin toss results in tails. Essentially, we are assigning 1 to heads and 0 to tails.<\/p>\n\n\n\n

\u2234 P(Y=1|\u03b8)=\u03b8 ; P(Y=0|\u03b8)=1\u2212\u03b8<\/strong><\/p>\n\n\n\n

Based on our above setup, Y can be modelled as a Bernoulli distribution<\/a> which we denote as<\/p>\n\n\n\n

Y \u223c Bernoulli (\u03b8)<\/strong><\/p>\n\n\n\n

I now briefly view our experimental setup through the lens of the frequentist and the Bayesian before proceeding with our estimation of the unknown parameter \u03b8<\/strong>.<\/p>\n\n\n\n

Two Perspectives on the Experiment Setup<\/strong><\/h2>\n\n\n\n

In classical statistics (i.e. the frequentist approach), our parameter \u03b8<\/strong> is a fixed but unknown value lying between 0<\/strong> and 1<\/strong>. The data we collect is one realization of a recurrent (i.e. repeating this n<\/strong>-toss experiment say N<\/strong> times) experiment. Classical estimation techniques like the method of maximum likelihood are used to arrive at \u03b8\u0304\u0302 (called ‘theta hat’), an estimate for the unknown parameter \u03b8<\/strong>. In statistics, we usually express an estimate by putting a hat over the name of the parameter. I dilate this idea in the next section. To restate what has been said previously, we observe that in the frequentist universe, the parameter is fixed but the data is varying.<\/p>\n\n\n\n

Bayesian statistics is fundamentally different. Here, the parameter \u03b8 is treated as a random variable since there is uncertainty about its value. It, therefore, makes sense for us to regard our parameter as a random variable which will have an associated probability distribution<\/a>. In order to apply Bayesian inference, we turn our attention to one of the fundamental laws of probability theory, Bayes’ Theorem<\/a> that we had seen previously.<\/p>\n\n\n\n

I use the mathematical form of Bayes’ Theorem as a way to establish a connection with Bayesian inference.<\/p>\n\n\n\n

\"Bayesian<\/figure>\n\n\n\n

To repeat what I said in my previous post, what makes this theorem so handy is it allows us to invert a conditional probability<\/strong>. So if we observe a phenomenon and collect data or evidence about it, the theorem helps us analytically define the conditional probability of different possible causes given the evidence.<\/em><\/p>\n\n\n\n

Let’s now apply this to our example by using the notations we had defined earlier. I label A = \u03b8<\/strong> and B = y<\/strong>. In the field of Bayesian statistics, there are special names used for each of these terms which I spell out below and use subsequently. (1)<\/a> can be rewritten as:<\/p>\n\n\n\n

\"Bayesian<\/figure>\n\n\n\n

where:<\/p>\n\n\n\n

P(\u03b8)<\/strong> is the prior probability<\/strong>. We express our belief about the cause \u03b8<\/strong> BEFORE observing the evidence Y<\/strong>. In our example, the prior would be quantifying our a priori belief on the fairness of the coin (here we can start with the assumption that it is an unbiased coin, so \u03b8 = 1\/2). P(Y|\u03b8)<\/strong> is the likelihood<\/strong>. Here is where the real action happens. This is the probability of the observed sample or evidence given the hypothesized cause. Let us, without loss of generality, assume that we obtain 5 heads in 8 coin tosses. Presuming the coin to be unbiased as specified above, the likelihood would be the probability of observing 5 heads in 8 tosses given that \u03b8 = 1\/2. P(\u03b8|Y)<\/strong> is the posterior probability<\/strong>. This is the probability of the underlying cause \u03b8 AFTER observing the evidence y. Here, we compute our updated or a posteriori belief on the bias of the coin after observing 5 heads in 8 coin tosses using Bayes’ theorem. P(Y)<\/strong> is the probability of the data or evidence<\/strong>. We sometimes also call this the marginal likelihood. This is obtained by taking the weighted sum (or integral) of the likelihood function of the evidence across all possible values of \u03b8. In our example, we would compute the probability of 5 heads in 8 coin tosses for all possible beliefs about \u03b8. This term is used to normalize the posterior probability. Since it is independent of the parameter to be estimated \u03b8, it is mathematically more tractable to express the posterior probability as:<\/p>\n\n\n\n

P(\u03b8|Y) \u221d P(Y|\u03b8) \u00d7 P(\u03b8) <\/strong>\u2026\u2026.(3)<\/p>\n\n\n\n

(3)<\/a> is the most important expression in Bayesian statistics<\/strong> and bears repeating. For clarity, I paraphrase what I said earlier. Bayesian inference allows us to turnaround conditional probabilities i.e. use the prior probabilities and the likelihood functions to provide a connecting link to the posterior probabilities i.e. P(\u03b8|Y)<\/strong> granted that we only know P(Y|\u03b8)<\/strong> and the prior, P(\u03b8<\/strong>). I find it helpful to view (3)<\/a> as:<\/p>\n\n\n\n

Posterior Probability \u221d Likelihood \u00d7 Prior Probability<\/strong> \u2026\u2026\u2026. (4)<\/p>\n\n\n\n

The experimental objective is to get an estimate of the unknown parameter \u03b8<\/strong> based on the outcome of n<\/strong> independent coin tosses. The coin tosses generate the sample or data y = (y1, y2, \u2026, yn), where yi is 1 or 0 based on the result of the ith<\/strong> coin toss.<\/p>\n\n\n\n

I now show the frequentist and Bayesian approaches to fulfilling this objective. Feel free to cursorily skim through the derivations I touch upon here if you are not interested in the mathematics behind it. You can still develop sufficient intuitions and learn to use Bayesian techniques in practice.<\/p>\n\n\n\n

Estimating <\/strong>\u03b8: The Frequentist Approach<\/strong>
We compute the joint probability function using the maximum likelihood estimation (MLE) approach. The probability of the outcome for a single coin toss can be elegantly expressed as:<\/p>\n\n\n\n

\"Bayesian<\/figure>\n\n\n\n

For a given value of \u03b8<\/em>, the joint probability of the outcome for n independent coin tosses is the product of the probability of each individual outcome:<\/p>\n\n\n\n

\"Bayesian<\/figure>\n\n\n\n

As we can see in (4)<\/a>, the expression worked out is a function of the unknown parameter \u03b8 given the observations from our experiment. This function of \u03b8 is called the likelihood function and is usually referred to in the literature as:<\/p>\n\n\n\n

\"Bayesian<\/figure>\n\n\n\n

OR<\/p>\n\n\n\n

\"Bayesian<\/figure>\n\n\n\n

We would like to compute the value of \u03b8<\/em> which is most likely to have yielded the observed set of outcomes. This is called the maximum likelihood estimate<\/em>, \u03b8\u0304\u0302 (‘theta hat’). For analytically computing it, we trivially take the first order derivative of (6)<\/a> with respect to the parameter and set it equal to zero. It is prudent to also take the second derivative and check the sign of its value at \u03b8 = \u03b8\u0304\u0302  to ensure that the estimate is indeed the maxima. We often customarily take the log of the likelihood function since it greatly simplifies the determination of the maximum likelihood estimator \u03b8\u0304\u0302 . It should therefore not surprise you that the literature is replete with log likelihood functions and their solutions.<\/p>\n\n\n\n

Estimating \u03b8: The Bayesian Approach<\/strong><\/p>\n\n\n\n

I now change the notations we have used so far to make them a little more precise mathematically. I will use these notations throughout this series now. The reason for this alteration is so that we can suitably ascribe each term with symbols that remind us of their random nature. There is uncertainty over the values of \u03b8, Y, etc., we, therefore, regard them as random variables and assign them corresponding probability distributions which I do below.<\/p>\n\n\n\n

Notations for the Density and Distribution Functions<\/strong><\/h2>\n\n\n\n