Asset Classes

Free investment financial education

Language

Multilingual content from IBKR

Close Navigation
Learn more about IBKR accounts
Multiple Linear Regression using Tensorflow Optimization

Multiple Linear Regression using Tensorflow Optimization

Posted February 21, 2023 at 1:00 pm

Sang-Heon Lee
SHLee AI Financial Model

This post implements the optimization based estimation of multiple linear regression model using Tensorflow. With this example, we can learn basic implementations of functions in Python and a numerical optimization in Tensorflow.

Optimization based Linear Regression using Tensorflow

This post deals with an optimization based linear regression model using Python and Tensorflow to learn some basic functionalities in them. This regression also can be easily estimated by using the standard matrix formula as in the previous post.

Minimization of Residual Sum of Squares

Multiple linear regression model has the following expression. (t=1,2,…,n)

To estimate the regression coefficients β, we minimize the sum of squared residuals directly by using numerical optimization in Tensorflow.

Python code using Tensorflow

To use a numerical optimization such as Nelder-Mead algorithm, we need to import tensorflow_probability package.

import tensorflow_probability as tfp


To use a numerical optimization, we also define an objective function using def command in Python, which returns a residual sum of squares. In fact we define two objective functions : objfunc_w_args(beta, y, X) and objfunc_wo_args(beta). The former is with another arguments except a vector of parameters (beta) but the latter is without another arguments. Therefore from these two types of objective functions, there are some modifications when we call a numerical optimization function in Tensorflow (tfp.optimizer.nelder_mead_minimize).

# -*- coding: utf-8 -*-
"""
#========================================================#
# Quantitative ALM, Financial Econometrics & Derivatives 
# ML/DL using R, Python, Tensorflow by Sang-Heon Lee 
#
# https://kiandlee.blogspot.com
#--------------------------------------------------------#
# Linear Regression model using Tensorflow Optimization
#========================================================#
"""
 
import numpy as np
from sklearn import datasets, linear_model
import tensorflow as tf
import tensorflow_probability as tfp # optimizer
 
 
"""""""""""""""""""""""""""""""""""""""""
        Load the diabetes dataset
"""""""""""""""""""""""""""""""""""""""""
X, y = datasets.load_diabetes(return_X_y=True)
nr, nc = X.shape; print (nr, nc) 
nparam = nc+1 # number of parameters
 
v_row_name = np.hstack(
    [["const"], ["X"+str(i) for i in range(1,nc+1)]])
 
# regression estimation using sklearn
mod = linear_model.LinearRegression() # object
mod.fit(X, y) # estimation or tradining
param = np.hstack([mod.intercept_, mod.coef_])
 
 
"""""""""""""""""""""""""""""""""""""""""
        Objective Function : RSS
"""""""""""""""""""""""""""""""""""""""""
# with another arguments such as data
def objfunc_w_args(beta, y, X):
    n = len(beta)
    mbeta = tf.reshape(tf.cast(beta,tf.float64),[n,1])
    err = y - tf.matmul(X, mbeta)
    # element-wise product
    RSS = tf.reduce_sum(tf.math.multiply(err, err))
    return RSS
 
# without any arguments except a vector of parameters
def objfunc_wo_args(beta):
    n = len(beta)
    mbeta = tf.reshape(tf.cast(beta,tf.float64),[n,1])
    err = my - tf.matmul(m1X, mbeta)
    return tf.reduce_sum(err ** 2)
 
 
"""""""""""""""""""""""""""""""""""""""""
     Data Preparation for Tensorflow
"""""""""""""""""""""""""""""""""""""""""
my  = tf.constant(y, shape=[nr, 1])
mX  = tf.constant(X, shape=[nr, nc])
m1  = tf.cast(tf.ones([nr, 1]), tf.float64)
m1X = tf.concat([m1, mX], 1); # 1, X
 
 
"""""""""""""""""""""""""""""""""""""""""
             Optimization
"""""""""""""""""""""""""""""""""""""""""
# initial guess for parameters
start = tf.constant([1.,1.,1.,1.,1.,1.,1.,1.,1.,1.,1.], 
                    dtype=tf.float64)  
 
print("\n========== w/o arguments ==========")
optim_results_wo = tfp.optimizer.nelder_mead_minimize(
    objfunc_wo_args, initial_vertex=start,
    func_tolerance=1e-15, position_tolerance=1e-15)
print(optim_results_wo.position)
print(optim_results_wo.converged)
 
print("\n========== with arguments ==========")
optim_results_w = tfp.optimizer.nelder_mead_minimize(
    lambda beta: objfunc_w_args(beta, my, m1X), 
    initial_vertex=start,
    func_tolerance=1e-15, position_tolerance=1e-15)
print(optim_results_w.position)  
print(optim_results_w.converged)
 
print("\n========== sklearn ==========")
print(param)

As can be seen in the output below, results from the numerical optimization is nearly the same as the result from sklearn package which is based on matrix formula since there are negligible numerical errors.

Concluding Remarks

This post dealt with how to define functions and run optimizations in Tensorflow. Now that we have learn some basic how-to regarding Tensorflow, a state space model with Kalman filtering and estimation of parameter with numerical optimization will be discussed in the next post.

Originally posted on SH Fintech Modeling Blog.

Disclosure: Interactive Brokers

Information posted on IBKR Campus that is provided by third-parties does NOT constitute a recommendation that you should contract for the services of that third party. Third-party participants who contribute to IBKR Campus are independent of Interactive Brokers and Interactive Brokers does not make any representations or warranties concerning the services offered, their past or future performance, or the accuracy of the information provided by the third party. Past performance is no guarantee of future results.

This material is from SHLee AI Financial Model and is being posted with its permission. The views expressed in this material are solely those of the author and/or SHLee AI Financial Model and Interactive Brokers is not endorsing or recommending any investment or trading discussed in the material. This material is not and should not be construed as an offer to buy or sell any security. It should not be construed as research or investment advice or a recommendation to buy, sell or hold any security or commodity. This material does not and is not intended to take into account the particular financial conditions, investment objectives or requirements of individual customers. Before acting on this material, you should consider whether it is suitable for your particular circumstances and, as necessary, seek professional advice.

IBKR Campus Newsletters

This website uses cookies to collect usage information in order to offer a better browsing experience. By browsing this site or by clicking on the "ACCEPT COOKIES" button you accept our Cookie Policy.