{"id":204013,"date":"2024-04-09T12:20:16","date_gmt":"2024-04-09T16:20:16","guid":{"rendered":"https:\/\/ibkrcampus.com\/?p=204013"},"modified":"2024-04-09T12:20:17","modified_gmt":"2024-04-09T16:20:17","slug":"r-statix-the-power-of-correlations-in-r","status":"publish","type":"post","link":"https:\/\/www.interactivebrokers.com\/campus\/ibkr-quant-news\/r-statix-the-power-of-correlations-in-r\/","title":{"rendered":"R-Statix: The Power of Correlations in R"},"content":{"rendered":"\n

First edition.
Roberto Delgado Castro
Data Scientistl and Compliance Officer
robdelcas @ gmail.com<\/p>\n\n\n\n

Introduction<\/h2>\n\n\n\n

One of the most known and used statistical tools is correlations.<\/p>\n\n\n\n

Correlations are performed to find out levels of relations between two or more objects, variables or situations, in order to extract insights that bring explanations about the reasons why such objects or variables are, or not, related to each other.<\/p>\n\n\n\n

The usage of correlations, in general terms, brings along the development of quantitative coefficients. The most famous are the Pearson and Spearman coefficients. These metrics are used to quantify the correlation analysis.<\/p>\n\n\n\n

The correlation analysis could be applied to find relevant insights in daily business tasks or projects. Furthermore, it could be applied to find explanations for why determined business phenomena occur. Beyond, correlations could be used to solve problems in a wide variety of issues that appear daily in organizations, no matter their size, nature, type of operations, location or origin.<\/p>\n\n\n\n

This article is based on the contents of my most recently published book: Correlations in R<\/strong>. (Free download from https:\/\/roberto-delgado.com\/<\/a>)<\/em><\/p>\n\n\n\n

Correlation<\/h2>\n\n\n\n

Shao, Shirvan and Alamer (2022) express that \u201cunderstanding the association between theoretical constructs is at the heart of quantitative research. Researchers use correlation to understand how two or more variables are associated. The relationship between variables is usually obtained by assessing how measures\/scales that represent the variables are correlated. Analysts rarely use single items to represent a complex phenomenon because single items cannot appropriately capture the complexity inherent in theoretical concepts\u201d .<\/p>\n\n\n\n

In addition, Prasad (2023) indicates that \u201ccorrelation and regression are the techniques which are used to investigate if there is a relationship between two quantitative variables. Correlation answers three questions\u2014is there a relationship, what is the strength of relationship and direction of the relationship? Regression expresses this relationship in a mathematical form so that the equation can be used for predicting other values. However, correlation does not deal with causation, that is even a high degree of correlation cannot be used to confirm which is the cause and which is the effect variable. However, this condition is clearly defined in \u201cregression or connection between two or more things. A more formal definition says, correlation is a statistical method used to assess a possible association between two or more variables\u201d.<\/p>\n\n\n\n

In these same ideas, Makowski, Ben-Shachar, Patil and L\u00fcdecke (2020), point out that \u201ccorrelations tests are arguably one of the most commonly used statistical procedures, and are used as a basis in many applications such as exploratory data analysis, structural modelling, data engineering etc.\u201d<\/p>\n\n\n\n

Therefore, according to the mentioned authors, the heart of correlation is finding some levels of association between two or more variables. At this point, it is possible to break down its definition.<\/p>\n\n\n\n

First, a statistical-quantitative method or procedure must be performed. Second, researchers must find two or more variables, that could be objects, data sets or any other phenomenon. Third, such an association could be explained by the calculation of some coefficients. Fourth, the variables under analysis, in the majority of cases, are data sets or time-series-data made of information that have been collected within time.<\/p>\n\n\n\n

Normality tests<\/h2>\n\n\n\n

Khatun (2021) expresses that \u201cnormality tests are used in different sectors\u201d. Also, Das and Imon (2016), expresses that \u201cin statistics it is conventional to assume that the observations are normal. The entire statistical framework is grounded on this assumption and if this assumption is violated the inference breaks down. For this reason it is essential to check or test this assumption before any statistical analysis of data\u201d.<\/p>\n\n\n\n

Also, Ahmad and Khan (2015), points that \u201cstandard statistical procedures often require data to be normally distributed and the results of these methods will be inappropriate when the assumption of normality is not satisfied. Therefore, the postulation of normality is strictly required before proceeding statistical analysis. Many parametric methods (like correlation, regression, t \u2013test, analysis of variance etc) require normality assumption. The assumption of normality is one of the most important assumptions of parametric procedures because of its extensive range of practical applications\u201d. <\/p>\n\n\n\n

In addition, Ahmand and Khan also pointed that \u201cresearchers developed many tests for the comparison of normality assumption in different years; some tests were modified for attaining better performance. Firstly, Pearson (1900) developed chi-square test for detection of non-normality. Kolmogrov and Smirnov (1933) suggested formal test for normality. The chi square test, based on acumulative distribution function, and can be used for any univariant distribution. After two decades, Anderson and Darling proposed their test for normality. Kuiper (1960) brought out the test of normality. Afterwards, Shapiro and Wilk (1965) suggested test of normality. In 1968, Ajne normality test was developed. After two years later, a modification of Kuiper and Ajne tests was proposed by Stephens (1970). D\u2019Agostino (1972) introduced another test of normality. In the same year, modification of Kolmogrov Smirnov was proposed by Stephens. Four years later, Vasicek\u2019s test of normality proposed by Vasicek (1976). Jarque and Bera designed a test of normality in 1987\u201d.<\/p>\n\n\n\n

Therefore, Hernandez (2021) expresses that \u201c determining whether or not a data sample has been obtained from a normally-distributed population is a common practice in statistics and data analysis. Up to this date, several dozens of methods have been proposed in the scientific literature for testing normality\u201d.<\/p>\n\n\n\n

According to the bibliography reviewed so far related with normality tests, from a pure statistical point of view, before applying any quantitative processing applied to a determined data set or variable, it is recommended to perform any kind of normality test, in order to determine whether or not such data set (variable) is normal or not.<\/p>\n\n\n\n

Why?<\/p>\n\n\n\n

Dagnino (2014) states that “many times, when two variables that change together are measured, it is possible to determine which is independent and which is dependent. In these circumstances, it is only possible to describe the strength of association between them, since not causal predictions or estimates can be made\u201d.<\/p>\n\n\n\n

From this point on, the work of comparing and relating two variables constitutes one of the most used and common scientific works in the world of research and statistics. The collection of data for the construction of time series on particular variables, and the development of coefficients that quantify the association between them, are one of the cornerstones in what is known as Linear Regression.<\/p>\n\n\n\n

Fiallos (2021) states that “the most common numerical index used to measure a correlation is the “Pearson Coefficient”. Pearson’s Coefficient (also called the product-momentum correlation coefficient), is represented by the symbol “r” and provides a numerical measure of the correlation between two quantitative variables. This coefficient has the following characteristics:<\/p>\n\n\n\n

    \n
  1. It tells us whether two variables are correlated or not.<\/li>\n\n\n\n
  2. It tells us whether the apparent relationship is positive or negative.<\/li>\n\n\n\n
  3. The coefficient sign denotes the strength or intensity of the correlation between the variables”.<\/li>\n<\/ol>\n\n\n\n

    It is particularly important to note that, if the data collected is parametric and numerical and follow a normal distribution trend, Pearson’s Coefficient should be used. On the other hand, if you are working with data that is not parametric or follows a normal distribution trend, it is best to use, among others, the so-called Spearman’s Coefficient. As Pinilla and Rico (2021) put it, “analyses based on Pearson’s Coefficient are known as parametric, and those based on Spearman’s Coefficient are known as non-parametric”.<\/p>\n\n\n\n

    Correlation coefficients: Pearson and Spearman<\/h2>\n\n\n\n

    According to Schober, Boer and Schwarte (2018), \u201cthe Pearson correlation coefficient is typically used for jointly normally distributed data (data that follow a bivariate normal distribution). For nonnormally distributed continuous data, for ordinal data, or for data with relevant outliers, a Spearman rank correlation can be used as a measure of a monotonic association. Both correlation coefficients are scaled such that they range from \u20131 to +1, where 0 indicates that there is no linear or monotonic association, and the relationship gets stronger and ultimately approaches a straight line (Pearson correlation) or a constantly increasing or decreasing curve (Spearman correlation) as the coefficient approaches an absolute value of 1. Hypothesis tests and confidence intervals can be used to address the statistical significance of the results and to estimate the strength of the relationship in the population from which the data were sampled\u201d.<\/p>\n\n\n\n

    Specifically in R<\/em>, due to the functions of the package RSTATIX, Shapiro-Wilk normality tests and Pearson and Spearman coefficients could be calculated using a certain code structure. Within the next sections of this book, a proper code structure is proposed in order to perform such calculations.<\/p>\n\n\n\n

    RStatix package<\/h2>\n\n\n\n

    According with the formal prospect, \u201cRSTATIX package is a pipe-friendly framework for basic statistical tests\u201d. Provides a simple and intuitive pipe-friendly framework, coherent with the ‘tidyverse’ design philosophy, for performing basic statistical tests, including t-test, Wilcoxon test, ANOVA, Kruskal- Wallis and correlation analyses. The output of each test is automatically transformed into a tidy data frame to facilitate visualization. The package contains helper functions for identifying univariate and multivariate outliers, assessing normality and homogeneity of variances\u201d.<\/p>\n\n\n\n

    Correlations in R: proposed check list using R-Statix<\/h2>\n\n\n\n

    The following image shows the suggested process of correlation analysis:<\/p>\n\n\n\n

    Image 1
    Correlation analysis in R
    Recommended process diagram<\/strong><\/p>\n\n\n\n

    \"\"<\/figure>\n\n\n\n