{"id":138442,"date":"2022-05-31T09:00:00","date_gmt":"2022-05-31T13:00:00","guid":{"rendered":"https:\/\/ibkrcampus.com\/?p=138442"},"modified":"2022-11-21T09:54:58","modified_gmt":"2022-11-21T14:54:58","slug":"data-manipulation-and-visualization-techniques-in-julia-part-iv","status":"publish","type":"post","link":"https:\/\/www.interactivebrokers.com\/campus\/ibkr-quant-news\/data-manipulation-and-visualization-techniques-in-julia-part-iv\/","title":{"rendered":"Data Manipulation and Visualization Techniques in Julia \u2013 Part IV"},"content":{"rendered":"\n

Learn how to create new dataframes with Part I<\/a>, how to perform basic mathematical operations in Part II<\/a> and see Part III<\/a> for instructions on how to use the package RDatasets.jl.<\/em><\/p>\n\n\n\n

Dealing with missing data<\/h3>\n\n\n\n

Julia has a \u201cmissing\u201d object that is used for unavailable data. You can use skipmissing() function to perform operations ignoring the missing values.<\/p>\n\n\n\n

Output:<\/p>\n\n\n\n

a<\/th>b<\/th><\/tr>
Int64?<\/th>String?<\/th><\/tr><\/thead>
1<\/td>Apple<\/td><\/tr>
missing<\/em><\/td>Orange<\/td><\/tr>
3<\/td>missing<\/em><\/td><\/tr>
7<\/td>Grapes<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n

You can use dropmissing() function to remove the missing values.<\/p>\n\n\n\n

a<\/th>b<\/strong><\/td><\/tr>
Int64<\/th>String<\/th><\/tr><\/thead>
1<\/td>Apple<\/td><\/tr>
7<\/td>Grapes<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n

More details for dealing with missing values can be found here.<\/a><\/p>\n\n\n\n

\n\n\n\n

Importing and exporting data as CSV and Excel files<\/h2>\n\n\n\n

Reading data is the first step in analysing any kind of data. Most of the information we come across is either in CSV or excel format, so we\u2019ll focus on these two. We will work with CSV.jl and XLSX.jl for dealing with CSV and Excel files.<\/p>\n\n\n\n

Reading and writing CSV files<\/h3>\n\n\n\n

We\u2019ll read a CSV file (infy.csv), as a dataframe, containing historical stock price data for Infosys downloaded from Yahoo finance for the period 21-Dec-2020 to 22-Dec-2021.<\/p>\n\n\n\n

Here\u2019s a summary for this data.<\/p>\n\n\n\n

variable<\/th>mean<\/th>min<\/th>median<\/th>max<\/th>nmissing<\/th>eltype<\/th><\/tr>
Symbol<\/th>Union\u2026<\/th>Any<\/th>Union\u2026<\/th>Any<\/th>Int64<\/th>DataType<\/th><\/tr><\/thead>
Date<\/td> <\/td>2020-12-22<\/td> <\/td>2021-12-21<\/td>0<\/td>Date<\/td><\/tr>
Open<\/td>20.5674<\/td>16.39<\/td>20.63<\/td>24.05<\/td>0<\/td>Float64<\/td><\/tr>
High<\/td>20.7164<\/td>16.69<\/td>20.775<\/td>24.5<\/td>0<\/td>Float64<\/td><\/tr>
Low<\/td>20.4097<\/td>16.36<\/td>20.51<\/td>23.94<\/td>0<\/td>Float64<\/td><\/tr>
Close<\/td>20.5685<\/td>16.58<\/td>20.725<\/td>24.22<\/td>0<\/td>Float64<\/td><\/tr>
Adj Close<\/td>20.3422<\/td>16.2664<\/td>20.5451<\/td>24.22<\/td>0<\/td>Float64<\/td><\/tr>
Volume<\/td>7.09982e6<\/td>1320600<\/td>6.43815e6<\/td>22911800<\/td>0<\/td>Int64<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n

Here, we calculate the range –<\/p>\n\n\n\n

Date<\/th>Open<\/th>High<\/th>Low<\/th>Close<\/th>Adj Close<\/th>Volume<\/th>range<\/th><\/tr>
Date<\/th>Float64<\/th>Float64<\/th>Float64<\/th>Float64<\/th>Float64<\/th>Int64<\/th>Float64<\/th><\/tr><\/thead>
2020-12-22<\/td>16.39<\/td>16.74<\/td>16.36<\/td>16.58<\/td>16.2664<\/td>6714400<\/td>0.379999<\/td><\/tr>
2020-12-23<\/td>16.9<\/td>16.93<\/td>16.57<\/td>16.59<\/td>16.2762<\/td>5913500<\/td>0.36<\/td><\/tr>
2020-12-24<\/td>16.68<\/td>16.69<\/td>16.52<\/td>16.6<\/td>16.286<\/td>1320600<\/td>0.170001<\/td><\/tr>
2020-12-28<\/td>16.73<\/td>16.84<\/td>16.72<\/td>16.77<\/td>16.4528<\/td>4239300<\/td>0.120001<\/td><\/tr>
2020-12-29<\/td>16.9<\/td>16.9<\/td>16.67<\/td>16.76<\/td>16.443<\/td>8473700<\/td>0.23<\/td><\/tr>
2020-12-30<\/td>16.87<\/td>17.0<\/td>16.83<\/td>16.93<\/td>16.6098<\/td>3877200<\/td>0.17<\/td><\/tr>
2020-12-31<\/td>17.01<\/td>17.03<\/td>16.89<\/td>16.95<\/td>16.6294<\/td>3693700<\/td>0.140002<\/td><\/tr>
2021-01-04<\/td>17.39<\/td>17.43<\/td>17.06<\/td>17.25<\/td>16.9237<\/td>12597600<\/td>0.370001<\/td><\/tr>
2021-01-05<\/td>17.32<\/td>17.67<\/td>17.32<\/td>17.65<\/td>17.3162<\/td>8109900<\/td>0.35<\/td><\/tr>
2021-01-06<\/td>17.4<\/td>17.79<\/td>17.34<\/td>17.73<\/td>17.3946<\/td>9136300<\/td>0.450001<\/td><\/tr>
2021-01-07<\/td>17.36<\/td>17.55<\/td>17.26<\/td>17.55<\/td>17.2181<\/td>10272000<\/td>0.289999<\/td><\/tr>
2021-01-08<\/td>18.07<\/td>18.61<\/td>18.02<\/td>18.59<\/td>18.2384<\/td>17802400<\/td>0.590001<\/td><\/tr>
2021-01-11<\/td>18.68<\/td>18.86<\/td>18.55<\/td>18.76<\/td>18.4052<\/td>12220600<\/td>0.310002<\/td><\/tr>
2021-01-12<\/td>18.92<\/td>18.94<\/td>18.54<\/td>18.6<\/td>18.2482<\/td>10629100<\/td>0.4<\/td><\/tr>
2021-01-13<\/td>19.03<\/td>19.07<\/td>18.4<\/td>18.43<\/td>18.0814<\/td>18409900<\/td>0.67<\/td><\/tr>
2021-01-14<\/td>18.57<\/td>18.65<\/td>18.14<\/td>18.22<\/td>17.8754<\/td>13286100<\/td>0.510001<\/td><\/tr>
2021-01-15<\/td>18.19<\/td>18.38<\/td>18.11<\/td>18.17<\/td>17.8263<\/td>7443000<\/td>0.269998<\/td><\/tr>
2021-01-19<\/td>18.08<\/td>18.18<\/td>17.95<\/td>18.12<\/td>17.7773<\/td>7179600<\/td>0.229999<\/td><\/tr>
2021-01-20<\/td>18.37<\/td>18.47<\/td>18.29<\/td>18.4<\/td>18.052<\/td>5408500<\/td>0.179998<\/td><\/tr>
2021-01-21<\/td>18.39<\/td>18.4<\/td>18.15<\/td>18.2<\/td>17.8558<\/td>7963400<\/td>0.25<\/td><\/tr>
2021-01-22<\/td>18.23<\/td>18.27<\/td>18.06<\/td>18.18<\/td>17.8361<\/td>5663500<\/td>0.210001<\/td><\/tr>
2021-01-25<\/td>18.15<\/td>18.22<\/td>17.84<\/td>17.92<\/td>17.5811<\/td>6012600<\/td>0.379999<\/td><\/tr>
2021-01-26<\/td>17.92<\/td>17.92<\/td>17.75<\/td>17.85<\/td>17.5124<\/td>5472600<\/td>0.17<\/td><\/tr>
2021-01-27<\/td>17.65<\/td>17.89<\/td>17.44<\/td>17.47<\/td>17.1396<\/td>11388300<\/td>0.449998<\/td><\/tr>
2021-01-28<\/td>17.46<\/td>17.75<\/td>17.41<\/td>17.64<\/td>17.3064<\/td>7877600<\/td>0.34<\/td><\/tr>
2021-01-29<\/td>17.16<\/td>17.23<\/td>16.88<\/td>16.88<\/td>16.5607<\/td>9671400<\/td>0.350001<\/td><\/tr>
2021-02-01<\/td>17.19<\/td>17.42<\/td>17.05<\/td>17.38<\/td>17.0513<\/td>5829200<\/td>0.370001<\/td><\/tr>
2021-02-02<\/td>17.45<\/td>17.51<\/td>17.34<\/td>17.44<\/td>17.1101<\/td>4119800<\/td>0.17<\/td><\/tr>
2021-02-03<\/td>17.6<\/td>17.75<\/td>17.49<\/td>17.65<\/td>17.3162<\/td>4677800<\/td>0.26<\/td><\/tr>
2021-02-04<\/td>17.54<\/td>17.64<\/td>17.36<\/td>17.59<\/td>17.2573<\/td>4439600<\/td>0.279998<\/td><\/tr>
\u22ee<\/td>\u22ee<\/td>\u22ee<\/td>\u22ee<\/td>\u22ee<\/td>\u22ee<\/td>\u22ee<\/td>\u22ee<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n

This updated dataframe can be saved using CSV.write() function.<\/p>\n\n\n\n

Reading and writing excel files<\/strong><\/p>\n\n\n\n

We\u2019ll use the XLSX.jl package in Julia to read and write excel files.<\/p>\n\n\n\n

Here\u2019s how it can be done –<\/p>\n\n\n\n

Date<\/th>Open<\/th>High<\/th>Low<\/th>Close<\/th>Adj Close<\/th>Volume<\/th><\/tr>
Any<\/th>Any<\/th>Any<\/th>Any<\/th>Any<\/th>Any<\/th>Any<\/th><\/tr><\/thead>
2020-12-22<\/td>16.39<\/td>16.74<\/td>16.36<\/td>16.58<\/td>16.2664<\/td>6714400<\/td><\/tr>
2020-12-23<\/td>16.9<\/td>16.93<\/td>16.57<\/td>16.59<\/td>16.2762<\/td>5913500<\/td><\/tr>
2020-12-24<\/td>16.68<\/td>16.69<\/td>16.52<\/td>16.6<\/td>16.286<\/td>1320600<\/td><\/tr>
2020-12-28<\/td>16.73<\/td>16.84<\/td>16.72<\/td>16.77<\/td>16.4528<\/td>4239300<\/td><\/tr>
2020-12-29<\/td>16.9<\/td>16.9<\/td>16.67<\/td>16.76<\/td>16.443<\/td>8473700<\/td><\/tr>
2020-12-30<\/td>16.87<\/td>17.0<\/td>16.83<\/td>16.93<\/td>16.6098<\/td>3877200<\/td><\/tr>
2020-12-31<\/td>17.01<\/td>17.03<\/td>16.89<\/td>16.95<\/td>16.6294<\/td>3693700<\/td><\/tr>
2021-01-04<\/td>17.39<\/td>17.43<\/td>17.06<\/td>17.25<\/td>16.9237<\/td>12597600<\/td><\/tr>
2021-01-05<\/td>17.32<\/td>17.67<\/td>17.32<\/td>17.65<\/td>17.3162<\/td>8109900<\/td><\/tr>
2021-01-06<\/td>17.4<\/td>17.79<\/td>17.34<\/td>17.73<\/td>17.3946<\/td>9136300<\/td><\/tr>
2021-01-07<\/td>17.36<\/td>17.55<\/td>17.26<\/td>17.55<\/td>17.2181<\/td>10272000<\/td><\/tr>
2021-01-08<\/td>18.07<\/td>18.61<\/td>18.02<\/td>18.59<\/td>18.2384<\/td>17802400<\/td><\/tr>
2021-01-11<\/td>18.68<\/td>18.86<\/td>18.55<\/td>18.76<\/td>18.4052<\/td>12220600<\/td><\/tr>
2021-01-12<\/td>18.92<\/td>18.94<\/td>18.54<\/td>18.6<\/td>18.2482<\/td>10629100<\/td><\/tr>
2021-01-13<\/td>19.03<\/td>19.07<\/td>18.4<\/td>18.43<\/td>18.0814<\/td>18409900<\/td><\/tr>
2021-01-14<\/td>18.57<\/td>18.65<\/td>18.14<\/td>18.22<\/td>17.8754<\/td>13286100<\/td><\/tr>
2021-01-15<\/td>18.19<\/td>18.38<\/td>18.11<\/td>18.17<\/td>17.8263<\/td>7443000<\/td><\/tr>
2021-01-19<\/td>18.08<\/td>18.18<\/td>17.95<\/td>18.12<\/td>17.7773<\/td>7179600<\/td><\/tr>
2021-01-20<\/td>18.37<\/td>18.47<\/td>18.29<\/td>18.4<\/td>18.052<\/td>5408500<\/td><\/tr>
2021-01-21<\/td>18.39<\/td>18.4<\/td>18.15<\/td>18.2<\/td>17.8558<\/td>7963400<\/td><\/tr>
2021-01-22<\/td>18.23<\/td>18.27<\/td>18.06<\/td>18.18<\/td>17.8361<\/td>5663500<\/td><\/tr>
2021-01-25<\/td>18.15<\/td>18.22<\/td>17.84<\/td>17.92<\/td>17.5811<\/td>6012600<\/td><\/tr>
2021-01-26<\/td>17.92<\/td>17.92<\/td>17.75<\/td>17.85<\/td>17.5124<\/td>5472600<\/td><\/tr>
2021-01-27<\/td>17.65<\/td>17.89<\/td>17.44<\/td>17.47<\/td>17.1396<\/td>11388300<\/td><\/tr>
2021-01-28<\/td>17.46<\/td>17.75<\/td>17.41<\/td>17.64<\/td>17.3064<\/td>7877600<\/td><\/tr>
2021-01-29<\/td>17.16<\/td>17.23<\/td>16.88<\/td>16.88<\/td>16.5607<\/td>9671400<\/td><\/tr>
2021-02-01<\/td>17.19<\/td>17.42<\/td>17.05<\/td>17.38<\/td>17.0513<\/td>5829200<\/td><\/tr>
2021-02-02<\/td>17.45<\/td>17.51<\/td>17.34<\/td>17.44<\/td>17.1101<\/td>4119800<\/td><\/tr>
2021-02-03<\/td>17.6<\/td>17.75<\/td>17.49<\/td>17.65<\/td>17.3162<\/td>4677800<\/td><\/tr>
2021-02-04<\/td>17.54<\/td>17.64<\/td>17.36<\/td>17.59<\/td>17.2573<\/td>4439600<\/td><\/tr>
\u22ee<\/td>\u22ee<\/td>\u22ee<\/td>\u22ee<\/td>\u22ee<\/td>\u22ee<\/td>\u22ee<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n

We can write an excel file using the writetable() function.<\/p>\n\n\n\n

Julia has in-built read() and write() open() close() functions to work with text files. More details can be found here<\/a>.<\/p>\n\n\n\n

Data can be written in .jld format as well. .jld is Julia\u2019s data format built using the JLD.jl package.<\/p>\n\n\n\n

Details for the following packages can be found here –<\/p>\n\n\n\n