{"id":108945,"date":"2021-10-29T09:40:00","date_gmt":"2021-10-29T13:40:00","guid":{"rendered":"https:\/\/ibkrcampus.com\/?p=108945"},"modified":"2022-11-21T09:48:50","modified_gmt":"2022-11-21T14:48:50","slug":"pickle-python-how-to-use-need-and-example","status":"publish","type":"post","link":"https:\/\/www.interactivebrokers.com\/campus\/ibkr-quant-news\/pickle-python-how-to-use-need-and-example\/","title":{"rendered":"Pickle Python – How to use, Need and Example"},"content":{"rendered":"\n

Excerpt<\/em><\/strong><\/p>\n\n\n\n

As analysts, we spend a lot of time processing, transforming and inferring data, we handle large amounts of data and devote a great deal of time to its analysis and manipulation.<\/p>\n\n\n\n

It is convenient to have a mechanism that allows us to save the processed data for future retrieval without going through the same costly process again. Pickle is a utility that allows us to save Python objects in a binary file. In other words:<\/p>\n\n\n\n

Pickle allows us to save time.<\/strong><\/p><\/blockquote>\n\n\n\n

The need for Pickle Python<\/h2>\n\n\n\n

When we process large amounts of data in our analysis and backtesting, the machine needs a few hours, if not days, to process all the information.<\/p>\n\n\n\n

The backtesting of a large portfolio of financial assets with historical data running into decades or the training of our ML algorithms are heavy processes from the point of view of the machine time needed to digest the data.<\/p>\n\n\n\n

Repeating this procedure over and over again, most of the time, is pointless and a waste of time and resources. So it is convenient to have a mechanism that allows us to save the processed data for future retrieval without having to repeat the same costly process.<\/p>\n\n\n\n

In Python there are multiple mechanisms and formats such as plain text files, binary files and structured and unstructured databases.<\/p>\n\n\n\n

Among the most popular plain text files are csv (comma separated values), json (JavaScript Object Notation) or xml (eXtended Markup Language).  The main feature of plain text files is that they are human-readable and can be exchanged between machines.<\/p>\n\n\n\n

Structured and unstructured databases are able to store large amounts of information, relate data to each other and provide fast and accurate answers to queries.<\/p>\n\n\n\n

Finally, we can use binary files to store the information. These files are not human-readable since they store bytes of information that can only be understood by machines.<\/p>\n\n\n\n

Their main characteristic is the speed of storage\/retrieval and the small size compared to the previous ones. Pickle is a utility that allows us to save Python objects in a binary file.<\/p>\n\n\n\n

\n\n\n\n

What is Pickle Python?<\/h2>\n\n\n\n

From the official documentation<\/a>, the technical explanation about Pickle Python is as follows:<\/p>\n\n\n\n

The<\/em> pickle<\/em><\/a> module implements binary protocols for serializing and de-serializing a Python object structure.<\/em><\/p>\n\n\n\n

\u201cPickling\u201d is the process whereby a Python object hierarchy is converted into a byte stream, and \u201cunpickling\u201d is the inverse operation, whereby a byte stream (from a<\/em> binary file<\/em><\/a> or<\/em> bytes-like object<\/em><\/a>) is converted back into an object hierarchy.<\/em><\/p>\n\n\n\n

Pickling (and unpickling) is alternatively known as \u201cserialization\u201d, \u201cmarshalling,\u201d<\/em> 1<\/sup><\/em><\/a> or \u201cflattening\u201d; however, to avoid confusion, the terms used here are \u201cpickling\u201d and \u201cunpickling\u201d.<\/em><\/p>\n\n\n\n

For simplicity, we can say that Pickle stores and retrieves Python objects to\/from the machine’s RAM.<\/p>\n\n\n\n

It is important to remember here that in Python even variables are objects and that regardless of where the data we are handling comes from, the information resides in the machine’s volatile memory, also called RAM (Random Access Memory).<\/p>\n\n\n\n

Unless we save this information in a storage system such as a hard disk, in any file format kind or database, the information is lost at the end of the Python session.<\/p>\n\n\n\n

\n\n\n\n

Example of Pickle Python<\/h2>\n\n\n\n

Let\u2019s suppose the following scenarios.<\/p>\n\n\n\n

Pickle Python Scenario 1<\/h3>\n\n\n\n
\"\"<\/figure>\n\n\n\n

Pickle Python example – High level abstraction<\/em><\/p>\n\n\n\n

The figure shows a very high level abstraction typically seen in a ML project.<\/p>\n\n\n\n

The ETL<\/strong> (Extraction, Transformation and Load) is the tool for:<\/p>\n\n\n\n