{"id":182051,"date":"2022-11-08T17:30:00","date_gmt":"2022-11-08T22:30:00","guid":{"rendered":"https:\/\/ibkrcampus.com\/traders-insight\/pyspark-a-beginners-guide-to-apache-spark-and-big-data\/"},"modified":"2023-02-13T17:03:58","modified_gmt":"2023-02-13T22:03:58","slug":"pyspark-a-beginners-guide-to-apache-spark-and-big-data","status":"publish","type":"post","link":"https:\/\/www.interactivebrokers.com\/campus\/ibkr-quant-news\/pyspark-a-beginners-guide-to-apache-spark-and-big-data\/","title":{"rendered":"PySpark \u2013 A Beginner&#8217;s Guide to Apache Spark and Big Data"},"content":{"rendered":"\n<p class=\"wp-block-paragraph\"><em>The article \u201cPySpark \u2013 A Beginner\u2019s Guide to Apache Spark and Big Data\u201d first appeared on&nbsp;<a href=\"https:\/\/algotrading101.com\/learn\/pyspark-guide\/\">AlgoTrading101 Blog<\/a>.<\/em><\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><em>Excerpt<\/em><\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"h-what-is-pyspark\">What is PySpark?<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">PySpark is a Python library that serves as an interface for Apache Spark.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"h-what-is-apache-spark\">What is Apache Spark?<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Apache Spark is an open-source distributed computing engine that is used for Big Data processing.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">It is a general-purpose engine as it supports Python, R, SQL, Scala, and Java.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"h-what-is-apache-spark-used-for\">What is Apache Spark used for?<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Apache Spark is often used with Big Data as it allows for distributed computing and it offers built-in data streaming, machine learning, SQL, and graph processing. It is often used by data engineers and data scientists.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"h-what-is-pyspark-used-for\">What is PySpark used for?<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">PySpark is used as an API for Apache Spark. This allows us to leave the Apache Spark terminal and enter our preferred Python programming IDE without losing what Apache Spark has to offer.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"h-is-apache-spark-free\">Is Apache Spark free?<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Apache Spark is an open-source engine and thus it is completely free to download and use.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"h-why-should-i-use-apache-spark\">Why should I use Apache Spark?<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Apache Spark offers distributed computing<\/li>\n\n\n\n<li>Apache Spark is easy to use<\/li>\n\n\n\n<li>Apache Spark is free<\/li>\n\n\n\n<li>Offer advanced analytics<\/li>\n\n\n\n<li>Is a very powerful engine<\/li>\n\n\n\n<li>Offers machine learning, streaming, SQL, and graph processing modules<\/li>\n\n\n\n<li>Is applicable to various programming languages like Python, R, Java\u2026<\/li>\n\n\n\n<li>Has a good community and is advancing as a product<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"h-why-shouldn-t-i-use-apache-spark\">Why shouldn\u2019t I use Apache Spark?<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Apache Spark can have scaling problems with compute-intensive jobs<\/li>\n\n\n\n<li>It can consume a lot of memory<\/li>\n\n\n\n<li>Can have issues with small files<\/li>\n\n\n\n<li>Is constrained by the number of available ML algorithms<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"h-why-should-i-use-pyspark\">Why should I use PySpark?<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>PySpark is easy to use<\/li>\n\n\n\n<li>PySpark can handle synchronization errors<\/li>\n\n\n\n<li>The learning curve isn\u2019t steep as in other languages like Scala<\/li>\n\n\n\n<li>Can easily handle big data<\/li>\n\n\n\n<li>Has all the pros of Apache Spark added to it<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"h-why-shouldn-t-i-use-pyspark\">Why shouldn\u2019t I use PySpark?<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>PySpark can be less efficient as it uses Python<\/li>\n\n\n\n<li>It is slow when compared to other languages like Scala<\/li>\n\n\n\n<li>It can be replaced with other libraries like Dask that easily integrate with Pandas (depends on the problem and dataset)<\/li>\n\n\n\n<li>Suffers from all the cons of Apache Spark<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"h-what-are-some-apache-spark-alternatives\">What are some Apache Spark alternatives?<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Apache Spark can be replaced with some alternatives and they are the following:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Apache Hadoop<\/li>\n\n\n\n<li>Google BigQuery<\/li>\n\n\n\n<li>Amazon EMR<\/li>\n\n\n\n<li>IBM Analytics Engine<\/li>\n\n\n\n<li>Apache Flink<\/li>\n\n\n\n<li>Lumify<\/li>\n\n\n\n<li>Presto<\/li>\n\n\n\n<li>Apache Pig<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"h-what-are-some-apache-spark-clients\">What are some Apache Spark clients?<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Some of the programming clients that has Apache Spark APIs are the following:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><a><\/a><a href=\"https:\/\/spark.apache.org\/docs\/latest\/api\/python\/index.html\">Python<\/a><\/li>\n\n\n\n<li><a href=\"https:\/\/spark.apache.org\/docs\/latest\/api\/scala\/org\/apache\/spark\/index.html\">Scala<\/a><\/li>\n\n\n\n<li><a href=\"https:\/\/spark.apache.org\/docs\/latest\/api\/java\/index.html\">Java<\/a><\/li>\n\n\n\n<li><a href=\"https:\/\/spark.apache.org\/docs\/latest\/api\/R\/index.html\">R<\/a><\/li>\n\n\n\n<li><a href=\"https:\/\/spark.apache.org\/docs\/latest\/api\/sql\/index.html\">SQL<\/a><\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"h-how-to-get-started-with-apache-spark\">How to get started with Apache Spark?<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">In order to get started with Apache Spark and the PySpark library, we will need to go through multiple steps. This can be a bit confusing if you have never done something similar but don\u2019t worry. We will do it together!<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"h-prerequisites\">Prerequisites<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">The first things that we need to take care of are the prerequisites that we need in order to make Apache Spark and PySpark work. These prerequisites are Java 8, Python 3, and something to extract .tar files.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Let\u2019s see what Java version are you rocking on your computer. If you\u2019re on Windows like me, go to Start, type&nbsp;<code>cmd<\/code>, and enter the Command Prompt. When there, type the following command:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>java -version<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">And you\u2019ll get a message similar to this one that will specify your Java version:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>java version \"1.8.0_281\"<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">If you didn\u2019t get a response you don\u2019t have Java installed. If your java is outdated ( &lt; 8) or non-existent, go over to the following&nbsp;<a href=\"https:\/\/java.com\/en\/download\/\">link<\/a>&nbsp;and download the latest version.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">If you, for some reason, don\u2019t have Python installed here is a&nbsp;<a href=\"https:\/\/www.python.org\/\">link<\/a>&nbsp;to download it. And lastly, for the extraction of .tar files, I use&nbsp;<a href=\"https:\/\/www.7-zip.org\/download.html\">7-zip<\/a>. You can use anything that does the job.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Download and set-up<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Go over to the following&nbsp;<a href=\"https:\/\/spark.apache.org\/downloads.html\">link<\/a>&nbsp;and download the 3.0.3. Spark release that is pre-built for Apache Hadoop 2.7.<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img decoding=\"async\" data-src=\"\/campus\/wp-content\/uploads\/sites\/2\/2023\/02\/AlgoTrading101-pyspark-1.jpg\" alt=\" class=\" class=\"wp-image-165525 lazyload\" src=\"data:image\/svg+xml;base64,PHN2ZyB3aWR0aD0iMSIgaGVpZ2h0PSIxIiB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciPjwvc3ZnPg==\" \/><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">Now click the blue link that is written under number 3 and select one of the mirrors that you would like to download from. While it is downloading create a folder named Spark in your root drive (C:).<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Go into that folder and extract the downloaded file into it. The next thing that you need to add is the winutils.exe file for the underlying Hadoop version that Spark will be utilizing.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">To do this, go over to the following&nbsp;<a href=\"https:\/\/github.com\/cdarlint\/winutils\">GitHub page<\/a>&nbsp;and select the version of Hadoop that we downloaded. After that, scroll down until you see the winutils.exe file. Click on it and download it.<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img decoding=\"async\" data-src=\"\/campus\/wp-content\/uploads\/sites\/2\/2023\/02\/AlgoTrading101-pyspark-2.png\" alt=\" class=\" class=\"wp-image-165527 lazyload\" src=\"data:image\/svg+xml;base64,PHN2ZyB3aWR0aD0iMSIgaGVpZ2h0PSIxIiB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciPjwvc3ZnPg==\" \/><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">Now create a new folder in your root drive and name it \u201cHadoop\u201d, then create a folder inside of that folder and name it \u201cbin\u201d. Inside the bin folder paste the winutils.exe file that we just downloaded.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Now for the final steps, we need to configure our environmental variables. Environmental variables allow us to add Spark and Hadoop to our system PATH. This way we can call Spark in Python as they will be on the same PATH.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Click Start and type \u201cenvironment\u201d. Then select the \u201cEdit the system environment variables\u201d option. A new window will pop up and in the lower right corner of it select \u201cEnvironment Variables\u201d.<\/p>\n\n\n\n<figure class=\"wp-block-image size-large img-twothird\"><img decoding=\"async\" data-src=\"\/campus\/wp-content\/uploads\/sites\/2\/2023\/02\/AlgoTrading101-pyspark-3.jpg\" alt=\"=\"\" src=\"data:image\/svg+xml;base64,PHN2ZyB3aWR0aD0iMSIgaGVpZ2h0PSIxIiB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciPjwvc3ZnPg==\" class=\"lazyload\"><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">A new window will appear that will show your environmental variables. In my case, I already have Spark there:<\/p>\n\n\n\n<figure class=\"wp-block-image size-large img-twothird\"><img decoding=\"async\" data-src=\"\/campus\/wp-content\/uploads\/sites\/2\/2023\/02\/AlgoTrading101-pyspark-4.jpg\" alt=\"=\"\" src=\"data:image\/svg+xml;base64,PHN2ZyB3aWR0aD0iMSIgaGVpZ2h0PSIxIiB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciPjwvc3ZnPg==\" class=\"lazyload\"><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">To add it there, click on \u201cNew\u201d. Then set the name to be \u201cSPARK_HOME\u201d and for the Variable value add the path where you downloaded your spark. It should be something like this&nbsp;<code>C:Sparkspark...<\/code>&nbsp;Click OK.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">For the next step be sure to be careful and not change your Path. Click on the \u201cPath\u201d in your user variables and then select \u201cEdit\u201d. A new window will appear, click on the \u201cNew\u201d button and then write this&nbsp;<code>%SPARK_HOME%bin<\/code><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">You\u2019ve successfully added Spark to your PATH! Now, repeat this process for both Hadoop and Java. The only things that will change will be their locations and the end name that you give to them.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Your end product should look like this:<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img decoding=\"async\" data-src=\"\/campus\/wp-content\/uploads\/sites\/2\/2023\/02\/AlgoTrading101-pyspark-5.png\" alt=\" class=\" class=\"wp-image-165536 lazyload\" src=\"data:image\/svg+xml;base64,PHN2ZyB3aWR0aD0iMSIgaGVpZ2h0PSIxIiB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciPjwvc3ZnPg==\" \/><\/figure>\n\n\n\n<h3 class=\"wp-block-heading\">Launch Spark<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Now let us launch our Spark and see it in its full glory. Start a new command prompt and then enter spark-shell to launch Spark. A new window will appear with Spark up and running.<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img decoding=\"async\" data-src=\"\/campus\/wp-content\/uploads\/sites\/2\/2023\/02\/AlgoTrading101-pyspark-6.jpg\" alt=\" class=\" class=\"wp-image-165540 lazyload\" src=\"data:image\/svg+xml;base64,PHN2ZyB3aWR0aD0iMSIgaGVpZ2h0PSIxIiB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciPjwvc3ZnPg==\" \/><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">Now open up your browser and write&nbsp;<code>https:\/\/localhost:4040\/<\/code>&nbsp;or whatever the name of your system is. This will open up the Apache Spark UI where you will be able to see all the information you might need.<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img decoding=\"async\" data-src=\"\/campus\/wp-content\/uploads\/sites\/2\/2023\/02\/AlgoTrading101-pyspark-7.jpg\" alt=\" class=\" class=\"wp-image-165541 lazyload\" src=\"data:image\/svg+xml;base64,PHN2ZyB3aWR0aD0iMSIgaGVpZ2h0PSIxIiB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciPjwvc3ZnPg==\" \/><\/figure>\n\n\n\n<h3 class=\"wp-block-heading\">What are the main components of Apache Spark?<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">There are several components that make Apache Spark and they are the following:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Spark Core&nbsp;<\/strong>\u2013 is the main part of Apache Spark that provides in-built memory computing and does all the basic I\/O functions, memory management, and much more.<\/li>\n\n\n\n<li><strong>Spark Streaming&nbsp;<\/strong>\u2013 allows for data streaming that can go up to a couple of gigabytes per second.<\/li>\n\n\n\n<li><strong>Spark SQL<\/strong>&nbsp;\u2013 allows the use of SQL (Structured Query Language) for easier data manipulation and analysis.<\/li>\n\n\n\n<li><strong>MlLib<\/strong>&nbsp;\u2013 packs several machine learning models that can be used in several programming languages.<\/li>\n\n\n\n<li><strong>GraphX<\/strong>&nbsp;\u2013 provides several methods for implementing graph theory to your dataset (i.e. network analysis).<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">What is the Apache Spark RDD?<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Apache Spark RDD (Resilient Distributed Dataset) is a data structure that serves as the main building block. An RDD can be seen as an immutable and partitioned set of data values that can be processed on a distributed system.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">To conclude, they are resilient because they are immutable, distributed as they have partitions that can be processed in a distributed manner, and datasets as they hold our data.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">How to use PySpark in Jupyter Notebooks?<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">To use PySpark in your Jupyter notebook, all you need to do is to install the PySpark pip package with the following command:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>pip install pyspark<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">As your Python is located on your system PATH it will work with your Apache Spark. If you want to use something like&nbsp;<a href=\"https:\/\/algotrading101.com\/learn\/google-colab-guide\/\">Google Colab<\/a>&nbsp;you will need to run the following block of code that will set up Apache Spark for you:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>!apt-get install openjdk-8-jdk-headless -qq &gt; \/dev\/null\n!wget -q https:\/\/www-us.apache.org\/dist\/spark\/spark-3.0.3\/spark-3.0.3-<em>bin<\/em>-hadoop2.7.tgz\n!tar xf spark-3.0.3-<em>bin<\/em>-hadoop2.7.tgz\n!pip install -q findspark\nimport os\nos.environ&#91;\"JAVA_HOME\"] = \"\/usr\/lib\/jvm\/java-8-openjdk-amd64\"\nos.environ&#91;\"SPARK_HOME\"] = \"\/content\/spark-3.0.3-bin-hadoop2.7\"\nimport findspark\nfindspark.init()<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">If you want to use Kaggle like we\u2019re going to do, you can just go straight to the \u201cpip install pyspark\u201d command as Apache Spark will be ready for use.<a><\/a><\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><em>Visit AlgoTrading101 Blog for additional insight on this topic: <a href=\"https:\/\/algotrading101.com\/learn\/pyspark-guide\/\">https:\/\/algotrading101.com\/learn\/pyspark-guide\/<\/a>.<\/em><\/p>\n","protected":false},"excerpt":{"rendered":"<p>PySpark is a Python library that serves as an interface for Apache Spark. Apache Spark is an open-source distributed computing engine that is used for Big Data processing.<\/p>\n","protected":false},"author":815,"featured_media":182059,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":"","jetpack_post_was_ever_published":false},"categories":[339,343,349,338],"tags":[14079,5121,14080,596,11956,14081,14082,14083,595,487,14084,8684],"contributors-categories":[13746],"class_list":["post-182051","post","type-post","status-publish","format-standard","has-post-thumbnail","category-data-science","category-programing-languages","category-python-development","category-ibkr-quant-news","tag-apache-spark","tag-big-data","tag-graphx","tag-java","tag-jupyter-notebooks","tag-kaggle","tag-mllib","tag-pyspark","tag-python","tag-r","tag-scala","tag-sql","contributors-categories-algotrading101"],"pp_statuses_selecting_workflow":false,"pp_workflow_action":"current","pp_status_selection":"publish","acf":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO Premium plugin v26.9 (Yoast SEO v27.9) - https:\/\/yoast.com\/product\/yoast-seo-premium-wordpress\/ -->\n<title>PySpark \u2013 A Beginner&#8217;s Guide to Apache Spark and Big Data<\/title>\n<meta name=\"description\" content=\"PySpark is a Python library that serves as an interface for Apache Spark. Apache Spark is an open-source distributed computing engine that is used for...\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.interactivebrokers.com\/campus\/wp-json\/wp\/v2\/posts\/182051\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"PySpark \u2013 A Beginner&#039;s Guide to Apache Spark and Big Data\" \/>\n<meta property=\"og:description\" content=\"PySpark is a Python library that serves as an interface for Apache Spark. Apache Spark is an open-source distributed computing engine that is used for Big Data processing.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.interactivebrokers.com\/campus\/ibkr-quant-news\/pyspark-a-beginners-guide-to-apache-spark-and-big-data\/\" \/>\n<meta property=\"og:site_name\" content=\"IBKR Campus US\" \/>\n<meta property=\"article:published_time\" content=\"2022-11-08T22:30:00+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2023-02-13T22:03:58+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.interactivebrokers.com\/campus\/wp-content\/uploads\/sites\/2\/2023\/02\/python-blue-dots-opaque.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"1000\" \/>\n\t<meta property=\"og:image:height\" content=\"563\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"Igor Radovanovic\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:title\" content=\"PySpark \u2013 A Beginner&#039;s Guide to Apache Spark and Big Data\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Igor Radovanovic\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\n\t    \"@context\": \"https:\\\/\\\/schema.org\",\n\t    \"@graph\": [\n\t        {\n\t            \"@type\": \"NewsArticle\",\n\t            \"@id\": \"https:\\\/\\\/www.interactivebrokers.com\\\/campus\\\/ibkr-quant-news\\\/pyspark-a-beginners-guide-to-apache-spark-and-big-data\\\/#article\",\n\t            \"isPartOf\": {\n\t                \"@id\": \"https:\\\/\\\/www.interactivebrokers.com\\\/campus\\\/ibkr-quant-news\\\/pyspark-a-beginners-guide-to-apache-spark-and-big-data\\\/\"\n\t            },\n\t            \"author\": {\n\t                \"name\": \"Igor Radovanovic\",\n\t                \"@id\": \"https:\\\/\\\/ibkrcampus.com\\\/campus\\\/#\\\/schema\\\/person\\\/b43b33f424bad38d84a7b78eb0193592\"\n\t            },\n\t            \"headline\": \"PySpark \u2013 A Beginner&#8217;s Guide to Apache Spark and Big Data\",\n\t            \"datePublished\": \"2022-11-08T22:30:00+00:00\",\n\t            \"dateModified\": \"2023-02-13T22:03:58+00:00\",\n\t            \"mainEntityOfPage\": {\n\t                \"@id\": \"https:\\\/\\\/www.interactivebrokers.com\\\/campus\\\/ibkr-quant-news\\\/pyspark-a-beginners-guide-to-apache-spark-and-big-data\\\/\"\n\t            },\n\t            \"wordCount\": 880,\n\t            \"publisher\": {\n\t                \"@id\": \"https:\\\/\\\/ibkrcampus.com\\\/campus\\\/#organization\"\n\t            },\n\t            \"image\": {\n\t                \"@id\": \"https:\\\/\\\/www.interactivebrokers.com\\\/campus\\\/ibkr-quant-news\\\/pyspark-a-beginners-guide-to-apache-spark-and-big-data\\\/#primaryimage\"\n\t            },\n\t            \"thumbnailUrl\": \"https:\\\/\\\/www.interactivebrokers.com\\\/campus\\\/wp-content\\\/uploads\\\/sites\\\/2\\\/2023\\\/02\\\/python-blue-dots-opaque.jpg\",\n\t            \"keywords\": [\n\t                \"Apache Spark\",\n\t                \"Big Data\",\n\t                \"GraphX\",\n\t                \"Java\",\n\t                \"Jupyter Notebooks\",\n\t                \"Kaggle\",\n\t                \"MlLib\",\n\t                \"PySpark\",\n\t                \"Python\",\n\t                \"R\",\n\t                \"Scala\",\n\t                \"SQL\"\n\t            ],\n\t            \"articleSection\": [\n\t                \"Data Science\",\n\t                \"Programming Languages\",\n\t                \"Python Development\",\n\t                \"Quant\"\n\t            ],\n\t            \"inLanguage\": \"en-US\"\n\t        },\n\t        {\n\t            \"@type\": \"WebPage\",\n\t            \"@id\": \"https:\\\/\\\/www.interactivebrokers.com\\\/campus\\\/ibkr-quant-news\\\/pyspark-a-beginners-guide-to-apache-spark-and-big-data\\\/\",\n\t            \"url\": \"https:\\\/\\\/www.interactivebrokers.com\\\/campus\\\/ibkr-quant-news\\\/pyspark-a-beginners-guide-to-apache-spark-and-big-data\\\/\",\n\t            \"name\": \"PySpark \u2013 A Beginner's Guide to Apache Spark and Big Data\",\n\t            \"isPartOf\": {\n\t                \"@id\": \"https:\\\/\\\/ibkrcampus.com\\\/campus\\\/#website\"\n\t            },\n\t            \"primaryImageOfPage\": {\n\t                \"@id\": \"https:\\\/\\\/www.interactivebrokers.com\\\/campus\\\/ibkr-quant-news\\\/pyspark-a-beginners-guide-to-apache-spark-and-big-data\\\/#primaryimage\"\n\t            },\n\t            \"image\": {\n\t                \"@id\": \"https:\\\/\\\/www.interactivebrokers.com\\\/campus\\\/ibkr-quant-news\\\/pyspark-a-beginners-guide-to-apache-spark-and-big-data\\\/#primaryimage\"\n\t            },\n\t            \"thumbnailUrl\": \"https:\\\/\\\/www.interactivebrokers.com\\\/campus\\\/wp-content\\\/uploads\\\/sites\\\/2\\\/2023\\\/02\\\/python-blue-dots-opaque.jpg\",\n\t            \"datePublished\": \"2022-11-08T22:30:00+00:00\",\n\t            \"dateModified\": \"2023-02-13T22:03:58+00:00\",\n\t            \"description\": \"PySpark is a Python library that serves as an interface for Apache Spark. Apache Spark is an open-source distributed computing engine that is used for Big Data processing.\",\n\t            \"inLanguage\": \"en-US\",\n\t            \"potentialAction\": [\n\t                {\n\t                    \"@type\": \"ReadAction\",\n\t                    \"target\": [\n\t                        \"https:\\\/\\\/www.interactivebrokers.com\\\/campus\\\/ibkr-quant-news\\\/pyspark-a-beginners-guide-to-apache-spark-and-big-data\\\/\"\n\t                    ]\n\t                }\n\t            ]\n\t        },\n\t        {\n\t            \"@type\": \"ImageObject\",\n\t            \"inLanguage\": \"en-US\",\n\t            \"@id\": \"https:\\\/\\\/www.interactivebrokers.com\\\/campus\\\/ibkr-quant-news\\\/pyspark-a-beginners-guide-to-apache-spark-and-big-data\\\/#primaryimage\",\n\t            \"url\": \"https:\\\/\\\/www.interactivebrokers.com\\\/campus\\\/wp-content\\\/uploads\\\/sites\\\/2\\\/2023\\\/02\\\/python-blue-dots-opaque.jpg\",\n\t            \"contentUrl\": \"https:\\\/\\\/www.interactivebrokers.com\\\/campus\\\/wp-content\\\/uploads\\\/sites\\\/2\\\/2023\\\/02\\\/python-blue-dots-opaque.jpg\",\n\t            \"width\": 1000,\n\t            \"height\": 563,\n\t            \"caption\": \"PySpark \u2013 A Beginner's Guide to Apache Spark and Big Data\"\n\t        },\n\t        {\n\t            \"@type\": \"WebSite\",\n\t            \"@id\": \"https:\\\/\\\/ibkrcampus.com\\\/campus\\\/#website\",\n\t            \"url\": \"https:\\\/\\\/ibkrcampus.com\\\/campus\\\/\",\n\t            \"name\": \"IBKR Campus US\",\n\t            \"description\": \"Financial Education from Interactive Brokers\",\n\t            \"publisher\": {\n\t                \"@id\": \"https:\\\/\\\/ibkrcampus.com\\\/campus\\\/#organization\"\n\t            },\n\t            \"potentialAction\": [\n\t                {\n\t                    \"@type\": \"SearchAction\",\n\t                    \"target\": {\n\t                        \"@type\": \"EntryPoint\",\n\t                        \"urlTemplate\": \"https:\\\/\\\/ibkrcampus.com\\\/campus\\\/?s={search_term_string}\"\n\t                    },\n\t                    \"query-input\": {\n\t                        \"@type\": \"PropertyValueSpecification\",\n\t                        \"valueRequired\": true,\n\t                        \"valueName\": \"search_term_string\"\n\t                    }\n\t                }\n\t            ],\n\t            \"inLanguage\": \"en-US\"\n\t        },\n\t        {\n\t            \"@type\": \"Organization\",\n\t            \"@id\": \"https:\\\/\\\/ibkrcampus.com\\\/campus\\\/#organization\",\n\t            \"name\": \"Interactive Brokers\",\n\t            \"alternateName\": \"IBKR\",\n\t            \"url\": \"https:\\\/\\\/ibkrcampus.com\\\/campus\\\/\",\n\t            \"logo\": {\n\t                \"@type\": \"ImageObject\",\n\t                \"inLanguage\": \"en-US\",\n\t                \"@id\": \"https:\\\/\\\/ibkrcampus.com\\\/campus\\\/#\\\/schema\\\/logo\\\/image\\\/\",\n\t                \"url\": \"https:\\\/\\\/www.interactivebrokers.com\\\/campus\\\/wp-content\\\/uploads\\\/sites\\\/2\\\/2024\\\/05\\\/ibkr-campus-logo.jpg\",\n\t                \"contentUrl\": \"https:\\\/\\\/www.interactivebrokers.com\\\/campus\\\/wp-content\\\/uploads\\\/sites\\\/2\\\/2024\\\/05\\\/ibkr-campus-logo.jpg\",\n\t                \"width\": 669,\n\t                \"height\": 669,\n\t                \"caption\": \"Interactive Brokers\"\n\t            },\n\t            \"image\": {\n\t                \"@id\": \"https:\\\/\\\/ibkrcampus.com\\\/campus\\\/#\\\/schema\\\/logo\\\/image\\\/\"\n\t            },\n\t            \"publishingPrinciples\": \"https:\\\/\\\/www.interactivebrokers.com\\\/campus\\\/about-ibkr-campus\\\/\",\n\t            \"ethicsPolicy\": \"https:\\\/\\\/www.interactivebrokers.com\\\/campus\\\/cyber-security-notice\\\/\"\n\t        },\n\t        {\n\t            \"@type\": \"Person\",\n\t            \"@id\": \"https:\\\/\\\/ibkrcampus.com\\\/campus\\\/#\\\/schema\\\/person\\\/b43b33f424bad38d84a7b78eb0193592\",\n\t            \"name\": \"Igor Radovanovic\",\n\t            \"url\": \"https:\\\/\\\/www.interactivebrokers.com\\\/campus\\\/author\\\/igor-radovanovic\\\/\"\n\t        }\n\t    ]\n\t}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"PySpark \u2013 A Beginner&#8217;s Guide to Apache Spark and Big Data","description":"PySpark is a Python library that serves as an interface for Apache Spark. Apache Spark is an open-source distributed computing engine that is used for...","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.interactivebrokers.com\/campus\/wp-json\/wp\/v2\/posts\/182051\/","og_locale":"en_US","og_type":"article","og_title":"PySpark \u2013 A Beginner's Guide to Apache Spark and Big Data","og_description":"PySpark is a Python library that serves as an interface for Apache Spark. Apache Spark is an open-source distributed computing engine that is used for Big Data processing.","og_url":"https:\/\/www.interactivebrokers.com\/campus\/ibkr-quant-news\/pyspark-a-beginners-guide-to-apache-spark-and-big-data\/","og_site_name":"IBKR Campus US","article_published_time":"2022-11-08T22:30:00+00:00","article_modified_time":"2023-02-13T22:03:58+00:00","og_image":[{"width":1000,"height":563,"url":"https:\/\/www.interactivebrokers.com\/campus\/wp-content\/uploads\/sites\/2\/2023\/02\/python-blue-dots-opaque.jpg","type":"image\/jpeg"}],"author":"Igor Radovanovic","twitter_card":"summary_large_image","twitter_title":"PySpark \u2013 A Beginner's Guide to Apache Spark and Big Data","twitter_misc":{"Written by":"Igor Radovanovic"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"NewsArticle","@id":"https:\/\/www.interactivebrokers.com\/campus\/ibkr-quant-news\/pyspark-a-beginners-guide-to-apache-spark-and-big-data\/#article","isPartOf":{"@id":"https:\/\/www.interactivebrokers.com\/campus\/ibkr-quant-news\/pyspark-a-beginners-guide-to-apache-spark-and-big-data\/"},"author":{"name":"Igor Radovanovic","@id":"https:\/\/ibkrcampus.com\/campus\/#\/schema\/person\/b43b33f424bad38d84a7b78eb0193592"},"headline":"PySpark \u2013 A Beginner&#8217;s Guide to Apache Spark and Big Data","datePublished":"2022-11-08T22:30:00+00:00","dateModified":"2023-02-13T22:03:58+00:00","mainEntityOfPage":{"@id":"https:\/\/www.interactivebrokers.com\/campus\/ibkr-quant-news\/pyspark-a-beginners-guide-to-apache-spark-and-big-data\/"},"wordCount":880,"publisher":{"@id":"https:\/\/ibkrcampus.com\/campus\/#organization"},"image":{"@id":"https:\/\/www.interactivebrokers.com\/campus\/ibkr-quant-news\/pyspark-a-beginners-guide-to-apache-spark-and-big-data\/#primaryimage"},"thumbnailUrl":"https:\/\/www.interactivebrokers.com\/campus\/wp-content\/uploads\/sites\/2\/2023\/02\/python-blue-dots-opaque.jpg","keywords":["Apache Spark","Big Data","GraphX","Java","Jupyter Notebooks","Kaggle","MlLib","PySpark","Python","R","Scala","SQL"],"articleSection":["Data Science","Programming Languages","Python Development","Quant"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/www.interactivebrokers.com\/campus\/ibkr-quant-news\/pyspark-a-beginners-guide-to-apache-spark-and-big-data\/","url":"https:\/\/www.interactivebrokers.com\/campus\/ibkr-quant-news\/pyspark-a-beginners-guide-to-apache-spark-and-big-data\/","name":"PySpark \u2013 A Beginner's Guide to Apache Spark and Big Data","isPartOf":{"@id":"https:\/\/ibkrcampus.com\/campus\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.interactivebrokers.com\/campus\/ibkr-quant-news\/pyspark-a-beginners-guide-to-apache-spark-and-big-data\/#primaryimage"},"image":{"@id":"https:\/\/www.interactivebrokers.com\/campus\/ibkr-quant-news\/pyspark-a-beginners-guide-to-apache-spark-and-big-data\/#primaryimage"},"thumbnailUrl":"https:\/\/www.interactivebrokers.com\/campus\/wp-content\/uploads\/sites\/2\/2023\/02\/python-blue-dots-opaque.jpg","datePublished":"2022-11-08T22:30:00+00:00","dateModified":"2023-02-13T22:03:58+00:00","description":"PySpark is a Python library that serves as an interface for Apache Spark. Apache Spark is an open-source distributed computing engine that is used for Big Data processing.","inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.interactivebrokers.com\/campus\/ibkr-quant-news\/pyspark-a-beginners-guide-to-apache-spark-and-big-data\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.interactivebrokers.com\/campus\/ibkr-quant-news\/pyspark-a-beginners-guide-to-apache-spark-and-big-data\/#primaryimage","url":"https:\/\/www.interactivebrokers.com\/campus\/wp-content\/uploads\/sites\/2\/2023\/02\/python-blue-dots-opaque.jpg","contentUrl":"https:\/\/www.interactivebrokers.com\/campus\/wp-content\/uploads\/sites\/2\/2023\/02\/python-blue-dots-opaque.jpg","width":1000,"height":563,"caption":"PySpark \u2013 A Beginner's Guide to Apache Spark and Big Data"},{"@type":"WebSite","@id":"https:\/\/ibkrcampus.com\/campus\/#website","url":"https:\/\/ibkrcampus.com\/campus\/","name":"IBKR Campus US","description":"Financial Education from Interactive Brokers","publisher":{"@id":"https:\/\/ibkrcampus.com\/campus\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/ibkrcampus.com\/campus\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/ibkrcampus.com\/campus\/#organization","name":"Interactive Brokers","alternateName":"IBKR","url":"https:\/\/ibkrcampus.com\/campus\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/ibkrcampus.com\/campus\/#\/schema\/logo\/image\/","url":"https:\/\/www.interactivebrokers.com\/campus\/wp-content\/uploads\/sites\/2\/2024\/05\/ibkr-campus-logo.jpg","contentUrl":"https:\/\/www.interactivebrokers.com\/campus\/wp-content\/uploads\/sites\/2\/2024\/05\/ibkr-campus-logo.jpg","width":669,"height":669,"caption":"Interactive Brokers"},"image":{"@id":"https:\/\/ibkrcampus.com\/campus\/#\/schema\/logo\/image\/"},"publishingPrinciples":"https:\/\/www.interactivebrokers.com\/campus\/about-ibkr-campus\/","ethicsPolicy":"https:\/\/www.interactivebrokers.com\/campus\/cyber-security-notice\/"},{"@type":"Person","@id":"https:\/\/ibkrcampus.com\/campus\/#\/schema\/person\/b43b33f424bad38d84a7b78eb0193592","name":"Igor Radovanovic","url":"https:\/\/www.interactivebrokers.com\/campus\/author\/igor-radovanovic\/"}]}},"jetpack_featured_media_url":"https:\/\/www.interactivebrokers.com\/campus\/wp-content\/uploads\/sites\/2\/2023\/02\/python-blue-dots-opaque.jpg","_links":{"self":[{"href":"https:\/\/ibkrcampus.com\/campus\/wp-json\/wp\/v2\/posts\/182051","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/ibkrcampus.com\/campus\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/ibkrcampus.com\/campus\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/ibkrcampus.com\/campus\/wp-json\/wp\/v2\/users\/815"}],"replies":[{"embeddable":true,"href":"https:\/\/ibkrcampus.com\/campus\/wp-json\/wp\/v2\/comments?post=182051"}],"version-history":[{"count":0,"href":"https:\/\/ibkrcampus.com\/campus\/wp-json\/wp\/v2\/posts\/182051\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/ibkrcampus.com\/campus\/wp-json\/wp\/v2\/media\/182059"}],"wp:attachment":[{"href":"https:\/\/ibkrcampus.com\/campus\/wp-json\/wp\/v2\/media?parent=182051"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/ibkrcampus.com\/campus\/wp-json\/wp\/v2\/categories?post=182051"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/ibkrcampus.com\/campus\/wp-json\/wp\/v2\/tags?post=182051"},{"taxonomy":"contributors-categories","embeddable":true,"href":"https:\/\/ibkrcampus.com\/campus\/wp-json\/wp\/v2\/contributors-categories?post=182051"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}