IBKR Quant Blog


1 2 3 4 5 2 11


Quant

Getting Started with Neural Networks for Algorithmic Trading - Part III


In this post Kris will continue with the coding and implementation of a Perceptron from Scratch

Let’s now ask our perceptron to learn a slightly more difficult problem. Using the same iris data set, this time we remove the setosa species and train a perceptron to classify virginica and versicolor on the basis of their petal lengths and petal widths. When we plot these species in their feature space, we get this:

Iris

This looks like a slightly more difficult problem, as this time, the difference between the two classifications is not as clear cut. Let’s see how our perceptron performs on this data set.

We’re also going to introduce the concept of the learning rate, which is important to understand if you decide to pursue neural networks beyond the perceptron. The learning rate controls the speed with which weights are adjusted during training. We simply scale the adjustment by the learning rate: a high learning rate means that weights are subject to bigger adjustments. Sometimes this is a good thing, for example, when the weights are far from their optimal values. But sometimes this can cause the weights to oscillate back and forth between two high-error states without ever finding a better solution. In that case, a smaller learning rate is desirable, which can be thought of as fine tuning of the weights.

Finding the best learning rate is largely a trial and error process, but a useful approach is to reduce the learning rate as training proceeds. In the example below, we do that by scaling the learning rate by the inverse of the epoch number.

Here’s a plot of our error rate after training in this manner for 400 epochs:

Epoch

You can see that training proceeds much less smoothly and takes a lot longer than last time, which is a consequence of the classification problem being more difficult. Also note that the error rate is never reduced to zero, that is, the perceptron is never able to perfectly classify this data set. Here’s a plot of the decision boundary, which demonstrates where the perceptron makes the wrong predictions:

Iris-2

Here’s the code for this perceptron:

perceptron

Visit Robot Wealth website to Download the code and data used in this post

 

Learn more about Robot Wealth here: https://robotwealth.com/

This article is from Robot Wealth and is being posted with Robot Wealth’s permission. The views expressed in this article are solely those of the author and/or Robot Wealth and IB is not endorsing or recommending any investment or trading discussed in the article. This material is for information only and is not and should not be construed as an offer to sell or the solicitation of an offer to buy any security. To the extent that this material discusses general market activity, industry or sector trends or other broad-based economic or political conditions, it should not be construed as research or investment advice. To the extent that it includes references to specific securities, commodities, currencies, or other instruments, those references do not constitute a recommendation by IB to buy, sell or hold such security. This material does not and is not intended to take into account the particular financial conditions, investment objectives or requirements of individual customers. Before acting on this material, you should consider whether it is suitable for your particular circumstances and, as necessary, seek professional advice.

 


16849




Quant

Are Drones A Legitimate Source of Alpha?


By Tony Fong, Author at Quandl Blog

With predictions claiming that the global UAV drones market could reach up to USD 127 billion by 2020, it is clear that the drones industry is tapping into the world’s demand for more data—at faster speeds and greater precision. While many still associate drone technology with the military market, which indeed accounted for almost three-quarters of the US market total in 2015, demand for commercial drone technology has been slowly catching up in the last decade, with a predicted market size of USD 1.05 billion by 2020 and expected growth at a CAGR of 57%.1

Established and Emerging Markets

The agriculture sector is by far the largest contributor to the recent surge in the commercial drones market, making up 76% of the revenue generated by the commercial UAV industry in 2015. In precision farming, UAVs assist by apportioning pesticides and fertilizers to only areas that require them, resulting in significant cost savings. Technavio’s market research study adds that in agriculture, “a $300 UAV can be used to check for disease and irrigation needs, while the same task performed with manned aircraft will cost at least $1,000 per hour.”2

The insurance industry is one of the latest markets to tap into the potential of drone aerial data collection. According to Cognizant, UAVs can improve productivity by up to 50%, by helping adjustors assess damage in difficult to reach areas both at the time of binding the account and after a claim is made.

Technological Advancements

Compared to satellite technology, which has been amassing valuable data for at least half a century, modern drones have been on the aerial data collection scene for only about two decades. Indeed, prior to 2005, the Federal Aviation Administration (FAA) barely differentiated between drones and model aircrafts flown by hobbyists.

But in the last several years the FAA has come a long way, developing sensible regulations that will make way for more widespread industry adoption of UAVs. This according to Murray Wu, CEO of Reforges, a Toronto-based startup offering drone fleets for hire to commercial businesses.

Much as the boom in satellite technology has provided stakeholders with actionable data insights, drones appear poised to do the same.

Unlike satellites, however, which can be finicky, limited by slow imagery refresh rates, (and expensive to purchase, maintain, and upgrade) drones are able to capture data in non-ideal conditions, such as through rain, clouds, darkness, or even concrete walls. They are versatile, modular, and relatively inexpensive compared to other aircraft, unmanned and otherwise.

This is not to say that drones will replace satellites altogether. In many ways, UAVs can be described as micro-satellites, collecting data on a smaller scale and with greater agility. Many strategies still benefit from the macro-perspective that satellite data provides.

Some also point to the flaws in current drone technology – especially as they relate to accurate data gathering. Jonathan Rupprecht is a lawyer who specializes in drones, and a pilot. He authored Drones: Their Many Civilian Uses and the U.S. Laws Surrounding Them. He says that the drone industry needs to communicate more with the aviation industry, because they have figured out things

like aerodynamic efficiency, which would mean better flight times for drones, and GPS, which would reduce drone “fly-aways” due to errant GPS signals. Rupprecht also points out that drones use LIDAR, a laser range finder that doesn’t have “very accurate or calibrated inertia measuring units (IMU) to measure changes in movement.” This can be somewhat addressed by using cameras, but the concern remains that investors could be trading on inaccurate data.

Still, commodities traders like Anurag Bhatia from Minance Capital claim to already be using drones to predict agricultural commodity prices such as wheat, cotton, sugar and maize. It’s undoubtedly early, but where there’s a will, there’s a way.

From Hardware to Software

The drone market can be broken into four major product types: 1) the UAV itself, which includes large and small fixed-wing aircrafts, rotorcrafts, hybrid and other vehicle methods of lift, such as lighter-than-air; 2) payloads, such as cameras and other sensors, as well as other separately sold parts; 3) ground stations and controls; and 4) software/data analytics, which focuses on the collection, storage, and processing of data. It is this last category, of course, that holds promise for alpha generation.

Today, more and more companies are moving to the data processing end of the industry, introducing software to help operators process and manage the data that drones collect. Ciara Bracken-Roche, a PhD candidate at Queen’s University and a researcher in drone technologies and data collection, describes the shift: “Ten years ago, the main focus of many UAV developers and operators was to get the technology to where it needed to be: trying to reduce the size of the UAV; improving the payload; and making drones more autonomous with capabilities like sense-and-avoid. In the last couple of years, there has been a move towards data collection and analytics optimization.” Here are several key players that have been making an impact in the world of UAV data analytics:

Sky-Futures

The UK-based Sky-Futures was one of the first drone firms to receive money from Airware’s Commercial Drone Fund. Sky-Futures focuses on oil and gas inspection services (onshore and offshore) and its cloud application, the Sky Futures Inspection Portal, hosts the HD video, still and thermal imagery that its drones produce during inspections.

RedBird

The France-based RedBird, a pioneer in drone data analytics for mining, quarrying, and construction industries was founded in 2013. Its cloud platform Cardinal, optimizes resources and ensures safety. Like Sky-Futures, RedBird was one of the first recipients of Airware’s Commercial Drone Fund. In FY2014-2015, the company received $3.19 million funding from three different investors and partnered with Caterpillar, a leading manufacturer of construction and mining equipment.

In September 2016, Airware announced the acquisition of Redbird. “Construction and excavation sites … must be managed and operated digitally in order to compete in today’s markets. Commercial drone technology brings these operations into the digital world while offering faster, cheaper and higher-quality data and analytics,” said Jonathan Downey, founder & CEO of Airware. “By … incorporating Redbird’s powerful analytics tools on top of the Airware platform, we are helping enterprises make this transition into the digital age.”

Gamaya

Gamaya, a Swiss data analytics startup that currently focuses on the soybean, corn, and sugarcane industries, recently raised $3.2 million in a Series A financing round, with the lofty goal of mitigating food scarcity through sustainable agricultural processes. The investors that participated in the round include chairman and former CEO of Nestlé, Peter Brabeck-Letmathe; the Sandoz Foundation; and VI Partners, a venture capital firm.

Gamaya uses its patented hyperspectral imaging for precision farming applications. The data captured by its drones includes alerts for disease, pests, and yield predictions. The imaging technology is so sophisticated that it can detect what is invisible to the human eye and then translate the data into action maps and recommendations for farmers. “Our future is dependent on whether we will be able to leverage technology and innovation in order to dramatically improve the efficiency of our food production methodology,” says Gamaya CEO Yosef Akhtman. “Our solution helps to increase production efficiency by providing farmers with full situational awareness of their farmland and crops.”

Kespry

Kespry, a Silicon Valley startup, builds commercial drone systems that allow its clients to capture, view, and analyze aerial imagery and survey data. The company recently partnered with NVIDIA on a deep-learning module to advance its drone system’s autonomous features. With the resulting Jetson TX1 supercomputing module, Kespry’s drone systems can now analyze data in real time. In June 2016, Kespry announced the close of a Series B equity-financing round of $16 million.

Drone Data as a Source of Alpha: It’s Early

The sky really is the limit when it comes to the potential for drone data usage. There is no telling what industry UAVs will revolutionize next. Consequently, it is no surprise to hear that there is a lot of interest in the idea of selling data to investors and hedge funds. One of Reforges’s investors, for instance, is a subsidiary of a big financial institution that is looking to eventually sell drone data to capital markets. Many drone manufacturers jump at the idea of monetizing the data captured by their UAVs. If there’s a market for their data, why not sell it?

Companies like Kespry, however, are not so sure. “The industry is still at an embryonic stage,” remarks VP Marketing at Kespry, David Shearer, “Our focus is more on providing our clients with what they want: a vertically integrated service that focuses on field to finish.” Many UAV data companies worry that the selling of data could compromise the trust and relationship that is necessary for the client-building phase.

Ultimately, there is no real consensus on who owns the data that UAVs collect: the clients who commission the data or the drone companies who collect it or the property owners whose area is being observed. Lawyer Rupprecht points out that the laws for data collection differ by state. In Florida, for example, you could get sued for flying a drone over a private property and gathering data that could not have been observed on the ground at eye level. So much for orange futures.

Consequently, if drone companies want to sell the data that they have collected on behalf of their clients, they may have to reserve the rights during the initial negotiation phase. For example, a UAV company could theoretically offer its clients a discount on their services in exchange for proprietary rights. (AgTech companies are doing this too: Farmobile offers “profit-sharing” to customers whose data gets resold). According to Ian Glenn, Founder and CEO/CTO of ING Robotic Aviation Inc., “This could save the client money and provide the UAV company with more flexibility.”

Tammer has said in the past: we’re already sharing our data every time we log in to Google. The advertising economy is based on this premise. Drones are collecting valuable data – we know that. And they offer significant benefits over other earth observation techniques. But like AgTech, it will be difficult to leverage the data until a paradigm shift occurs in data sharing practices.

 

1 Technavio, Global Commercial-Purpose Drones Market: 2015-2019, 5.

2 Technavio, Global Commercial-Purpose Drones Market: 2015-2019, 30.

 

 

Learn more about Quandl here: https://www.quandl.com

Quandl’s mission is to extract value from the world's data.

We believe that data is one of the most important resources of the 21st century. It is ubiquitous. Data now arises from virtually everything humans and businesses do, every minute of every day. But how much of it has tangible value to capital markets? That is what we ask ourselves every day.

We want to inspire customers to make new discoveries and incorporate them into trading strategies. We find opportunities in places people otherwise wouldn’t think about. We believe there are new and better ways to understand the complex information that creates markets and market movement.

Quandl’s diverse team members have backgrounds in finance, technology, data science and astrophysics, just to name a few. Our team is purpose-built to extract signal from a world increasingly full of noise. We uncover valuable data that is difficult to find, shape it and share it with our clients.

We believe that data, and alternative data in particular, is going to become the primary driver of active investment performance over the next decade.

 

 

This article is from Quandl and is being posted with Quandl’s permission. The views expressed in this article are solely those of the author and/or Quandl and IB is not endorsing or recommending any investment or trading discussed in the article. This material is for information only and is not and should not be construed as an offer to sell or the solicitation of an offer to buy any security. To the extent that this material discusses general market activity, industry or sector trends or other broad-based economic or political conditions, it should not be construed as research or investment advice. To the extent that it includes references to specific securities, commodities, currencies, or other instruments, those references do not constitute a recommendation by IB to buy, sell or hold such security. This material does not and is not intended to take into account the particular financial conditions, investment objectives or requirements of individual customers. Before acting on this material, you should consider whether it is suitable for your particular circumstances and, as necessary, seek professional advice.


17419




Quant

Machine Learning for Trading


This video covers the basic concepts of Machine Learning and how it can be applied in trading.

 

Learn more about Quantra by QuantInsti here https://quantra.quantinsti.com/

This article is from Quantra by QuantInsti and is being posted with their permission. The views expressed in this article are solely those of the author and/or Quantra by QuantInsti and IB is not endorsing or recommending any investment or trading discussed in the article. This material is for information only and is not and should not be construed as an offer to sell or the solicitation of an offer to buy any security. To the extent that this material discusses general market activity, industry or sector trends or other broad-based economic or political conditions, it should not be construed as research or investment advice. To the extent that it includes references to specific securities, commodities, currencies, or other instruments, those references do not constitute a recommendation by IB to buy, sell or hold such security. This material does not and is not intended to take into account the particular financial conditions, investment objectives or requirements of individual customers. Before acting on this material, you should consider whether it is suitable for your particular circumstances and, as necessary, seek professional advice.


17369




Quant

Six Reasons to Learn R For Business


By Matt Dancho, Business Science

Data science for business (DS4B) is the future of business analytics yet it is really difficult to figure out where to start. The last thing you want to do is waste time with the wrong tool. Making effective use of your time involves two pieces: (1) selecting the right tool for the job, and (2) efficiently learning how to use the tool to return business value.


REASON 1: R HAS THE BEST OVERALL QUALITIES

DS4B

 

There are a number of tools available business analysis/business intelligence (with DS4B being a subset of this area). Each tool has its pros and cons, many of which are important in the business context. We can use these attributes to compare how each tool stacks up against the others! We did a qualitative assessment using several criteria:

  • Business Capability (1 = Low, 10 = High)
  • Ease of Learning (1 = Difficult, 10 = Easy)
  • Cost (Free/Minimal, Low, High)
  • Trend (0 = Fast Decline, 5 = Stable, 10 = Fast Growth)

Further discussion on the assessment is available on Business Science website.

What we saw was particularly interesting. A trendline developed exposing a tradeoff between learning curve and DS4B capability rating. The most flexible tools are more difficult to learn but tend to have higher business capability. Conversely, the “easy-to-learn” tools are often not the best long-term tools for business or data science capability. Our opinion is go for capability over ease of use.

Of the top tools in capability, R has the best mix of desirable attributes including high data science for business capability, low cost, and it’s growing very fast. The only downside is the learning curve. The rest of the article explains why R is so great for business.

REASON 2: R IS DATA SCIENCE FOR NON-COMPUTER SCIENTISTS

If you are seeking high-performance data science tools, you really have two options: R or Python. When starting out, you should pick one. It’s a mistake to try to learn both. Your choice comes down to what’s right for you. The difference between the R and Python has been described in numerous infographics and debates online, but the most overlooked reason is person-programming language fit. Don’t understand what we mean? Let’s break it down.

  • Fact 1: Most people interested in learning data science for business are not computer scientists. They are business professionals, non-software engineers (e.g. mechanical, chemical), and other technical-to-business converts. This is important because of where each language excels.
     
  • Fact 2: Most activities in business and finance involve communication. This comes in the form of reports, dashboards, and interactive web applications that allow decision makers to recognize when things are not going well and to make well-informed decisions that improve the business.

Now that we recognize what’s important, let’s learn about the two major players in data science.

ABOUT PYTHON

Python is a general service programming language developed by software engineers that has solid programming libraries for math, statistics and machine learning. Python has best-in-class tools for pure machine learning and deep learning, but lacks much of the infrastructure for subjects like econometrics and communication tools such as reporting. Because of this, Python is well-suited for computer scientists and software engineers.

ABOUT R

R is a statistical programming language developed by scientists that has open source libraries for statistics, machine learning, and data science. R lends itself well to business because of its depth of topic-specific packages and its communication infrastructure. R has packages covering a wide range of topics such as econometrics, finance, and time series. R has best-in-class tools for visualization, reporting, and interactivity, which are as important to business as they are to science. Because of this, R is well-suited for scientists, engineers and business professionals.

WHAT SHOULD YOU DO?

Don’t make the decision tougher than what it is. Think about where you are coming from:

  • Are you a computer scientist or software engineer? If yes, you might consider choosing Python.
     
  • Are you an analytics professional or mechanical/industrial/chemical engineer looking to get into data science? If yes, you might consider choosing R.


Think about what you are trying to do:

  • Are you trying to build a self-driving car? If yes, you might consider choosing Python.
     
  • Are you trying to communicate business analytics throughout your organization? If yes, you might consider choosing R.

REASON 3: LEARNING R IS EASY WITH THE TIDYVERSE

Learning R used to be a major challenge. Base R was a complex and inconsistent programming language. Structure and formality was not the top priority as in other programming languages. This all changed with the “tidyverse”, a set of packages and tools that have a consistently structured programming interface.

When tools such as dplyr and ggplot2 came to fruition, it made the learning curve much easier by providing a consistent and structured approach to working with data. As Hadley Wickham and many others continued to evolve R, the tidyverse came to be, which includes a series of commonly used packages for data manipulation, visualization, iteration, modeling, and communication. The end result is that R is now much easier to learn.

tidyverse

Source: tidyverse.org

R continues to evolve in a structured manner, with advanced packages that are built on top of the tidyverse infrastructure. A new focus is being placed on modeling and algorithms, which we are excited to see.
Further, the tidyverse is being extended to cover topical areas such as text (tidytext) and finance (tidyquant). For newcomers, this should give you confidence in selecting this language. R has a bright future.

REASON 4: R HAS BRAINS, MUSCLE, AND HEART

Saying R is powerful is actually an understatement. From the business context, R is like Excel on steroids! But more important than just muscle is the combination of what R offers: brains, muscle, and heart.

R HAS BRAINS

R implements cutting-edge algorithms including:

  • H2O (h2o) - High-end machine learning package
     
  • Keras/TensorFlow (kerastensorflow) - Go-to deep learning packages
     
  • xgboost - Top Kaggle algorithm
     
  • And many more!

These tools are used everywhere from AI products to Kaggle Competitions, and you can use them in your business analyses.

R HAS MUSCLE

R has powerful tools for:

  • Vectorized Operations - R uses vectorized operations to make math computations lightning fast right out of the box
     
  • Loops (purrr)
     
  • Parallelizing operations (parallelfuture)
     
  • Speeding up code using C++ (Rcpp)
     
  • Connecting to other languages (rJavareticulate)
     
  • Working With Databases - Connecting to databases (dbplyrodbcbigrquery)
     
  • Handling Big Data - Connecting to Apache Spark (sparklyr)
     
  • And many more!

R HAS HEART

We already talked about the infrastructure, the tidyverse, that enables the ecosystem of applications to be built using a consistent approach. It’s this infrastructure that brings life into your data analysis. The tidyverse enables:

  • Data manipulation (dplyrtidyr)
     
  • Working with data types (stringr for strings, lubridate for date/datetime, forcats for categorical/factors)
     
  • Visualization (ggplot2)
     
  • Programming (purrrtidyeval)
     
  • Communication (Rmarkdownshiny)

REASON 5: R IS BUILT FOR BUSINESS

Two major advantages of R versus every other programming language is that it can produce business-ready reports and machine learning-powered web applications. Neither Python or Tableau or any other tool can currently do this as efficiently as R can. The two capabilities we refer to are rmarkdown for report generation and shiny for interactive web applications.

RMARKDOWN

Rmarkdown is a framework for creating reproducible reports that has since been extended to building blogs, presentations, websites, books, journals, and more. It’s the technology that’s behind this blog, and it allows us to include the code with the text so that anyone can follow the analysis and see the output right with the explanation. What’s really cool is that the technology has evolved so much. Here are a few examples of its capability:

SHINY

Google Trend Index

 

Source: shiny.rstudio.com

Shiny is a framework for creating interactive web applications that are powered by R. Shiny is a major consulting area for us as four of five assignments involve building a web application using shiny.
It’s not only powerful, it enables non-data scientists to gain the benefit of data science via interactive decision making tools. Here’s an example of a Google Trend app built with shiny.

REASON 6: R COMMUNITY SUPPORT

Being a powerful language alone is not enough. To be successful, a language needs community support. We’ll hit on two ways that R excels in this respects: CRAN and the R Community.

CRAN: COMMUNITY-PROVIDED R PACKAGES

CRAN is like the Apple App store, except everything is free, super useful, and built for R. With over 14,000 packages, it has most everything you can possibly want from machine learning to high-performance computing to finance and econometrics! The task views cover specific areas and are one way to explore R’s offerings. CRAN is community-driven, with top open source authors such as Hadley Wickham and Dirk Eddelbuettel leading the way. Package development is a great way to contribute to the community especially for those looking to showcase their coding skills and give back!

COMMUNITY SUPPORT

You begin with R because of its capability, you stay with R because of its community. The R Community is the coolest part. It’s tight-knit, opinionated, fun, silly, and highly knowledgeable… all of the things you want in a high performing team. Visit Business Science website for a list of Conferences, Social Channels, and Meetups

CONCLUSION

R has a wide range of benefits making it our obvious choice for Data Science for Business (DS4B). That’s not to say that Python isn’t a good choice as well, but, for the wide-range of needs for business, there’s nothing that compares to R. In this article we saw why R is a great choice. In the next article we’ll show you how to learn R in 12 weeks.

 

About the Author

Matt Dancho is an R developer and R/Finance 2017 speaker. He is the Founder and CEO of Business Science.

Business Science specializes in “ROI-driven data science”. Their focus is machine learning and data science in business and financial applications. Business Science team builds web applications and automated reports to put machine learning in the hands of decision makers. Visit the Business Science University to learn more about their virtual workshops on how to apply data science and machine learning in real-world business applications. They focus on modeling problems, creating interactive data products, and distributing solutions within an organization.


This article is from Matt Dancho and is being posted with Matt Dancho’s permission. The views expressed in this article are solely those of the author and/or Matt Dancho and IB is not endorsing or recommending any investment or trading discussed in the article. This material is for information only and is not and should not be construed as an offer to sell or the solicitation of an offer to buy any security. To the extent that this material discusses general market activity, industry or sector trends or other broad-based economic or political conditions, it should not be construed as research or investment advice. To the extent that it includes references to specific securities, commodities, currencies, or other instruments, those references do not constitute a recommendation by IB to buy, sell or hold such security. This material does not and is not intended to take into account the particular financial conditions, investment objectives or requirements of individual customers. Before acting on this material, you should consider whether it is suitable for your particular circumstances and, as necessary, seek professional advice.


17336




Quant

Blockchain FAQ #1: What is a fork in the Blockchain?


Blockchain

 

In the standard software industry parlance, a software fork is just a copy of an existing project. Considering the current pre-eminence of open-source software development, a software fork can very often be carried out without explicit permission from the developer of the original project. In case of the blockchain, it is different. A blockchain fork is when a blockchain splits into two separate chains.

First, let’s revisit blockchain in an elementary manner. In this technology, a network of computers, commonly called “nodes”, maintain a shared, verifiable, and immutable record of data. Two or more block records, commonly called “blocks”, are linked together while adhering to a predetermined protocol in this distributed database. Each node has the complete information of the blockchain, and each node can be considered as a ledger. This is why blockchain is also often referred to as “Distributed Ledger Technology” (DLT). Unlike one central server, (or any other method of ensuring every update goes through one central point of authority) every node in a blockchain has the same authority. Every node can add a new block if consensus on the validity of the transaction can be achieved. To achieve this consensus, a node has to carry out significant number-crunching operation at high speed to solve a cryptographic puzzle and provide evidence of that. This evidence is called “proof of work” (POW). In summary, to add a new block, a node has to achieve consensus, and it is here that the concept of fork enters our discussion.

Take for example a blockchain where the existing protocol mandates blocks of 1 megabyte (MB) size. “Miners” (i.e., a combination of user, software, and specialized hardware) in this blockchain will continue to follow this 1 MB mandate so that they can get rewarded for their mining. What happens when a miner decides to create a 2 MB block instead? Initially, the other miners will ignore this block since it does not conform to the rules. Remember that the other miners are looking for rewards, for which they will need to adhere to the rules. Let’s assume that our adventurous miner continues to create 2 MB blocks on top of the earlier 2 MB blocks he/she had created. If the other miners continue to ignore these blocks, then eventually the chain of the 2 MB blocks will be the shortest chain and will be orphaned by the network. If this is just one miner trying to break the rules, then the transgression will have a negligible economic impact in terms of whatever that blockchain is dealing with (e.g., BitcoinEthereum etc.). Note that at this point, there is already a split in the network (i.e., there are now multiple chains, one with 1 MB blocks and another shorter chain of 2 MB blocks). A fork has occurred!

What if 25% of the miners in the network feel that 2 MB blocks should be created instead of 1 MB ones and start adding blocks on top of the 2 MB chain of blocks? Now we have a fork of a real economic significance!

Now, to actually change the rules in this blockchain and allow 2 MB blocks instead of 1 MB, we will need a “hard fork”. A hard fork is when the software is upgraded to introduce a new rule, and as a result, the network will no longer be compatible with the older software. If a miner continues to run the old software, then he/she will see the new transactions as invalid. In order to mine valid blocks in the new chain, an upgradation is must for that node.

When the Ethereum community was split over their decentralized autonomous organization (DAO) and became two entities (Ethereum and Ethereum Classic); it was a split over rules, and it was a hard fork.

On the contrary, let’s consider that some miners wanted to create 500 kilobytes (KB) blocks instead of the 1 MB ones in our blockchain. If this change is done, the nodes that did not upgrade will continue to see the 500 k blocks as valid, since 500 k is less than 1 MB. However, if these non-upgraded nodes continue to mine blocks, their new blocks will be rejected by the upgraded nodes. This is “soft fork” and to implement this majority of hash power, the majority of the computing power in the network needs to support the change.

There can also be unplanned forks, which are usually the result of not understanding the software complexity or changes made without sufficient planning, review, and testing.

In summary, there can be different types of forks in a blockchain network:

  • Soft forks – these invalidate previously valid blocks, and can cause security risks for non-upgrading nodes;
  • Hard forks – make previously invalid blocks valid, and the non-upgrading nodes become incompatible;
  • Unplanned forks – results of inadequately understood software complexity or rushed changes, and can cause serious vulnerability in the network.

Learn more about Byte Academy here.

This article is from Byte Academy and is being posted with Byte Academy’s permission. The views expressed in this article are solely those of the author and/or Byte Academy and IB is not endorsing or recommending any investment or trading discussed in the article. This material is for information only and is not and should not be construed as an offer to sell or the solicitation of an offer to buy any security. To the extent that this material discusses general market activity, industry or sector trends or other broad-based economic or political conditions, it should not be construed as research or investment advice. To the extent that it includes references to specific securities, commodities, currencies, or other instruments, those references do not constitute a recommendation by IB to buy, sell or hold such security. This material does not and is not intended to take into account the particular financial conditions, investment objectives or requirements of individual customers. Before acting on this material, you should consider whether it is suitable for your particular circumstances and, as necessary, seek professional advice.


17309




1 2 3 4 5 2 11

Disclosures

We appreciate your feedback. If you have any questions or comments about IBKR Quant Blog please contact ibkrquant@ibkr.com.

The material (including articles and commentary) provided on IBKR Quant Blog is offered for informational purposes only. The posted material is NOT a recommendation by Interactive Brokers (IB) that you or your clients should contract for the services of or invest with any of the independent advisors or hedge funds or others who may post on IBKR Quant Blog or invest with any advisors or hedge funds. The advisors, hedge funds and other analysts who may post on IBKR Quant Blog are independent of IB and IB does not make any representations or warranties concerning the past or future performance of these advisors, hedge funds and others or the accuracy of the information they provide. Interactive Brokers does not conduct a "suitability review" to make sure the trading of any advisor or hedge fund or other party is suitable for you.

Securities or other financial instruments mentioned in the material posted are not suitable for all investors. The material posted does not take into account your particular investment objectives, financial situations or needs and is not intended as a recommendation to you of any particular securities, financial instruments or strategies. Before making any investment or trade, you should consider whether it is suitable for your particular circumstances and, as necessary, seek professional advice. Past performance is no guarantee of future results.

Any information provided by third parties has been obtained from sources believed to be reliable and accurate; however, IB does not warrant its accuracy and assumes no responsibility for any errors or omissions.

Any information posted by employees of IB or an affiliated company is based upon information that is believed to be reliable. However, neither IB nor its affiliates warrant its completeness, accuracy or adequacy. IB does not make any representations or warranties concerning the past or future performance of any financial instrument. By posting material on IB Quant Blog, IB is not representing that any particular financial instrument or trading strategy is appropriate for you.