After you do feature engineering, feature importance is a key step before deploying a strategy backtesting code. Boruta-Shap comes as a viable source for that purpose. However, this algorithm might take a lot of time to run with large datasets. This unique article provides us with an estimation of the mentioned algorithm using CPU parallelism and GPU to make it run faster. Code will be implemented using the XGBoost library and futures library for CPU parallelism.<\/p>\n\n\n\n
We will cover:<\/p>\n\n\n\n
The Boruta-Shap algorithm is a good technique for feature selection, especially in machine learning<\/a> and data science<\/a> applications, is the Boruta-Shap algorithm. Boruta-Shap combines the Boruta feature selection process with the Shapley values to enhance feature importance assessment.<\/p>\n\n\n\n The Boruta-Shap algorithm works in the following way:<\/p>\n\n\n\n The Boruta-Shap algorithm has the following benefits.<\/p>\n\n\n\n Visit QuantInsti website to watch this clip: “Industry expert and renowned author, Dr. Ernest Chan talks about Financial Data Science & Feature Engineering and shares his knowledge:”<\/em> https:\/\/blog.quantinsti.com\/boruta-shap-gpu-python\/<\/a><\/em> <\/p>\n\n\n\n Despite Boruta-Shap’s strength, its computational cost can be high, particularly for large datasets with many characteristics. To solve this, I’ve included a Boruta-Shap code that uses the CPU and GPU in tandem to expedite the Boruta-Shap’s execution. Cool, right?<\/p>\n\n\n\n This approach drastically cuts computation time by effectively allocating the workload and utilizing the parallel processing powers of both CPUs and GPUs.<\/p>\n\n\n\n Let’s dissect the code. Depending on the number of cores available in your CPU, the code will group the number of trials in buckets and each bucket will be run in parallel. We use a modified version of the code provided by Moosa Ali (2022), who implements the CPU-based algorithm.<\/p>\n\n\n\n Let\u2019s code!<\/p>\n\n\n\n import_libraries.py <\/a>hosted with \u2764 by GitHub<\/a><\/p>\n\n\n\n The following function is responsible for computing the minimum number of trials needed as a threshold to accept an input feature as a selected feature based on the probability mass function (pmf) and a significance level. It iterates over the pmf and accumulates the probabilities<\/a> until the cumulative probability exceeds the significance level.<\/p>\n\n\n\n get_tail_items_func.py <\/a>hosted with \u2764 by GitHub<\/a><\/p>\n\n\n\n The next function selects features based on the number of hits they receive during the trials. It categorizes features into two zones:<\/p>\n\n\n\n choose_features_func.py <\/a>hosted with \u2764 by GitHub<\/a><\/p>\n\n\n\n The following last function is the main function implementing the Boruta-Shap algorithm. It takes input data X and target variable y, along with optional parameters such as trials, workers, significance_level, and seed.<\/p>\n\n\n\n Find below what the function does:<\/p>\n\n\n\n gpu_cpu_based_boruta_shap_func.py <\/a>hosted with \u2764 by GitHub<\/a><\/p>\n\n\n\n References<\/strong><\/p>\n\n\n\n You have learned how to create the Boruta-Shap algorithm using both the CPU and GPU. You\u2019ll see a great difference, compared with using only the CPU, if you use a dataframe with many observations. Besides, the higher the number of threads and cores, the better the parallelism and the quicker the loop will run.<\/p>\n\n\n\n What\u2019s next? You would ask.<\/em> In case you want to learn more about machine learning, keep track of this learning track<\/a>! You\u2019ll learn the basics of machine learning in finance.<\/p>\n\n\n\n Now that you’ve grasped the power of Boruta Shap for identifying key features, you might be wondering how to put it into practice for real-world problems. Here’s where things get exciting! This Machine Learning & Deep Learning for Trading<\/a> course by Quantra helps you learn these techniques for building advanced trading strategies. You’ll not only learn the theory behind Boruta Shap but also gain hands-on experience implementing it to select the most impactful features for your own trading algorithms.<\/p>\n\n\n\n It’s the perfect next step to turn your newfound knowledge into action!
\n\n\n\nHow the Boruta-Shap algorithm works<\/h2>\n\n\n\n
\n
\n\n\n\nSignificance of Boruta-Shap<\/h2>\n\n\n\n
\n
Accelerating Boruta-Shap Algorithm<\/h2>\n\n\n\n
\n\n\n\nA CPU-and-GPU-based algorithm to run quicker the Boruta-Shap algorithm<\/h2>\n\n\n\n
# Import the necessary libraries\nimport scipy as sp\nimport numpy as np\nimport pandas as pd\nimport shap\nfrom xgboost import XGBRFClassifier\nfrom xgboost import XGBRFRegressor\nfrom sklearn.preprocessing import LabelEncoder\nfrom concurrent import futures<\/pre>\n\n\n\n
# Set the minimum number of trials as a threshold number to accept an input feature as a selected feature\ndef get_tail_items(pmf, significance_level=0.05):\n # Set total to zero\n total = 0\n # Create a loop based on the probability mass function\n for i, x in enumerate(pmf):\n # Increment the total variable with the probability \u201cx\u201d of i\n total += x\n # If total is higher than the significance level\n if total >= significance_level:\n # Break the code\n break\n # Return i\n return i<\/pre>\n\n\n\n
\n
# select features from n number of trials\ndef choose_features(feature_hits, TRIALS, thresh):\n # Define the boundaries for the green zone\n # Define the green zone threshold\n green_zone_thresh = TRIALS - thresh\n # Define the blue zone upper threshold\n blue_zone_upper = green_zone_thresh\n # Define the blue zone lower threshold\n blue_zone_lower = thresh\n\n\n # Select the input features as green whenever their hits are higher than the green zone threshold\n green_zone = [key for key, value in feature_hits.items() if value >= green_zone_thresh]\n # Select the input features as blue whenever their hits are between the blue zone lower threshold and the blue zone upper threshold \n blue_zone = [key for key, value in feature_hits.items() if (value >= blue_zone_lower and value < blue_zone_upper)]\n return green_zone, blue_zone<\/pre>\n\n\n\n
\n
def boruta_shap_algorithm(X, y, trials=20, workers=2, significance_level=0.05, seed=2024):\n # Set the seed\n np.random.seed(seed) \n\n\n # Assert that the number of samples of both data match\n assert X.shape[0] == y.shape[0], \"X and y dimensions don't coincide\"\n\n\n # Set a dictionary to save the number of hits for each feature\n features_hits = {feature:0 for feature in X.columns}\n\n\n # Create the names of all the features shuffled\n shuffled_col_names = [str(column+'_shuffle') for column in X.columns]\n\n\n # Set the train and test X data\n X_train, X_test = X.iloc[:int(0.8*len(X))], X.iloc[int(0.8*len(X)):]\n\n\n # Set the label enconder object\n le = LabelEncoder()\n\n\n # Transform the y series to a prediction features useful for the machine-learning model\n label_encoded = le.fit_transform(y)\n\n\n # Transform the encoded label into a Pandas series\n y = pd.Series(data=label_encoded, index=y.index, name='y')\n\n\n # Set the y train data\n y_train = y.iloc[:int(0.8*len(y))]\n \n # define the model\n classifier = XGBRFClassifier(n_estimators=100, subsample=1, colsample_bynode=1, tree_method='gpu_hist', random_state=seed) \n\n\n # Define a function to compute the number of times the features\n def features_hits_func(trial):\n # Set the seed for the trial\n np.random.seed(seed+trial)\n\n\n # Set the X train data for the shuffled features\n X_shuffle_train = X_train.apply(np.random.permutation)\n # Set the names for the X train shuffled features\n X_shuffle_train.columns = shuffled_col_names\n # Set the X-test data for the shuffled features\n X_shuffle_test = X_test.apply(np.random.permutation)\n # Set the names for the X-test shuffled features\n X_shuffle_test.columns = shuffled_col_names\n\n\n # Set the whole input features for the Boruta-Shap algorithm training\n X_boruta_train = pd.concat([X_train, X_shuffle_train], axis=1)\n # Set the whole input features for the Boruta-Shap algorithm test data\n X_boruta_test = pd.concat([X_test, X_shuffle_test], axis=1)\n\n\n # Fit the model\n model = classifier.fit(X_boruta_train, y_train)\n\n\n # Set the explainer object\n explainer = shap.TreeExplainer(model)\n\n\n # Get the Shap values for each feature\n shap_values = explainer.shap_values(X_boruta_test)\n\n\n # Set the mean value of each feature's Shap values\n features_importance = np.array(np.abs(shap_values).mean(0))\n # Set a dataframe with the above features' importance\n features_importance_df = pd.DataFrame(data=features_importance, index=X_boruta_test.columns, columns=['Values'])\n\n\n # Subset the feature importance dataframe with the non-shuffled features\n feature_imp_X = features_importance_df.iloc[:len(X.columns)]\n # Subset the feature importance dataframe with the shuffled features\n feature_imp_shuffled = features_importance_df.iloc[len(X.columns):]\n\n\n # Add one hit in case the feature is better than the best Shap value of all the shuffled features\n for feature in feature_imp_X.index:\n features_hits[feature] += int(feature_imp_X.loc[feature,'Values'] > feature_imp_shuffled['Values'].max())\n\n\n # Define a function to run multiple trials as per the maximum number of cores available in your CPU\n def multithreading_loop(function, params_list):\n # Set the number of lists we'll have as per the number of cores\n num_lists = int(np.floor(len(params_list)\/workers))\n # Set the params list to be used to loop\n params_list_for_loop = params_list[:int(num_lists*workers)]\n # If the number of trials in the above list is higher than the num_lists\n if len(params_list)>int(num_lists*workers):\n # Create the last params list to be used to multithread the computations\n last_params_list = params_list[int(num_lists*workers):]\n\n\n # For each list of trials\n for i in range(0,num_lists):\n # Use the number of cores for the futures library executor\n with futures.ThreadPoolExecutor(workers) as executor:\n # Run the features_hits_func function to compute the hits in parallel\n list(executor.map(function, params_list_for_loop[int(workers*i):int(workers*(i+1))]))\n # Once you finish the above, run the last trials to be computed in parallel\n if len(params_list)>int(num_lists*workers):\n # Use the number of cores for the futures library executor\n with futures.ThreadPoolExecutor(len(last_params_list)) as executor:\n # Run the features_hits_func function to compute the hits in parallel\n list(executor.map(function, last_params_list))\n\n\n # Set the range for the number of trails as a list\n trails_list = [*range(trials)]\n\n\n # Run the loop to compute the trails in parallel in buckets\n multithreading_loop(features_hits_func, trails_list) \n \n # Calculate the probability mass function: Get the Binomial distribution in \"trials\" number of buckets\n pmf = [sp.stats.binom.pmf(x, trials, .5) for x in range(trials + 1)]\n\n\n # Set the minimum number of trials as the threshold to classify an input feature as a selected feature\n thresh = get_tail_items(pmf, significance_level)\n \n # green are the accepted features, blue are the tentative features\n green, blue = choose_features(features_hits, trials, thresh)\n\n\n # If there are green features\n if len(green) != 0:\n # Return the green features\n return green\n # If there aren't green features\n else:\n # If there are blue features\n if len(blue) != 0:\n # Return the blue features\n return blue\n # If there aren't blue features\n else:\n # Return all the features\n return X.columns.tolist()<\/pre>\n\n\n\n\n
Conclusion<\/h3>\n\n\n\n
Well, you can use the above code to get the feature importance before you backtest a strategy. We suggest you use the Boruta-Shap algorithm before you optimize a strategy’s parameters. You can find the source file below.<\/p>\n\n\n\n
Happy Learning!<\/p>\n\n\n\n