{"id":248792,"date":"2026-07-01T11:00:54","date_gmt":"2026-07-01T15:00:54","guid":{"rendered":"https:\/\/ibkrcampus.com\/campus\/?p=248792"},"modified":"2026-07-01T11:10:32","modified_gmt":"2026-07-01T15:10:32","slug":"a-poor-persons-transformer","status":"publish","type":"post","link":"https:\/\/www.interactivebrokers.com\/campus\/ibkr-quant-news\/a-poor-persons-transformer\/","title":{"rendered":"A Poor Person&#8217;s Transformer"},"content":{"rendered":"\n<p class=\"wp-block-paragraph\"><em>The article &#8220;A Poor Person&#8217;s Transformer&#8221; was originally published on <a href=\"https:\/\/predictnow.ai\/a-poor-persons-transformer\/\">PredictNow.ai<\/a>.<\/em><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">For those of us who grew up before GenAI became a thing (e.g. Ernie), we often use tree-based algorithms for supervised learning. Trees work very well with heterogeneous and tabular feature sets, and by limiting the number of nodes or the depth of a branch, there is feature selection by default. With neural networks (NN), before deep learning came around, it was quite common to perform feature selection using L1 regularization \u2013 i.e. adding a L1 penalty term to the objective function in order to encourage some of the network parameters to become zero. However, L1 regularizations are quite tedious when we have millions or billions of parameters in a deep neural network. In its place, transformers and attention became the go-to technique for feature selection in a deep neural network (see Chapter 5 of our&nbsp;<a href=\"https:\/\/amzn.to\/4jrX2kj\">book<\/a>.) But beyond making feature selection practical for DNN, the attention mechanism provides one important benefit that is absent from traditional regularization or feature selection methods (such as MDA, SHAP, or LIME, see Chan &amp; Man https:\/\/arxiv.org\/abs\/2005.12483):&nbsp;<em>the selected features depend on each sample<\/em>. They aren\u2019t selected globally like traditional feature selection methods do. In other words, the features are selected based on their values themselves. In the language of transformers, we use&nbsp;<em>self-attention<\/em>&nbsp;for feature selection.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Transformers are usually illustrated with textual input. For e.g., a sentence containing 4 features (words\/tokens) \u201cI\u201d, \u201cam\u201d, \u201ca\u201d, \u201cstudent\u201d. Let\u2019s call this input feature vector X. In DNN, each feature may be a vector (e.g. we may use a d-dimensional vector to represent a word\/token), as opposed to a scalar. So X may actually have dimension n \u00d7 d, where n is the number of features (<em>not&nbsp;<\/em>the number of samples!) and d is the dimension of each feature. A financial application where this can be useful is when one feature (row) vector captures the daily return of a stock, its P\/E, dividend yield, \u2026, up d types of features, at a snapshot in time t. Another feature vector captures the same information at time t-1, and so on, up to a lookback of n. So if you have n lookback periods, the feature matrix has dimension n \u00d7 d. But in many financial applications, each feature is just a real-valued scalar such as the daily return of a single stock. So X=[r(t), r(t-1), \u2026, r(t-n+1)]<sup>T<\/sup>.&nbsp;<strong>This is the simple example we will use in our Poor Person\u2019s version of transformer: d=1, and X is just a column vector with dimension n \u00d7 1.<\/strong><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Now, in ordinary transformers, the next step is to&nbsp;<em>transform&nbsp;<\/em>X into 3 different vectors \/ matrices: Q (query, with width d<sub>q<\/sub>), K (key, with height d<sub>k<\/sub>), and V (value, with height d<sub>v<\/sub>). An element in Q is like \u201cwhat this feature is looking for in other features that can provide as context\u201d, an element in K is like \u201cthis is the context that this feature can provide, and element in V is like \u201cthis is a feature in a new representation\u201d.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">In a typical transformer with self-attention, for each input vector X, the Q, K and V values are calculated as linear transformation of X:<\/p>\n\n\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-full\"><img decoding=\"async\" width=\"768\" height=\"81\" data-src=\"https:\/\/www.interactivebrokers.com\/campus\/wp-content\/uploads\/sites\/2\/2026\/07\/predict-ai-transformer.png\" alt=\"A Poor Person\u2019s Transformer\" class=\"wp-image-248803 lazyload\" data-srcset=\"https:\/\/ibkrcampus.com\/campus\/wp-content\/uploads\/sites\/2\/2026\/07\/predict-ai-transformer.png 768w, https:\/\/ibkrcampus.com\/campus\/wp-content\/uploads\/sites\/2\/2026\/07\/predict-ai-transformer-700x74.png 700w, https:\/\/ibkrcampus.com\/campus\/wp-content\/uploads\/sites\/2\/2026\/07\/predict-ai-transformer-300x32.png 300w\" data-sizes=\"(max-width: 768px) 100vw, 768px\" src=\"data:image\/svg+xml;base64,PHN2ZyB3aWR0aD0iMSIgaGVpZ2h0PSIxIiB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciPjwvc3ZnPg==\" style=\"--smush-placeholder-width: 768px; aspect-ratio: 768\/81;\" \/><\/figure>\n<\/div>\n\n\n<p class=\"wp-block-paragraph\">The W<sup>Q<\/sup>, W<sup>K<\/sup>, W<sup>V<\/sup>&nbsp;matrices themselves are learned parameters, learned based on the ultimate objective of this NN (e.g. classification, regression, or optimization), but the resulting<strong>&nbsp;attention score is computed as the function of the input sample X<\/strong>. The W\u2019s all have heights n, but widths d<sub>q,&nbsp;<\/sub>d<sub>k<\/sub>, and d<sub>v&nbsp;<\/sub>respectively, though d<sub>q,&nbsp;<\/sub>d<sub>k&nbsp;<\/sub>are often set to be the same dimension. The intuition behind these Q, K, V is we want some linear mixtures of the original feature matrix X that best represent it, reminiscent of the familiar PCA. In the example of the n \u00d7 d financial feature matrix we described above, we want to linearly project the return and fundamentals of a stock to some \u201cprincipal component\u201d vector, while preserving the distinctness of each lagged snapshot of these features since the projection is row-wise. I.e. Q, K, V have same height as X and so each row still represents a specific snapshot in time, as seen in the figure below which illustrates the building blocks of a transformer with self-attention.<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img decoding=\"async\" width=\"768\" height=\"560\" data-src=\"https:\/\/www.interactivebrokers.com\/campus\/wp-content\/uploads\/sites\/2\/2026\/07\/predict-ai-transformer-diagram.jpg\" alt=\"A Poor Person\u2019s Transformer\" class=\"wp-image-248808 lazyload\" data-srcset=\"https:\/\/ibkrcampus.com\/campus\/wp-content\/uploads\/sites\/2\/2026\/07\/predict-ai-transformer-diagram.jpg 768w, https:\/\/ibkrcampus.com\/campus\/wp-content\/uploads\/sites\/2\/2026\/07\/predict-ai-transformer-diagram-700x510.jpg 700w, https:\/\/ibkrcampus.com\/campus\/wp-content\/uploads\/sites\/2\/2026\/07\/predict-ai-transformer-diagram-300x219.jpg 300w\" data-sizes=\"(max-width: 768px) 100vw, 768px\" src=\"data:image\/svg+xml;base64,PHN2ZyB3aWR0aD0iMSIgaGVpZ2h0PSIxIiB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciPjwvc3ZnPg==\" style=\"--smush-placeholder-width: 768px; aspect-ratio: 768\/560;\" \/><\/figure>\n\n\n\n<p class=\"has-text-align-center wp-block-paragraph\">Source: https:\/\/sebastianraschka.com\/blog\/2023\/self-attention-from-scratch.html<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The figure below shows specifically a transformer with n=4, d=4, and d<sub>q<\/sub>=d<sub>k<\/sub>=d<sub>v<\/sub>=2. It shows also how the Q and K matrices are multiplied together, scaled by sqrt(d<sub>k<\/sub>) to prevent the magnitude from exploding, and fed through a softmax function to turn them into attention scores in [0, 1], in a process called \u201cScaled Dot-Product Attention\u201d (for more details, see again Chapter 5 of&nbsp;<a href=\"https:\/\/amzn.to\/4lU6jmX\">our book<\/a>).<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Why sqrt(d<sub>k<\/sub>)? We will quote&nbsp;<a href=\"https:\/\/papers.ssrn.com\/sol3\/papers.cfm?abstract_id=3554486\">Cong et. al.<\/a>&nbsp;\u201cAssume that the components of q and k are independent random variables with mean 0 and variance 1. Then their dot product,<\/p>\n\n\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-full\"><img decoding=\"async\" width=\"300\" height=\"92\" data-src=\"https:\/\/www.interactivebrokers.com\/campus\/wp-content\/uploads\/sites\/2\/2026\/07\/predict-ai-transformer-2.png\" alt=\"A Poor Person\u2019s Transformer\" class=\"wp-image-248814 lazyload\" src=\"data:image\/svg+xml;base64,PHN2ZyB3aWR0aD0iMSIgaGVpZ2h0PSIxIiB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciPjwvc3ZnPg==\" style=\"--smush-placeholder-width: 300px; aspect-ratio: 300\/92;\" \/><\/figure>\n<\/div>\n\n\n<p class=\"wp-block-paragraph\">has mean 0 and variance d<sub>k<\/sub>. Why softmax? Softmax function normalizes the scaled dot-product into a matrix where each row is the normalized weights (i.e. they sum to 1) which are the attention weights applicable to the feature value matrix V. To wit,<\/p>\n\n\n<div class=\"wp-block-image\">\n<figure class=\"aligncenter size-full is-resized\"><img decoding=\"async\" width=\"300\" height=\"88\" data-src=\"https:\/\/www.interactivebrokers.com\/campus\/wp-content\/uploads\/sites\/2\/2026\/07\/predict-ai-transformer-diagram-2.png\" alt=\"A Poor Person\u2019s Transformer\" class=\"wp-image-248818 lazyload\" style=\"--smush-placeholder-width: 300px; aspect-ratio: 300\/88;width:300px;height:auto\" src=\"data:image\/svg+xml;base64,PHN2ZyB3aWR0aD0iMSIgaGVpZ2h0PSIxIiB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciPjwvc3ZnPg==\" \/><\/figure>\n<\/div>\n\n\n<figure class=\"wp-block-image size-full\"><img decoding=\"async\" width=\"853\" height=\"300\" data-src=\"https:\/\/www.interactivebrokers.com\/campus\/wp-content\/uploads\/sites\/2\/2026\/07\/predict-ai-transformer-diagram-3.jpg\" alt=\"A Poor Person\u2019s Transformer\" class=\"wp-image-248820 lazyload\" data-srcset=\"https:\/\/ibkrcampus.com\/campus\/wp-content\/uploads\/sites\/2\/2026\/07\/predict-ai-transformer-diagram-3.jpg 853w, https:\/\/ibkrcampus.com\/campus\/wp-content\/uploads\/sites\/2\/2026\/07\/predict-ai-transformer-diagram-3-700x246.jpg 700w, https:\/\/ibkrcampus.com\/campus\/wp-content\/uploads\/sites\/2\/2026\/07\/predict-ai-transformer-diagram-3-300x106.jpg 300w, https:\/\/ibkrcampus.com\/campus\/wp-content\/uploads\/sites\/2\/2026\/07\/predict-ai-transformer-diagram-3-768x270.jpg 768w\" data-sizes=\"(max-width: 853px) 100vw, 853px\" src=\"data:image\/svg+xml;base64,PHN2ZyB3aWR0aD0iMSIgaGVpZ2h0PSIxIiB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciPjwvc3ZnPg==\" style=\"--smush-placeholder-width: 853px; aspect-ratio: 853\/300;\" \/><\/figure>\n\n\n\n<p class=\"has-text-align-center wp-block-paragraph\">Source: https:\/\/sebastianraschka.com\/blog\/2023\/self-attention-from-scratch.html. Here n=4, d=2.<br><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">But, in our Poor Person\u2019s transformer, W is just a scalar, and Q, K, and V are all just 1-dimensional vectors. So we might as well&nbsp;<strong>eliminate this step and replace them all by the vector X<\/strong>. Note that this doesn\u2019t collapse the matrix QK<sup>T&nbsp;<\/sup>into a scalar or vector. It is still a n \u00d7 n matrix formed by XX<sup>T<\/sup>. Each feature i is still multiplying feature j to form the attention matrix element A(i, j). The elements of each row of A sums to 1 as in all attention matrices. If you ask \u201cWhat is the feature importance score of feature j\u201d, you can sum over all the values of column j, since column j represents the&nbsp;<em>key&nbsp;<\/em>feature j.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">So if feature importance scores or feature selection are all you are after, we are done. But usually we are interested in downstream applications. In our 1 stock n-returns example, we might be interested in using these n daily returns, with proper feature weights, to predict the next day\u2019s return. In this case, all we need to do it to multiply the attention matrix with V, which in our case is equal to X, to create the \u201ccontext vector\u201d Z=AV=AX. The context vector is an attention-weighted version of our original feature vector X. Downstream, we can use Z as input to a MLP for supervised learning, such as predicting the next day\u2019s return, or for optimization via reinforcement learning.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Does this work? You can ask ChatGPT or some other favorite chatbot to create a program based on this blog post and try it out. Let us know how the results look in the comments!<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>P.S.<\/strong><\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><br>You may get excited by this feature selection method and think we should throw in a bunch of \u201cheterogeneous\u201d features such as volatility, P\/E, earning yield, \u2026 of the stock to see if they work better. Unfortunately, the Poor Person\u2019s self-attention method discussed above doesn\u2019t work very well with features that cannot embedded in the same space. For example, it is nonsensical to add together A(i, j)=volatility * P\/E and A(i+1, j)=dividend * P\/E to form the feature importance score of P\/E. To do that, we need to do some normalization and embedding. Also, maybe we want to tell the transformer that r(t), r(t-1), \u2026 is a time series and the features are time-ordered. All topics for the next blog post!<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Trees work very well with heterogeneous and tabular feature sets, and by limiting the number of nodes or the depth of a branch, there is feature selection by default. <\/p>\n","protected":false},"author":186,"featured_media":183462,"comment_status":"open","ping_status":"closed","sticky":true,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":"","jetpack_post_was_ever_published":false},"categories":[339,338,341],"tags":[21842,2105,21409,9839,852,15593,21844,21843],"contributors-categories":[13719],"class_list":["post-248792","post","type-post","status-publish","format-standard","has-post-thumbnail","category-data-science","category-ibkr-quant-news","category-quant-development","tag-attention-mechanism","tag-deep-learning","tag-feature-selection","tag-financial-time-series","tag-machine-learning","tag-neural-networks","tag-scaled-dot-product-attention","tag-transformer","contributors-categories-predictnow-ai"],"pp_statuses_selecting_workflow":false,"pp_workflow_action":"current","pp_status_selection":"publish","acf":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO Premium plugin v26.9 (Yoast SEO v28.0) - https:\/\/yoast.com\/product\/yoast-seo-premium-wordpress\/ -->\n<title>A Poor Person&#8217;s Transformer | IBKR Quant<\/title>\n<meta name=\"description\" content=\"Trees work very well with heterogeneous and tabular feature sets, and by limiting the number of nodes or the depth of a branch, there is feature\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.interactivebrokers.com\/campus\/wp-json\/wp\/v2\/posts\/248792\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"A Poor Person\u2019s Transformer\" \/>\n<meta property=\"og:description\" content=\"Trees work very well with heterogeneous and tabular feature sets, and by limiting the number of nodes or the depth of a branch, there is feature selection by default.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.interactivebrokers.com\/campus\/ibkr-quant-news\/a-poor-persons-transformer\/\" \/>\n<meta property=\"og:site_name\" content=\"IBKR Campus US\" \/>\n<meta property=\"article:published_time\" content=\"2026-07-01T15:00:54+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2026-07-01T15:10:32+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.interactivebrokers.com\/campus\/wp-content\/uploads\/sites\/2\/2023\/02\/abstract-quant-blue-purple-nodes.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"1000\" \/>\n\t<meta property=\"og:image:height\" content=\"563\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"Contributor Author\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Contributor Author\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"7 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\n\t    \"@context\": \"https:\\\/\\\/schema.org\",\n\t    \"@graph\": [\n\t        {\n\t            \"@type\": \"NewsArticle\",\n\t            \"@id\": \"https:\\\/\\\/ibkrcampus.com\\\/campus\\\/ibkr-quant-news\\\/a-poor-persons-transformer\\\/#article\",\n\t            \"isPartOf\": {\n\t                \"@id\": \"https:\\\/\\\/ibkrcampus.com\\\/campus\\\/ibkr-quant-news\\\/a-poor-persons-transformer\\\/\"\n\t            },\n\t            \"author\": {\n\t                \"name\": \"Contributor Author\",\n\t                \"@id\": \"https:\\\/\\\/ibkrcampus.com\\\/campus\\\/#\\\/schema\\\/person\\\/e823e46b42ca381080387e794318a485\"\n\t            },\n\t            \"headline\": \"A Poor Person&#8217;s Transformer\",\n\t            \"datePublished\": \"2026-07-01T15:00:54+00:00\",\n\t            \"dateModified\": \"2026-07-01T15:10:32+00:00\",\n\t            \"mainEntityOfPage\": {\n\t                \"@id\": \"https:\\\/\\\/ibkrcampus.com\\\/campus\\\/ibkr-quant-news\\\/a-poor-persons-transformer\\\/\"\n\t            },\n\t            \"wordCount\": 1308,\n\t            \"commentCount\": 0,\n\t            \"publisher\": {\n\t                \"@id\": \"https:\\\/\\\/ibkrcampus.com\\\/campus\\\/#organization\"\n\t            },\n\t            \"image\": {\n\t                \"@id\": \"https:\\\/\\\/ibkrcampus.com\\\/campus\\\/ibkr-quant-news\\\/a-poor-persons-transformer\\\/#primaryimage\"\n\t            },\n\t            \"thumbnailUrl\": \"https:\\\/\\\/www.interactivebrokers.com\\\/campus\\\/wp-content\\\/uploads\\\/sites\\\/2\\\/2023\\\/02\\\/abstract-quant-blue-purple-nodes.jpg\",\n\t            \"keywords\": [\n\t                \"attention mechanism\",\n\t                \"Deep Learning\",\n\t                \"feature selection\",\n\t                \"Financial Time Series\",\n\t                \"Machine Learning\",\n\t                \"Neural Networks\",\n\t                \"scaled dot-product attention\",\n\t                \"transformer\"\n\t            ],\n\t            \"articleSection\": [\n\t                \"Data Science\",\n\t                \"Quant\",\n\t                \"Quant Development\"\n\t            ],\n\t            \"inLanguage\": \"en-US\",\n\t            \"potentialAction\": [\n\t                {\n\t                    \"@type\": \"CommentAction\",\n\t                    \"name\": \"Comment\",\n\t                    \"target\": [\n\t                        \"https:\\\/\\\/ibkrcampus.com\\\/campus\\\/ibkr-quant-news\\\/a-poor-persons-transformer\\\/#respond\"\n\t                    ]\n\t                }\n\t            ]\n\t        },\n\t        {\n\t            \"@type\": \"WebPage\",\n\t            \"@id\": \"https:\\\/\\\/ibkrcampus.com\\\/campus\\\/ibkr-quant-news\\\/a-poor-persons-transformer\\\/\",\n\t            \"url\": \"https:\\\/\\\/ibkrcampus.com\\\/campus\\\/ibkr-quant-news\\\/a-poor-persons-transformer\\\/\",\n\t            \"name\": \"A Poor Person's Transformer | IBKR Campus US\",\n\t            \"isPartOf\": {\n\t                \"@id\": \"https:\\\/\\\/ibkrcampus.com\\\/campus\\\/#website\"\n\t            },\n\t            \"primaryImageOfPage\": {\n\t                \"@id\": \"https:\\\/\\\/ibkrcampus.com\\\/campus\\\/ibkr-quant-news\\\/a-poor-persons-transformer\\\/#primaryimage\"\n\t            },\n\t            \"image\": {\n\t                \"@id\": \"https:\\\/\\\/ibkrcampus.com\\\/campus\\\/ibkr-quant-news\\\/a-poor-persons-transformer\\\/#primaryimage\"\n\t            },\n\t            \"thumbnailUrl\": \"https:\\\/\\\/www.interactivebrokers.com\\\/campus\\\/wp-content\\\/uploads\\\/sites\\\/2\\\/2023\\\/02\\\/abstract-quant-blue-purple-nodes.jpg\",\n\t            \"datePublished\": \"2026-07-01T15:00:54+00:00\",\n\t            \"dateModified\": \"2026-07-01T15:10:32+00:00\",\n\t            \"description\": \"Trees work very well with heterogeneous and tabular feature sets, and by limiting the number of nodes or the depth of a branch, there is feature\",\n\t            \"inLanguage\": \"en-US\",\n\t            \"potentialAction\": [\n\t                {\n\t                    \"@type\": \"ReadAction\",\n\t                    \"target\": [\n\t                        \"https:\\\/\\\/ibkrcampus.com\\\/campus\\\/ibkr-quant-news\\\/a-poor-persons-transformer\\\/\"\n\t                    ]\n\t                }\n\t            ]\n\t        },\n\t        {\n\t            \"@type\": \"ImageObject\",\n\t            \"inLanguage\": \"en-US\",\n\t            \"@id\": \"https:\\\/\\\/ibkrcampus.com\\\/campus\\\/ibkr-quant-news\\\/a-poor-persons-transformer\\\/#primaryimage\",\n\t            \"url\": \"https:\\\/\\\/www.interactivebrokers.com\\\/campus\\\/wp-content\\\/uploads\\\/sites\\\/2\\\/2023\\\/02\\\/abstract-quant-blue-purple-nodes.jpg\",\n\t            \"contentUrl\": \"https:\\\/\\\/www.interactivebrokers.com\\\/campus\\\/wp-content\\\/uploads\\\/sites\\\/2\\\/2023\\\/02\\\/abstract-quant-blue-purple-nodes.jpg\",\n\t            \"width\": 1000,\n\t            \"height\": 563,\n\t            \"caption\": \"Hull-White 2-factor Model: 3) Simulation\"\n\t        },\n\t        {\n\t            \"@type\": \"WebSite\",\n\t            \"@id\": \"https:\\\/\\\/ibkrcampus.com\\\/campus\\\/#website\",\n\t            \"url\": \"https:\\\/\\\/ibkrcampus.com\\\/campus\\\/\",\n\t            \"name\": \"IBKR Campus US\",\n\t            \"description\": \"Financial Education from Interactive Brokers\",\n\t            \"publisher\": {\n\t                \"@id\": \"https:\\\/\\\/ibkrcampus.com\\\/campus\\\/#organization\"\n\t            },\n\t            \"potentialAction\": [\n\t                {\n\t                    \"@type\": \"SearchAction\",\n\t                    \"target\": {\n\t                        \"@type\": \"EntryPoint\",\n\t                        \"urlTemplate\": \"https:\\\/\\\/ibkrcampus.com\\\/campus\\\/?s={search_term_string}\"\n\t                    },\n\t                    \"query-input\": {\n\t                        \"@type\": \"PropertyValueSpecification\",\n\t                        \"valueRequired\": true,\n\t                        \"valueName\": \"search_term_string\"\n\t                    }\n\t                }\n\t            ],\n\t            \"inLanguage\": \"en-US\"\n\t        },\n\t        {\n\t            \"@type\": \"Organization\",\n\t            \"@id\": \"https:\\\/\\\/ibkrcampus.com\\\/campus\\\/#organization\",\n\t            \"name\": \"Interactive Brokers\",\n\t            \"alternateName\": \"IBKR\",\n\t            \"url\": \"https:\\\/\\\/ibkrcampus.com\\\/campus\\\/\",\n\t            \"logo\": {\n\t                \"@type\": \"ImageObject\",\n\t                \"inLanguage\": \"en-US\",\n\t                \"@id\": \"https:\\\/\\\/ibkrcampus.com\\\/campus\\\/#\\\/schema\\\/logo\\\/image\\\/\",\n\t                \"url\": \"https:\\\/\\\/www.interactivebrokers.com\\\/campus\\\/wp-content\\\/uploads\\\/sites\\\/2\\\/2024\\\/05\\\/ibkr-campus-logo.jpg\",\n\t                \"contentUrl\": \"https:\\\/\\\/www.interactivebrokers.com\\\/campus\\\/wp-content\\\/uploads\\\/sites\\\/2\\\/2024\\\/05\\\/ibkr-campus-logo.jpg\",\n\t                \"width\": 669,\n\t                \"height\": 669,\n\t                \"caption\": \"Interactive Brokers\"\n\t            },\n\t            \"image\": {\n\t                \"@id\": \"https:\\\/\\\/ibkrcampus.com\\\/campus\\\/#\\\/schema\\\/logo\\\/image\\\/\"\n\t            },\n\t            \"publishingPrinciples\": \"https:\\\/\\\/www.interactivebrokers.com\\\/campus\\\/about-ibkr-campus\\\/\",\n\t            \"ethicsPolicy\": \"https:\\\/\\\/www.interactivebrokers.com\\\/campus\\\/cyber-security-notice\\\/\"\n\t        },\n\t        {\n\t            \"@type\": \"Person\",\n\t            \"@id\": \"https:\\\/\\\/ibkrcampus.com\\\/campus\\\/#\\\/schema\\\/person\\\/e823e46b42ca381080387e794318a485\",\n\t            \"name\": \"Contributor Author\",\n\t            \"url\": \"https:\\\/\\\/www.interactivebrokers.com\\\/campus\\\/author\\\/contributor-author\\\/\"\n\t        }\n\t    ]\n\t}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"A Poor Person&#8217;s Transformer | IBKR Quant","description":"Trees work very well with heterogeneous and tabular feature sets, and by limiting the number of nodes or the depth of a branch, there is feature","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.interactivebrokers.com\/campus\/wp-json\/wp\/v2\/posts\/248792\/","og_locale":"en_US","og_type":"article","og_title":"A Poor Person\u2019s Transformer","og_description":"Trees work very well with heterogeneous and tabular feature sets, and by limiting the number of nodes or the depth of a branch, there is feature selection by default.","og_url":"https:\/\/www.interactivebrokers.com\/campus\/ibkr-quant-news\/a-poor-persons-transformer\/","og_site_name":"IBKR Campus US","article_published_time":"2026-07-01T15:00:54+00:00","article_modified_time":"2026-07-01T15:10:32+00:00","og_image":[{"width":1000,"height":563,"url":"https:\/\/www.interactivebrokers.com\/campus\/wp-content\/uploads\/sites\/2\/2023\/02\/abstract-quant-blue-purple-nodes.jpg","type":"image\/jpeg"}],"author":"Contributor Author","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Contributor Author","Est. reading time":"7 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"NewsArticle","@id":"https:\/\/ibkrcampus.com\/campus\/ibkr-quant-news\/a-poor-persons-transformer\/#article","isPartOf":{"@id":"https:\/\/ibkrcampus.com\/campus\/ibkr-quant-news\/a-poor-persons-transformer\/"},"author":{"name":"Contributor Author","@id":"https:\/\/ibkrcampus.com\/campus\/#\/schema\/person\/e823e46b42ca381080387e794318a485"},"headline":"A Poor Person&#8217;s Transformer","datePublished":"2026-07-01T15:00:54+00:00","dateModified":"2026-07-01T15:10:32+00:00","mainEntityOfPage":{"@id":"https:\/\/ibkrcampus.com\/campus\/ibkr-quant-news\/a-poor-persons-transformer\/"},"wordCount":1308,"commentCount":0,"publisher":{"@id":"https:\/\/ibkrcampus.com\/campus\/#organization"},"image":{"@id":"https:\/\/ibkrcampus.com\/campus\/ibkr-quant-news\/a-poor-persons-transformer\/#primaryimage"},"thumbnailUrl":"https:\/\/www.interactivebrokers.com\/campus\/wp-content\/uploads\/sites\/2\/2023\/02\/abstract-quant-blue-purple-nodes.jpg","keywords":["attention mechanism","Deep Learning","feature selection","Financial Time Series","Machine Learning","Neural Networks","scaled dot-product attention","transformer"],"articleSection":["Data Science","Quant","Quant Development"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/ibkrcampus.com\/campus\/ibkr-quant-news\/a-poor-persons-transformer\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/ibkrcampus.com\/campus\/ibkr-quant-news\/a-poor-persons-transformer\/","url":"https:\/\/ibkrcampus.com\/campus\/ibkr-quant-news\/a-poor-persons-transformer\/","name":"A Poor Person's Transformer | IBKR Campus US","isPartOf":{"@id":"https:\/\/ibkrcampus.com\/campus\/#website"},"primaryImageOfPage":{"@id":"https:\/\/ibkrcampus.com\/campus\/ibkr-quant-news\/a-poor-persons-transformer\/#primaryimage"},"image":{"@id":"https:\/\/ibkrcampus.com\/campus\/ibkr-quant-news\/a-poor-persons-transformer\/#primaryimage"},"thumbnailUrl":"https:\/\/www.interactivebrokers.com\/campus\/wp-content\/uploads\/sites\/2\/2023\/02\/abstract-quant-blue-purple-nodes.jpg","datePublished":"2026-07-01T15:00:54+00:00","dateModified":"2026-07-01T15:10:32+00:00","description":"Trees work very well with heterogeneous and tabular feature sets, and by limiting the number of nodes or the depth of a branch, there is feature","inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/ibkrcampus.com\/campus\/ibkr-quant-news\/a-poor-persons-transformer\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/ibkrcampus.com\/campus\/ibkr-quant-news\/a-poor-persons-transformer\/#primaryimage","url":"https:\/\/www.interactivebrokers.com\/campus\/wp-content\/uploads\/sites\/2\/2023\/02\/abstract-quant-blue-purple-nodes.jpg","contentUrl":"https:\/\/www.interactivebrokers.com\/campus\/wp-content\/uploads\/sites\/2\/2023\/02\/abstract-quant-blue-purple-nodes.jpg","width":1000,"height":563,"caption":"Hull-White 2-factor Model: 3) Simulation"},{"@type":"WebSite","@id":"https:\/\/ibkrcampus.com\/campus\/#website","url":"https:\/\/ibkrcampus.com\/campus\/","name":"IBKR Campus US","description":"Financial Education from Interactive Brokers","publisher":{"@id":"https:\/\/ibkrcampus.com\/campus\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/ibkrcampus.com\/campus\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/ibkrcampus.com\/campus\/#organization","name":"Interactive Brokers","alternateName":"IBKR","url":"https:\/\/ibkrcampus.com\/campus\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/ibkrcampus.com\/campus\/#\/schema\/logo\/image\/","url":"https:\/\/www.interactivebrokers.com\/campus\/wp-content\/uploads\/sites\/2\/2024\/05\/ibkr-campus-logo.jpg","contentUrl":"https:\/\/www.interactivebrokers.com\/campus\/wp-content\/uploads\/sites\/2\/2024\/05\/ibkr-campus-logo.jpg","width":669,"height":669,"caption":"Interactive Brokers"},"image":{"@id":"https:\/\/ibkrcampus.com\/campus\/#\/schema\/logo\/image\/"},"publishingPrinciples":"https:\/\/www.interactivebrokers.com\/campus\/about-ibkr-campus\/","ethicsPolicy":"https:\/\/www.interactivebrokers.com\/campus\/cyber-security-notice\/"},{"@type":"Person","@id":"https:\/\/ibkrcampus.com\/campus\/#\/schema\/person\/e823e46b42ca381080387e794318a485","name":"Contributor Author","url":"https:\/\/www.interactivebrokers.com\/campus\/author\/contributor-author\/"}]}},"jetpack_featured_media_url":"https:\/\/www.interactivebrokers.com\/campus\/wp-content\/uploads\/sites\/2\/2023\/02\/abstract-quant-blue-purple-nodes.jpg","_links":{"self":[{"href":"https:\/\/ibkrcampus.com\/campus\/wp-json\/wp\/v2\/posts\/248792","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/ibkrcampus.com\/campus\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/ibkrcampus.com\/campus\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/ibkrcampus.com\/campus\/wp-json\/wp\/v2\/users\/186"}],"replies":[{"embeddable":true,"href":"https:\/\/ibkrcampus.com\/campus\/wp-json\/wp\/v2\/comments?post=248792"}],"version-history":[{"count":18,"href":"https:\/\/ibkrcampus.com\/campus\/wp-json\/wp\/v2\/posts\/248792\/revisions"}],"predecessor-version":[{"id":248847,"href":"https:\/\/ibkrcampus.com\/campus\/wp-json\/wp\/v2\/posts\/248792\/revisions\/248847"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/ibkrcampus.com\/campus\/wp-json\/wp\/v2\/media\/183462"}],"wp:attachment":[{"href":"https:\/\/ibkrcampus.com\/campus\/wp-json\/wp\/v2\/media?parent=248792"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/ibkrcampus.com\/campus\/wp-json\/wp\/v2\/categories?post=248792"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/ibkrcampus.com\/campus\/wp-json\/wp\/v2\/tags?post=248792"},{"taxonomy":"contributors-categories","embeddable":true,"href":"https:\/\/ibkrcampus.com\/campus\/wp-json\/wp\/v2\/contributors-categories?post=248792"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}