{"id":93556,"date":"2021-06-30T11:30:58","date_gmt":"2021-06-30T15:30:58","guid":{"rendered":"https:\/\/ibkrcampus.com\/?p=93556"},"modified":"2024-05-20T16:25:59","modified_gmt":"2024-05-20T20:25:59","slug":"towards-better-keras-modeling-part-vii","status":"publish","type":"post","link":"https:\/\/www.interactivebrokers.com\/campus\/ibkr-quant-news\/towards-better-keras-modeling-part-vii\/","title":{"rendered":"Towards Better Keras Modeling \u2013 Part VII"},"content":{"rendered":"\n

The Alpha Scientist demonstrates the Relative Importance of Features. See the previous installment<\/a> in this series to learn about Multivariate Effects<\/em>.<\/p>\n\n\n\n

Relative Importance of Features<\/h2>\n\n\n\n

With so many findings, where do we start? I’ll run a quick random forest regression model and test the relative significance of each hyperparameter in overall model performance:<\/p>\n\n\n\n

In [70]:<\/p>\n\n\n\n

\nfrom sklearn.preprocessing import MinMaxScaler
\nX = df[[‘first_neuron’,’hidden_neuron’,’hidden_layers’,’dropout’]]
\nscaler = MinMaxScaler()
\ny = scaler.fit_transform(df[[‘val_loss_improvement’]])

\n\nfrom sklearn.ensemble import RandomForestRegressor

\n\nreg = RandomForestRegressor(max_depth=3,n_estimators=100)
\nreg.fit(X,y)
\npd.Series(reg.feature_importances_,index=X.columns).\\
\nsort_values(ascending=True).plot.barh(color=’grey’,title=’Feature Importance of Hyperparameters’)\n<\/p>\n\n\n\n

Out [70]:<\/p>\n\n\n\n

\n<matplotlib.axes._subplots.AxesSubplot at 0x7f4231951710>\n<\/p>\n\n\n\n

\"Towards<\/figure>\n\n\n\n

It appears that number of hidden layers is – by far – most important, followed by size of hidden layers. Of course, if we drop to zero hidden layers, then first layer size become supremely important.<\/p>\n\n\n\n

For the next hyperparameter sweep, I’ll focus on larger layer sizes – and fewer layers.<\/p>\n\n\n\n

In [2]:<\/p>\n\n\n\n

\n## Experiment 2:
\nfrom keras.models import Sequential
\nfrom keras.layers import Dropout, Dense
\nfrom keras.callbacks import TensorBoard
\nfrom talos.model.early_stopper import early_stopper

\n\n\n# track performance on tensorboard
\ntensorboard = TensorBoard(log_dir=’.\/logs’,
\n histogram_freq=0,batch_size=10000,
\n write_graph=False,
\n write_images=False)

\n\n\n# (1) Define dict of parameters to try
\np = {‘first_neuron’:[100,200,400,800,1600,3200],
\n ‘hidden_neuron’:[100, 200, 400, 800 ],
\n ‘hidden_layers’:[0,1],
\n ‘batch_size’: [10000],
\n ‘optimizer’: [‘adam’],
\n ‘kernel_initializer’: [‘uniform’], #’normal’
\n ‘epochs’: [100], # increased in case larger dimensions take longer to train
\n ‘dropout’: [0.0,0.25],
\n ‘last_activation’: [‘sigmoid’]}

\n\n# (2) create a function which constructs a compiled keras model object
\ndef numerai_model(x_train, y_train, x_val, y_val, params):
\n print(params)

\n\n model = Sequential()

\n \n \n ## initial layer
\n model.add(Dense(params[‘first_neuron’], input_dim=x_train.shape[1],
\n activation=’relu’,
\n kernel_initializer = params[‘kernel_initializer’] ))
\n model.add(Dropout(params[‘dropout’]))

\n \n ## hidden layers
\n for i in range(params[‘hidden_layers’]):
\n print (f”adding layer {i+1}”)
\n model.add(Dense(params[‘hidden_neuron’], activation=’relu’,
\n kernel_initializer=params[‘kernel_initializer’]))
\n model.add(Dropout(params[‘dropout’]))

\n \n \n ## final layer
\n model.add(Dense(1, activation=params[‘last_activation’],
\n kernel_initializer=params[‘kernel_initializer’]))

\n \n model.compile(loss=’binary_crossentropy’,
\n optimizer=params[‘optimizer’],
\n metrics=[‘acc’])

\n \n history = model.fit(x_train, y_train,
\n validation_data=[x_val, y_val],
\n batch_size=params[‘batch_size’],
\n epochs=params[‘epochs’],
\n callbacks=[tensorboard,early_stopper(params[‘epochs’], patience=10)], #,ta.live(),
\n verbose=0)
\n return history, model

\n\n# (3) Run a “Scan” using the params and function created above

\n\nt = ta.Scan(x=X_train.values,
\n y=y_train.values,
\n model=numerai_model,
\n params=p,
\n grid_downsample=1.00,
\n dataset_name=’numerai_example’,
\n experiment_no=’2′)\n\n\n<\/p>\n\n\n\n

There we have it. Is this optimal? Almost certainly not. But I now have a much better understanding of how the model performs at various geometries – and have spent relatively little time performing plug-and-chug parameter tweaking.<\/p>\n\n\n\n

At this point, I’ll build and train a single model with parameter values that showed most successful in the hyperparameter sweeps.<\/p>\n\n\n\n

There are infinite possibilities for further optimizations, which I won’t explore here. For instance:<\/p>\n\n\n\n