Neural networks for forecasting

A function with knobs, and a way to turn them

A neural network is just a function with many adjustable numbers, the weights. A layer multiplies the inputs by a weight matrix and adds a bias; a non-linearity such as ReLU (max(x, 0)) is applied; the result feeds the next layer. Stacked this way, such functions can approximate very complicated relationships. Training means choosing the weights so that predictions match known targets: measure the mismatch with a loss (for forecasting a number, the mean squared error), compute how the loss changes with each weight, and nudge every weight a little in the direction that lowers it. That nudge, repeated thousands of times, is gradient descent.

# A one-hidden-layer network: relu(x @ w1 + b1) @ w2 + b2
import numpy as np

def relu(x):
    return np.maximum(x, 0.0)

def mlp_forward(x, w1, b1, w2, b2):
    return relu(x @ w1 + b1) @ w2 + b2

x = np.array([[1.0, 2.0], [0.5, -1.0]])                       # two samples, two features
w1 = np.array([[1.0, -1.0], [0.5, 1.0]])                      # two hidden units
b1 = np.array([0.0, 0.5])
w2 = np.array([1.0, 2.0])
print(relu(np.array([-1.5, 0.0, 2.0])))                       # [0. 0. 2.]
print(mlp_forward(x, w1, b1, w2, 0.1))                        # [5.1 0.1]   a prediction for each sample; the second sample's hidden values are switched off by the relu

Gradient descent on the simplest model

Training a network uses backpropagation to get the gradients, but the idea is already complete for a linear model y ~ x @ w + b. The mean squared error has gradient 2/n * x.T @ err with respect to w and 2 * mean(err) with respect to b, where err = prediction - target. Start from zeros, subtract the learning rate times the gradient, repeat. Too large a learning rate overshoots and diverges; too small crawls. Standardise the features first (zero mean, unit variance) so one learning rate suits all of them. Libraries such as PyTorch compute the gradients automatically, but they do exactly this loop.

# Gradient descent on a line: start at zero and walk downhill on the mean squared error
import numpy as np

def mse_loss(pred, target):
    return float(np.mean((pred - target) ** 2))

def train_linear(x, y, lr, epochs):
    n, d = x.shape
    w, b = np.zeros(d), 0.0
    for _ in range(epochs):
        err = x @ w + b - y
        w = w - lr * 2 / n * x.T @ err
        b = b - lr * 2 * float(np.mean(err))
    return w, float(b)

x = np.array([[0.0], [1.0], [2.0], [3.0]])
y = np.array([1.0, 3.0, 5.0, 7.0])                          # exactly y = 2x + 1
for epochs in (0, 10, 2000):
    w, b = train_linear(x, y, 0.05, epochs)
    print(epochs, np.round(w, 3), round(b, 3), round(mse_loss(x @ w + b, y), 6))
# 0 [0.] 0.0 21.0          the zero start
# 10 [2.009] 0.958 0.00092   already close
# 2000 [2.] 1.0 0.0          the line y = 2x + 1 is found

Why forecasting returns is hard for a network

A network can fit almost anything, and that is the danger with financial data: returns are mostly noise, so a flexible model memorises the noise of the training period and fails on the next one. Overfitting shows as a training error far below the error on data it has not seen. The defences are the ones from the earlier topics: compare against a simple baseline (a linear model, or predicting zero); evaluate only on later data with walk-forward validation, never shuffling time; use few features and a small network; add regularisation (weight decay, dropout) and early stopping; and average several networks started from different seeds. If the network cannot clearly beat the linear baseline out of sample, use the linear model.

# Overfitting in miniature: a flexible fit to pure noise looks great in-sample and useless afterwards
import numpy as np

rng = np.random.default_rng(5)
x_train, x_test = rng.normal(size=(30, 12)), rng.normal(size=(1000, 12))      # 12 features that mean nothing
y_train, y_test = rng.normal(0, 0.01, size=30), rng.normal(0, 0.01, size=1000)  # next-day returns: pure noise

w = np.linalg.lstsq(x_train, y_train, rcond=None)[0]                          # a model with 12 free weights fitted to 30 points
train_mse = np.mean((x_train @ w - y_train) ** 2)
test_mse = np.mean((x_test @ w - y_test) ** 2)
baseline_mse = np.mean(y_test ** 2)                                           # predict zero
print(train_mse < baseline_mse, test_mse > baseline_mse)                      # True True   better in-sample, worse than zero out of sample

# The learning rate matters: a small one settles, one that is too large overshoots and the error blows up
import numpy as np

x = np.array([[1.0], [2.0], [3.0]])
y = np.array([2.0, 4.0, 6.0])

def final_errors(lr, epochs):
    w, b, errors = np.zeros(1), 0.0, []
    for _ in range(epochs):
        err = x @ w + b - y
        errors.append(float(np.mean(err ** 2)))
        w = w - lr * 2 / len(y) * x.T @ err
        b = b - lr * 2 * float(np.mean(err))
    return errors

small, large = final_errors(0.05, 30), final_errors(0.5, 30)
print(small[-1] < small[0])                          # True   the error shrinks
print(large[-1] > 1000 * large[0])                   # True   the error explodes: the steps are too big

• A layer is relu(x @ w + b); a network stacks layers, and the loss (mean squared error for a forecast) measures how far predictions are from targets.
• Training is gradient descent: for the linear model w -= lr * 2/n * x.T @ err and b -= lr * 2 * mean(err); scale features, tune the learning rate.
• Returns are mostly noise: compare with a linear baseline, validate walk-forward, use small networks with regularisation, and stop when the out-of-sample gain disappears.

Next up: reinforcement learning for trading, learning a policy from rewards instead of labelled answers.

Lab · what's graded

• Switch off negatives with relu25 pts
• Run the network forward with mlp_forward25 pts
• Measure the error with mse_loss25 pts
• Learn by descending with train_linear25 pts