Feature engineering for financial data

Give the model comparable inputs

A price of 250 means nothing to a model: it depends on the stock, and it trends upward over time, so the values of 2020 never resemble those of 2010. Feature engineering turns raw data into inputs that are comparable across time and across assets: returns instead of prices, volatility instead of ranges, standardised scores instead of levels. Good features carry more of the result than the choice of model, and each must describe only what was known on the day.

import numpy as np

prices = np.array([100.0, 101.0, 99.0, 102.0, 104.0, 103.0, 106.0])

window = 3
momentum = np.full(len(prices), np.nan)
momentum[window:] = prices[window:] / prices[:-window] - 1          # the return over the last 3 days
print(np.round(momentum, 4))                                        # nan for the first 3 days: no price 3 days back

Windows that look back

Most features are statistics of a trailing window: the last n values up to and including today. Momentum compares today with n days ago; rolling volatility is the standard deviation of the last n returns; the rolling z-score says how many window-standard-deviations today lies from the window mean. With sliding_window_view you get all windows as rows of a matrix and compute the statistic along an axis, no loop. The first n - 1 days have no complete window and stay nan.

import numpy as np

prices = np.array([100.0, 101.0, 99.0, 102.0, 104.0, 103.0, 106.0])
returns = np.diff(prices) / prices[:-1]
window = 3

windows = np.lib.stride_tricks.sliding_window_view(returns, window)        # every 3-day window as a row
vol = np.full(len(returns), np.nan)
vol[window - 1:] = windows.std(axis=1, ddof=1)
print(np.round(vol, 4))                                                    # nan nan then one value per complete window

p_windows = np.lib.stride_tricks.sliding_window_view(prices, window)
z = np.full(len(prices), np.nan)
z[window - 1:] = (prices[window - 1:] - p_windows.mean(axis=1)) / p_windows.std(axis=1, ddof=1)
print(np.round(z, 3))                                                      # how unusual each price is for its last 3 days

# Features put prices of different stocks on the same footing
import numpy as np

cheap = np.array([5.0, 5.2, 5.1, 5.4, 5.5])
dear = np.array([500.0, 520.0, 510.0, 540.0, 550.0])           # ten times the price, the same movements
print(cheap[-1] / cheap[-3] - 1, dear[-1] / dear[-3] - 1)        # identical momentum, about 0.0784 for both
print(dear[-1] - dear[-3], cheap[-1] - cheap[-3])                # price changes (40.0 and 0.4) would not be comparable

The target looks forward

What the model predicts is the target, usually the return over the next h days: prices[t + h] / prices[t] - 1. It is the only column allowed to look forward, and it is nan for the last h days because the future has not happened yet; drop those rows before training. A common mistake is a feature that contains the target by accident, such as a centred window, a statistic computed on all the data, or a forward return reused as an input. The test is simple: a feature on day t must not change when later days are added.

import numpy as np

def forward_return(prices, horizon):
    out = np.full(len(prices), np.nan)
    out[:-horizon] = prices[horizon:] / prices[:-horizon] - 1
    return out

def momentum(prices, window):
    out = np.full(len(prices), np.nan)
    out[window:] = prices[window:] / prices[:-window] - 1
    return out

prices = np.array([100.0, 101.0, 99.0, 102.0, 104.0, 103.0, 106.0])
print(np.round(momentum(prices, 2), 4))                 # looks back: known on each day
print(np.round(forward_return(prices, 2), 4))           # looks forward: the target, nan for the last 2 days

# The prefix test: a feature computed on the first 5 days equals the first 5 values of the full computation
print(np.allclose(momentum(prices[:5], 2), momentum(prices, 2)[:5], equal_nan=True))      # True

• Turn prices into comparable features: momentum prices[t] / prices[t - n] - 1, rolling volatility, rolling z-score, each over a trailing window including today.
• sliding_window_view gives every window as a row, so statistics need no loop; the first n - 1 days stay nan.
• The forward return prices[t + h] / prices[t] - 1 is the target, never a feature; check each feature with the prefix test.

Next up: walk-forward validation, testing a model in the order that time actually flows.

Lab · what's graded

• Measure the trend with momentum25 pts
• Track the recent risk with rolling_vol25 pts
• Say how unusual today is with rolling_zscore25 pts
• Define the target with forward_return25 pts