A smooth line that has seen the future
A filter turns an input sequence xₜ into an output yₜ as a weighted sum of recent inputs and recent outputs. The exponential moving average is the simplest: yₜ=αxₜ+(1-α)yₜ₋₁. In scipy.signal a filter is a pair of coefficient lists, b for the inputs and a for the outputs, and yₜ=b₀xₜ-a₁yₜ₋₁ here with b = [alpha] and a = [1, -(1 - alpha)]. signal.lfilter(b, a, x) runs it forward in time, exactly like a live system, so yₜ depends only on xₛ with s≤t. By default it starts from rest (zero state), which makes the first values wrong for a price near 100; pass zi = signal.lfilter_zi(b, a) * x[0] to start from the first value instead. The result agrees with pd.Series(x).ewm(alpha=alpha, adjust=False).mean() to machine precision.
signal.filtfilt(b, a, x) runs the same filter forwards and then backwards. The lag cancels, so the smooth line sits right on the data, which looks perfect on a chart, but every output now depends on inputs from after it. Used to build a trading signal it is a leak. Take a trend signal, the log price minus its average, and correlate it with the next day's return: for SPY over 2,501 days and α=0.1 the causal signal has a correlation of -0.055 (a faint mean reversion), the zero-phase one -0.33; for α=0.3 the figures are -0.076 and -0.54. No trader could earn those zero-phase numbers, because the filtered value already contains tomorrow's price. The test of a signal is simple: truncate the data at some date and check that the values before it do not change.
lfilteris live,filtfiltis hindsight: only the first can run on a trading desk.- Set the initial state:
lfilter_zi(b, a) * x[0]removes the start-up transient. - Truncation test: cut the data at a date; a valid signal does not change before it.
Lab · what's graded
- Filter coefficients10 pts
- The causal average20 pts
- The pandas version10 pts
- The zero-phase average15 pts
- Signals and tomorrow's return25 pts
- The SPY study20 pts