Detecting outliers

A value that cannot be right

A daily return of +30% on a large-cap stock is either a takeover or a data error such as a missing split or a misplaced decimal point. An outlier is a value far enough from the others to need a decision: keep it, fix it, or remove it. Never delete one just because it is inconvenient, since the real shocks are exactly what risk models must see. The first step is always to make them visible with a rule that is explicit and repeatable.

The z-score rule, and why it can fail

The classic rule flags values more than 2 or 3 standard deviations from the mean: z = (x - mean) / std. It is simple and it works on clean data. But the outlier itself inflates the mean and especially the standard deviation, so it hides from the rule. In the example below the +30% day gets a z-score of only 2.85, and with a threshold of 3 it would not be flagged. The more extreme the error, the more it protects itself.

import pandas as pd

returns = pd.Series([0.004, -0.002, 0.001, 0.003, -0.004, 0.002, 0.30, -0.001, 0.0, 0.002])

z = (returns - returns.mean()) / returns.std()
print(z.round(2).tolist())
# [-0.28, -0.34, -0.31, -0.29, -0.36, -0.3, 2.85, -0.33, -0.32, -0.3]

print(returns.mean().round(4), returns.std().round(4))   # 0.0305 0.0947   one day moved the mean tenfold
print(returns.drop(6).std().round(4))                    # 0.0026          the std without that day

Robust rules: IQR and MAD

Robust statistics are barely moved by the outliers they hunt. The interquartile range IQR = Q3 - Q1 measures the spread of the middle half of the data, and values beyond Q1 - 1.5 * IQR or Q3 + 1.5 * IQR are outside the fences of a box plot. The MAD (median absolute deviation) replaces the standard deviation by a median, and the modified z-score 0.6745 * (x - median) / MAD is read like a normal z-score (3.5 is a common cutoff). Both flag the +30% day without trouble.

# The fences of a box plot
q1, q3 = returns.quantile(0.25), returns.quantile(0.75)
spread = q3 - q1
lower, upper = q1 - 1.5 * spread, q3 + 1.5 * spread
print(round(lower, 4), round(upper, 4))                    # -0.006 0.008
print(returns[(returns < lower) | (returns > upper)].to_dict())   # {6: 0.3}

# The modified z-score, built on the median
median = returns.median()
mad = (returns - median).abs().median()
modified_z = 0.6745 * (returns - median) / mad
print(modified_z.round(1).tolist())   # [0.8, -1.2, -0.2, 0.5, -1.9, 0.2, 100.7, -0.8, -0.5, 0.2]

Winsorizing: clip instead of delete

Deleting a row also deletes the information in its other columns and breaks the spacing of dates. Winsorizing keeps every row and pulls the extreme values back to chosen quantiles, for instance the 1% and 99% ones: anything below the 1% quantile becomes the 1% quantile, anything above the 99% one becomes the 99% one. In pandas that is clip(lower, upper) with the limits taken from quantile. The shape of the series survives and the extremes lose their pull on averages and regressions.

low, high = returns.quantile(0.1), returns.quantile(0.9)
print(round(low, 4), round(high, 4))             # -0.0022 0.0336

clipped = returns.clip(lower=low, upper=high)
print(clipped.round(4).tolist())
# [0.004, -0.002, 0.001, 0.003, -0.0022, 0.002, 0.0336, -0.001, 0.0, 0.002]
#                                 ^ raised                  ^ the 30% day pulled back to the 90% quantile

# Flag with a rule, then look at what is flagged before deciding anything
mask = z.abs() > 2
print(returns[mask].to_dict())                 # {6: 0.3}        the +30% day
print(returns[~mask].std().round(4))           # 0.0026          the volatility of the remaining days

• The z-score rule abs((x - mean) / std) > k is simple, but outliers inflate the mean and std and can hide.
• Robust rules use quantiles and medians: IQR fences (Q1 - 1.5 IQR, Q3 + 1.5 IQR) and the MAD-based modified z-score.
• clip(lower, upper) with quantile limits (winsorizing) keeps the rows; never delete a shock without checking what it is.

Next up: corporate actions, the splits and dividends that make an old price incomparable with today's.

Lab · what's graded

• Flag the far-away returns with zscore_outliers25 pts
• Compute the fences of the box plot with iqr_bounds25 pts
• Flag outliers with a robust score using mad_outliers25 pts
• Tame the extremes with winsorize25 pts