Holes in the data
Real price tables have holes: a stock did not trade that day, a feed dropped a quote, a market was on holiday. pandas marks them as NaN ("not a number"). Before choosing what to do, look: df.isna() gives a True/False table, .sum() counts the holes per column, and .mean() gives their share. A column that is 40% empty is a different problem from one missing a single value.
import pandas as pd
nan = float("nan")
df = pd.DataFrame(
{"AAPL": [184.0, nan, 186.5, 187.0, nan], "MSFT": [410.0, 411.0, nan, nan, 415.0]},
index=pd.bdate_range("2026-03-02", periods=5),
)
print(df.isna().sum().to_dict()) # {'AAPL': 2, 'MSFT': 2} holes per column
print(df.isna().mean().round(2).to_dict()) # {'AAPL': 0.4, 'MSFT': 0.4} share of holesCarry forward, or drop
For prices, the natural fill is the last known price: until a new quote arrives, the market is still at the old one. That is ffill() (forward fill). A long gap probably means the data is stale, so ffill(limit=2) fills at most two days in a row. NaN at the very start stay NaN, because there is no earlier price to carry. The alternative is to drop: dropna() removes every row with a hole, and dropna(subset=["AAPL"]) only looks at the columns you care about. Dropping is safe but shrinks the data, and it changes the spacing between dates.
msft = df["MSFT"]
print(msft.ffill().tolist()) # [410.0, 411.0, 411.0, 411.0, 415.0]
print(msft.ffill(limit=1).tolist()) # [410.0, 411.0, 411.0, nan, 415.0] only one day carried
print(len(df.dropna())) # 1 only one row is complete
print(len(df.dropna(subset=["AAPL"]))) # 3 rows where AAPL is knownInterpolation, and what never to fill
interpolate() fills a gap with a straight line between the known values on either side. It suits smooth quantities (a yield curve, a volatility level) and rarely prices, which jump. Be careful at the edges: by default interpolate() also copies the last value into trailing NaN, which invents data. limit_area="inside" restricts it to gaps between two known values. Never fill with 0 blindly: a price of 0 is a crash, a return of 0 is a flat day, and fillna(0) on the wrong table silently changes every average you compute afterwards.
gappy = pd.Series([nan, nan, 10.0, nan, nan, 16.0, nan])
print(gappy.interpolate().tolist())
# [nan, nan, 10.0, 12.0, 14.0, 16.0, 16.0] the last 16.0 is invented
print(gappy.interpolate(limit_area="inside").tolist())
# [nan, nan, 10.0, 12.0, 14.0, 16.0, nan] only between known values# Filling with 0 is a decision, not a default: it changes the statistics
print(gappy.fillna(0).tolist()) # [0.0, 0.0, 10.0, 0.0, 0.0, 16.0, 0.0]
print(gappy.mean()) # 13.0 NaN are skipped
print(gappy.fillna(0).mean()) # 3.7142857142857144 the zeros drag the average downbfill() and interpolate() use values from after the gap, so at the date of the gap they use information that did not exist yet. Inside a backtest that is look-ahead bias. Use ffill() for anything a strategy sees in real time, and keep interpolation for cleaning historical data you analyse as a whole.df.isna().sum()counts the holes per column; look at the data before deciding how to treat it.ffill(limit=n)carries the last known value (the right choice for prices);dropna(subset=...)removes the incomplete rows.interpolate(limit_area="inside")bridges interior gaps only; never fill with 0 or look into the future by habit.
Next up: detecting outliers, finding the values that are present but cannot be right.
Lab · what's graded
- Count the holes in each column with
count_missing25 pts - Carry the last price forward with
forward_fill25 pts - Keep only the rows you can trust with
complete_rows25 pts - Bridge the gaps inside the data with
fill_gaps25 pts