pandas DataFrames

A table of labelled columns

A DataFrame is a two-dimensional table: a set of columns, each a Series, that share one index of row labels. It is the workhorse of market-data analysis: one row per symbol or per day, one column per field (price, volume, return). Build one from a dictionary of columns, pd.DataFrame({"price": [...], "volume": [...]}, index=symbols), or from a list of row dictionaries. Each column has its own dtype (prices are float64, volumes int64), unlike a NumPy array that has a single dtype for everything.

import pandas as pd

df = pd.DataFrame(
    {"price": [184.25, 410.2, 171.0], "volume": [200, 50, 300]},
    index=["AAPL", "MSFT", "GOOG"],
)
print(df)
#        price  volume
# AAPL  184.25     200
# MSFT  410.20      50
# GOOG  171.00     300

print(df.shape, df.columns.tolist())              # (3, 2) ['price', 'volume']
print(df.dtypes.astype(str).to_dict())            # {'price': 'float64', 'volume': 'int64'}

rows = [{"symbol": "AAPL", "price": 184.25}, {"symbol": "MSFT", "price": 410.2}]
print(pd.DataFrame(rows))                         # from a list of row dictionaries
#   symbol   price
# 0   AAPL  184.25
# 1   MSFT  410.20

Columns, new columns and filtered rows

df["price"] is one column, a Series; df[["price"]] with a list is a one-column DataFrame. A new column comes from arithmetic on columns: df.assign(notional=df["price"] * df["volume"]) returns a new frame with the extra column and leaves df untouched (writing df["notional"] = ... modifies df itself). A Boolean condition on a column filters rows, df[df["volume"] >= 100], and df.sort_values("volume", ascending=False) orders them, with .head(n) keeping the first n. All of these work on whole columns at once: never loop over rows with iterrows() when a column operation exists.

import pandas as pd

df = pd.DataFrame(
    {"price": [184.25, 410.2, 171.0], "volume": [200, 50, 300]},
    index=["AAPL", "MSFT", "GOOG"],
)

bigger = df.assign(notional=df["price"] * df["volume"])   # a new frame
print(bigger)
#        price  volume  notional
# AAPL  184.25     200   36850.0
# MSFT  410.20      50   20510.0
# GOOG  171.00     300   51300.0
print(df.columns.tolist())                                # ['price', 'volume']  df is unchanged

print(df[df["volume"] >= 100])                            # rows that pass the test
#        price  volume
# AAPL  184.25     200
# GOOG  171.00     300

print(df.sort_values("volume", ascending=False).head(2))  # the two biggest volumes
#        price  volume
# GOOG  171.00     300
# AAPL  184.25     200

Summaries and a look at the data

Always look at a table before trusting it. df.head() shows the first rows, df.shape gives (rows, columns), df.dtypes lists the type of each column, df.isna().sum() counts missing values per column and df.describe() summarises the numeric columns (count, mean, standard deviation, minimum, quartiles, maximum). Column methods such as df.mean() give one value per column; df.sum(axis=1) gives one per row. A dtype of object or str where you expected numbers usually means text slipped into the column.

import pandas as pd

df = pd.DataFrame(
    {"price": [184.25, 410.2, 171.0], "volume": [200, 50, 300]},
    index=["AAPL", "MSFT", "GOOG"],
)

print(df.describe().loc[["count", "mean", "max"]])
#         price      volume
# count    3.00    3.000000
# mean   255.15  183.333333
# max    410.20  300.000000

print(df.mean().to_dict())                 # {'price': 255.15, 'volume': 183.33333333333334}
print(df.sum(axis=1).tolist())             # [384.25, 460.2, 471.0]   one value per row
print(df.isna().sum().to_dict())           # {'price': 0, 'volume': 0}

# Changing a copy never changes the original (pandas 3 copies on write)
cheap = df[df["price"] < 200].copy()
cheap.loc["AAPL", "price"] = 0.0
print(df.loc["AAPL", "price"])             # 184.25

import pandas as pd

rows = [
    {"symbol": "AAPL", "price": 184.25, "volume": 200},
    {"symbol": "MSFT", "price": 410.2, "volume": 50},
]
df = pd.DataFrame(rows).set_index("symbol")      # make a column the row labels
print(df)
#          price  volume
# symbol
# AAPL    184.25     200
# MSFT    410.20      50

print(df.rename(columns={"price": "px"}).columns.tolist())   # ['px', 'volume']
print(df.drop(columns="volume").columns.tolist())             # ['price']
print(df.reset_index().columns.tolist())                      # ['symbol', 'price', 'volume']

• pd.DataFrame({column: values}, index=labels); df["c"] is a Series, df[["c"]] a DataFrame; each column has its own dtype.
• assign adds a column to a new frame; df[mask] filters rows; sort_values(...).head(n) picks the top n.
• Look first: shape, dtypes, head(), describe(), isna().sum(); never edit the caller's frame.

Next up: selecting data with loc and iloc, the two precise ways to pick rows and columns by label or by position.

Lab · what's graded

• Build a table of prices and volumes with make_frame25 pts
• Add a traded-value column with add_notional25 pts
• Keep only the liquid rows with filter_liquid25 pts
• Rank the rows by any column with top_by25 pts