Counting by range
The mean and the standard deviation squeeze a thousand returns into two numbers. A histogram keeps the shape: it cuts the range of values into bins and counts how many observations land in each one. Tall bars in the middle and short bars at the ends is the familiar bell; an unbalanced or heavy-ended shape is a warning. np.histogram(values, bins=edges) returns the counts and works on a whole array. Bin i takes the values from edges[i] up to, but not including, edges[i + 1]; only the last bin also includes its right edge.
import numpy as np
returns = np.array([-0.021, -0.012, -0.008, -0.004, -0.002, 0.000, 0.001, 0.003, 0.004, 0.006, 0.009, 0.015, 0.032])
edges = np.array([-0.03, -0.01, 0.0, 0.01, 0.03]) # four bins: -3%..-1%, -1%..0%, 0%..1%, 1%..3%
counts, _ = np.histogram(returns, bins=edges)
print(counts.tolist(), counts.sum(), len(returns)) # [2, 3, 6, 1] 12 13 the 3.2% return is outside every bin
# bins=4 asks numpy for 4 equal-width bins between the minimum and the maximum
counts, edges4 = np.histogram(returns, bins=4)
print(counts.tolist(), edges4.round(4).tolist()) # [3, 6, 3, 1] [-0.021, -0.0078, 0.0055, 0.0188, 0.032]Frequency or density
Counts depend on how many observations you have and on the width of the bins, so two histograms with different sample sizes cannot be compared. A density fixes that: divide each count by the total number counted and by the bin width, so the area of the bars adds up to 1. A density can then be compared with a theoretical curve, such as the normal bell, or with another sample. np.histogram(values, bins=edges, density=True) does it, and so does counts / (counts.sum() * np.diff(edges)).
counts, _ = np.histogram(returns, bins=edges)
density = counts / (counts.sum() * np.diff(edges))
print(density.round(2).tolist()) # [8.33, 25.0, 50.0, 4.17]
print((density * np.diff(edges)).sum()) # 1.0 the areas add up to 1Drawing it, and reading the tails
ax.hist(values, bins=10) counts and draws in one call: one bar per bin (ax.patches), whose heights are the counts. Always label the axes, since a histogram of returns is meaningless without "Daily return" under it. The shape is what matters, and the tails are what hurt: counting the share of observations more than k standard deviations from the mean, np.mean(np.abs(x - mean) > k * std), tells you how often extremes happen. For a normal distribution about 4.6% of the values lie beyond 2 standard deviations and 0.27% beyond 3. Markets are worse than that, as the next topics show.
import matplotlib
matplotlib.use("Agg")
import matplotlib.pyplot as plt
fig, ax = plt.subplots()
ax.hist(returns, bins=5)
ax.set_xlabel("Daily return")
ax.set_ylabel("Frequency")
print(len(ax.patches), [round(p.get_height()) for p in ax.patches]) # 5 [2, 4, 5, 1, 1] one bar per bin; the heights are the counts
# How often are the returns more than 1 standard deviation from the mean?
mean, std = returns.mean(), returns.std(ddof=1)
print(round(float(np.mean(np.abs(returns - mean) > 1 * std)), 3)) # 0.308 about 4 of 13 days# The number of bins changes the picture: compare a few
for bins in (2, 4, 8):
print(bins, np.histogram(returns, bins=bins)[0].tolist())
# 2 [9, 4]
# 4 [3, 6, 3, 1]
# 8 [1, 2, 2, 4, 2, 1, 0, 1]np.histogram(values, bins=edges)counts the values in each bin; the last bin includes its right edge.- A density divides by the total counted and the bin width, so the areas add up to 1 and samples become comparable.
ax.hist(values, bins=n)draws it;np.mean(np.abs(x - mean) > k * std)measures the tails. Label the axes and compare with the same edges.
Next up: mean and variance of returns, the two numbers that summarise a distribution, computed from their definitions.
Lab · what's graded
- Count the returns in each bin with
bin_counts25 pts - Turn the counts into a density with
density25 pts - Measure the tails with
fraction_beyond25 pts - Draw the histogram with
plot_histogram25 pts