Descriptive statistics and robust estimators

Summarising a messy sample

Descriptive statistics compress a sample into a few numbers: where it sits (location), how spread out it is (scale) and what shape it has. The usual summaries are the mean $\bar x=\frac1n\sum x_i$ and the sample standard deviation $s=\sqrt{\frac{1}{n-1}\sum(x_i-\bar x)^2}$ , with $n-1$ in the denominator because the mean has already been estimated from the same data. They are optimal for clean, bell-shaped data and fragile otherwise.

\operatorname{MAD}=\operatorname{median}_i\,|x_i-\operatorname{median}(x)|

(1)

Real return data has outliers: a crash day, a bad tick, a merger. A single extreme value drags the mean and inflates the standard deviation without limit, which is why they have a breakdown point of zero. Robust alternatives resist that: the median for location, the median absolute deviation (MAD) for scale, and the trimmed mean, which drops a fraction of the smallest and largest values before averaging. Winsorising is similar but clips the extremes instead of discarding them.

One bad dayTake the sample

2,\,3,\,3,\,4,\,100

. The mean is

22.4

, nowhere near any observation, because of the 100. The median is

3

. Dropping the smallest and largest value gives a trimmed mean of

(3+3+4)/3=3.33

. The absolute deviations from the median are

1,0,0,1,97

, so the MAD is

1

. Compare the sample standard deviation of about 43.4: one value turned a typical spread of 1 into a number 40 times larger.

Location: mean (fragile), median and trimmed mean (robust).
Scale: standard deviation (fragile), MAD and interquartile range (robust).
Robust is not free: for clean normal data the mean is more precise than the median, so choose by the data you expect.

Do not delete outliers, decide about themA crash day is real data, and a risk model that trims it away reports a calm world. A bad tick is an error and should be fixed at the source. Look at each outlier, label it as error or event, and then choose a method on purpose: robust statistics describe the typical day, while the mean and standard deviation remain the right tools when the extremes are exactly what you care about, as in risk.

Lab · what's graded

The mean10 pts
The median10 pts
A trimmed mean15 pts
The median absolute deviation15 pts
Which estimators are robust15 pts
A sample standard deviation15 pts
One extreme day20 pts

Summarising a messy sample

\operatorname{MAD}=\operatorname{median}_i\,|x_i-\operatorname{median}(x)|

(1)

One bad dayTake the sample

2,\,3,\,3,\,4,\,100

. The mean is

22.4

, nowhere near any observation, because of the 100. The median is

3

. Dropping the smallest and largest value gives a trimmed mean of

(3+3+4)/3=3.33

. The absolute deviations from the median are

1,0,0,1,97

, so the MAD is

1

. Compare the sample standard deviation of about 43.4: one value turned a typical spread of 1 into a number 40 times larger.

Location: mean (fragile), median and trimmed mean (robust).
Scale: standard deviation (fragile), MAD and interquartile range (robust).
Robust is not free: for clean normal data the mean is more precise than the median, so choose by the data you expect.

Lab · what's graded

The mean10 pts
The median10 pts
A trimmed mean15 pts
The median absolute deviation15 pts
Which estimators are robust15 pts
A sample standard deviation15 pts
One extreme day20 pts

Descriptive statistics and robust estimators

Summarising a messy sample

Lab · what's graded

Log in to run your code.

Descriptive statistics and robust estimators

Summarising a messy sample

Lab · what's graded

Log in to run your code.