A probability, a baseline and a fair test
Logistic regression turns a linear score z=b₀+b₁x₁+…+bₚxₚ into a probability p=1/(1+e^(-z)). The score is the log of the odds: lnp/(1-p)=z, so one extra unit of xⱼ multiplies the odds of an up day by e^bⱼ. Put a StandardScaler in front, inside a Pipeline, and the unit becomes one standard deviation, which makes the odds ratios of different features comparable. The model is fitted by minimising the log loss -1/nΣ(yln p+(1-y)ln(1-p)) plus a penalty controlled by C (a small C means a strong penalty). The fitted object gives predict_proba for probabilities and predict for the class at the threshold 0.5.
A direction model must be judged against what it could achieve by doing nothing. About 55% of SPY days are up days, so always predicting 'up' is right 55% of the time: that is the baseline, and it must come from the train labels, because the test labels are not known when you predict. Accuracy depends on the threshold; the AUC does not, and equals the probability that a random up day gets a higher score than a random down day (0.5 means no skill). The log loss punishes confident mistakes. With four features known at the close, split in time (70% for training, the later 30% for testing), the lab model gets an accuracy of 55.3% against 55.1% for the baseline, an AUC of 0.501, and a log loss of 0.688, barely under the 0.690 of a coin. Odds ratios just under 1 say that yesterday's rise slightly lowers the odds of another, a real but untradable effect.
- Judge against a baseline taken from the train labels: predicting the majority class already scores about 55%.
- AUC and log loss do not depend on one threshold; 0.5 and 0.69 are the values of no skill.
- Scale inside the pipeline, then e^bⱼ is the odds ratio per one standard deviation.
Lab · what's graded
- Splitting in time10 pts
- The pipeline10 pts
- Three scores20 pts
- The majority-class baseline15 pts
- Odds ratios15 pts
- The SPY direction study30 pts