Learn in fit, apply in transform
A transformer is an estimator with a second job. fit(X) learns numbers from the data (a mean, a quantile, a vocabulary), stores them in attributes that end with an underscore, and returns self. transform(X) then applies those stored numbers to any data, new or old, without looking at that data to decide anything. This split is the whole point: the numbers are learned once, on the training rows, and the test rows are only transformed. A winsorizer is a good first example. fit learns one lower and one upper quantile per column, and transform clips with np.clip between them. If transform recomputed the quantiles on the data it receives, every dataset would be clipped to its own range and the training information would never reach the test set. Inherit from TransformerMixin and BaseEstimator and you also get fit_transform and the parameter machinery for free.
Real tables mix numbers and categories, so scikit-learn offers ColumnTransformer([(name, transformer, columns), ...]): each branch handles its columns and the outputs are glued side by side. Here the numeric branch is make_pipeline(Winsorizer(0.02, 0.98), StandardScaler()) and the categorical branch is OneHotEncoder(handle_unknown='ignore'), which turns an unseen category into a row of zeros instead of an error. sparse_threshold=0 forces a dense array. To name the output columns, every step must answer get_feature_names_out, which is why your winsorizer has to implement it: it simply returns the names it was given. The golden rule is to call fit_transform on the training rows and only transform on the test rows. On the stock panel (26 symbols, 10,296 rows) the split at 1 January 2023 leaves 5,564 training rows and 4,732 test rows, and about 2.2% of the test momentum values fall outside the training 2%–98% range and are clipped to the training bounds.
- Learn once, apply anywhere:
fitstores numbers on the training rows,transformonly applies them. - One preprocessor, two branches:
ColumnTransformersends numbers and categories to their own steps;handle_unknown="ignore"survives new categories. - Implement the contract fully: a transformer that cannot name its output columns breaks
get_feature_names_outfor the whole pipeline.
Lab · what's graded
- The constructor5 pts
- Fit and transform20 pts
- Feature names and pipelines10 pts
- The two-branch preprocessor15 pts
- Split by date10 pts
- Fit on the past only10 pts
- The panel study30 pts