Machine Learning II: Generalization and Model Selection
Holdout and cross-validation
Ben Dai
CUHK · Department of Statistics and Data Science
Generalization
01
A model can fit its training data beautifully and still fail on new examples.
Three familiar models make different assumptions
fewer learned values more stored detail
SIMPLE Global mean
Use one learned number for every user–item pair.
\[\widehat r_{ui}=\mu\]
MORE STRUCTURE Three-mean average
Average global, user, and item means, with fallback for missing IDs.
personalization without storing every answer
MAXIMUM DETAIL Memorization + fallback
Store every observed rating; use the global mean for an unseen pair.
perfect recall of the training rows
A model that remembers more can reduce training error . That alone does not tell us which model predicts unseen pairs best.
[Teaching] - These are deliberately familiar recommender examples. No new prediction algorithm is needed to motivate validation. - Memorization is a thought experiment: for a pair already in the training triples, return its stored rating exactly.
Memorization wins on the training ratings
TRAINING TRIPLES
user item rating
\(u_1\) \(i_2\) 4
\(u_2\) \(i_1\) 5
\(u_1\) \(i_4\) 2
store exact answers →
LOOKUP ON SEEN PAIRS
\((u_1,i_2)\) → \(\widehat r=4\)
\((u_2,i_1)\) → \(\widehat r=5\)
\((u_1,i_4)\) → \(\widehat r=2\)
training RMSE = 0
NOW PREDICT A NEW PAIR \((u_4,i_7)\) → \(\mu\)
No stored answer: fall back to the global mean.
Perfect reconstruction is not generalization: the training rows cannot reveal how the lookup rule behaves on a new pair.
Model selection needs fresh known ratings
FIT training triples
learn the global mean, user/item summaries, or stored lookup table
produce candidate predictors →
global mean
three-mean average
memorization + fallback
predict →
FRESH PAIRS ratings known to us
but never shown while fitting the candidates
TRAINING RMSE Can it reconstruct ratings it learned from?
VALIDATION RMSE Can it predict held-out ratings?
We call those fresh, scored rows the validation set . They let us compare candidates before the final test.
Train and validation reveal model fit
candidate training RMSE validation RMSE diagnosis
global mean underfitting
three-mean average useful fit
memorization + fallback overfitting
Underfitting The model misses useful patterns in both training and validation data.
Overfitting The model learns training details that do not transfer to held-out ratings.
Validation turns “fits the data” into the question we actually care about: does the learned pattern transfer?
[Teaching] - The RMSE labels are qualitative; they show the diagnostic pattern rather than claiming a benchmark value. - The middle row represents a reasonable baseline candidate, not a universal winner on every dataset.
Three fits to the same noisy data
The overfit curve learns the noise around the trend . Those deviations may not repeat on new data.
[Teaching] - Hold the observations fixed while comparing the three fitted curves. The dashed trend is visible only because this is a schematic example; with real data, we do not know it. - The flat fit misses the trend. The smooth fit captures it without matching every observation. The jagged fit has near-zero training error but follows noise. - Link this picture back to the preceding training-versus-validation RMSE table.
Model selection with Ridge regression
02
Keep the prediction rule fixed; use validation to choose the penalty strength.
Ridge regression: four questions
Return to the California housing model from ML I. Ridge(alpha=…) adds a penalty on large coefficients.
1 · MODEL
\(\widehat y=b+x^{\mathsf T}w\)
Predict from the feature values.
2 · HYPERPARAMETER
\(\alpha\geq 0\)
Choose the penalty strength before fitting.
3 · FITTING CRITERION \[\min_{b,w}\;\sum_{i=1}^{n}\bigl(y_i-b-x_i^{\mathsf T}w\bigr)^2+\alpha\lVert w\rVert_2^2\]
Fit the training rows; penalize the coefficients \(w\) , not the intercept \(b\) .
4 · LEARNED PARAMETERS \(\widehat b,\widehat w\)
After fit: intercept_ and coef_.
[Teaching] - Reuse the four-question template from ML I. Ridge has the same prediction rule as LinearRegression; only its fitting criterion changes. - Alpha is set before each fit. Coefficients and intercept are learned separately for each alpha from the fitting rows. - The objective is the training criterion. Held-out RMSE is used later to choose alpha; it is not added to this objective.
[Sources] - Ridge objective, alpha, fitted attributes: https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.Ridge.html.
What does \(\alpha\) change?
\(\alpha=0\) No penalty
The fitting objective becomes ordinary least squares. Coefficients may follow training noise.
moderate \(\alpha\) Some shrinkage
Smaller coefficients can improve prediction on new rows.
large \(\alpha\) Strong shrinkage
Too much penalty can miss useful relationships.
We do not know the best \(\alpha\) from the training loss alone. Compare held-out RMSE.
[Teaching] - This is a qualitative pattern, not a guarantee that the middle setting wins on every dataset. - At alpha zero the fitting objective is ordinary least squares. In scikit-learn, use LinearRegression for an actual unpenalized fit rather than Ridge(alpha=0).
[Sources] - Alpha and zero-penalty guidance: https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.Ridge.html.
Holdout validation chooses \(\alpha\)
FIT SPLIT fit Ridge for each \(\alpha\)
learn a new \(\widehat b\) and \(\widehat w\) each time
predict held-out rows
VALIDATION SPLIT compare RMSE
score each fitted Ridge model without refitting it
\(\alpha=0.1\) higher validation RMSE
\(\alpha=1\) lowest validation RMSE
\(\alpha=100\) higher validation RMSE
In this illustration, choose \(\alpha=1\) . These rankings are schematic; the best value depends on the data.
[Teaching] - This is a schematic comparison, not a measured benchmark. Each alpha needs a fresh fit on the same fitting rows. - The validation labels choose alpha. They must not be used to fit its coefficients. Keep final test labels outside the comparison. - This Ridge example follows the same fit/validation distinction introduced with the rating baselines.
One validation split can give a noisy ranking
SPLIT A
best: \(\alpha=0.1\)
one held-out sample favors weaker shrinkage
SPLIT B
best: \(\alpha=1\)
another sample favors stronger shrinkage
One validation sample the score depends on which rows were held out
One \(\alpha\) ranking a close comparison may reverse on another sample
Natural extension let every block take a turn as the validation set
K-fold CV rotates the validation role
ROUND 1 VALID FIT FIT FIT FIT RMSE1
ROUND 2 FIT VALID FIT FIT FIT RMSE2
ROUND 3 FIT FIT VALID FIT FIT RMSE3
ROUND 4 FIT FIT FIT VALID FIT RMSE4
ROUND 5 FIT FIT FIT FIT VALID RMSE5
\[\mathrm{CV\ RMSE}=\frac{1}{5}\sum_{j=1}^{5}\mathrm{RMSE}_j\]
For each \(\alpha\) , fit Ridge five times and average its validation RMSE. Choose \(\alpha\) using these averages.
[Sources] - Cross-validation process: https://scikit-learn.org/stable/modules/cross_validation.html.
[Teaching] - Present K-fold CV as repeated holdout validation. The only new idea is rotating which block plays the validation role. - For every alpha, repeat all five fitting rounds. Keep the same folds across candidate alpha values so their scores are comparable. - The secret test set stays outside CV.
After validation, refit and test once
FITTING ROUNDS fit Ridge for each \(\alpha\)
HOLDOUT OR CV SCORES select \(\alpha\)
selection finished
ALL DEVELOPMENT DATA refit Ridge with that \(\alpha\)
learn new coefficients from every development row
evaluate once
TEST final RMSE
no more model changes afterward
More fitting data helps after the model choice has been made.
[Teaching] - Whether selection used holdout or CV, refit only after alpha is chosen. The coefficients in the final model differ from those in individual folds. - Keep final test labels out of both selection and refitting. - The next two slides show how GridSearchCV performs the candidate comparisons and final refit in sklearn.
GridSearchCV puts the CV loop in sklearn
from sklearn.linear_model import Ridgefrom sklearn.model_selection import GridSearchCV= GridSearchCV(= Ridge(),= {"alpha" : [0.1 , 1 , 100 ]},= "neg_root_mean_squared_error" ,= 5 ,= True ,
MODEL Ridge()
Fit a fresh Ridge model on the fitting rows of each fold.
CANDIDATES three \(\alpha\) values
param_grid lists the settings we want to compare.
SELECTION five-fold RMSE
fit compares all candidates, selects one, then refits it on all training rows.
[Teaching] - The housing arrays X_train_housing and y_train_housing come from ML I. The held-out housing test arrays do not enter this call. - This uses the same Ridge estimator students just saw. alpha is a constructor setting, so param_grid can name it directly. - GridSearchCV tries three alpha values on five folds: 15 fold fits, followed by one final refit. The default refit=True is written explicitly to connect with the preceding slide. - sklearn scorers are maximized, so negative RMSE is used here. Explain the sign on the next slide.
[Sources] - GridSearchCV arguments, cross-validation, and refit: https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html. - Scorer name: https://scikit-learn.org/stable/modules/model_evaluation.html#scoring-string-names.
Read the selected Ridge model
CHOSEN SETTING best_params_
Which candidate \(\alpha\) won?
CV PERFORMANCE -best_score_
Change sklearn’s negative score back to positive RMSE.
FINAL FIT best_estimator_
The selected Ridge model is already refitted on all training rows.
from sklearn.metrics import root_mean_squared_error= search.best_params_["alpha" ]= - search.best_score_= search.best_estimator_= final_model.predict(X_test_housing)= root_mean_squared_error(y_test_housing, test_pred)
Choose with CV, report on test once. Do not call fit again: search.fit(…) already did the final refit.
[Teaching] - Do not claim that a particular alpha wins here; the result comes from running the search on the housing training split. - best_score_ is the mean of the best candidate’s negative fold RMSE scores. Negating it gives the average fold RMSE shown conceptually on the previous slide. - best_estimator_ is a fitted Ridge model because refit=True. search.predict(X_test_housing) would also work, but the explicit attribute makes the refit step visible. - The test RMSE is for final evaluation, not another candidate-selection score.
[Sources] - Fitted attributes and prediction after refit: https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html.
Diagnose Ridge with fitting and CV error
A 200-row sample of the housing training data makes the effect visible. Each point averages five fitting or validation folds.
The lowest fitting error is at weak regularization. The lowest CV error among these candidates is at \(\alpha=100\) .
Course housing training data · 200 rows sampled with seed 42 · 5-fold CV · illustrative diagnostic, not the final full-data search
[Teaching] - Ask students to point to the smallest training error, then the smallest validation error. Those are different choices. - The curves are measured, not schematic. random.Random(42).sample selects 200 rows from dataset/housing/train.csv; GridSearchCV with five default regression folds and return_train_score=True reproduces the plotted values. - Do not claim alpha 100 is best for the full housing training set. The preceding ridge_search uses all training rows for the actual selection.
[Sources] - return_train_score and cv_results_: https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html.
Read the curve before selecting
WEAK PENALTY · \(\alpha=0.1\) fit 0.6818 · CV 0.8039
The fit is better than validation. A low training score alone is not the goal.
BEST OF THIS GRID · \(\alpha=100\) fit 0.7506 · CV 0.7802
Sacrificing some fitting accuracy improves validation prediction here.
TOO STRONG · \(\alpha=10{,}000\) fit 1.1625 · CV 1.1686
Both errors are high: useful relationships are being suppressed.
A validation curve is a diagnostic for this sample . The full-data search may select another \(\alpha\) ; it must still keep the final test set untouched.
[Teaching] - The values are mean fold RMSE, rounded to four decimals. Training and validation folds have different sample sizes. - Ask: if the full training set is much larger, must this curve keep the same minimum? No. Use the full-data search for the final alpha. - This example is evidence for interpreting train/validation gaps, not a claim that CV curves always have a U shape.
Apply the workflow to rating prediction
03
The data roles stay the same when we return to user–item ratings.
Netflix Prize: data roles in practice
KNOWN RATINGS Training → fit
Use these ratings while developing the model.
PUBLIC FEEDBACK Quiz → leaderboard
Labels are hidden, but the score can influence later choices.
FINAL CHECK Test → final score
Keep it outside the model-selection loop.
Diagram: Pantelis Monogioudis, Netflix Prize lecture notes . Split roles: Koren (2009) , Section II.
[Teaching] - Transfer the fit/validation/test roles from Ridge to the Netflix Prize. This is an example of split roles, not a Ridge experiment. - The image is the same Netflix split diagram used in the baseline lecture. Its training and probe ratings were available during development; quiz and test ratings were hidden. - Quiz feedback is public, so repeated use can influence model choice. The final test stays outside that loop.
[Sources] - Diagram: https://pantelis.github.io/cs301/docs/common/lectures/recommenders/netflix/. - Split roles and public/final score distinction: https://www2.seas.gwu.edu/~simhaweb/champalg/cf/papers/KorenBellKor2009.pdf, Section II.
For rating data, split pairs before fitting
OBSERVED TRIPLES
user item rating
u1 i2 4
u2 i1 5
u1 i4 2
u3 i2 3
fold
FOLD TRAIN learn model values
global mean, user means, or item means from these triples only
predict
FOLD VALIDATION pairs \((u,i)\) → ratings \(\widehat r_{ui}\)
compare with held-out ratings only after prediction
Compute learned quantities after the fold split. Computing them earlier leaks validation information into training.
[Teaching] - The fold rule applies to both datasets: fit every transformation and learned model value using only that fold’s fitting rows. - For the familiar rating baselines, compute the means from fitting triples only.
Which future pair are we trying to predict?
PAIR-LEVEL HOLDOUT Usually known IDs, new pair
Remove selected ratings. Other ratings from these users and items may remain in fitting data.
Mostly asks: can we predict another interaction?
USER HOLDOUT Entire user is new
Keep every rating from selected users out of fitting. Their user means cannot be learned.
Asks: how do we serve a new user?
ITEM HOLDOUT Entire item is new
Keep every rating for selected items out of fitting. Their item means cannot be learned.
Asks: how do we serve a new item?
Choose the validation split to match the prediction situation . A random pair split can still contain a few unseen IDs, but it does not deliberately test cold start.
[Teaching] - The words “known” and “new” are always relative to the fitting part of a fold, not the full CSV file. - Pair-level holdout does not guarantee warm validation rows. Rare users or items may appear only in the held-out rows. - Whole-user or whole-item holdout is a conceptual split design here; no new sklearn splitter API is required today.
[Source] - Group-based CV keeps a group entirely in one side of a split: https://scikit-learn.org/stable/modules/cross_validation.html.
Our pair-level fold is mostly warm
In fold 1 of the course’s 0/1 rating data, “known” means the ID appears in the other four fitting folds.
97.5% of validation ratings have both a known user and a known item
validation pair
count
specific means available
known user + known item
9,980
user and item
new user + known item
35
item only
known user + new item
217
user only
new user + new item
1
none; use global mean
An overall pair-level RMSE is dominated by the warm rows. Report the cold cases separately if new IDs matter to the application.
[Teaching] - Counts come from dataset/b-rs/train.csv using the first of the five default regression folds created by GridSearchCV(cv=5). Fold 1 has 10,233 validation ratings. - This pair-level fold has 253 cold validation rows, only 2.5% of the fold. Overall RMSE therefore says little about cold-start quality. - “Available means” means user/item means learned from fitting rows. The global mean always exists.
Pick the split before the candidate
For each deployment question, which validation design would you use?
A · EXISTING CATALOG Will a returning user like another known film?
Hold out pairs ; score predictions for known users and items.
B · NEW MEMBERS What can we recommend before a user has rated anything?
Hold out entire users ; the user-mean path needs a fallback.
C · NEW FILMS How will newly added films perform?
Hold out entire items ; the item-mean path needs a fallback.
The same three-mean model can receive different scores under these splits because the question changes , not because the formula changes.
[Teaching] - Pause before revealing the answers. Ask which learned mean is unavailable in B and C. - A whole-user/item holdout may create other cold cases too. If a deployment setting needs both new, define that explicitly. - This discussion sets up later cold-start models; today’s weight selection still uses the stated pair-level folds.
Three-mean RS: the weights are the choice
Return to the three-mean predictor from the previous lecture.
LEARNED IN EACH FOLD Three means
Compute the global, user, and item means from that fold’s fitting ratings. For an unseen ID, use that fold’s global mean.
CHOSEN BY CROSS-VALIDATION Three weights
Try several user / item / global weight triples. Every candidate uses the same five folds.
GridSearchCV changes the weights ; each fresh fit learns means only from its fitting fold.
[Teaching] - Equal weights from the previous lecture are one candidate, not a required choice. - The mean values are learned separately from each fold’s fitting rows. The three weights are hyperparameters selected using validation RMSE. - If a validation pair has an unseen user or item, replace only the unavailable mean with the global mean from that fold’s fitting rows. - The notebook supplies ThreeMeanRS with the same sklearn fit / predict structure students used in ML I. Its constructor exposes weights to GridSearchCV.
GridSearchCV selects the RS weights
from sklearn.model_selection import GridSearchCV= [(1 / 3 , 1 / 3 , 1 / 3 ), (0.5 , 0.3 , 0.2 ),0.3 , 0.5 , 0.2 ), (0.2 , 0.2 , 0.6 )]= GridSearchCV(= ThreeMeanRS(),= {"weights" : weight_grid},= "neg_root_mean_squared_error" ,= 5 ,= True ,
ESTIMATOR ThreeMeanRS()
Each fit learns means from four folds; predict scores the fifth.
CANDIDATES four weight triples
weights is a constructor setting, just like Ridge’s alpha.
FINAL FIT best_estimator_
The winner is refitted on all training ratings.
[Teaching] - ThreeMeanRS is supplied in the notebook. It follows the fit / predict estimator structure from ML I and exposes weights in its constructor. - The search runs four candidates on five pair-level folds. For every fit, the means come only from the fitting rows, so validation labels cannot leak into them. - The cv=5 setting matches the Ridge example. Every candidate sees the same five pair-level folds; no manual fold indices are needed. - With refit=True, the selected weights and means fitted on all training ratings are in search.best_estimator_.
[Source] - GridSearchCV and refit behavior: https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html.
Five-fold CV compares four weight candidates
For the course’s 0/1 rating split , GridSearchCV reports each candidate’s five validation RMSEs and their mean. Lower is better.
Candidate
User / item / global
Fold 1
Fold 2
Fold 3
Fold 4
Fold 5
CV mean
Equal
1/3, 1/3, 1/3
0.4658
0.4674
0.4681
0.4676
0.4663
0.4670
User-heavy
0.5, 0.3, 0.2
0.4618
0.4637
0.4647
0.4645
0.4623
0.4634
Item-heavy
0.3, 0.5, 0.2
0.4644
0.4667
0.4665
0.4655
0.4646
0.4656
Global-heavy
0.2, 0.2, 0.6
0.4751
0.4757
0.4768
0.4765
0.4758
0.4760
Choose (0.5, 0.3, 0.2) among these four. best_estimator_ already has the means refitted on all training ratings.
Course dataset/b-rs/train.csv · pair-level 5-fold CV via GridSearchCV(cv=5) · RMSE rounded to four decimals
[Teaching] - Each row is one hyperparameter candidate, not a different fitting dataset. Keep the five folds fixed when comparing rows. - The numbers use the 51,161 ratings in dataset/b-rs/train.csv and GridSearchCV(cv=5, scoring='neg_root_mean_squared_error'). For each fold, ThreeMeanRS.fit computes means from the other four folds and uses that fold’s global mean for unavailable user/item means. The table negates sklearn’s scores and rounds only for display. - The selected weights match the illustrative 0.5 / 0.3 / 0.2 weighting in the previous lecture’s notebook. This table supplies the validation step that example deferred. - “Best” means best of the four listed candidates on these folds; the test ratings are not used for selecting weights.
How an estimator plugs into sklearn
We supply an estimator; sklearn reuses its methods during model selection.
→
GridSearchCVrepeats for every candidate and fold
SET alpha or weights
constructor setting
→
ESTIMATOR SLOT Ridge or YourRS
__init__(…)fit(X_fit, y_fit)predict(X_valid)
→
VALIDATE RMSE
on held-out rows
→
OUTPUT best_estimator_
Ready for predict(X_test).
Define YourRS once. With BaseEstimator and the same fit / predict interface, it can use sklearn’s CV and search tools.
[Teaching] - This is a synthesis of components already seen: ID encoding from the warm-up and ML I, sklearn-style estimators from ML I, and GridSearchCV from this lecture. - The highlighted estimator is the interchangeable node. GridSearchCV fits fresh copies on each fitting fold, scores held-out rows, selects constructor settings, and refits the winner. It does not automatically repeat the separate ID encoding step. - alpha belongs to Ridge; weights belongs to the supplied ThreeMeanRS class. YourRS is the generic version of that example: constructor settings, fold-specific fit, and predict let the same search machinery work with a new estimator. - For preprocessing that learns values from training data, the same fold boundary applies: fit it on each fitting fold only. A pipeline would automate that later; it is not needed for today’s ID lookup example.
[Sources] - Custom estimator contract and BaseEstimator parameter methods: https://scikit-learn.org/stable/developers/develop.html. - GridSearchCV CV and refit behavior: https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html.
The complete workflow
01 Define
inputs, target, metric
→
02 Develop
fit candidate estimators
→
03 Validate
holdout or cross-validation
→
04 Refit
best choice on all development data
→
05 Test once
predict and report RMSE
FIT learns from known answers
VALIDATE chooses without touching test
TEST checks the final frozen workflow
Next: use the same workflow to develop matrix factorization models.
[Teaching] - Ask students to map Ridge alpha and three-mean weights onto each of the five visible steps.
