Machine Learning I: Models and Estimators

Today’s question

WE CAN ALREADY PREDICT global mean · user mean · item mean

How do we move from hand-designed averages to a general way of learning prediction rules from data?

BEFOREChoose a formula

Then calculate its averages from the training triples.

→

TODAYChoose a model family

Then learn its parameters from examples.

→

THE REAL CHALLENGEGeneralize

Work well on ratings the model has never seen.

ML I · A language for prediction

01

Start with $X$ and $y$. Use four questions to understand a prediction model and how it learns.

Rating prediction is supervised learning

RECOMMENDER SYSTEM

user$u$

item$i$

→

rating$r_{ui}$

HOUSING REGRESSION · EXAMPLE INPUTS

income$x_1$

location$x_2$

→

value$y$

Supervised learning: learn a mapping from inputs to a known numerical target, then apply it to new inputs.

One row becomes one learning example

RATING DATA

user	item	rating
1960	670	4
1346	152	4
785	1741	4

$X$ contains the inputs; $y$ contains the answer.

ONE EXAMPLE

$x_1=(1960,670)$

the user–item pair

paired with

$y_1=4$

the observed rating

The row number is not a feature. It only identifies an observation.

Four questions decode every estimator

1 · MODEL$f_{\theta,h}(x)$

What family of prediction rules can this estimator represent?

2 · LEARNED PARAMETERS$\theta$

What quantities will fit learn from the training data?

3 · HYPERPARAMETERS$h$

What choices are fixed before fit and control the model or fitting rule?

4 · LOSS / CRITERION$L(y,\widehat y)$

What does the fitting procedure regard as a better solution?

BEFORE FITchoose settings\[h\]

→

DURING FITlearn from training data\[\widehat\theta\]guided by the loss or fitting criterion

→

AFTER FITpredict new rows\[\widehat y=f_{\widehat\theta,h}(x_{\mathrm{new}})\]

ML principles through California housing

02

Use one dataset to understand a model family, training loss, and evaluation on new examples.

California housing: the data for our model

ROWS20,640

California block groups

FEATURES8

demographic and geographic measurements

TARGETMedHouseVal

median house value in units of $100,000

MedIncHouseAgeAveRoomsAveBedrmsPopulationAveOccupLatitudeLongitude

X_train.shape(13828, 8)y_train.shape(13828,)

Source: scikit-learn California housing dataset documentation.

Linear regression combines the features

\[f_{\theta}(x)=\theta_0+\sum_{j=1}^{8}\theta_j x_j\]

$\theta_0$intercept

the model’s reference level

$\theta_j$coefficient

how prediction changes with feature $j$, holding others fixed

$\widehat y$prediction

a weighted sum calculated for one row

Here each row has eight housing features. A choice of nine numbers $\theta_0,\ldots,\theta_8$ gives one prediction rule.

Least squares fits the housing coefficients

ONE TRAINING BLOCK GROUP \[e_k(\theta)=y_k-f_\theta(x_k), \qquad \ell_k(\theta)=e_k(\theta)^2\] ALL TRAINING BLOCK GROUPS \[\widehat\theta=\arg\min_{\theta\in\mathbb R^9} \frac{1}{n}\sum_{k=1}^{n}\left(y_k-f_\theta(x_k)\right)^2\]

minimum is the smallest training loss.

arg minimum is the coefficient vector that achieves it.

Training chooses the intercept and eight coefficients that minimize the squared errors on the training rows.

Training and test answer different questions

TRAINING SET How should the model fit?

X_train and y_train are available
estimates parameters
may be reused during fitting

TEST INPUT AND LABEL What may the model see?

X_test is available for prediction
y_test does not estimate parameters
y_test is opened only for final evaluation

Generalization is performance on new examples drawn from the task we care about.

From model to scikit-learn estimator

03

First understand the class and its fit/predict contract. Then use LinearRegression on the housing data.

One contract for every prediction model

CONSTRUCTmodel = Estimator(…)

Choose the model and its settings.

model.fit(X, y)

LEARNfitted estimator

Store patterns learned from the training data.

model.predict(X_new)

APPLYprediction vector

Return one prediction for each new row.

LinearRegression()KNeighborsRegressor()YourRegressor()can share the same interface

The contract specifies how we use a model. Each estimator supplies its own learning and prediction rules.

Class, estimator, and fitted estimator

PYTHON CLASS LinearRegression

__init__(settings)
fit(X, y)
predict(X_new)

Defines settings and methods; fit later stores learned attributes.

call the classcreates

ESTIMATOR INSTANCE model = LinearRegression( fit_intercept=True)

available nowfit_intercept

not learned yetcoef_

A configured Python object, ready to receive data.

model.fit(X, y)updates

FITTED ESTIMATOR model

configurationfit_intercept

learned from datacoef_

learned from dataintercept_

The same object now contains fitted state.

model.fit(X, y) is modelTruefit conventionally returns self.

Read `LinearRegression` with the four questions

MODEL$f_{\theta}(x)=\theta_0+\sum_{j=1}^{8}\theta_jx_j$

One intercept and eight feature coefficients.

LEARNED PARAMETERS$\widehat\theta_0,\ldots,\widehat\theta_8$

Stored after fit as intercept_ and coef_.

HYPERPARAMETERfit_intercept=True

Chosen before fitting; controls whether the model includes an intercept.

LOSS / CRITERION$\sum_{k=1}^{n}(y_k-f_{\theta}(x_k))^2$

Ordinary least squares uses the training block groups.

THE FITTED RULEchoose fit_intercept → fit nine values from the training rows → predict new house values

Fit and predict with California housing

from sklearn.linear_model import LinearRegression

model = LinearRegression(fit_intercept=True)
model.fit(X_train, y_train)

model.coef_        # eight learned coefficients
model.intercept_   # learned intercept

y_pred = model.predict(X_test)

fit_interceptchosen before fittingControls whether the model includes $\theta_0$.

coef_, intercept_learned during fitThe trailing underscore marks fitted attributes.

y_predone value per test rowpredict sees eight features and no target.

model is one object: fit stores the learned values; predict applies them to new block groups.

Many errors become one RMSE

Illustrative test rows; house values in units of $100,000

TRUE$y$2.41.83.22.6

−

PREDICTED$\widehat y$2.12.02.72.8

→

ERROR$e$0.3−0.20.5−0.2

01square

0.09, 0.04, 0.25, 0.04

02average

$(0.42)/4=0.105$

03square root

$\mathrm{RMSE}=0.324$

root_mean_squared_error(y_test, y_pred) calculates RMSE from the true and predicted test values.

Recommender baselines: same template, new models

04

Now $X$ contains user–item pairs and $y$ contains ratings. Answer the four questions before coding each mean rule.

`GlobalMeanRS`: specify before coding

MODEL$f_c(u,i)=c$

The family contains every constant prediction rule.

HYPERPARAMETERSnone

No setting is chosen before fitting.

LOSS / CRITERION$J(c)=\sum_{j=1}^{n}(r_j-c)^2$

$r_1,\ldots,r_n$ are the training ratings. Compare constants by total squared error.

FIT · OPTIMIZE

MINIMIZE THE LOSS\[\widehat\mu=\arg\min_{c\in\mathbb R}J(c)\]

→

THE FITTED VALUE\[\widehat\mu=\frac{1}{n}\sum_{j=1}^{n}r_j\]

LEARNED PARAMETER AFTER fit$\widehat\mu$

stored as global_mean_ · predict returns it once per new row

Now implement `GlobalMeanRS`

import numpy as np
from sklearn.base import BaseEstimator, RegressorMixin
from sklearn.utils.validation import check_is_fitted

class GlobalMeanRS(RegressorMixin, BaseEstimator):
    def fit(self, X, y):
        self.global_mean_ = np.asarray(y).mean()
        return self

    def predict(self, X):
        check_is_fitted(self, "global_mean_")
        return np.full(len(X), self.global_mean_)

FITone learned valueglobal_mean_ comes from y.

PREDICTone value per rowX determines only how many predictions to return.

CHECKfit must happen firstOtherwise scikit-learn raises NotFittedError.

`UserMeanRS`: specify before coding

MODEL$f_{\mathbf a}(u,i)=a_u$

One candidate constant per user; use the global rule for an unseen user.

HYPERPARAMETERSnone

The grouping and fallback rule are fixed in this definition.

LOSS / CRITERION$J_u(c)=\sum_{i\in I_u}(r_{ui}-c)^2$

$I_u$ contains items rated by user $u$ in training.

FIT · EACH USER $u$

MINIMIZE USER LOSS\[\widehat a_u=\arg\min_{c\in\mathbb R}J_u(c)\]

→

USER’S RATING MEAN\[\widehat a_u=\frac{1}{|I_u|}\sum_{i\in I_u}r_{ui}\]

LEARNED PARAMETERS AFTER fit$\widehat\mu,\{\widehat a_u\}$

stored as global_mean_ and user_means_ · retrieve by user ID

User mean learns a lookup vector

GROUP TRAINING RATINGS BY USER

$u_0$423mean = 3.0

$u_1$543mean = 4.0

$u_2$2mean = 2.0

store

FITTED ATTRIBUTES

global_mean_ = 3.3

user ID        0    1    2
user_means_ = [3.0, 4.0, 2.0]

Only a user absent from training uses global_mean_.

Every observed user receives a mean. The global mean handles users absent from training.

Implement `UserMeanRS`: `fit`

class UserMeanRS(RegressorMixin, BaseEstimator):
    def fit(self, X, y):
        X, y = np.asarray(X), np.asarray(y)
        users = X[:, 0].astype(int)
        self.global_mean_ = y.mean()
        self.user_means_ = np.full(
            users.max() + 1, self.global_mean_
        )

        for user in set(users):
            ratings = y[users == user]
            self.user_means_[user] = ratings.mean()
        return self

global_mean_fallback learned by fit

user_means_NumPy lookup vector learned by fit

users == userselect that user’s ratings

Input: a two-column array with nonnegative user ID in column 0.

Complete `UserMeanRS`: `predict`

def predict(self, X):
    check_is_fitted(
        self, ["global_mean_", "user_means_"]
    )
    users = np.asarray(X)[:, 0].astype(int)
    y_pred = np.full(len(users), self.global_mean_)

    valid = (users >= 0) & (users < len(self.user_means_))
    y_pred[valid] = self.user_means_[users[valid]]
    return y_pred

check_is_fittedrequire fit first

y_predstart every row at the global mean

validwhich user IDs have a safe lookup index?

y_pred[valid]replace fallback with learned user means

Add this method inside the UserMeanRS class from the previous slide.

`ItemMeanRS`: transfer the same template

MODEL$f_{\mathbf b}(u,i)=b_i$

One candidate constant per item; use the global rule for an unseen item.

HYPERPARAMETERSnone

The grouping and fallback rule are fixed in this definition.

LOSS / CRITERION$J_i(c)=\sum_{u\in U_i}(r_{ui}-c)^2$

$U_i$ contains users who rated item $i$ in training.

FIT · EACH ITEM $i$

MINIMIZE ITEM LOSS\[\widehat b_i=\arg\min_{c\in\mathbb R}J_i(c)\]

→

ITEM’S RATING MEAN\[\widehat b_i=\frac{1}{|U_i|}\sum_{u\in U_i}r_{ui}\]

LEARNED PARAMETERS AFTER fit$\widehat\mu,\{\widehat b_i\}$

stored as global_mean_ and item_means_ · retrieve by item ID

One workflow can use every estimator

models = [
    GlobalMeanRS(),
    UserMeanRS(),
]

for model in models:
    fitted_model = model.fit(X_train, y_train)
    y_pred = fitted_model.predict(X_new)

    print(type(model).__name__, y_pred.shape)

OUTPUT · $m$ = NUMBER OF ROWS IN X_new

GlobalMeanRS  (m,)
UserMeanRS    (m,)

Same inputsX_train, y_train, X_new

Same methodsfit and predict

Same output contractone prediction per new row

Model-specific logic stays inside each class. Workflow code depends only on the shared estimator interface.

One interface—and one way to read every estimator

ASK FOUR QUESTIONS model learned parameters hyperparameters loss / criterion

then use

fit(X, y)predict(X_new)

next lecture

ML IIWhich fitted estimator will generalize?

Holdout validation and cross-validation compare candidates without using the final test labels.

Exit ticket: for ItemMeanRS, name the learned attribute and explain how predict handles an unseen item.

Machine Learning I: Models and Estimators

Today’s question

ML I · A language for prediction

Rating prediction is supervised learning

One row becomes one learning example

Four questions decode every estimator

ML principles through California housing

California housing: the data for our model

Linear regression combines the features

Least squares fits the housing coefficients

Training and test answer different questions

From model to scikit-learn estimator

One contract for every prediction model

Class, estimator, and fitted estimator

Read LinearRegression with the four questions

Fit and predict with California housing

Many errors become one RMSE

Recommender baselines: same template, new models

GlobalMeanRS: specify before coding

Now implement GlobalMeanRS

UserMeanRS: specify before coding

User mean learns a lookup vector

Implement UserMeanRS: fit

Complete UserMeanRS: predict

ItemMeanRS: transfer the same template

One workflow can use every estimator

One interface—and one way to read every estimator

Read `LinearRegression` with the four questions

`GlobalMeanRS`: specify before coding

Now implement `GlobalMeanRS`

`UserMeanRS`: specify before coding

Implement `UserMeanRS`: `fit`

Complete `UserMeanRS`: `predict`

`ItemMeanRS`: transfer the same template