Neural Networks

Recall: the linear model

\[\widehat y_i=x_i^\top w+b,\qquad J(w,b)=\frac1n\sum_{i=1}^n\bigl(y_i-x_i^\top w-b\bigr)^2\]

MODELone linear layer

Each feature contributes through one learned coefficient.

OBJECTIVEmean squared error

Training chooses \(w,b\) to minimize the average squared error on the training rows.

Same data, same loss, same training question: minimize an average loss over the training set.

From linear to nonlinear

\[h=A(W_1x+b_1),\qquad \widehat y=W_2h+b_2\]

LINEAR MODEL\(x^\top w+b\)

One fixed transformation; features enter additively.

NEURAL MODEL\(f_\theta(x)\)

Several learned transformations, with nonlinearities between them.

The objective is unchanged. We only make the prediction rule \(f_\theta\) more flexible.

Why use a nonlinear activation?

WITHOUT NONLINEARITYTwo linear layers collapse to one

\[W_2(W_1x+b_1)+b_2=Wx+c\]

WITH NONLINEARITYThe composition can bend

\[h=A(W_1x+b_1),\quad\widehat y=W_2h+b_2\]

ReLU is zero for negative inputs and rises linearly for positive inputs.

ReLU, \(A(z)=\max(0,z)\), is the default nonlinearity in this course: cheap, simple, and enough to stop layers collapsing.

The training problem

\[J(\theta)=\frac1n\sum_{i=1}^n L\bigl(y_i,f_\theta(x_i)\bigr)\]

DATA\((x_i,y_i)\)

\(n\) training examples: features \(x_i\) and target \(y_i\).

MODEL\(f_\theta\)

A prediction rule with parameters \(\theta\).

LOSS\(L(y,\widehat y)\)

How wrong one prediction is; for ratings, squared error.

OBJECTIVE\(J(\theta)\)

The average loss over the training set; training minimizes it.

Everything that follows is about minimizing \(J(\theta)\): define \(f_\theta\), choose \(L\), and update \(\theta\).

Full-batch gradient descent

\[\theta^{(t+1)}=\theta^{(t)}-\eta\,\frac1n\sum_{i=1}^n\nabla_\theta L\bigl(y_i,f_\theta(x_i)\bigr)\]

CURRENT\(\theta^{(t)}\)

The network’s weights right now.

DIRECTIONgradient

How the loss changes when each weight moves.

STEP SIZE\(\eta\)

The learning rate controls how far we move.

This is exact, but one step uses all \(n\) training examples. That is expensive when \(n\) is large.

One small gradient step

For \(J(w)=(w-3)^2\), start at \(w=0\) and choose \(\eta=0.2\).

\[J'(w)=2(w-3),\qquad w_{\mathrm{new}}=0-0.2(-6)=1.2\]

point	weight	loss
before	\(0\)	\(9\)
after one step	\(1.2\)	\(3.24\)
minimum	\(3\)	\(0\)

The parabola of J(w) = (w-3)^2 with a tangent at w=0 and one step to w=1.2.

The same idea updates all network weights at once; the gradient is then a large vector, not a single slope.

Gradient descent in higher dimensions

Minimize \(L(\theta_1,\theta_2)=\theta_1^2+\theta_2^2\) with \(\eta=0.2\) from \(\theta^{(0)}=(2,1)\).

\[\nabla L(\theta)=\begin{pmatrix}2\theta_1\\2\theta_2\end{pmatrix},\qquad \theta^{(t+1)}=\theta^{(t)}-\eta\nabla L(\theta^{(t)})\]

\(t\)	\(\theta_1\)	\(\theta_2\)	\(L\)
0	2.00	1.00	5.00
1	1.20	0.60	1.80
2	0.72	0.36	0.65
3	0.43	0.22	0.23
4	0.26	0.13	0.08

The gradient is a vector: one partial derivative per parameter. A single learning rate must balance all directions; that is why \(\eta\) is so important.

Why use a mini-batch?

Write the per-example loss as \(\ell_i(\theta)=L(y_i,f_\theta(x_i))\)
The full gradient is an average over all \(n\) examples.

\[\nabla J(\theta)=\frac1n\sum_{i=1}^n\nabla_\theta\ell_i(\theta),\qquad \mathbb{E}_B\bigl[\nabla J_B(\theta)\bigr]=\nabla J(\theta)\]

FULL GRADIENTexact, cost \(O(n)\)

One update needs all \(n\) rows.

BATCH GRADIENTunbiased, cost \(O(b)\)

A uniformly sampled batch estimates the full gradient.

TRADE-OFFnoise vs cost

Small \(b\): noisy but cheap. Large \(b\): expensive but less noisy.

A mini-batch turns one expensive exact step into many cheap noisy steps. The noise is the price of not touching all \(n\) rows.

SGD: one batch at a time

A batch \(B\) is a subset of the training rows. Replace the full average by the average over \(B\).

\[\nabla J_B(\theta)=\frac1{|B|}\sum_{j\in B}\nabla_\theta L\bigl(y_j,f_\theta(x_j)\bigr),\qquad \theta\leftarrow\theta-\eta\nabla J_B(\theta)\]

FULL BATCHall \(n\) rows

Exact gradient, but one step costs a full pass over the data.

MINI-BATCH\(b\) rows

The usual SGD compromise: a noisy but cheap gradient, many updates per epoch.

SINGLE SAMPLE\(b=1\)

Very noisy gradient, but the cheapest possible update.

EPOCHone pass

Uses every training row once on average; with batch size \(b\), about \(n/b\) updates.

A batch is the set of rows used for one update; an epoch is one pass over all rows.

One batch, one update; one epoch, many updates

batch 14 rows → update 1

→

batch 24 rows → update 2

→

batch 34 rows → update 3

Illustrative example: 12 rows, batch size 4, so one epoch has 3 updates.

A batch gives one gradient step. An epoch is one pass over all rows, so it gives about \(n/b\) steps.

SGD terminology

term	meaning
loss \(L\)	error for one prediction
objective \(J\)	average loss over the training set
gradient	derivative of the objective with respect to the parameters
batch	the subset of rows used for one update
batch size \(b\)	number of rows in that subset
iteration / step	one gradient update, using one batch
epoch	one pass over the training set; about \(n/b\) updates
learning rate \(\eta\)	step size in the parameter update
shuffle	reshuffle row order each epoch, changing the batches

A batch is not a model and not a dataset: it is the set of rows used for one SGD step.

From matrix factorization to a learned interaction

BACK TO RECOMMENDATION

\[\widehat r_{ui}=p_u^\top q_i\qquad\longrightarrow\qquad\widehat r_{ui}=f_\theta(p_u,q_i)\]

MATRIX FACTORIZATIONFixed interaction rule

Learn two vectors and combine them with an inner product.

NEURAL MODELLearn the interaction rule

A network can combine the same inputs through several linear and nonlinear steps.

The ratings, the loss, and the validation discipline stay the same. The function class changes.

The four questions and their implementation modules

Q1 · DATAexamples → batches

TensorDataset + DataLoader

Q2 · MODELlayers → \(f_\theta\)

nn.Linear, nn.ReLU, nn.Sequential, nn.Module

Q3 · LOSSprediction → number

nn.MSELoss

Q4 · OPTIMIZATIONgradient → update

torch.optim.SGD + training loop

Each question has a code home. Evaluation is the check after the four modules: validation and RMSE/MAE.

Q1 · Data: what goes into the model?

MODULE 1 · DATA

QUESTIONWhat examples, and how are they batched?

Rows contain features and a target; training consumes small batches, not the whole table.

IMPLEMENTATIONTensorDataset + DataLoader

Convert rows to tensors, pair \(x_i\) with \(y_i\), then yield mini-batches.

The data module’s output is a stream of (xb, yb) batches with a known shape.

The PyTorch data pipeline

from torch.utils.data import DataLoader, TensorDataset

X = torch.tensor(X_train, dtype=torch.float32)
y = torch.tensor(y_train, dtype=torch.float32).unsqueeze(1)
loader = DataLoader(TensorDataset(X, y),
                    batch_size=64, shuffle=True)

TENSOR\((n,8)\) and \((n,1)\)

Features are float32; targets match the model output shape.

DATASETone example per row

TensorDataset pairs each \(x_i\) with its \(y_i\).

LOADERmini-batches

shuffle=True reshuffles every epoch; batch_size is a hyperparameter.

The loader yields (xb, yb) with shapes (64,8) and (64,1); the training loop consumes those batches.

Q2 · Model: what is \(f_\theta\)?

MODULE 2 · MODEL

QUESTIONWhat prediction rule does the network represent?

A composition of learned linear maps and fixed nonlinear activations.

IMPLEMENTATIONnn.Module + layers

Pick bricks, check their shapes, then snap them together with Sequential or a custom Module.

Think of a model as a row of bricks: each brick transforms the tensor and passes it to the next.

A neural network is a chain of transformations

input
\(x\)

→

hidden layer 1
features

→

hidden layer 2
combinations

→

output
\(\widehat y\)

LINEAR STEP\(Wx+b\)

Mix the inputs and shift the result.

NONLINEAR STEPactivation \(A(\cdot)\)

Let the next layer respond differently in different input regions.

“Deep” means we compose several layers; each layer creates the input for the next.

Start with one neuron

\[z=w_1x_1+w_2x_2+b,\qquad h=A(z)\]

INPUTS\(x_1,x_2\)

Values supplied for one example.

LEARNED\(w_1,w_2,b\)

Weights and bias adjusted during fitting.

OUTPUT\(h\)

The transformed signal sent forward.

A layer contains many neurons; the layer’s outputs form a vector.

Stack neurons into layers

A network with three inputs, four neurons in the first hidden layer, three in the second, and one output.

\[W_1\in\mathbb R^{4\times3},\qquad W_2\in\mathbb R^{3\times4},\qquad W_3\in\mathbb R^{1\times3}\]

Each edge is a weight, and each node adds a bias before applying \(A\). This toy network has 35 learned numbers: \((4\cdot3+4)+(3\cdot4+3)+(1\cdot3+1)\).

The model in one line

\[f_0(x)=x,\qquad f_\ell(x)=A_\ell\bigl(W_\ell f_{\ell-1}(x)+b_\ell\bigr),\quad \ell=1,\ldots,L\]

HIDDEN LAYERSUsually nonlinear

They build an internal representation of the input.

OUTPUT LAYERChosen for the task

For numerical rating prediction, a linear output is a simple starting point.

The network’s output is \(\widehat y=f_L(x)\); training decides the values inside its weight matrices and bias vectors.

Inside a `nn.Linear` brick

ONE LAYERnn.Linear(8, 32)W: 32 × 8 · b: 32

in (B,8) → out (B,32)

\[h=xW^\top+b\]

object	shape
input \(x\)	\((B,8)\)
weight \(W\)	\((32,8)\)
bias \(b\)	\((32,)\)
output \(h\)	\((B,32)\)

A Linear brick has \(8\cdot32+32=288\) learned numbers; the bias is broadcast across the batch.

Build the model brick by brick

LINEARLinear(8,32)288 weights

(B,8) → (B,32)

→

SHAPEReLUno weights

(B,32) → (B,32)

→

LINEARLinear(32,16)528 weights

(B,32) → (B,16)

→

SHAPEReLUno weights

(B,16) → (B,16)

→

LINEARLinear(16,1)17 weights

(B,16) → (B,1)

model = nn.Sequential(
    nn.Linear(8, 32), nn.ReLU(),
    nn.Linear(32, 16), nn.ReLU(),
    nn.Linear(16, 1),
)

Sequential is the baseplate: it snaps the bricks in order. ReLU keeps the shape; Linear changes it.

A model class

class MLP(nn.Module):
    def __init__(self):
        super().__init__()
        torch.manual_seed(0)  # pin initialization
        self.net = nn.Sequential(
            nn.Linear(8, 32), nn.ReLU(),
            nn.Linear(32, 16), nn.ReLU(),
            nn.Linear(16, 1),
        )

    def forward(self, x):
        return self.net(x)

init registers the bricks; forward defines the path. The notebook uses this class in both the pure-PyTorch and skorch sections.

The layers we use

layer	role	learned parameters
`nn.Linear(d_in, d_out)`	affine map \(xW^\top+b\)	\(d_{\rm in}d_{\rm out}+d_{\rm out}\)
`nn.ReLU()`	elementwise nonlinearity	none
`nn.Sequential(…)`	container that calls layers in order	none
`nn.Module`	base class for a custom forward pass	depends on layers
`nn.Embedding(n, K)`	lookup table for integer IDs	\(nK\) (next lecture)

For the housing model, Linear + ReLU + Sequential are enough. Embedding will replace the one-hot lookup in NCF.

Branch and merge with `nn.Module`

class ParallelMLP(nn.Module):
    def __init__(self):
        super().__init__()
        self.up = nn.Sequential(
            nn.Linear(8, 3), nn.ReLU(), nn.Linear(3, 1))
        self.down = nn.Sequential(
            nn.Linear(8, 2), nn.ReLU(), nn.Linear(2, 1))

    def forward(self, x):
        return self.up(x) + self.down(x)

WHY Module?Sequential is one straight path

Two branches cannot be expressed as a single Sequential.

MERGE\(f(x)=\psi_{\mathrm{up}}(x)+\psi_{\mathrm{down}}(x)\)

The forward pass computes each branch, then adds them.

Use Sequential for a stack; use Module when the computation branches or combines inputs—as NCF will.

The shape contract

OUTPUTLinear(8,32)

out (B,32)

→

MISMATCHLinear(16,1)

expects (B,16)

32 ≠ 16 → mat1 and mat2 shapes cannot be multiplied

Before training, check every brick’s output shape against the next brick’s input shape. The first dimension is the batch size; the second is the feature width.

Parameters are learned; hyperparameters are chosen

	examples	who decides?
Parameters \(\theta\)	\(W_\ell\) and \(b_\ell\)	The optimizer, using training data
Hyperparameters	\(L\), \(d_\ell\), \(A\), \(\eta\), batch size, epochs	We choose; validation compares candidates

MORE CAPACITYMore layers or wider layers

More learned numbers and more flexibility.

THE TRADEOFFFlexibility is not free

Training error usually falls; validation decides whether the extra capacity helps.

Architecture choices define the model family; training only changes the parameters inside it.

Why can a network be flexible?

WHAT IT SAYSA wide enough network can approximate

With enough hidden units, a feedforward network can approximate a broad class of continuous functions on a bounded region.

WHAT IT DOES NOT SAYExistence is not a guarantee

It does not promise that our data, optimizer, or validation set will find a good approximation.

The theorem explains why the model family is expressive. Generalization still has to be measured on held-out ratings.

Q3 · Loss: what do we minimize?

MODULE 3 · LOSS

QUESTIONWhat objective turns predictions into a number?

The optimizer needs a scalar that says how wrong the current model is.

IMPLEMENTATIONnn.MSELoss

Squared error between the prediction and the observed target.

The loss is for the optimizer; metrics such as RMSE and MAE are for us.

The housing example: data, model, loss, metric

component	choice in the example
Data	California housing: 8 numerical features; target is median house value
Model	PyTorch MLP with ReLU hidden layers and a linear output
Loss	Mean squared error, used by the optimizer
Metrics	RMSE and MAE, reported for interpretation
Preprocessing	Standardize features; wrap tensors in a `DataLoader`; keep the test set untouched

\[\min_\theta\ \frac1n\sum_{i=1}^n\bigl(y_i-f_\theta(x_i)\bigr)^2\]

The loss is what training minimizes. A metric is what we report; they need not be the same function.

Q4 · Optimization: how do we update \(\theta\)?

MODULE 4 · OPTIMIZATION

QUESTIONHow do we use gradients to update all parameters?

Compute the loss on a mini-batch, backpropagate, and take a step.

IDEAgradient → step

One mini-batch gives a noisy gradient; the learning rate sets the step size.

“SGD” in practice means mini-batch SGD: many small updates, not one update on the full dataset.

Backpropagation supplies the gradient

forward pass
predictions

→

loss
prediction error

→

backward pass
chain rule

→

optimizer
update weights

\[\frac{\partial J}{\partial W_1}=\frac{\partial J}{\partial f_L}\cdot\frac{\partial f_L}{\partial f_{L-1}}\cdots\frac{\partial f_1}{\partial W_1}\]

The framework computes these derivatives automatically. We still need to specify the model, loss, optimizer, and data.

Backpropagation, layer by layer

Forward pass computes h1, h2, and yhat; backward pass sends gradients from the loss back through the layers and fills the parameter gradients.

Forward stores the intermediate activations; backward reuses them to fill one gradient for every weight and bias.

Pure PyTorch: the training loop

loss_fn = nn.MSELoss()
optimizer = torch.optim.SGD(model.parameters(), lr=0.01)

for epoch in range(epochs):
    model.train()
    for xb, yb in train_loader:
        optimizer.zero_grad()
        loss = loss_fn(model(xb), yb)
        loss.backward()
        optimizer.step()

zero_grad clears old gradients; backward applies autograd; step updates \(\theta\leftarrow\theta-\eta\nabla_\theta J\).

skorch: the same loop with `.fit`

from skorch import NeuralNetRegressor

model = NeuralNetRegressor(
    module=MLP,
    criterion=nn.MSELoss,
    optimizer=torch.optim.SGD,
    max_epochs=100,
    lr=0.01,
    batch_size=64,
    iterator_train__shuffle=True,
)

model.fit(X_train, y_train)
y_pred = model.predict(X_test)

skorch is a thin sklearn wrapper: .fit runs the loop above; .history stores the training and validation curves.

SGD: the details that matter

LEARNING RATEstep size

Too small: slow or stuck. Too large: oscillates or diverges. Start with 0.01 or 0.001.

BATCH SIZEgradient noise

Larger batches give smoother gradients but use more memory; smaller batches update more often.

SHUFFLEnew batch order

Reshuffle each epoch so the model does not memorize the order of the training rows.

EPOCHSpasses over data

One epoch uses every training row once on average; validation decides how many epochs are useful.

“SGD” in practice means mini-batch SGD. Tune the learning rate and batch size on validation, never on the test set.

After the four modules: did it generalize?

THE CHECK · EVALUATION

QUESTIONDo the learned weights help unseen examples?

Training loss only says how well the model fits the rows it has already seen.

IMPLEMENTATIONvalidation loop

Evaluate on held-out rows, track the curve, and keep the test set for the final report.

Training loss is not the goal. Validation decides which model to report.

Validation answers a different question

TRAINING LOSSAre weight updates fitting seen examples?

The optimizer uses these answers to change \(\theta\).

VALIDATION LOSSDo those changes help unseen examples?

Never use these target values in gradient updates.

A network can keep lowering training loss while validation loss starts rising. That pattern signals overfitting.

Watch the validation curve

Training loss keeps falling; validation loss reaches a minimum near epoch 9 and then rises.

UNDERFITboth losses high

USEFUL FITvalidation near its minimum

OVERFITtraining falls, validation rises

The same U-shape appears when we increase model complexity. More flexibility helps only until validation stops improving.

K-fold cross-validation

fit 1validationtraintrain

fit 2trainvalidationtrain

fit 3traintrainvalidation

FIT 3 MODELSeach row is one fit

Every data fold is validation once; the other folds train.

SCOREaverage the 3 scores

Usually more stable than one holdout split.

K-fold uses each row for validation once; the test set remains untouched.

Cross-validation vs a validation split

K-FOLDRefit K times

Accurate model selection, but expensive for large networks.

HOLDOUTOne validation split

Standard for deep learning: fast, though noisier.

GRID SEARCHCompare hyperparameters

Fit each candidate on the CV folds; keep the test set for the final report.

Cross-validation is still correct; a single holdout is faster but noisier. GridSearchCV automates the comparison.

Tune hyperparameters with `GridSearchCV`

from sklearn.model_selection import GridSearchCV

search_net = NeuralNetRegressor(
    module=MLP,
    criterion=nn.MSELoss,
    optimizer=torch.optim.SGD,
    batch_size=64,
)
search_net.set_params(train_split=False, verbose=0)

param_grid = {
    'lr': [0.01],
    'max_epochs': [80, 100],
    'batch_size': [32, 64],
}

gs = GridSearchCV(
    search_net, param_grid,
    cv=3, scoring='neg_root_mean_squared_error',
    refit=True,
)
gs.fit(X_train, y_train)
final_net = gs.best_estimator_

refit=True refits the best settings on all training rows; gs.best_estimator_ is the tuned model used for the final test prediction.

What to tune—and what to monitor

ARCHITECTUREDepth and width

How many transformations and hidden units?

OPTIMIZATIONLearning rate and batch size

How do training updates behave?

GENERALIZATIONValidation curve

When should we stop, and which design predicts best?

Start with a small network and a held-out validation set; increase complexity only when the evidence supports it.

Common failure patterns

symptom	likely cause	next check
loss oscillates	learning rate too large	reduce step size
loss hardly moves	learning rate too small or inputs poorly scaled	inspect features and gradients
train improves, validation worsens	overfitting	simpler model or more data
validation looks excellent	possible leakage	audit the split and preprocessing
loss becomes NaN	unstable update or unscaled inputs	reduce step size; standardize features

Debug with both training and validation evidence—not a single final accuracy number.

Rules of thumb

choice	starting point
Output layer	Match the task: linear for regression; sigmoid or softmax for classification
Hidden layers	First layer ≈ half the input width; halve thereafter (128, 64, 32, …)
Activation	ReLU is a good default
Epochs	Start with 20 to check convergence; then use 100 and choose by validation
Batch size	Powers of 2; start with 16–64, larger for big or imbalanced data
Inputs	Standardize numerical features before fitting

These are starting points, not laws. Tune systematically and let validation choose.

In-class practice: build and diagnose a housing network

Open nb/STAT3009_NN.ipynb and use the California housing data. Work in pairs for 20–25 minutes.

Standardize the 8 input features; keep a test set aside.
Build an MLP with hidden widths 256 → 128 → 64 → 32 → 16, each followed by ReLU, then a linear output.
Use MSE loss; report RMSE and MAE.
Use the notebook’s NeuralNetRegressor with batch size 64 for at most 50 epochs; compare validation RMSE.
Plot the validation RMSE curve and predict the test set.

Compare the training and validation curves. Which epoch had the best validation RMSE?

Practice debrief

DID IT LEARN?training loss falls

If not: check the learning rate and shapes.

DID IT GENERALIZE?validation has a minimum

If not: overfitting, leakage, or too little data.

WHAT WOULD YOU CHANGE?one thing at a time

Try learning rate, batch size, or depth; compare validation curves.

A model is not finished when training loss is low; it is finished when validation and the test set agree.

Carry the template forward

question	in this lecture	implementation
Model	layers + activations	`nn.Module`
Hyperparameters	depth, width, lr, batch size	choices in Data / Model / Optimization
Criterion	MSE on training data	`nn.MSELoss`
Learned	weights + biases	`torch.optim.SGD`

\[\text{MF vectors}\;\longrightarrow\;\text{embedding layers}\;\longrightarrow\;\text{neural interaction}\]

Next: use embeddings to turn user and item IDs into network inputs. That is neural collaborative filtering.

What you should be able to do now

BUILDa model from layers

Choose bricks, check shapes, and wrap them in Sequential or Module.

DISTINGUISHparameters from hyperparameters

Know what training learns and what validation chooses.

EXPLAINone SGD update

Gradient descent, mini-batches, and the chain rule behind backpropagation.

DIAGNOSEa validation curve

Use validation curves to compare models and keep the test set untouched.

Next: use embedding layers to turn user and item IDs into network inputs. That is neural collaborative filtering.