Course Overview & Software Setup

STAT3009 · Recommender Systems

Ben Dai

CUHK · Department of Statistics and Data Science

Every platform chooses what to show next

01 · COURSE PURPOSE

products music connections films investments

AMAZON · ALIBABA · NETFLIX · SPOTIFY · LINKEDIN · FINANCE

Recommendation turns many possible choices into a useful next decision.

The semester moves from baselines to neural recommenders

COURSE MAP
FOUNDATIONS

Frame the problem

Interaction data
Evaluation
Mean baselines

LATENT STRUCTURE

Learn hidden preferences

Cross-validation
SVD and matrix factorization
MovieLens

NEURAL RS

Add expressive models

Neural networks
Neural matrix factorization
Side information

STATISTICS×MACHINE LEARNING×CODE

How this course works

02
CUHK emblem

Understand the expectations before we begin the technical work.

You will learn, build, and compete

01

LEARN

Concepts in slides and notes

02

BUILD

Live Python in Colab and Jupyter

03

COMPETE

In-class practice and Kaggle

Course homepage ↗·GitHub repository ↗

Bring foundations—not prior recommender-system experience

WHAT HELPS

We build from tools you have already seen

You do not need to arrive as a recommender-systems expert.

Python-library tutorials are provided when we need them.

01Statistics
Linear and ridge regression; hypothesis testing
02Programming
Python, NumPy, Pandas, basic scikit-learn
03Mathematics
Linear algebra, calculus, probability

Assessment rewards implementation

15%Homework
40%Open-book
in-class KaggleApproximately mid-semester
45%Final in-class
coding quizLast class of the semester

Use AI as a tutor—not a substitute for understanding

AI-ASSISTED · STUDENT-OWNED
USE IT TOQuestion · compare · debug

Ask for an explanation of an error, compare two approaches, or improve the clarity of your Markdown.

VERIFY ITRun · inspect · test

Execute every cell, read the output, check assumptions, and test the suggestion on the actual data.

OWN ITExplain · modify · reproduce

You remain responsible for every submitted result and should be able to adapt the code yourself.

Not evidence of learning: submitting generated work that you cannot reproduce, justify, or debug.

Follow each assessment’s stated tool and disclosure rules. During an assessment, use only the tools explicitly permitted.

Building a knowledge landscape

Know where an idea fits, what it depends on, and what you need to learn next.

Foundation Connection in this course A question you can ask
Linear algebra
Vectors and inner products 		Matrix factorization
Represents users and items with vectors 		What does the predicted score represent?
Probability and statistics
Sampling and uncertainty 		Model evaluation
Uses held-out data as evidence 		Will this result extend to new ratings?
Coding
Arrays, indexing, functions 		Computational experiments
Turns a model into data operations 		Does the program implement the intended experiment?

A model needs a representation, an implementation, and evidence about its predictions.

Use this map to locate an AI suggestion and identify the knowledge you need to use it.

Mathematics and coding in the AI era

Mathematics and coding turn your knowledge landscape into something you can reason with and use.

Mathematics

Define the problem and judge the evidence.

  • Connect models to their assumptions and objectives.
  • Understand what an evaluation result supports.

Does a lower rating error imply a better recommendation list?

Coding

Understand and adapt the computation.

  • Trace how a modeling idea becomes an experiment.
  • Modify the workflow when data or requirements change.

Which observations went into fitting and evaluation?

Use these foundations to guide your questions and extend your understanding with AI.

Performance on familiar questions

A student can solve every question on the practice sheet.

What would convince you that they can solve new questions?

Try a separate set of questions that was not used for practice.

The best way to learn code is to wrestle with it

Practice illustration
PRACTICE
Debugging illustration
DEBUG
Persistence illustration
PERSIST

Bring a laptop. Read code, run it, break it, fix it—and make it yours.

Software preparation

03
CUHK emblem

Use notebooks to keep executable code and readable reasoning together.

Three layers make a Colab notebook work

ONE COMPUTATIONAL WORKFLOW
LANGUAGEPython

The instructions and objects we write.

lives inside
DOCUMENTJupyter Notebook

An .ipynb file containing code, text, and saved output.

runs through
SERVICEGoogle Colab

A hosted notebook interface and computational runtime.

Colab is based on Jupyter; Python is the primary language executed by its kernel.

A notebook mixes Python and Markdown

JUPYTER NOTEBOOK
PYTHONCompute

Import packages, define objects, run models, and create output.

MARKDOWNExplain

Write headings, equations, links, interpretation, and task status.

Open the course notebook in Colab ↗

STAT3009 · live notebook

JupyterLab notebook preview

Source: Jupyter Project

Use Colab on the web or in VS Code

SAME NOTEBOOK · SAME COLAB COMPUTE
COLAB WEB

Browser first

  1. Open the course notebook below
  2. Sign in with Google
  3. Save a copy in your Drive
  4. Run a cell with Shift + Enter

Open the course notebook ↗

COLAB FOR VS CODE

Editor first

  1. Install the official Colab extension
  2. Download the course notebook as .ipynb
  3. Open it in VS Code
  4. Choose Select Kernel → Colab
  5. Select Auto Connect and sign in

Install the VS Code extension ↗

In Colab: File → Download → Download .ipynb gives you a local notebook file.

Notebook, kernel, and Python environment

AN ANALOGY: MIXING A GLASS OF SUGAR WATER

Notebook

Instructions & experiment log

The .ipynb file stores code, explanations, and saved results.

Kernel

The person doing the experiment

Runs the code and holds variables in memory, like the current mixture.

Python environment

Workbench & available tools

Provides the Python interpreter and installed packages the kernel uses.

Restart the kernel

Start again with an empty cup. Variables disappear. Saved notebook contents remain.

%pip install pandas

Add a tool to the active workbench. The package goes into the kernel’s environment.

A notebook remembers execution—not visual order

THE KERNEL HOLDS INVISIBLE STATE
01 · RUN CELLdiscount = 0.10

kernel memory → discount = 0.10

02 · EDIT, BUT DO NOT RUNdiscount = 0.20

screen → 0.20 · memory → 0.10

03 · RUN THE NEXT CELL100 * (1 - discount)

output → 90.0, not 80.0

ON SCREENSaved cells

What the notebook currently displays.

IN MEMORYExecuted state

What the kernel will actually use.

BEFORE SHARING OR SUBMITTING Restart session → Run all → Verify outputs

A fresh Colab runtime may also require the package-setup cells again.

Markdown turns plain text into structure

WHAT YOU TYPE 
## Preparation

**Goal:** verify the software setup.

- [x] install the packages
- [ ] finish the practice

[Python docs](https://docs.python.org/)

Inline code: `print(version)`

Equation: $y = \beta_0 + \beta_1 x$
WHAT THE NOTEBOOK RENDERS

Preparation

Goal: verify the software setup.

  • install the packages
  • finish the practice

Python docs

Inline code: print(version)

y = β₀ + β₁x

Markdown also handles technical notation

### Prediction error

Use `predict(...)` to obtain predictions.

$$
\operatorname{RMSE}
= \sqrt{\frac{1}{n}\sum_{i=1}^{n}
(y_i-\hat y_i)^2}
$$

| symbol | meaning |
|---|---|
| y | observed rating |
| y_hat | predicted rating |
inline codewrap a short expression in backticks
display mathwrap an equation in double dollar signs
pipe tableuse vertical bars to separate columns
blank linesseparate blocks and prevent rendering surprises

Python refresher

04
CUHK emblem

Review the small set of patterns we will use from the first lab onward.

Packages extend what Python can do

%pip install -q numpy pandas seaborn rehline

import numpy as np
import pandas as pd
import seaborn as sns
import rehline

print("NumPy:", np.__version__)
print("Pandas:", pd.__version__)
%pip installinstalls into the current notebook environment
importloads a package in the current session
as np · as pdcreates conventional short aliases
seaborn · rehlinematch the dependencies listed in the combined course notebook

Assignment connects a name to an object

course = "STAT3009"
week = 1
threshold = 3.5
ready = True

# inspect and compare
print(type(course))
print(week == 1)
name = valueassigns an object to a name
str · int · float · boolare common scalar types
==compares two values
#starts a comment

Expressions transform values

score = 82
bonus = 5

adjusted = score + bonus
passed = adjusted >= 60
in_range = 0 <= adjusted <= 100

message = f"adjusted score: {adjusted}"

print(message)
print(passed, in_range)
+ · - · * · /perform arithmetic operations
>= · <= · == · !=produce Boolean values
chained comparisonexpresses a range naturally
f”{name}“inserts an expression into text

Lists and dictionaries organize values

ratings = [4, 5, 3, 4]
student = {"name": "Ada", "ready": True}

print(ratings[0])        # first item
print(ratings[-1])       # last item
print(ratings[1:3])      # positions 1 and 2
print(student["name"])  # look up a key
listordered values indexed by position
dictkey–value pairs indexed by key
[start:stop]includes start, excludes stop
-1refers to the last item

A for loop visits each value once

liked = []

for rating in ratings:
    if rating >= 4:
        liked.append(rating)

for index, rating in enumerate(ratings):
    print(index, rating)
for item in valuesiterates over a sequence
if conditionruns a block only when true
.append(…)adds one item to a list
enumerate(…)provides both position and value

Indentation is part of Python syntax

for rating in ratings:
    if rating >= 4:
        label = "liked"
    else:
        label = "other"
    print(label)

print("finished")
:starts an indented block
4 spacesis the recommended indentation
same levelmeans the same block
dedentleaves the current block

Do not mix tabs and spaces. Python uses indentation to decide which statements belong together.

Functions package repeated logic

def select_liked(values, threshold=4):
    liked = []
    for value in values:
        if value >= threshold:
            liked.append(value)
    return liked

select_liked(ratings)
def name(…):defines a function
parametersreceive inputs
local namesexist inside the function
returnsends one result back

NumPy arrays have shape and dtype

ratings = np.array([4, 5, 3, 4], dtype=float)

print("ndim:", ratings.ndim)
print("shape:", ratings.shape)
print("size:", ratings.size)
print("dtype:", ratings.dtype)

matrix = np.arange(12).reshape(3, 4)
print("matrix shape:", matrix.shape)
ndarraystores homogeneous values efficiently
ndimcounts the number of axes
shape · sizedescribe dimensions and element count
dtyperecords the common value type

Axis tells NumPy which direction to summarize

matrix = np.arange(12).reshape(3, 4)

print(matrix.mean())         # one overall mean
print(matrix.mean(axis=0))  # one mean per column
print(matrix.mean(axis=1))  # one mean per row

column_totals = matrix.sum(axis=0)
row_totals = matrix.sum(axis=1)
print(column_totals, row_totals)
axis omittedsummarizes all elements
axis=0collapses rows; keeps columns
axis=1collapses columns; keeps rows
check shapepredicts the size of the result

Boolean masks filter NumPy arrays

ratings = np.array([4, 5, 3, 4, 2])

mask = ratings >= 4
liked = ratings[mask]

middle = ratings[
    (ratings >= 3) & (ratings <= 4)
]
share_liked = mask.mean()

print(mask, liked, middle, share_liked, sep="\n")
comparisoncreates an array of True and False
array[mask]keeps values where the mask is True
& · |combine array conditions element by element
mask.mean()computes the share of True values

Pandas adds labels to tabular objects

df = pd.DataFrame({
    "student": ["A", "B", "C"],
    "score": [82, 95, 74],
})

selected = df[["student", "score"]]
high_scores = df.loc[df["score"] >= 80]
df["centered"] = (
    df["score"] - df["score"].mean()
)

display(selected, high_scores, df)
DataFramea labeled table
columnsselect named variables
.locfilters rows explicitly
assignmentcreates or updates a column

Pandas can sort and summarize groups

df["tutorial"] = ["A", "A", "B"]

ordered = df.sort_values(
    "score", ascending=False
)

summary = (
    df.groupby("tutorial")
      .agg(n=("score", "size"),
           mean_score=("score", "mean"))
      .reset_index()
)

display(ordered, summary)
.sort_values(…)orders rows by one or more columns
.groupby(…)splits rows into meaningful groups
.agg(…)calculates several summaries at once
method chainreads as a sequence of transformations

Missing values require an explicit decision

scores = pd.DataFrame({
    "student": ["A", "B", "C", "D"],
    "score": [82, None, 74, 91],
})

print(scores.isna().sum())
complete = scores.dropna(subset=["score"])
scores["score"] = scores["score"].fillna(
    scores["score"].median()
)

display(complete, scores)
.isna()locates missing entries
.dropna(…)removes rows under a stated rule
.fillna(…)replaces missing entries
document the choicebecause it changes the analysis

Recommendation setup

05
CUHK emblem

Turn observed user–item interactions into predictions for unseen pairs.

The recommendation feedback loop

A user generates clicks and purchases, the system tracks those events, a recommendation engine learns from them, and recommendations return to the user
Source: Voice Tech Podcast, “Recommendation Engine in AI”

A recommender learns from actions produced by earlier recommendations.

1

Observe behaviorClicks, purchases, and ratings reveal user preferences.

2

Record eventsThe tracker stores interactions for later learning.

3

Learn preferencesThe model turns interaction history into scores.

4

Serve recommendationsThe system returns selected items to each user.

New actions become the next round of training data.

Recommendation serving and learning

Training and serving must use consistent feature definitions.

This course focuses on rating prediction

Data

Sparse user–item ratings

Identify the observed feedback and the missing pairs.

Models

Baselines, factorization, neural models

Learn a function that predicts a rating for a user–item pair.

Evaluation

Prediction on held-out ratings

Compare models on validation data and report final test performance.

The predicted scores can then support candidate ranking in the wider system.

Real datasets use different names for the same roles

THREE DATASETS · ONE ABSTRACTION
Dataset Fields in user / item / rating order Actual record (u, i, r)
Netflix user_id · movie_id · rating (1960, 670, 4)
CiaoDVD userID · itemID · rating (FA8D7A, 79CBAD, 5.0)
Book-Crossing User-ID · ISBN · Book-Rating (276726, 0155061224, 5)

(u, i, rui) is a modeling vocabulary: IDs identify who and what; feedback records the observed preference.

Load the table, then inspect the rows

import pandas as pd

base = (
    "https://raw.githubusercontent.com/"
    "statmlben/CUHK-STAT3009/main/"
    "dataset/netflix/"
)

fields = ["movie_id", "user_id", "rating"]
train = pd.read_csv(
    base + "train.csv", usecols=fields
)
test = pd.read_csv(
    base + "test.csv", usecols=fields
)
train.head()
train.head()
movie_id user_id rating
0 670 1960 4
1 152 1346 4
2 1741 785 4
3 3032 686 5
4 536 1894 4

One row records one observed user–movie rating.

A few checks reveal the dataset structure

# rows and columns
print("shape:", train.shape)

# storage types
display(train.dtypes)

# users and movies across both splits
both = pd.concat([train, test])
ids = ["user_id", "movie_id"]
display(both[ids].nunique())

# response scale
display(train["rating"].agg(["min", "max"]))
SHAPE51,161 × 3

training observations × selected fields

CATEGORIES2,000 users · 3,568 movies

unique IDs across train and test

RATING RANGE1–5

integer-valued explicit feedback

MODEL TABLE2 IDs · 1 target

the columns needed for rating prediction

Netflix data separates IDs from ratings

STORAGE TYPE ≠ MODELING ROLE
Field Example Pandas Meaning
user_id 1960 int64 categorical user identifier
movie_id 670 int64 categorical item identifier
rating 4 int64 ordered score and prediction target

An integer ID is a label—not a quantity. User 1960 is not “larger” than user 20.

User and movie IDs form X; rating is y

features = ["user_id", "movie_id"]

X_train = train[features].to_numpy()
y_train = train["rating"].to_numpy()

X_test = test[features].to_numpy()
y_test = test["rating"].to_numpy()
INPUTX_train · 51,161 × 2

Each row is [user_id, movie_id].

TARGETy_train · 51,161 values

Each value is the corresponding rating.

PREDICTION TASKTwo IDs → one score

Learn fθ(u, i) → r̂ui.

Most user–movie pairs are unobserved

Movie 0 Movie 1 Movie 2 Movie 3
User 0 4 ? 5 ?
User 1 ? 2 ? 4
User 2 5 ? 4 ?
User 3 ? 3 ? 5
OBSERVED(u, i) ∈ Ω

We know rui. Split observed ratings before model selection.

UNKNOWN(u, i) ∉ Ω

We estimate the missing preference as r̂ui.

Missing ≠ zero.
A blank cell means “not observed,” not “disliked.”

New pairs and new IDs are different problems

Test case What training provides In this Netflix split
Known IDs, new pair History for both IDs, but no rating for this pair Use the learned user and movie information
New user No training history for this user 74 test users are absent from training
New movie No training ratings for this movie 584 test movies are absent from training

For an unseen ID, define a fallback such as the training-set mean. Later models can use side information.

Three minutes: make the data contract explicit

first = train.iloc[0][
    ["user_id", "movie_id", "rating"]
]

features = ["user_id", "movie_id"]
X = train[features].to_numpy()
y = train["rating"].to_numpy()

display(first)
print("X:", X.shape, "y:", y.shape)
01

Write the first observation as (u, i, rui).

02

Report the shapes of X and y.

03

Explain why an integer ID is a label, not a continuous measurement.

04

Complete the sentence: “A missing rating means ___, not zero.”

Run it · compare with a neighbor · explain one answer aloud

Training, validation, and test ratings

Training

Ωtr

Fit model parameters.

Use the fitting portion of train.csv.

Validation

Ωval

Choose the method and its settings.

Reserve a portion of train.csv.

Test

Ωte

Report final predictive performance.

Keep test.csv untouched during model selection.

After choosing settings, refit on all of train.csv, then evaluate once on test.csv.

The task is to predict held-out ratings accurately

EVALUATE ON HELD-OUT PAIRS

Compare predictions with known ratings in Ωte.

RMSE=1|Ωte|∑(u,i)∈Ωte(rui−r̂ui)2\operatorname{RMSE}=\sqrt{\frac{1}{|\Omega^{\mathrm{te}}|}\sum_{(u,i)\in\Omega^{\mathrm{te}}}(r_{ui}-\hat r_{ui})^2}

Training error and prediction on new ratings

This code measures training fit. What evidence would support prediction on new ratings? 

# X_train: user and movie IDs
# y_train: observed ratings
model.fit(X_train, y_train)

pred = model.predict(X_train)
rmse = np.sqrt(
    np.mean((y_train - pred) ** 2)
)
print(rmse)
Read the codeFitting and evaluation use the same observations.
Interpret the numberThis is training RMSE, measuring fit to the training ratings.
Identify the missing evidenceTraining RMSE alone does not establish accuracy on new ratings.
Change the experimentUse validation ratings to choose settings. Keep test ratings for the final report.

Coding reveals which data was used. Mathematics explains what the result can support.

Ranking unrated candidates by predicted score

RATING PREDICTION → CANDIDATE SCORES → RANKED LIST
COURSE FOCUSLearn the scoring function

Estimate ratings accurately from sparse observations and evaluate with held-out RMSE.

SYSTEM USETurn scores into an ordered list

Rank candidates by predicted preference, then apply later serving constraints.

Observed ratings → rating model → predicted scores → ranked list → user actions → new observations

Offline accuracy is a gate—not the final outcome

MODEL → SYSTEM → PRODUCT
01 · MODELHeld-out RMSE

Can predicted ratings generalize to unseen user–item pairs?

02 · SYSTEMLatency · reliability · coverage

Can the service respond quickly and consistently for real users?

03 · PRODUCTSatisfaction · engagement · retention

Does the full experience improve the intended user outcome?

The objective shapes the behavior of the system. A better model metric does not automatically imply a better product.

Ready for the first lab?

01 Bring a laptop 02 Open the notebook 03 Run the setup cell 04 Trace ratings → scores → ranked list → new feedback

bendai.org/CUHK-STAT3009 ↗