Lecture 1: Introduction

Deep Learning

• Machine Learning: learn an optimal model $H$ from data through training/optimization.
• Deep Learning $\subset$ ML: optimize multiple processing layers to approximate a function.

Data splits

Set	Role
Training	optimization
Validation	tuning during training
Testing	unbiased final evaluation

Classical ML vs. Neural Networks: classical ML relies on hand-crafted features; neural networks learn features, which makes them robust and scalable.

Artificial Neuron

$$y=f(\sum _{i=1}^n{x}_i{w}_i+b)$$

Integration → Activation → Output. Training = learning weights $w$ and biases $b$.

TIP

Without a non-linear activation, a neural network is just a linear model.

Linear Regression

$$\hat{y}(x)=p_0+p_{1}x+\cdots +p_m{x}^m=\overrightarrow{p}\cdot \overrightarrow{x}$$

Least squares:

$$J=\sum _{i=1}^n(\overrightarrow{p}\cdot {\overrightarrow{x}}_i-y_i{)}^2\Rightarrow {\overrightarrow{p}}^{\ast }=(X^{\mathrm{\top }}X{)}^{-1}{X}^{\mathrm{\top }}\overrightarrow{y}$$

Ridge regression (adds an L2 penalty):

$$J=(y-Xp{)}^{\mathrm{\top }}(y-Xp)+\lambda \Vert p{\Vert }^2\Rightarrow p=(X^{\mathrm{\top }}X+\lambda I{)}^{-1}{X}^{\mathrm{\top }}y$$

	Least squares	Ridge
Bias	unbiased	biased
Variance	high	low