Make It Easy To Learn Deep Learning Basics Math Concepts with PyTorch

Quick Navigation

Difficulty: Beginner
Estimated Time: 20-30 minutes
Prerequisites: Python basics, PyTorch installed, basic linear algebra, high-school calculus

You can see this video to get a big picture of where this math is used.

1. Data Encoding

One-Hot Encoding

One-hot encoding transforms categorical variables into binary vectors.

In deep learning, one-hot encoding is mainly used to represent categorical data so that neural networks can process it numerically. Here's how it's typically applied:

1. Classification Targets (Labels)

For supervised learning, especially multi-class classification, one-hot vectors are used for the labels.

Example: Classifying colors (Red, Blue, Green)

import torch
import torch.nn as nn
# Suppose we have 3 classes: Red, Blue, Green
labels = ['Red', 'Blue', 'Green', 'Red']
# Map categories to indices
category_to_idx = {'Red':0, 'Blue':1, 'Green':2}
indices = torch.tensor([category_to_idx[c] for c in labels])
# Convert to one-hot vectors
one_hot_labels = torch.nn.functional.one_hot(indices, num_classes=3).float()
print(one_hot_labels)

Output:

tensor([[1., 0., 0.],
[0., 1., 0.],
[0., 0., 1.],
[1., 0., 0.]])

When your features are categorical (like color, country, product type), you one-hot encode them so the network can process them numerically.

Example: Input features

features = ['Red', 'Blue', 'Green']
indices = torch.tensor([category_to_idx[f] for f in features])
one_hot_features = torch.nn.functional.one_hot(indices, num_classes=3).float()
# Feed into a neural network
model = nn.Sequential(
nn.Linear(3, 5),
nn.ReLU(),
nn.Linear(5, 2) # output 2 classes
)
output = model(one_hot_features)
print(output)

Summary of use in deep learning:

• One-hot for labels in multi-class classification.
• One-hot for categorical input features.

Dummy encoding

Dummy encoding is essentially the same as one-hot encoding: each category is represented by a vector where one position is 1 and the rest are 0. The term "dummy" comes from statistics/ML literature, but in deep learning we just treat it as a one-hot vector.

Here's a PyTorch example:

import torch
# Example categorical feature
features = ['Cat', 'Dog', 'Bird', 'Cat', 'Dog']
# Create a mapping: category index
unique_categories = list(set(features))
category_to_idx = {cat: idx for idx, cat in enumerate(unique_categories)}
# Convert categories to indices
indices = torch.tensor([category_to_idx[f] for f in features])
# Dummy/one-hot encoding
dummy_features = torch.nn.functional.one_hot(indices, num_classes=len(unique_categories)).float()
print(dummy_features)

Possible output:

tensor([[0., 0., 1.],
[0., 1., 0.],
[1., 0., 0.],
[0., 0., 1.],
[0., 1., 0.]])

Usage in a neural network

import torch.nn as nn
# Example model
model = nn.Sequential(
nn.Linear(len(unique_categories), 4),
nn.ReLU(),
nn.Linear(4, 2) # output 2 classes
)
# Forward pass
output = model(dummy_features)
print(output)

Explanation:

• Each category becomes a dummy vector (one-hot).
• .float() converts it to float since PyTorch models expect floats as input.
• This is used as input features for the model.

2. Matrix Operations

Transpose

Transpose a matrix:

data = torch.tensor([[1, 2], [3, 4]])
data_T = data.T # equivalent to data.transpose(0,1)
print(data_T)

Usage: the transpose of the input is required to align dimensions for matrix multiplication.

Dot Product and Matrix Multiplication

Dot product: scalar product between vectors

x = torch.tensor([1,2,3])
y = torch.tensor([4,5,6])
print(torch.dot(x,y)) # 1*4 + 2*5 + 3*6 = 32

Matrix multiplication: A@B or torch.matmul(A,B)

If A is M×N and B is N×K, the product is valid and results in M×K.

A = torch.randn(2,3)
B = torch.randn(3,4)
C = A @ B
print(C.shape) # torch.Size([2, 4])

3. Convolution

Convolutions are key operations for CNNs. Example in 1D:

1D Convolution Example

2D Convolution Example (image)

import torch.nn.functional as F
input = torch.randn(1,1,5) # batch, channel, width
kernel = torch.randn(1,1,3)
output = F.conv1d(input, kernel)
print(output)

4. Softmax and Probabilities

Softmax converts a vector into a probability distribution:

Each element of the softmax output represents the probability of that specific class being the correct one.

For example, if the softmax output is [0.66, 0.24, 0.10]:

• First element 66% chance it's class 1
• Second element 24% chance it's class 2
• Third element 10% chance it's class 3

All probabilities always sum to 1.

x = torch.tensor([2.0, 1.0, 0.1])
prob = torch.nn.functional.softmax(x, dim=0)
print(prob, prob.sum()) # sum = 1

• Each output represents the probability of belonging to a class.
• The sum of all outputs is always 1.

5. Entropy

Entropy measures unpredictability:

Entropy is like a confusion meter: if the model is really sure, entropy is low; if it's confused, entropy is high.

• If probability = 0 or 1 entropy = 0 (no surprise)
• If probability = 0.5 entropy = 1 (maximum surprise)

import torch
p = torch.tensor([0.5,0.5])
entropy = -(p*torch.log2(p)).sum()
print(entropy) # 1.0

Cross Entropy

Cross-entropy measures how different two probability distributions are — usually the true labels vs. the model's predictions. It's widely used as a loss function in classification.

Cross-entropy is like a punishment score for the model: the more it disagrees with the truth, the higher the score.

The better the model predicts, the smaller the cross-entropy.

In PyTorch:

import torch.nn.functional as F
pred = torch.tensor([0.9, 0.1])
target = torch.tensor([1.0, 0.0])
loss = F.binary_cross_entropy(pred, target)
print(loss)

6. Argmax and Argmin

Argmax and argmin are like shortcuts to find which input gives the biggest or smallest output.

• Argmax = "which choice is the biggest?"
• Argmin = "which choice is the smallest?"

In deep learning:

Argmax Predicted Class

• After a softmax layer, the model outputs probabilities for each class.
• argmax picks the class with the highest probability.

Argmin Loss Minimization

• During training, optimizers try to minimize the loss function.
• argmin can be used theoretically to find parameters that give the smallest loss.

x = torch.tensor([1,3,2])
print(torch.argmax(x)) # 1
print(torch.argmin(x)) # 0

Mean and variance

Mean and variance describe where the data is centered and how spread it is. They are fundamental in deep learning.

Where they are used in Deep Learning

Data normalization / standardization

Helps models train faster and more stably.

Batch Normalization

• Computes mean and variance per batch.
• Keeps activations stable during training.

Weight initialization

• Variance controls how signals flow through layers
• Avoids exploding or vanishing gradients.

Loss analysis & uncertainty

• Variance shows how confident or unstable predictions are.

Matrix example (batch of data)

x = torch.tensor([
[1.0, 2.0, 3.0],
[4.0, 5.0, 6.0]
])
mean_all = torch.mean(x)
var_all = torch.var(x, unbiased=False)
print(mean_all) # global mean
print(var_all) # global variance

Mean and variance per dimension

Very common in deep learning.

mean_dim0 = torch.mean(x, dim=0) # per feature (columns)
var_dim0 = torch.var(x, dim=0, unbiased=False)
mean_dim1 = torch.mean(x, dim=1) # per sample (rows)
print(mean_dim0) # tensor([2.5, 3.5, 4.5])
print(var_dim0)
print(mean_dim1) # tensor([2., 5.])

• dim=0 statistics per feature
• dim=1 statistics per sample

Sampling Variability — Graph Explanation

What sampling variability means

Sampling variability is the fact that different samples from the same population give slightly different results (mean, proportion, etc.).

Even if the population is fixed, samples change statistics change.

Concrete example

Population mean = 50

Sample size = 10

• Sample means: 47, 52, 49, 54
• High variability

Sample size = 100

• Sample means: 49.8, 50.3, 50.1
• Low variability

Same population, different stability.

Why it matters in Deep Learning

• Mini-batch training = sampling variability
• Batch mean and variance change from batch to batch

Explains:

• larger batch size more stable training

Reproducible randomness via seeding

Reproducible randomness means random results that can be repeated exactly by fixing a seed.

A seed is the starting point of a random number generator.

• Same seed same "random" sequence
• Different seed different sequence

CPU + GPU (full setup)

import torch
seed = 42
torch.manual_seed(seed)
torch.cuda.manual_seed(seed)
torch.cuda.manual_seed_all(seed)
torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False

t-test

A t-test checks whether the difference between means is real or just due to random sampling noise. Link with Machine Learning:

• Compare two models
• Validate A/B experiments
• Check if an improvement is real
• Used in evaluation, not training
• Can be used to compare models' performance

Graph explanation

• Big t difference is real
• Small t could just be luck

2. Relationship with p-value

• p-value = probability that the observed difference is due to chance
• Large |t| small p-value difference is statistically significant
• Small |t| large p-value difference could be random

Example with PyTorch (independent samples)

import torch
from scipy import stats
torch.manual_seed(42)
# Model A data on 30 batches
acc_modelA = torch.normal(mean=0.91, std=0.02, size=(30,))
# Model B data on 30 batches (independent)
acc_modelB = torch.normal(mean=0.89, std=0.02, size=(30,))
# Convert to NumPy for scipy
accA_np = acc_modelA.numpy()
accB_np = acc_modelB.numpy()
# Independent two-sample t-test (Welch's)
t_stat, p_value = stats.ttest_ind(accA_np, accB_np, equal_var=False)
print(f"t-statistic = {t_stat:.3f}, p-value = {p_value:.3f}")

Interpret results

Suppose the output is:

t-statistic = 3.92, p-value = 0.00032

Interpretation:

t-statistic = 3.92

• The difference between Model A and B is 3.92 times larger than the expected variability
• Large t strong evidence of a real difference

p-value = 0.00032

• Only 0.032% probability that such a difference occurred by chance
• p-value < 0.05 statistically significant difference

Practical meaning

• Model A mean accuracy = 91%
• Model B mean accuracy = 89%
• Even a 2% difference is consistent across batches, confirmed by t-test

Conclusion: Model A performs significantly better than Model B.

Intuition of a Derivative

• A derivative measures how fast a function changes at a given point.
• Geometrically, it is the slope of the tangent line on the curve.
• Positive slope function is increasing
• Negative slope function is decreasing
• Zero slope local maximum, minimum, or flat point

Intuition for Deep Learning

• Derivatives = gradients
• Tell how weights should change to minimize loss
• Polynomial is a simple example of how function change is measured locally

Derivatives find minima

1. Basic principle

• Derivative = slope of the function
• At a minimum, slope = 0:
• Then check the second derivative to confirm it's a minimum:
• Intuition: slope zero + curve is concave up valley

Deep Learning usage

• Neural networks minimize loss functions
• Maxima are rarely used, unless maximizing reward (reinforcement learning)

Derivatives: Product Rule and Chain Rule

1. Product Rule

Used when you have two functions multiplied:

Explanation:

• First term: derivative of first function (x²) times second function (sin(x))
• Second term: first function (x²) times derivative of second function (cos(x))

In Deep Learning:

Product rule is implicitly used when computing gradients of weights in layers that involve element-wise multiplications, such as in certain activation functions or when combining inputs in custom layers.

PyTorch Example:

import torch
# Define inputs with requires_grad=True to track gradients
x = torch.tensor(2.0, requires_grad=True)
y = torch.tensor(3.0, requires_grad=True)
# Function: f = x * y
f = x * y
# Compute gradients
f.backward() # df/dx and df/dy are computed automatically
print(f'df/dx: {x.grad}, df/dy: {y.grad}') # df/dx = y, df/dy = x

# PyTorch automatically applies product rule when computing gradients.

2. Chain Rule

Used when you have a function inside another function: (f(g(x)))

In Deep Learning:

• Chain rule is fundamental for backpropagation.
• When calculating the gradient of the loss with respect to weights, the derivative of the output is multiplied layer by layer (output of one layer is input to the next).

Example:

This is a direct application of the chain rule.

Explanation:

• Take derivative of outer function (sin(u)) (cos(u))
• Multiply by derivative of inner function (x²) (2x)

PyTorch Example:

import torch
x = torch.tensor(2.0, requires_grad=True)
# Function: f(x) = (x^2)^3 = x^6
f = (x**2)**3
f.backward()
print(f'df/dx: {x.grad}') # df/dx = 6*x^5

# PyTorch automatically applies chain rule for nested functions.

3. Combined Example (Product + Chain Rule)

In Deep Learning:

• Combined rules appear when a neuron output involves products of functions of previous layers.
• Backpropagation through layers with multiple operations (multiplication, non-linear activation) uses both product and chain rules together to compute gradients efficiently.

Example pytorch:

import torch
x = torch.tensor(1.0, requires_grad=True)
# Function: f(x) = x^2 * exp(x^3)
f = x**2 * torch.exp(x**3)
f.backward()
print(f'df/dx: {x.grad}') # df/dx = 2x*e^(x^3) + x^2*3x^2*e^(x^3) = 2*x*exp(x**3) + 3*x**4*exp(x**3)

# This shows PyTorch automatically handles both product and chain rules during backpropagation.