Introduction to python Mathematical Optimization : Algorithms That Power the Future

Quick Navigation

Difficulty: Intermediate
Estimated Time: 15-25 minutes
Prerequisites: python-basics, linear-algebra, calculus-fundamentals, scipy-numpy

What is Mathematical Optimization?

Mathematical optimization is a powerful tool used to find the best possible solution in various real-world applications. Whether it's minimizing costs, maximizing efficiency, or finding the shortest path, optimization algorithms are at the core of artificial intelligence, finance, logistics, engineering, and more.

In this article, we'll explore key optimization algorithms, their use cases, and how to implement them in Python. Get ready for an exciting journey!

1 Linear Programming (LP) — Finding the Best Decision

Used for: Decision-making problems where constraints and objectives are linear. Common Algorithm: Simplex Method, Interior Point Methods

Example:

Supply Chain Optimization: Minimizing transportation costs Portfolio Optimization: Maximizing returns under budget constraints

Explaining Linear Programming (LP) — Simplex Method to a Child with a Visual Example

Imagine You Have a Lemonade Stand

You have two ingredients: lemons and sugar. You need to decide how many lemonades and how many lemon cakes to make.

But… there are some limits (constraints): 1 You only have 20 lemons 2 You only have 12 cups of sugar 3 You can't make negative amounts of lemonade or cakes (so x and y must be positive)

Your Goal? Maximize Your Profit

You want to find the best combination of lemonades and cakes to make the most money!

How the Simplex Method Works

1 We draw the constraints on a graph. 2 The shaded area (feasible region) shows all possible combinations that respect the rules. 3 We find the best point (corner of the feasible region) where profit is maximized!

The Graph Above The red, blue, and green lines are the limits (constraints). The gray shaded region is where all your possible solutions exist. The best point in this area gives the maximum profit!

This is how businesses decide how much to produce while making the most money!

Python Implementation (Simplex Method):

from scipy.optimize import linprog
c = [-1, -2]  # Objective function (Maximize)
A = [[2, 1], [1, 1], [1, 2]]  # Constraints
b = [20, 12, 16]
bounds = [(0, None), (0, None)]
result = linprog(c, A_ub=A, b_ub=b, bounds=bounds, method="highs")
print("Optimal solution:", result.x)

2 Nonlinear Programming (NLP) — Beyond Linear Problems

What is Nonlinear Programming (NLP)?

Nonlinear programming helps find the best solution when the objective function is curved or complex! Unlike linear programming (where everything is a straight line ), NLP deals with curves.

Used for: When the objective function or constraints are nonlinear. Common Algorithms: Gradient Descent, Newton's Method

Example:

Machine Learning Training: Adjusting weights to minimize loss Structural Engineering: Optimizing material use in bridges

Imagine You're Rolling a Ball Down a Hill

Think about a ball rolling on a bumpy hill. The ball wants to reach the lowest point (where it's most stable). But because the hill has curves and dips, the ball doesn't always move in a straight line.

This is how nonlinear problems work — they have curvy paths instead of straight ones!

What Are We Trying to Do?

Let's say you're playing a video game where you need to find the fastest way to collect coins .

If you go too fast , you might miss some coins. If you go too slow , the game timer might run out!

You need to find the best speed to win the most coins. That's nonlinear optimization!

How Does the Computer Help?

1 It starts at a random point (like a ball on a hill). 2 It checks the slope (is it going up or down? ). 3 It moves step by step to the best spot (where the game score is highest!).

Python Implementation (Gradient Descent):

import numpy as np
def f(x):
    return x**2 + 4*x + 4
def grad_f(x):
    return 2*x + 4
x = 10  # Initial guess
learning_rate = 0.1
for _ in range(100):
    x -= learning_rate * grad_f(x)
print("Optimal solution:", x)

3 Integer Programming (IP) — When You Need Whole Numbers

Used for: Problems where decision variables must be integers (e.g., scheduling, resource allocation). Common Algorithms: Branch and Bound, Branch and Cut

Imagine You're Building a Team of Robots!

You're the boss of a super cool robot company , and you need to decide:

How many robots should you build? How many AI assistants should you create?

BUT… You can't build half a robot! You can't have 2.5 AI assistants either!

You need to only use whole numbers (0, 1, 2, 3… and so on).

What's Happening in the Picture?

The blue dots are all possible choices (whole number solutions). The red star is the best choice — the one that gives the highest score (profit or efficiency).

Where Do We Use This in Real Life?

School Scheduling — Assigning teachers to classes Lego Building — Choosing how many blocks to use Pizza Delivery — Deciding how many pizzas to make

Integer programming helps us make the best decisions when we can't use fractions!

Python Implementation (Branch and Bound using PuLP):

from pulp import LpMaximize, LpProblem, LpVariable
model = LpProblem(name="integer-programming", sense=LpMaximize)
x = LpVariable(name="x", lowBound=0, cat="Integer")
y = LpVariable(name="y", lowBound=0, cat="Integer")
model += 3 * x + 2 * y
model += (2 * x + y <= 8)
model += (x + 2 * y <= 6)
model.solve()
print("Optimal solution: x =", x.varValue, "y =", y.varValue)

4 Quadratic Programming (QP) — When Your Function is Curved

Used for: Problems where the objective function is quadratic but constraints are linear. Common Algorithms: Interior Point Methods, Active Set Methods

Example:

Finance: Portfolio risk minimization Control System Optimization: Reducing fuel consumption in rockets

Quadratic Programming (QP) — When Your Function is Curved!

Imagine You're Jumping on a Trampoline!

When you jump up and down on a trampoline, you follow a curved path! You go up, up, up…, reach the highest point , and then come back down.

Or, if you throw a ball , it also follows a curve before landing!

This is how quadratic functions work — they curve up or down, and we want to find the best point (highest or lowest).

What's Happening in the Picture?

The purple curve is like the path of a bouncing ball. The red dot is the best point — the lowest point of the curve (where we find the best solution).

Where Do We Use This in Real Life?

Throwing a Basketball — Finding the perfect arc for scoring a point Building Bridges — Making sure they don't bend too much Stock Market — Finding the best time to buy or sell stocks

Quadratic programming helps us solve problems with curves — just like jumping, bouncing, and aiming arrows!

Python Implementation (Using CVXPY):

import cvxpy as cp
x = cp.Variable()
y = cp.Variable()
objective = cp.Minimize(x**2 + y**2 + x*y)
constraints = [x + y >= 1, x - y <= 2]
prob = cp.Problem(objective, constraints)
prob.solve()
print("Optimal solution: x =", x.value, "y =", y.value)

5 Global Optimization — Escaping Local Minima

Used for: When local optimization methods fail to find the global minimum. Common Algorithms: Simulated Annealing, Genetic Algorithms, Particle Swarm Optimization (PSO)

Example:

Route Planning: Finding the best delivery route AI Model Optimization: Hyperparameter tuning

Global Optimization — Escaping the Traps of Local Minima!

Imagine You're Hiking on a Mountain!

You are on a big mountain, and you want to find the lowest valley. But…

Some paths go down a little bit and feel like valleys (local minimum), but they are not the lowest place. The real lowest valley is far away, and you must keep searching for the best path!

This is how global optimization works — sometimes you get stuck in small dips, but the best solution is somewhere else!

What's Happening in the Picture?

The blue curve is like a hilly road with ups and downs. The orange dot is a local minimum — it feels like the lowest point, but it's not the best! The red dot is the global minimum — the lowest and best solution!

How Do We Escape Local Minima?

Simulated Annealing — Like melting metal to reshape it into something better! Genetic Algorithms — Like evolution, trying different solutions to find the best one! Particle Swarm Optimization — Like birds flying together to find the best food spot!

Where Do We Use Global Optimization?

AI Training — Helping computers learn the best way! Travel Planning — Finding the fastest and cheapest route! Game Strategy — Finding the best moves in chess!

Python Implementation (Simulated Annealing using SciPy):

from scipy.optimize import dual_annealing
import numpy as np
def objective(x):
    return x**2 + 4 * np.sin(5 * x)
bounds = [(-5, 5)]
result = dual_annealing(objective, bounds)
print("Optimal solution:", result.x)

6 Combinatorial Optimization — The Smartest Way to Arrange Things

Used for: Problems where the solution space is discrete. Common Algorithms: Dijkstra's Algorithm, Bellman-Ford Algorithm, Hungarian Algorithm

Example:

Shortest Path: GPS navigation Job Scheduling: Assigning workers to tasks

Combinatorial Optimization — The Smartest Way to Arrange Things!

Imagine You're a Delivery Driver!

You have 5 cities to visit, and you want to find the shortest path to deliver all the packages . But…

There are many different routes you can take! Some paths are shorter than others. You want to save time and fuel .

This is a combinatorial optimization problem — we need to find the best way to arrange things!

What's Happening in the Picture?

The blue circles are the cities . The gray lines are possible roads between the cities. The numbers show the distance (cost) of each road.

Our goal is to find the shortest path that visits all cities once and returns home!

How Do We Solve These Problems?

Dijkstra's Algorithm — Finds the shortest path in road networks. Traveling Salesman Problem (TSP) — Finds the best way to visit multiple places. Hungarian Algorithm — Assigns jobs to workers in the most efficient way.

Where Do We Use Combinatorial Optimization?

Delivery Services — Finding the best routes for Amazon & FedEx! Scheduling Flights — Organizing plane takeoffs and landings. Solving Puzzles — Creating the best strategy for solving Sudoku!

Python Implementation (Dijkstra's Algorithm using NetworkX):

import networkx as nx
G = nx.DiGraph()
G.add_weighted_edges_from([(1, 2, 7), (1, 3, 9), (1, 6, 14), (2, 3, 10), (3, 6, 2), (4, 5, 6)])
path = nx.dijkstra_path(G, source=1, target=5, weight="weight")
print("Shortest path:", path)

7 Reinforcement Learning-Based Optimization — AI Learning from Experience

Used for: Problems where an agent learns optimal actions through experience. Common Algorithms: Q-Learning, Deep Q-Networks (DQN)

Example:

Self-Driving Cars: Learning the best driving strategy Game AI: Learning how to play games optimally

Reinforcement Learning-Based Optimization — AI Learning from Experience

Imagine You're Playing a Maze Game!

You're a robot , and your goal is to reach the trophy ! But…

There are obstacles blocking some paths! You don't know the best way to get to the trophy at first! You need to try different moves, learn from mistakes, and improve!

This is how reinforcement learning works! Instead of knowing the answers, the AI tries, fails, learns, and gets better over time!

What's Happening in the Picture?

The robot starts in the bottom-left corner. The trophy is the goal (winning state!). The obstacles make it harder to find the best path. The AI learns by trial and error to find the best way to reach the goal!

How Does AI Learn? (Q-Learning Algorithm)

Try different moves Get rewards or penalties Remember what worked best Repeat until it finds the fastest way!

Python Implementation (Q-Learning Table using NumPy):

import numpy as np
Q = np.zeros((5, 2))
alpha = 0.1
gamma = 0.9
for episode in range(100):
    state = np.random.randint(0, 5)
    action = np.random.choice([0, 1])
    reward = np.random.randint(-10, 10)
    next_state = np.random.randint(0, 5)
    Q[state, action] += alpha * (reward + gamma * np.max(Q[next_state]) - Q[state, action])
print("Q-table:\n", Q)

Conclusion — The Power of Optimization!

Mathematical optimization is a game-changer in technology, science, and business. From simplifying logistics to building smarter AI, these algorithms help us solve complex problems efficiently and intelligently.

Want to go deeper? Try implementing these algorithms in your projects!

Did you enjoy this article? Let me know in the comments!