Optimization Techniques (ME-604 (B)) - Unit 1 Short Notes

Unit 1: Fundamentals of Optimization Techniques

1. Introduction to Optimization

Definition & Concept:

Optimization is the process of finding the best solution from all feasible alternatives. Mathematically, it involves:

• Minimizing or maximizing an objective function $f(x)$.

• Subject to constraints on the decision variables $$\displaystyle x = (x_1, x_2, ..., x_n) $$.

Components of an Optimization Problem:

1. Objective Function ($f(x)$): The goal to be optimized (e.g., minimize cost, maximize profit).

2. Decision Variables ($x$): Independent quantities that can be controlled.

3. Constraints:

   • Equality: $$\displaystyle h_i(x) = 0 $$

   • Inequality: $$\displaystyle g_j(x) \leq 0 $$ (or $\geq 0$)

   • Variable bounds: $$\displaystyle x_l \leq x \leq x_u $$

Classification:

Optimum Design Concept:

• Feasible Region: Set of all points satisfying constraints.

• Local Optimum: Best solution in a neighborhood.

• Global Optimum: Best solution over entire feasible region.

[!TIP]

Nonlinear problems may have multiple local optima; global optimum is the true best.

Need in Engineering:

• Cost reduction (material, manufacturing).

• Performance improvement (efficiency, strength).

• Resource efficiency (energy, time).

• Critical in industrial product development for competitive design.

2. Linear Programming (LP)

Formulation (Standard Form):

• Objective: Minimize $$\displaystyle c^T x $$

• Subject to: $$\displaystyle Ax = b $$, $x \geq 0$

  (All constraints as equalities; variables non-negative).

Assumptions:

1. Proportionality: Contribution proportional to variable value.

2. Additivity: Total effect is sum of individual effects.

3. Divisibility: Variables can take fractional values.

4. Certainty: All coefficients are known constants.

Applications:

• Production planning: Maximize profit given resource limits.

• Blending: Mix raw materials to meet specs at min cost.

• Resource allocation: Distribute limited resources optimally.

Steps to Solve LP:

1. Formulate the problem mathematically.

2. Graphical/Algebraic solution (2 variables) or Simplex method (multi-variable).

3. Sensitivity analysis: Study effect of parameter changes.

Graphical Method (2 Variables):

1. Plot all constraint lines.

2. Identify feasible region (common intersection).

3. Locate corner points (vertices of feasible region).

4. Evaluate objective at each corner point.

5. Optimal solution occurs at a corner point (if bounded).

Example:

Maximize $$\displaystyle Z = 3x_1 + 2x_2 $$

Subject to:

$$\displaystyle 2x_1 + x_2 \leq 100 $$

$$\displaystyle x_1 + 2x_2 \leq 100 $$

$$\displaystyle x_1, x_2 \geq 0 $$

Solution: Evaluate $Z$ at corners: (0,0), (50,0), (33.33,33.33), (0,50). Max at (33.33, 33.33).

Limitations:

• Only for 2 variables (visual).

• Not scalable; for $$\displaystyle n > 2 $$, use Simplex or software.

3. Unconstrained Optimization

Algorithms:

1. Direct Search (Derivative-free):

   • Evaluate $f(x)$ at trial points.

   • Examples: Pattern Search, Hooke-Jeeves.

2. Gradient-based: Use derivatives for faster convergence.

Newton's Method (Gradient-based):

• Derivation: Taylor series expansion of $f(x)$ around $$\displaystyle x_k $$:

$$f(x) \approx f(x_k) + \nabla f(x_k)^T (x - x_k) + \frac{1}{2} (x - x_k)^T \nabla^2 f(x_k) (x - x_k)$$

• Iterative Formula:

$$x_{k+1} = x_k - [\nabla^2 f(x_k)]^{-1} \nabla f(x_k)$$

where $\nabla f$ = gradient, $$\displaystyle \nabla^2 f $$ = Hessian matrix.

• Steps:

  1. Start at initial guess $$\displaystyle x_0 $$.

  2. Compute gradient $$\displaystyle \nabla f(x_k) $$ and Hessian $$\displaystyle \nabla^2 f(x_k) $$.

  3. Update $$\displaystyle x_{k+1} $$ using formula.

  4. Check convergence: $$\displaystyle \|\nabla f(x_k)\| < \epsilon $$.

• Convergence: Quadratic near optimum (very fast).

• Advantages: Fast convergence for smooth functions.

• Limitations:

  • Requires Hessian (costly for large $n$).

  • Sensitive to initial guess (may diverge).

  • Hessian must be positive definite near minimum.

[!TIP]

For large problems, use Quasi-Newton methods (e.g., BFGS) that approximate Hessian.

4. Constrained Optimization

Penalty Function Method:

• Concept: Transform constrained problem into unconstrained by adding a penalty term for constraint violation.

• General Form:

$$P(x, r) = f(x) + r \cdot Q(x)$$

where $$\displaystyle r > 0 $$ = penalty parameter, $Q(x)$ = penalty function (zero if feasible, positive if infeasible).

Types:

1. Exterior (Penalty):

   • Penalizes infeasible points.

   • $$\displaystyle Q(x) = \sum [\max(0, g_j(x))]^2 + \sum [h_i(x)]^2 $$

   • As $r \to \infty$, solution approaches constrained optimum.

2. Interior (Barrier):

   • Keeps iterates inside feasible region.

   • For $$\displaystyle g_j(x) \leq 0 $$, use $$\displaystyle Q(x) = -\sum \log(-g_j(x)) $$.

   • $r$ decreased gradually; solution approaches boundary.

Selection of $r$:

• Start with small $r$, increase gradually (exterior).

• Too small: slow convergence; too large: ill-conditioning.

Engineering Applications:

Design problems with stress, displacement, frequency constraints (e.g., minimize weight of a beam subject to stress $\leq$ allowable).

5. Modern Optimization Techniques

Traditional vs. Modern:

Steps in Modern Optimization:

1. Problem definition (objective, constraints, variables).

2. Algorithm selection (GA, PSO, etc.).

3. Parameter tuning (population size, mutation rate).

4. Execution (run algorithm).

5. Validation (check feasibility, robustness).

Genetic Algorithms (GA):

• Inspiration: Biological evolution (natural selection).

• Key Operators:

  1. Selection: Choose parents (e.g., roulette wheel, tournament).

  2. Crossover: Recombine parents' genes (e.g., single-point).

  3. Mutation: Random alter offspring (maintain diversity).

• Representation:

  • Binary: Strings of 0s/1s.

  • Real-coded: Floating-point vectors (better for continuous).

• Fitness Function: Evaluates solution quality (often $f(x)$ transformed).

• Parameters: Population size (50–200), crossover prob (0.6–0.9), mutation prob (0.01–0.1).

• Applications:

  • Multi-objective optimization (Pareto front).

  • Combinatorial problems (scheduling, routing).

Fuzzy Optimization:

• Fuzzy Sets: Elements have membership degree $$\displaystyle \mu_A(x) \in [0,1] $$ (vs. crisp 0/1).

• Handles: Vague objectives/constraints (e.g., "cost should be low").

• Approach:

  1. Fuzzify uncertain parameters (define membership functions).

  2. Solve fuzzy mathematical program (e.g., maximize satisfaction).

  3. Defuzzify if crisp solution needed.

• Applications:

  • Decision-making under uncertainty.

  • Supplier selection, risk assessment.

6. Engineering Applications and Computational Tools

Engineering Applications:

• Design Optimization: Minimize weight/cost subject to stress, deflection limits.

• Process Optimization: Maximize yield, minimize energy in chemical processes.

• Planning & Scheduling: Optimize production sequences, inventory.

• Case Study:

  Weight minimization of a welded beam subject to stress, buckling, deflection constraints → solved via GA or fmincon.

Role of MATLAB:

• Optimization Toolbox provides functions:

  • linprog: Linear programming.

  • fmincon: Constrained nonlinear optimization.

  • ga: Genetic Algorithm.

  • patternsearch: Direct search (Hooke-Jeeves).

• Typical Steps:

  1. Define objective function @(x) f(x).

  2. Define constraints (linear A,b,Aeq,beq; nonlinear nonlcon).

  3. Call solver: [x,fval] = fmincon(fun,x0,A,b,Aeq,beq,lb,ub,nonlcon).

  4. Analyze results.

• Example:

  Minimize $$\displaystyle f(x) = x_1^2 + x_2^2 $$

  Subject to $$\displaystyle x_1 + x_2 \geq 1 $$

  
  fun = @(x) x(1)^2 + x(2)^2;
  
  nonlcon = @(x) deal(-(x(1)+x(2)-1), []); % g(x) ≤ 0 → -(x1+x2-1) ≤ 0
  
  x0 = [0,0];
  
  [x,fval] = fmincon(fun,x0,[],[],[],[],[],[],nonlcon);
  
  

Integration with Other Tools:

• Python: scipy.optimize (minimize, linprog), pygmo (GA).

• Specialized Software:

  • ANSYS: Design optimization in FEA.

  • COMSOL: Parameter optimization in multiphysics simulations.

[!TIP]

For non-differentiable or noisy objectives (common in simulation), use GA or patternsearch in MATLAB.