Computational Intelligence (CS-702 (A)) Unit 1 Notes

How unit 1 is examined

This unit covers what CI is, its types and components, learning and parametric/nonparametric models, and feed-forward versus feedback networks. Marks sit in the learning model (14-mark short note), components, feed-forward/feedback networks and parametric models.

Introduction to Computational Intelligence

<span style="display:inline-block;padding:.16em .6em;border:1.5px solid currentColor;border-radius:999px;font-size:.68em;font-weight:700;letter-spacing:.06em;text-transform:uppercase;opacity:.75">Not asked since 2022</span>

Definition. <mark>Computational Intelligence (CI) is the study of nature-inspired, adaptive computing methods that solve problems too imprecise, uncertain or complex for exact mathematical models.</mark>

Key points.

CI is also called soft computing because it tolerates imprecision, uncertainty and partial truth.
It imitates nature: brains give neural networks, human reasoning gives fuzzy logic, evolution gives genetic algorithms.
CI methods learn or adapt from data and need no complete mathematical model of the problem.
Its goal is robust, low-cost, approximate solutions where hard computing is impractical.

Types of Computational Intelligence

Definition. The types (paradigms) of CI are the main nature-inspired families of methods.

Key points.

Neural networks learn input-output mappings from data, inspired by the brain.
Fuzzy systems reason with degrees of truth in $[0,1]$ using if-then rules.
Evolutionary computation (genetic algorithms) searches by selection, crossover and mutation.
Swarm intelligence (ant colony, particle swarm) uses collective behaviour of simple agents; rough sets handle vagueness through approximation.

Components of Computational Intelligence

Definition. <mark>Computational Intelligence is a set of nature-inspired techniques that gives machines adaptive, human-like problem solving, and its components are neural networks, fuzzy systems, evolutionary computation, swarm intelligence and rough sets.</mark>

Key points.

Artificial neural networks are layers of connected neurons that learn weights from examples, for example recognising handwritten digits.
Fuzzy systems use membership degrees and rules such as "if temperature is high then fan speed is fast", for example a washing machine controller.
Evolutionary computation, mainly genetic algorithms, evolves a population of candidate solutions using a fitness function, for example timetable scheduling.
Swarm intelligence such as ant colony and particle swarm optimisation uses cooperating agents, for example finding the shortest route.
Rough sets describe vague data by lower and upper approximations, for example reducing attributes in a decision table.
The components complement one another: neuro-fuzzy systems learn fuzzy rules with a network, and genetic algorithms can tune network weights.

Answer frame. Open by defining CI and its goal (approximate, adaptive solutions under uncertainty); draw a block diagram of the components joined by arrows to show hybridisation; develop points 1-5 with one example each; close with point 6 on hybrid systems.

Asked: [7 marks] (Dec 2020, Nov 2023) Discuss the various components of computational intelligence; explain all the components with the help of examples.

Concept of Learning/Training model

Definition. <mark>A learning (training) model adjusts its internal parameters using data so that its output error on the task becomes small, and it then generalises to unseen inputs.</mark>

Diagram. <figure class="ds-fig" style="margin:1.4rem 0;overflow-x:auto"><svg xmlns="http://www.w3.org/2000/svg" id="dsfig-u1-01" viewBox="0 0 467 217.6" width="467" height="217.6" role="img" aria-label="Training loop. D = training data, M = model, O = output, E = error, U = weight update"><style>#dsfig-u1-01 .e{stroke:#454C5A;stroke-width:1.4;fill:none}#dsfig-u1-01 .e.hi{stroke:#2340B8;stroke-width:2.6}#dsfig-u1-01 .n{fill:#FFFFFF;stroke:#16181D;stroke-width:1.4}#dsfig-u1-01 .n.hi{fill:#E3E9FC;stroke:#2340B8;stroke-width:2.2}#dsfig-u1-01 .n.rb-b{fill:#16181D;stroke:#16181D}#dsfig-u1-01 .n.rb-r{fill:#BD3227;stroke:#BD3227}#dsfig-u1-01 text{font-family:"JetBrains Mono",ui-monospace,Menlo,Consolas,monospace;font-size:13px}#dsfig-u1-01 .t{fill:#16181D;font-weight:500}#dsfig-u1-01 .t.inv{fill:#FFFFFF;font-weight:700}#dsfig-u1-01 .kd{stroke:#16181D;stroke-width:1.2}#dsfig-u1-01 .dot{fill:#16181D}#dsfig-u1-01 .ann{fill:#2340B8;font-size:11px;font-weight:700}#dsfig-u1-01 .lbl{fill:#6F7787;font-family:system-ui,-apple-system,sans-serif;font-size:12px;font-weight:700}#dsfig-u1-01 .ptr{fill:#2340B8;font-size:12px;font-weight:700}#dsfig-u1-01 .ah{fill:#454C5A}#dsfig-u1-01 .ah.hi{fill:#2340B8}#dsfig-u1-01 .wl rect{fill:#FFFFFF;stroke:#DCE0E7}#dsfig-u1-01 .wl .t{font-size:12px;font-weight:700}#dsfig-u1-01 .wl.hi rect{fill:#2340B8;stroke:#2340B8}#dsfig-u1-01 .wl.hi .t{fill:#FFFFFF}html.dark #dsfig-u1-01 .e{stroke:#B1B7C3}html.dark #dsfig-u1-01 .e.hi{stroke:#8FA3FF}html.dark #dsfig-u1-01 .n{fill:#161920;stroke:#E6E8ED}html.dark #dsfig-u1-01 .n.hi{fill:#1E2748;stroke:#8FA3FF}html.dark #dsfig-u1-01 .n.rb-b{fill:#E6E8ED;stroke:#E6E8ED}html.dark #dsfig-u1-01 .n.rb-r{fill:#FF7E71;stroke:#FF7E71}html.dark #dsfig-u1-01 .t{fill:#E6E8ED}html.dark #dsfig-u1-01 .t.inv{fill:#0F1115}html.dark #dsfig-u1-01 .kd{stroke:#E6E8ED}html.dark #dsfig-u1-01 .dot{fill:#E6E8ED}html.dark #dsfig-u1-01 .ann{fill:#8FA3FF}html.dark #dsfig-u1-01 .lbl{fill:#858D9C}html.dark #dsfig-u1-01 .ptr{fill:#8FA3FF}html.dark #dsfig-u1-01 .ah{fill:#B1B7C3}html.dark #dsfig-u1-01 .ah.hi{fill:#8FA3FF}html.dark #dsfig-u1-01 .wl rect{fill:#161920;stroke:#2A2E37}html.dark #dsfig-u1-01 .wl.hi rect{fill:#8FA3FF;stroke:#8FA3FF}html.dark #dsfig-u1-01 .wl.hi .t{fill:#0F1115}</style><defs><marker id="ah1" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="ah" d="M0,1 L9,5 L0,9 z"/></marker><marker id="ahh1" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="ah hi" d="M0,1 L9,5 L0,9 z"/></marker></defs><path class="e" d="M59,40 L148,40" marker-end="url(#ah1)"/><path class="e" d="M188,40 L277,40" marker-end="url(#ah1)"/><path class="e" d="M317,40 L406,40" marker-end="url(#ah1)"/><path class="e" d="M414,53.9 L312.4,162.3" marker-end="url(#ah1)"/><path class="e" d="M285,163.7 L183.4,55.3" marker-end="url(#ah1)"/><g class="wl"><rect x="335.4" y="99.8" width="54.3" height="18" rx="9"/><text class="t" x="362.5" y="108.8" dy=".35em" text-anchor="middle">adjust</text></g><circle class="n" cx="40" cy="40" r="18"/><text class="t" x="40" y="40" dy=".35em" text-anchor="middle">D</text><circle class="n" cx="169" cy="40" r="18"/><text class="t" x="169" y="40" dy=".35em" text-anchor="middle">M</text><circle class="n" cx="298" cy="40" r="18"/><text class="t" x="298" y="40" dy=".35em" text-anchor="middle">O</text><circle class="n" cx="427" cy="40" r="18"/><text class="t" x="427" y="40" dy=".35em" text-anchor="middle">E</text><circle class="n" cx="298" cy="177.6" r="18"/><text class="t" x="298" y="177.6" dy=".35em" text-anchor="middle">U</text></svg><figcaption style="font-size:.82em;opacity:.72;margin-top:.45rem">Training loop. D = training data, M = model, O = output, E = error, U = weight update</figcaption></figure>

Key points.

Training feeds examples to the model, compares its output with the target and computes an error.
The parameters (weights) are updated to reduce that error, for example by gradient descent $w \leftarrow w - \eta\,\partial E/\partial w$ with learning rate $\eta$.
Supervised learning uses labelled data (input with target), for example classification.
Unsupervised learning finds structure in unlabelled data, for example clustering.
Reinforcement learning learns from rewards and penalties received from the environment.
Data is split into a training set to fit the model and a test set to measure generalisation.
Overfitting (memorising training data) and underfitting (too simple a model) are the two failure modes.

Example. A network for pass/fail prediction is shown study hours with the known result, its error is measured, weights are updated, and the loop repeats until the error is small on both training and test data.

Also asked in the short note. Fitness function: a function $f(x)$ that scores how good a candidate solution is in a genetic algorithm; higher fitness means a better chance of selection, for example $f(x)=x^2$ for maximising $x^2$. Set approximation: a rough set $X$ is described by its lower approximation (objects surely in $X$) and upper approximation (objects possibly in $X$); the difference is the boundary region. Types of CI: see the types section above.

Answer frame. For "Learning model", open with the definition, draw the training loop, then develop points 1-7 and end with the example. For a short note on any three, give each a definition, one feature or formula and one example, about a third of a page each.

Pitfall: Writing only the name of each short-note item without a definition and example loses marks, because each of the three needs its own.

Asked: [14 marks] (Dec 2020) Write short notes on (any three): i) Learning Model ii) Fitness Function iii) Set approximation iv) Types of Computational Intelligence.

Parametric Models

Definition. <mark>A parametric model assumes a fixed functional form with a finite number of parameters, which are estimated from data.</mark>

Key points.

Working: choose a form such as $y=\beta_0+\beta_1 x$ or a normal distribution $N(\mu,\sigma^2)$, estimate the parameters from data (for example by least squares or maximum likelihood), then use the fitted model to predict.
The number of parameters stays fixed however much data is available, so the model is fast and needs little data but is biased if the assumed form is wrong.
Application in statistics: linear regression, and estimating the mean and variance of a normal population from a sample, where $\hat\mu=\frac{1}{n}\sum x_i$.
Example: heights are assumed normal; from a sample, $\hat\mu$ and $\hat\sigma$ define the whole distribution and give probabilities for any height.

Asked: [7 marks] (Nov 2023) Explain the working of a Parametric Model and its application in statistics.

Nonparametric Models

Definition. A nonparametric model makes no fixed-form assumption, so its complexity grows with the amount of data (distribution-free, instance-based).

Key points.

Examples are k-nearest neighbours, kernel density estimation and decision trees.
It is flexible and fits unknown shapes, but needs more data and is slower at prediction.
Compared with parametric models, which have a fixed number of parameters, it has a growing number of effective parameters.

Multilayer Networks: Feed Forward network

Definition. <mark>A feed-forward network is a multilayer neural network whose connections form no cycle, so signals travel only from the input layer through hidden layers to the output layer.</mark>

Diagram. <figure class="ds-fig" style="margin:1.4rem 0;overflow-x:auto"><svg xmlns="http://www.w3.org/2000/svg" id="dsfig-u1-02" viewBox="0 0 338 166" width="338" height="166" role="img" aria-label="Feed-forward network. I = input layer, H = hidden layer, O = output layer"><style>#dsfig-u1-02 .e{stroke:#454C5A;stroke-width:1.4;fill:none}#dsfig-u1-02 .e.hi{stroke:#2340B8;stroke-width:2.6}#dsfig-u1-02 .n{fill:#FFFFFF;stroke:#16181D;stroke-width:1.4}#dsfig-u1-02 .n.hi{fill:#E3E9FC;stroke:#2340B8;stroke-width:2.2}#dsfig-u1-02 .n.rb-b{fill:#16181D;stroke:#16181D}#dsfig-u1-02 .n.rb-r{fill:#BD3227;stroke:#BD3227}#dsfig-u1-02 text{font-family:"JetBrains Mono",ui-monospace,Menlo,Consolas,monospace;font-size:13px}#dsfig-u1-02 .t{fill:#16181D;font-weight:500}#dsfig-u1-02 .t.inv{fill:#FFFFFF;font-weight:700}#dsfig-u1-02 .kd{stroke:#16181D;stroke-width:1.2}#dsfig-u1-02 .dot{fill:#16181D}#dsfig-u1-02 .ann{fill:#2340B8;font-size:11px;font-weight:700}#dsfig-u1-02 .lbl{fill:#6F7787;font-family:system-ui,-apple-system,sans-serif;font-size:12px;font-weight:700}#dsfig-u1-02 .ptr{fill:#2340B8;font-size:12px;font-weight:700}#dsfig-u1-02 .ah{fill:#454C5A}#dsfig-u1-02 .ah.hi{fill:#2340B8}#dsfig-u1-02 .wl rect{fill:#FFFFFF;stroke:#DCE0E7}#dsfig-u1-02 .wl .t{font-size:12px;font-weight:700}#dsfig-u1-02 .wl.hi rect{fill:#2340B8;stroke:#2340B8}#dsfig-u1-02 .wl.hi .t{fill:#FFFFFF}html.dark #dsfig-u1-02 .e{stroke:#B1B7C3}html.dark #dsfig-u1-02 .e.hi{stroke:#8FA3FF}html.dark #dsfig-u1-02 .n{fill:#161920;stroke:#E6E8ED}html.dark #dsfig-u1-02 .n.hi{fill:#1E2748;stroke:#8FA3FF}html.dark #dsfig-u1-02 .n.rb-b{fill:#E6E8ED;stroke:#E6E8ED}html.dark #dsfig-u1-02 .n.rb-r{fill:#FF7E71;stroke:#FF7E71}html.dark #dsfig-u1-02 .t{fill:#E6E8ED}html.dark #dsfig-u1-02 .t.inv{fill:#0F1115}html.dark #dsfig-u1-02 .kd{stroke:#E6E8ED}html.dark #dsfig-u1-02 .dot{fill:#E6E8ED}html.dark #dsfig-u1-02 .ann{fill:#8FA3FF}html.dark #dsfig-u1-02 .lbl{fill:#858D9C}html.dark #dsfig-u1-02 .ptr{fill:#8FA3FF}html.dark #dsfig-u1-02 .ah{fill:#B1B7C3}html.dark #dsfig-u1-02 .ah.hi{fill:#8FA3FF}html.dark #dsfig-u1-02 .wl rect{fill:#161920;stroke:#2A2E37}html.dark #dsfig-u1-02 .wl.hi rect{fill:#8FA3FF;stroke:#8FA3FF}html.dark #dsfig-u1-02 .wl.hi .t{fill:#0F1115}</style><defs><marker id="ah2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="ah" d="M0,1 L9,5 L0,9 z"/></marker><marker id="ahh2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="ah hi" d="M0,1 L9,5 L0,9 z"/></marker></defs><path class="e" d="M59,40 L148,40" marker-end="url(#ah2)"/><path class="e" d="M55.8,50.5 L151.5,114.4" marker-end="url(#ah2)"/><path class="e" d="M55.8,115.5 L151.5,51.6" marker-end="url(#ah2)"/><path class="e" d="M59,126 L148,126" marker-end="url(#ah2)"/><path class="e" d="M187,46 L278.1,76.4" marker-end="url(#ah2)"/><path class="e" d="M187,120 L278.1,89.6" marker-end="url(#ah2)"/><circle class="n" cx="40" cy="40" r="18"/><text class="t" x="40" y="40" dy=".35em" text-anchor="middle">I1</text><circle class="n" cx="40" cy="126" r="18"/><text class="t" x="40" y="126" dy=".35em" text-anchor="middle">I2</text><circle class="n" cx="169" cy="40" r="18"/><text class="t" x="169" y="40" dy=".35em" text-anchor="middle">H1</text><circle class="n" cx="169" cy="126" r="18"/><text class="t" x="169" y="126" dy=".35em" text-anchor="middle">H2</text><circle class="n" cx="298" cy="83" r="18"/><text class="t" x="298" y="83" dy=".35em" text-anchor="middle">O</text></svg><figcaption style="font-size:.82em;opacity:.72;margin-top:.45rem">Feed-forward network. I = input layer, H = hidden layer, O = output layer</figcaption></figure>

Key points.

Each neuron computes $y=f\left(\sum w_i x_i + b\right)$ and passes it to the next layer only.
It has no memory; the output depends only on the present input.
It is trained by backpropagation, which propagates the error backwards to update weights.
Feedback (recurrent) networks contain cycles, so they keep a state, for example Hopfield and RNN.

Point	Feed-forward	Feedback
Structure	Acyclic, one direction	Has loops
Memory	None	Keeps state
Training	Backpropagation	Recurrent methods, BPTT
Dynamics	Static mapping	Evolves over time
Example	Multilayer perceptron	Hopfield network

Design for singular boundary value problems. Take $y''+\frac{2}{x}y'=f(x,y)$ with $y(0)=A$, $y'(0)=0$; the term $2/x$ is singular at $x=0$.

Network: one input $x$, one hidden layer, one output $N(x,p)$ with weights $p$.
Trial solution: $y_t(x)=A+x^2N(x,p)$, which satisfies both boundary conditions for any $p$.
Loss: $E(p)=\sum_i\left[y_t''(x_i)+\frac{2}{x_i}y_t'(x_i)-f(x_i,y_t)\right]^2$ over collocation points $x_i>0$.
Training: minimise $E$ by backpropagation until it is near zero.

Answer frame. For "feed forward or feedback", define both, draw a network, then give the comparison table. For "design", state the problem, draw the architecture, give steps 2-4, and close that the trained $y_t$ approximates the solution.

Asked: [7 marks] (Dec 2020) What is meant by Feed forward or Feedback Networks?

Asked: [7 marks] (Nov 2023) Design a Feed Forward Neural Network to solve Singular Boundary Value Problems.

Feedback network

Definition. A feedback (recurrent) network has connections that loop outputs back as inputs, so it holds a state.

Key points.

Because of the loops its output depends on past inputs, which gives memory.
The Hopfield network is a fully connected feedback network that stores patterns and recalls them from noisy input.
RNNs use feedback to process sequences such as text.

Last-minute revision

CI means nature-inspired adaptive computing that handles imprecision (soft computing).
Components: neural networks, fuzzy systems, evolutionary computation, swarm intelligence, rough sets.
Learning model: adjust parameters to reduce error, then generalise.
Update rule: $w \leftarrow w - \eta\,\partial E/\partial w$.
Supervised, unsupervised and reinforcement are the three learning types.
Parametric model: fixed form, finite parameters, for example $N(\mu,\sigma^2)$.
Nonparametric model: no fixed form, grows with data, for example k-NN.
Feed-forward is acyclic with no memory; feedback has cycles and memory.
Trial solution $y_t=A+x^2N(x,p)$ satisfies the boundary conditions.
Set approximation: lower (surely in), upper (possibly in).

Memory hooks

"NFERS": Neural, Fuzzy, Evolutionary, Rough, Swarm.
Parametric = "fixed pockets"; nonparametric = "growing pockets".
Feed-forward = one-way street; feedback = roundabout.
Train = predict, compare, correct, repeat.

Coverage checklist

Introduction to Computational Intelligence: no past question.
types of Computational Intelligence: covered in the Dec 2020 short note.
components of Computational Intelligence: Dec 2020, Nov 2023 (7 marks).
Concept of Learning/Training model: Dec 2020 short note (14 marks).
Parametric Models: Nov 2023 (7 marks).
Nonparametric Models: no past question.
Multilayer Networks: Feed Forward network: Dec 2020, Nov 2023 (7 marks each).
Feedback network: covered with the Dec 2020 question.

Account

Computational Intelligence (CS-702 (A)) - Unit 1 Short Notes

How unit 1 is examined

Introduction to Computational Intelligence

Types of Computational Intelligence

Components of Computational Intelligence

Concept of Learning/Training model

Parametric Models

Nonparametric Models

Multilayer Networks: Feed Forward network

Feedback network

Last-minute revision

Memory hooks

Coverage checklist