DATA SCIENCE (AD-404) Unit 3 Notes - RGPV

How unit 3 is examined

This unit covers statistical inference (hypothesis testing, estimation, confidence intervals), regression and regularization, Bayesian statistics, and Excel analysis tools. Marks come from VLOOKUP/XLOOKUP (9), correlation and regression (7-8), SUMIFS/SUMPRODUCT, logistic regression, and the low-weight theory topics (lasso, L1/L2, Bayesian, multiple testing).

Multiple hypothesis testing

Low weight

Definition. Multiple hypothesis testing means testing many hypotheses at once, which inflates the chance of at least one false positive, so the significance level must be corrected.

Key points.

With $m$ independent tests at level $\alpha$, the probability of at least one false positive is $1-(1-\alpha)^m$; for 20 tests at 0.05 it is about 64%.
Bonferroni correction controls the family-wise error rate by testing each hypothesis at $\alpha/m$; for $m=10$ and $\alpha=0.05$ the cut-off is $0.005$. It is simple but conservative.
The Benjamini-Hochberg method controls the false discovery rate (FDR): sort p-values $p_{(1)}\le\dots\le p_{(m)}$ and reject all up to the largest $k$ with $p_{(k)}\le \frac{k}{m}\alpha$. It is more powerful than Bonferroni.
Use Bonferroni when any false positive is costly; use FDR when screening thousands of items (genes, A/B variants).

Parameter estimation (asked with it). MLE picks the parameter that maximises the likelihood of the observed data; MAP maximises likelihood times a prior, so it adds prior belief (see next topic).

Asked: [7 marks] (Jun 2026) Explain Multiple Hypothesis Testing and Parameter Estimation Methods.

Parameter Estimation methods

Definition. Parameter estimation finds the value of a population parameter (mean, variance, weights) from sample data.

Key points.

Method of moments equates sample moments (mean, variance) to population moments and solves for the parameters.
Maximum likelihood estimation chooses $\hat\theta=\arg\max_\theta L(\theta)=\arg\max_\theta\sum_i\log p(x_i\mid\theta)$; for a normal sample, the MLE of the mean is the sample mean.
MAP estimation maximises $p(x\mid\theta)\,p(\theta)$, so it equals MLE plus a prior; a Gaussian prior on weights gives L2 regularization.
A point estimate gives one value; an interval estimate (confidence interval) gives a range.

Confidence intervals

Definition. A confidence interval is a range computed from a sample that would contain the true parameter in a stated percentage (confidence level) of repeated samples.

Formula. For a mean with known $\sigma$: $\bar x \pm z\,\dfrac{\sigma}{\sqrt n}$, with $z=1.96$ for 95% and $2.58$ for 99%. Margin of error $=z\,\sigma/\sqrt n$.

Key points.

A 95% level means 95% of such intervals would capture the true value, not that the parameter is 95% likely to lie in this one interval.
A larger sample $n$ or a smaller $\sigma$ narrows the interval; a higher confidence level widens it.
For small samples with unknown $\sigma$, use the $t$ value instead of $z$.

Example. $\bar x=50,\ \sigma=10,\ n=100$: margin $=1.96\times1=1.96$, so the 95% CI is $(48.04,\ 51.96)$.

Correlation & Regression analysis

Definition. Correlation measures the strength and direction of the linear relationship between two variables using a coefficient $r$ from $-1$ to $+1$; regression fits an equation that predicts a dependent variable from one or more independent variables.

Formula. Pearson correlation and simple linear regression:

$$r=\frac{\sum (x-\bar x)(y-\bar y)}{\sqrt{\sum (x-\bar x)^2\sum (y-\bar y)^2}},\qquad y=\beta_0+\beta_1x+\varepsilon,\quad \beta_1=\frac{\sum (x-\bar x)(y-\bar y)}{\sum (x-\bar x)^2}$$

Key points (correlation).

Positive correlation ($r>0$) means both variables rise together, negative ($r<0$) means one rises as the other falls, and zero ($r\approx0$) means no linear relationship.
$|r|$ near 1 is strong, about 0.5 is moderate, and below 0.3 is weak; $r^2$ is the share of variance explained.
Pearson measures linear association on numeric data; Spearman uses ranks, so it handles monotonic non-linear data and outliers.
A scatter plot shows the pattern: points rising along a line give $r$ near $+1$, a shapeless cloud gives $r$ near 0.
Correlation is not causation: two variables can move together because of a third factor.

Key points (regression role).

Regression predicts a numeric target (sales, price) from features, which is the core of predictive modelling in data science.
It quantifies relationships: each coefficient states the change in $y$ for a one-unit change in that feature, holding others fixed.
Types are simple linear (one feature), multiple linear (many features), polynomial, and logistic (for classes).
Applications include demand forecasting, house-price prediction, risk scoring and finding which factors matter.

Example. $x=1..5,\ y=2,4,5,4,5$: $\bar x=3,\bar y=4$, $\sum(x-\bar x)(y-\bar y)=6$, $\sum(x-\bar x)^2=10$, $\sum(y-\bar y)^2=6$. So $r=6/\sqrt{60}=$ 0.775 (strong positive), and $\beta_1=0.6$, $\beta_0=4-0.6\times3=2.2$, giving $\hat y=2.2+0.6x$.

Answer frame. Open with the definition of correlation (or regression); draw a scatter plot with a trend line; develop types and $r$ values, then the formula and interpretation; for the regression question, list types then applications; close by noting correlation does not imply causation.

Pitfall: Do not claim causation from a high $r$.

Asked: [8 marks] (Jun 2024) What is Correlation analysis? Explain. Asked: [7 marks] (Jun 2024) Explain the role of Regression Analysis in Data Science.

Logistic regression

Definition. Logistic regression is a classification algorithm that models the probability of a binary outcome by passing a linear combination of features through the sigmoid function.

Formula.

$$z=\beta_0+\beta_1x_1+\dots+\beta_nx_n,\qquad P(y=1\mid x)=\sigma(z)=\frac{1}{1+e^{-z}}$$

Cost (log loss) minimised over $m$ samples:

$$J(\beta)=-\frac1m\sum_{i}\big[y_i\log \hat p_i+(1-y_i)\log(1-\hat p_i)\big]$$

Key points.

Despite the name it is a classifier: the output is a probability between 0 and 1, never outside it.
The sigmoid is S-shaped with $\sigma(0)=0.5$; $\sigma(2)=0.88$ and $\sigma(-1)=0.27$.
A threshold (usually 0.5) turns the probability into a class: predict 1 if $\hat p\ge0.5$; the threshold can be moved to trade precision against recall.
Parameters are estimated by maximum likelihood, equivalent to minimising log loss, using gradient descent since there is no closed form.
The log-odds $\ln\frac{p}{1-p}=z$ is linear in the features, so $e^{\beta_j}$ is the odds ratio for feature $j$.
Evaluate with confusion matrix, accuracy, precision, recall, F1 and ROC-AUC.

Example (loan approval). Features are income, credit score and existing debt; label is 1 if approved. After training, an applicant with $z=2$ gets $\hat p=0.88$, so the loan is approved; one with $z=-1$ gets $0.27$ and is rejected.

Answer frame. Open with the definition; draw the sigmoid curve (S-shape, 0.5 at $z=0$, threshold line); develop points 1-6 in order with the formulas; close with loan-approval or spam-detection applications.

Asked: [7 marks] (Dec 2024, Jun 2025) Explain the concept of logistic regression. Explain logistic regression with an example of binary classification, such as predicting loan approval (Yes/No) using applicant features.

Shrinkage Methods

Definition. Shrinkage methods fit a regression while adding a penalty that pulls coefficients towards zero, trading a little bias for a large drop in variance.

Key points.

They reduce overfitting, especially with many or correlated features.
Ridge shrinks coefficients smoothly; lasso can shrink some exactly to zero.
The tuning parameter $\lambda$ controls the amount of shrinkage and is chosen by cross-validation.

Lasso Regression

Definition. Lasso (Least Absolute Shrinkage and Selection Operator) regression is linear regression with an L1 penalty, the sum of absolute coefficient values, which shrinks some coefficients exactly to zero.

Formula. $\displaystyle \min_\beta\ \sum_i (y_i-\hat y_i)^2+\lambda\sum_j|\beta_j|$

Key points.

The penalty term $\lambda\sum|\beta_j|$ punishes large coefficients; a larger $\lambda$ gives stronger shrinkage, and $\lambda=0$ gives ordinary least squares.
The diamond-shaped L1 constraint region touches the loss contours at corners, so coefficients become exactly zero, giving automatic feature selection.
The resulting model is sparse and easy to interpret.
Ridge (L2) uses $\lambda\sum\beta_j^2$, shrinks but never zeroes coefficients, and suits correlated features; lasso tends to keep one of a correlated group.

Asked: [7 marks] (Dec 2024) Discuss in detail about lasso regression.

Bayesian statistics

Definition. Bayesian statistics treats a parameter as a random variable with a probability distribution and updates its prior belief with observed data to get a posterior, using Bayes' theorem.

Formula. $P(\theta\mid D)=\dfrac{P(D\mid\theta)\,P(\theta)}{P(D)}$, i.e. posterior $\propto$ likelihood $\times$ prior.

Key points.

Prior $P(\theta)$ is belief before seeing data, likelihood $P(D\mid\theta)$ is how probable the data is for each $\theta$, and posterior is the updated belief.
Classical (frequentist) statistics treats the parameter as fixed but unknown and uses only sample data, with probability meaning long-run frequency.

Basis	Bayesian	Classical
Parameter	Random variable	Fixed constant
Prior knowledge	Used	Not used
Result	Posterior distribution, credible interval	Point estimate, p-value, confidence interval
Probability means	Degree of belief	Long-run frequency
Example	Disease 1% prevalent, test 90% sensitive, 5% false positive: posterior $=0.009/0.0585=0.154$	Test whether a coin is fair from 100 tosses using a p-value

Advantages are use of prior knowledge, a direct probability for a hypothesis and good performance on small data; applications are spam filtering, medical diagnosis and A/B testing.

Asked: [7 marks] (Jun 2026) Explain Bayesian Statistics. Compare Bayesian Statistics with Classical Statistics using examples.

L1 and L2 regularizations

Definition. Regularization adds a penalty on coefficient size to the loss function to prevent overfitting; L1 penalises $\sum|w|$ and L2 penalises $\sum w^2$.

Key points.

An overfitted model has large weights that fit noise; the penalty keeps weights small so the model generalises.
L1 (Lasso) loss is $L+\lambda\sum_j|w_j|$; it gives sparse weights and performs feature selection.
L2 (Ridge) loss is $L+\lambda\sum_j w_j^2$; it shrinks all weights smoothly but keeps them non-zero.

Point	L1 (Lasso)	L2 (Ridge)
Penalty	$\lambda\sum\lvert w\rvert$	$\lambda\sum w^2$
Sparsity	Yes, exact zeros	No
Feature selection	Built in	None
Use	Many irrelevant features	Correlated features

Applications: L1 in text classification with thousands of words, L2 in house-price regression and neural network weight decay.

Asked: [7 marks] (Jun 2026) What are L1 and L2 Regularization? Explain in detail.

POWERFUL DATA ANALYSIS—SUMIFS

Definition. SUMIFS adds the values in a sum range that satisfy all of several criteria at once (AND logic).

Syntax. =SUMIFS(sum_range, criteria_range1, criterion1, [criteria_range2, criterion2], ...)

Key points.

The sum range comes first, then each criteria range paired with its criterion; all ranges must be the same size.
Criteria can be text, numbers or operators such as ">5000", and wildcards like "N*".
Example: =SUMIFS(D2:D100, B2:B100, "North", C2:C100, ">=1000") totals sales in the North region with orders of at least 1000.
SUMPRODUCT multiplies matching elements of arrays and adds the results: =SUMPRODUCT(B2:B10, C2:C10) gives total revenue as price times quantity; with conditions, =SUMPRODUCT((A2:A10="North")*(B2:B10)*(C2:C10)).
Use SUMIFS for simple conditional totals and SUMPRODUCT for weighted sums or OR/array conditions.
Customer segmentation combines tools: UNIQUE lists the segments, SUMIFS totals spend per segment, INDEX+MATCH fetches each customer's attributes, and FILTER/SORT list the top segment's customers.

Answer frame. Open with the definition and syntax of each function; give one formula each; compare use cases; for segmentation, walk through UNIQUE, SUMIFS, INDEX+MATCH, FILTER in that order; close with the benefit of automated reports.

Asked: [8 marks] (Dec 2024) Explain the following data analysis tools. i) SUMIFS ii) SUMPRODUCT Asked: [7 marks] (Jun 2025) Explain how SUMIFS, INDEX + MATCH and Dynamic Array Formulas can be used for customer segmentation in Excel.

SUMPRODUCT

Definition. SUMPRODUCT multiplies corresponding entries of one or more arrays and returns the sum of the products.

Key points.

Syntax: =SUMPRODUCT(array1, array2, ...); arrays must have equal size.
Conditions work as 1/0 arrays: =SUMPRODUCT((A2:A10="North")*(B2:B10)).
It gives weighted averages, e.g. =SUMPRODUCT(marks, weights)/SUM(weights), without helper columns.

VLOOKUP | XLOOKUP

Definition. VLOOKUP searches for a value in the first column of a table and returns a value from a column to its right; XLOOKUP is its modern replacement that searches any column and returns from any column.

Syntax.

=VLOOKUP(lookup_value, table_array, col_index_num, [range_lookup])
=XLOOKUP(lookup_value, lookup_array, return_array, [if_not_found], [match_mode], [search_mode])

Key points.

VLOOKUP looks only in the leftmost column and returns from the column number counted from it; FALSE means exact match, TRUE (the default) means approximate match on sorted data.
Its column index breaks when columns are inserted, and it cannot look left.
XLOOKUP takes separate lookup and return ranges, so it looks in any direction and survives column insertion.
XLOOKUP defaults to exact match, has a built-in if_not_found argument instead of wrapping in IFERROR, and can search last-to-first.
Both are used to merge tables in analysis, for example fetching a customer's city or a product's price by ID before summarising.

Point	VLOOKUP	XLOOKUP
Direction	Right only	Left or right
Default match	Approximate	Exact
Column reference	Index number	Return range
Not found	`#N/A` (needs IFERROR)	`if_not_found` argument
Availability	All versions	Excel 365 / 2021

Example. =XLOOKUP(E2, A2:A100, C2:C100, "Not found") returns the price in column C for the product ID in E2; the VLOOKUP form is =VLOOKUP(E2, A2:C100, 3, FALSE).

Answer frame. Open with a definition of lookup functions; write both syntaxes; develop points 1-5; give the comparison table; close with a data-analysis use, then state that XLOOKUP is preferred.

Asked: [9 marks] (Jun 2023, Jun 2024) Discuss the usage of VLOOKUP and XLOOKUP operations used for data analysis. Explain the operations of VLOOKUP and XLOOKUP for data analysis.

INDEX + MATCH

Definition. INDEX + MATCH is a two-function lookup: MATCH finds the position of a value in a range and INDEX returns the item at that position.

Key points.

Syntax: =INDEX(return_range, MATCH(lookup_value, lookup_range, 0)), where 0 means exact match.
It can look left of the key column, and inserting columns does not break it.
It works in all Excel versions, so it is the standard alternative to VLOOKUP before XLOOKUP.

Handling Formula Errors

Definition. Formula errors are Excel messages that show a formula cannot compute; they are handled with IFERROR or IFNA.

Key points.

Common errors: #DIV/0! (division by zero), #N/A (value not found), #VALUE! (wrong data type), #REF! (deleted reference), #NAME? (unknown name).
=IFERROR(A2/B2, 0) returns 0 whenever the formula errors; IFNA catches only #N/A.
Fix the cause rather than hide it, since IFERROR can mask real mistakes.

Dynamic Array Formulas

Definition. A dynamic array formula returns multiple values that automatically spill into neighbouring cells.

Key points.

Functions include FILTER, SORT, UNIQUE, SEQUENCE and XLOOKUP; the result range is referred to as A2#.
=FILTER(A2:C100, C2:C100>1000) lists only rows above 1000 and updates automatically.
A #SPILL! error appears when cells in the spill range are not empty.

Circular References

Definition. A circular reference occurs when a formula refers, directly or through other cells, back to its own cell.

Key points.

Example: typing =A1+B1 in cell A1; Excel warns and shows 0.
Find it under Formulas, Error Checking, Circular References.
Iterative calculation (File, Options, Formulas) allows deliberate circularity, e.g. interest on a balance, with a set iteration count.

Formula Auditing

Definition. Formula auditing is Excel's toolset to trace, check and debug formulas.

Key points.

Trace Precedents shows cells that feed the selected formula; Trace Dependents shows cells that use it.
Show Formulas (Ctrl + `) displays every formula instead of results.
Error Checking and Evaluate Formula step through a calculation to find where it goes wrong.

Pivoting

Definition. Pivoting reorganises and summarises data by rotating rows into columns, so records can be grouped and aggregated (sum, count, average) along chosen dimensions.

Key points.

A pivot table is built from a dataset with fields placed in Rows, Columns, Values and Filters areas, and it refreshes when the source changes.
Types: simple pivot with one row field and one value; multi-dimensional pivot with rows and columns fields (e.g. region by product); pivot with calculated fields and groupings (dates by month); and long-to-wide pivoting (reshape) versus wide-to-long (melt, unpivot).
Use case: total sales by region and month without writing formulas. In pandas, df.pivot_table(index="Region", columns="Month", values="Sales", aggfunc="sum").
A pivot chart shows the result graphically.

Asked: [5 marks] (Jun 2023) What is pivoting? Explain the types of pivoting.

Last-minute revision

Multiple testing: Bonferroni uses $\alpha/m$; Benjamini-Hochberg controls FDR.
MLE maximises likelihood; MAP maximises likelihood times prior.
95% CI $=\bar x\pm1.96\,\sigma/\sqrt n$.
Pearson $r$ lies in $[-1,1]$; $r^2$ is the variance explained; Spearman uses ranks.
Regression line $y=\beta_0+\beta_1x$; example data give $r=0.775$, $\hat y=2.2+0.6x$.
Logistic regression: $\sigma(z)=1/(1+e^{-z})$, threshold 0.5, log-loss cost, trained by maximum likelihood.
Lasso is L1 ($\lambda\sum|\beta|$, zeros coefficients); ridge is L2 ($\lambda\sum\beta^2$, shrinks only).
Bayes: posterior $\propto$ likelihood $\times$ prior.
SUMIFS(sum_range, criteria_range, criterion, ...); SUMPRODUCT multiplies arrays then sums.
VLOOKUP looks right only, default approximate; XLOOKUP looks both ways, default exact, has if_not_found.
INDEX(return, MATCH(value, lookup, 0)); IFERROR(formula, value).

Memory hooks

Lasso = Loses features (zeros); Ridge = Reduces size only.
"V for Vertical, Very limited": VLOOKUP goes right only; "X for anywhere".
Bonferroni = divide alpha by the number of tests.
Sigmoid: S-curve, 0.5 at zero.
Pivot = rotate and summarise.

Coverage checklist

Multiple hypothesis testing: Jun 2026 (7 marks) covered.
Parameter Estimation methods: covered (asked with Jun 2026 question).
Confidence intervals: covered, unasked.
Correlation & Regression analysis: Jun 2024 (8 and 7 marks) covered.
logistic regression: Dec 2024, Jun 2025 covered.
Shrinkage Methods: covered, unasked.
Lasso Regression: Dec 2024 covered.
Bayesian statistics: Jun 2026 covered.
L1 and L2 regularizations: Jun 2026 covered.
POWERFUL DATA ANALYSIS—SUMIFS: Dec 2024, Jun 2025 covered.
SUMPRODUCT: covered, with Dec 2024 question.
VLOOKUP | XLOOKUP: Jun 2023, Jun 2024 covered.
INDEX + MATCH: covered, with Jun 2025 question.
Handling Formula Errors: covered, unasked.
Dynamic Array Formulas: covered, with Jun 2025 question.
Circular References: covered, unasked.
Formula Auditing: covered, unasked.
Pivoting: Jun 2023 covered.

Account

DATA SCIENCE (AD-404) - Unit 3 Short Notes

How unit 3 is examined

Multiple hypothesis testing

Parameter Estimation methods

Confidence intervals

Correlation & Regression analysis

Logistic regression

Shrinkage Methods

Lasso Regression

Bayesian statistics

L1 and L2 regularizations

POWERFUL DATA ANALYSIS—SUMIFS

SUMPRODUCT

VLOOKUP | XLOOKUP

INDEX + MATCH

Handling Formula Errors

Dynamic Array Formulas

Circular References

Formula Auditing

Pivoting

Last-minute revision

Memory hooks

Coverage checklist