Data Science (IT-702 (A)) - Unit 4 Short Notes

UNIT 4: ADVANCED DATA ANALYTICS, BIG DATA, AND BUSINESS INTELLIGENCE

I. BIG DATA FUNDAMENTALS

Characteristics of Big Data (5 Vs)

Data Types:

• Structured: Fixed schema (SQL databases, spreadsheets).

• Unstructured: No fixed schema (emails, social media, images).

• Semi-structured: Partial schema (JSON, XML, NoSQL).

Processing Considerations

• Scalability: Horizontal scaling (adding machines) vs. vertical (upgrading hardware).

• Distributed Systems: Partition data across clusters (e.g., HDFS).

• Parallel Processing: MapReduce, Spark for simultaneous computation.

• Challenges: Storage cost, latency, data integration, security.

Phases of Big Data Analytics

1. Data Ingestion: Collecting data from sources (APIs, logs, streams).

2. Storage: Distributed storage (HDFS, cloud object stores).

3. Processing: Cleaning, transforming (MapReduce, Spark).

4. Analysis: ML, statistical modeling, mining.

5. Visualization: Dashboards, reports (Tableau, Power BI).

6. Analytic Sandbox: Isolated environment for experimentation without affecting production.

!TIP: Analytic sandbox allows testing new models/scripts safely—key for iterative analysis.

II. HADOOP ECOSYSTEM

HDFS Architecture

• NameNode: Master server; manages file system metadata (directory tree, block locations). Single point of failure (mitigated by HA).

• DataNode: Slave nodes; store actual data blocks (default 128 MB). Report status to NameNode.

• Secondary NameNode: Checkpoints NameNode metadata (not backup). Merges fsimage and edits logs.

• Fault Tolerance: Data replicated (default 3x) across racks. Automatic re-replication on node failure.

!TIP: Secondary NameNode ≠ Backup NameNode. It only aids checkpointing.

MapReduce Programming Model

1. Map Phase: (key, value) → list(key, value)

   Example (WordCount): (docID, text) → (word, 1) for each word.

2. Shuffle & Sort: Groups values by key (framework handles).

3. Reduce Phase: (key, list(value)) → (key, aggregated)

   Example: (word, [1,1,...]) → (word, sum).

• Job Scheduling (Fair Scheduler): Allocates resources to multiple jobs/queues to ensure fairness, not just FIFO.

HBase Storage Mechanism

• Column-Family Store: NoSQL, column-oriented. Tables have column families (groups of columns).

• Regions: Tables split into regions (range of row keys). RegionServer manages regions.

• Scalability: Automatically shards/redistributes regions as data grows.

• Consistency: Strong consistency per row; eventual across rows.

Pig & Hive

• Pig Flow: Script → Parser → Logical Plan → Optimizer → Physical Plan → MapReduce.

• Hive UDF: Write Java class implementing UDF interface; add JAR; use in HiveQL.

III. DATA MINING AND ASSOCIATION ANALYSIS

Apriori Algorithm

• Goal: Find frequent itemsets (items co-occurring often).

• Key Metrics:

  • Support: P(itemset) = (transactions containing itemset) / (total transactions).

  • Confidence: P(B \| A) = support(A∪B) / support(A).

  • Lift: confidence(A→B) / support(B); >1 indicates positive correlation.

• Steps:

  1. Set min support threshold.

  2. Generate candidate 1-itemsets, prune by support.

  3. Iteratively generate k-itemsets from (k-1)-itemsets (join step), prune by support.

  4. Use frequent itemsets to generate rules meeting min confidence.

• Advantages: Simple, easy to implement.

• Limitations: Multiple DB scans, combinatorial explosion for low support.

Association Rules

• Interestingness: Lift > 1, high confidence, actionable, non-obvious.

• Coincidental Rules: High support but no causal link (e.g., "diapers → beer" may be spurious if both bought together only on weekends).

• Applications: Market basket analysis, cross-selling, recommendation systems.

Clustering (K-means)

• Algorithm:

  1. Choose k initial centroids (randomly or k-means++).

  2. Assign each point to nearest centroid (Euclidean distance).

  3. Recompute centroids as mean of assigned points.

  4. Repeat 2–3 until convergence (centroids stable).

• Elbow Method: Plot within-cluster sum of squares (WCSS) vs. k; choose k at "elbow" (point of diminishing returns).

• Use Cases: Customer segmentation, image compression, anomaly detection.

IV. MACHINE LEARNING FOR ADVANCED ANALYTICS

A. Supervised Learning

Logistic Regression

• Binary Classification: Predicts probability P(Y=1|X).

• Model: log(odds) = β₀ + β₁x₁ + ... + βₙxₙ, where odds = P/(1-P).

• Cost Function: Log Loss (cross-entropy):

$$J(\theta) = -\frac{1}{m} \sum_{i=1}^{m} \left[ y^{(i)} \log(h_\theta(x^{(i)})) + (1-y^{(i)}) \log(1-h_\theta(x^{(i)})) \right]$$

• Example: Loan approval → features: income, credit score, debt. Output: probability of default.

Decision Trees

• Structure: Root → internal nodes (splits) → leaf nodes (predictions).

• Splitting Criteria:

  • Gini Impurity: G = 1 - Σ(p_i²); minimize weighted Gini after split.

  • Entropy: H = -Σ(p_i log₂ p_i); maximize Information Gain = H(parent) - Σ(weighted H(children)).

• Pruning: Remove branches with low predictive power to prevent overfitting (cost-complexity pruning).

• Applications: Credit scoring, medical diagnosis.

Random Forest

• Ensemble (Bagging): Multiple decision trees trained on bootstrapped samples.

• Prediction: Majority vote (classification) or average (regression).

• Feature Importance: Mean decrease in impurity (MDI) or permutation importance.

• Advantages: Reduces overfitting, handles non-linearities, robust to outliers.

Gradient Boosting

• Boosting: Sequential trees; each corrects errors of previous.

• Process:

  1. Fit initial tree to residuals.

  2. Compute pseudo-residuals (negative gradient of loss).

  3. Fit new tree to residuals; add to ensemble with learning rate η.

• Loss Functions: Deviance (classification), squared error (regression).

• Example: XGBoost, LightGBM.

Naive Bayes

• Bayes' Theorem: P(Y|X) = [P(X|Y) P(Y)] / P(X).

• Independence Assumption: Features conditionally independent given Y.

  P(X|Y) = Π P(x_i|Y).

• Types:

  • Gaussian: Continuous features (assume normal distribution).

  • Multinomial: Discrete counts (e.g., text word frequencies).

  • Bernoulli: Binary features.

• Example: Spam detection → words as features; P(spam|words).

B. Unsupervised Learning

• K-means: (See above).

• Association Rules: (See above).

C. Dimensionality Reduction

t-SNE (t-Distributed Stochastic Neighbor Embedding)

• Goal: Visualize high-dim data in 2D/3D by preserving local similarities.

• How:

  1. Convert pairwise similarities to probabilities (high-dim: Gaussian; low-dim: t-distribution).

  2. Minimize KL divergence between probability distributions.

• Perplexity: Roughly equivalent to number of nearest neighbors; balances local/global structure.

• Example: Visualizing word embeddings (e.g., BERT) or image features.

!TIP: t-SNE is for visualization, not feature extraction. Results vary with perplexity; try multiple values.

V. BUSINESS INTELLIGENCE AND ANALYTICS

BI Concepts

• Definition: Technologies, processes, tools to transform raw data into meaningful information for strategic decisions.

• Evolution: From static reports → interactive dashboards → predictive/prescriptive analytics.

• Core Objectives: Improve decision speed/quality, identify opportunities, optimize operations.

Types of BI Analytics

BI Tools Comparison

BI for Strategic Decision-Making

• Dashboards: Real-time KPIs (e.g., revenue, customer satisfaction).

• KPIs: Quantifiable metrics aligned with goals (e.g., conversion rate, CLV).

• Data-Driven Culture: Decisions based on data, not intuition; requires training & accessible tools.

Data Analysis Methods

• Predictive: Uses ML/statistical models (e.g., regression, classification) to forecast.

• Prescriptive: Recommends actions using optimization, simulation (e.g., "Offer discount to X segment to maximize profit").

Excel for BI

• Advanced Functions:

  • SUMIFS(sum_range, criteria_range1, criteria1, ...) → Conditional sum.

  • INDEX(array, row_num, [col_num]) + MATCH(lookup_value, lookup_array, 0) → Flexible lookup (left lookup, two-way).

  • Dynamic Arrays: FILTER(array, include, [if_empty]), SORT(array, [sort_index], [sort_order]), UNIQUE(array).

• Pivot Tables: Summarize large data; drag-and-drop grouping, filtering, calculated fields.

• Scenario Manager: What-if analysis; save sets of input values (e.g., best/worst case).

• Data Validation: Restrict cell inputs (list, date range, custom formula) to prevent errors.

!TIP: XLOOKUP (newer) replaces VLOOKUP; handles vertical/horizontal, default exact match, no column index number.

VI. ETHICS, BIASES, AND SECURITY

Unfair Discrimination

• Definition: Treating groups differently based on protected attributes (race, gender) leading to adverse outcomes.

• Examples:

  • Lending: Loan denial for minority groups despite similar credit scores.

  • Hiring: Algorithm filtering out resumes with "women's" in name.

• Implications: Legal (e.g., Equal Credit Opportunity Act), social (perpetuating inequality), reputational.

Reinforcing Human Biases

• Sources:

  • Data Bias: Historical discrimination reflected in training data (e.g., policing data biased against minorities).

  • Algorithm Bias: Model amplifies patterns (e.g., word embeddings: "doctor" → male, "nurse" → female).

  • Feedback Loops: Biased predictions influence future data (e.g., predictive policing → more patrols in biased areas → more arrests → reinforces bias).

• Mitigation:

  • Pre-processing: Remove sensitive attributes, re-weight samples.

  • In-processing: Add fairness constraints (e.g., demographic parity).

  • Post-processing: Adjust thresholds per group.

  • Diverse teams, bias audits, transparency.

Ethical Issues in Data Science & BI

• Privacy: Invasive data collection, re-identification risks.

• Transparency: "Black box" models; lack of explainability.

• Accountability: Who is responsible for erroneous decisions?

• Fairness: Equitable outcomes across groups.

Security Issues

• Data Breaches: Unauthorized access to sensitive data (PII, financial).

• Access Control: Role-based permissions, least privilege.

• Encryption: At rest (AES) and in transit (TLS).

• Secure Handling: Anonymization, secure APIs, audit logs.

VII. ADVANCED STATISTICAL METHODS

Maximum Likelihood Estimation (MLE)

• Concept: Find parameters θ that maximize likelihood L(θ|data) = P(data|θ).

• Steps:

  1. Write likelihood function (product of PDFs for i.i.d. data).

  2. Take log (log-likelihood; easier to maximize).

  3. Differentiate w.r.t. θ, set to zero, solve.

• Example: For normal distribution, MLE of mean = sample mean, variance = sample variance (biased).

Hypothesis Testing Errors

• Trade-off: Decreasing α increases β (for fixed sample size).

• Multiple Testing: Increases family-wise error rate (FWER); use Bonferroni correction (α' = α/k for k tests).

!TIP: In medical testing: Type I = false alarm (unnecessary treatment), Type II = missed disease (no treatment).

\boxed{\text{End of Unit 4 Notes}}
Aligned with RGPV IT-702(A) past papers (2023–2025). Focus on definitions, algorithms, comparisons, and examples for 7-mark questions.