Data Science (IT-702 (A)) - Unit 5 Short Notes

1. Introduction to Data Science

• Definition: An interdisciplinary field that uses scientific methods, processes, algorithms, and systems to extract knowledge and insights from structured and unstructured data.

• Scope: Data acquisition, storage, processing, analysis, visualization, and deployment of data-driven solutions.

• Applications:

  • Healthcare: Predictive diagnostics, drug discovery.

  • Finance: Fraud detection, algorithmic trading.

  • Social Media: Sentiment analysis, recommendation systems.

• Role of Data Scientist: Combines skills in statistics, programming, and domain expertise to collect, clean, analyze, and interpret complex datasets.

• Data Science vs. Data Analysis:

  • Data Science: Broader scope; includes predictive modeling, machine learning, and big data technologies.

  • Data Analysis: Focuses on interpreting historical data to provide actionable insights; often a subset of data science.

[!TIP]

Exam Focus: Be ready to define Data Science and list applications across 3-4 domains with specific examples.

2. Data Types and Characteristics

2.1 Structured Data

• Definition: Data organized in a fixed schema (rows and columns), typically stored in relational databases or spreadsheets.

• Examples: SQL tables, Excel sheets, CSV files.

• Characteristics:

  • Predefined data model (schema-on-write).

  • Easy to query and analyze.

  • Less flexible for schema changes.

2.2 Unstructured Data

• Definition: Data without a predefined format or organization.

• Examples: Text documents, images, audio files, videos, social media posts.

• Importance in Modern Applications:

  • Healthcare: Medical imaging (X-rays, MRIs) for diagnosis.

  • Finance: Transaction logs, call center audio for fraud detection.

  • Social Media: Posts, tweets, images for sentiment analysis.

• Challenges: Requires advanced techniques (NLP, computer vision) for processing.

2.3 Semi-structured Data

• Definition: Data with some organizational properties (tags, markers) but not a rigid schema.

• Examples: JSON, XML, NoSQL databases (MongoDB).

• Comparison:

  | Feature | Structured | Semi-structured | Unstructured | |---------|------------|-----------------|--------------| | Schema | Fixed (schema-on-write) | Flexible (schema-on-read) | None | | Storage | RDBMS | NoSQL, files | Files, blobs | | Querying | SQL | Specialized query languages | Complex (NLP, CV) |

2.4 Big Data Fundamentals

• The 3 Vs:

  1. Volume: Size of data (TB, PB, EB).

  2. Variety: Different types (structured, unstructured, semi-structured).

  3. Velocity: Speed of data generation and processing (real-time streams).

• Additional Dimensions:

  • Veracity: Uncertainty, quality, trustworthiness of data.

  • Value: Extracting meaningful insights for decision-making.

• Challenges: Storage, processing, analysis, scalability, privacy.

[!TIP]

Exam Focus: Distinguish structured vs. unstructured with domain-specific examples. Explain all 5 Vs of Big Data.

3. Data Visualization

3.1 Traditional Charts and Graphs

• Pie Chart: Shows parts of a whole; use for categorical data with few categories (<6).

• Bar Graph: Compares categories; vertical/horizontal, grouped, stacked.

• Histogram: Shows distribution of continuous data; bins represent ranges.

• Box Plot: Displays five-number summary (min, Q1, median, Q3, max); identifies outliers.

• Scatter Plot: Reveals relationship between two continuous variables (correlation).

3.2 Advanced Visualization Techniques

• t-SNE (t-Distributed Stochastic Neighbor Embedding):

  • Reduces high-dimensional data to 2D/3D for visualization.

  • Preserves local structure; ideal for visualizing clusters in unstructured data (e.g., word embeddings from text).

• Rug Plot: Marginal tick marks along an axis showing individual data points; often added to density plots.

3.3 Python Visualization Libraries

• Matplotlib: Basic plotting (pie, bar, histogram, scatter). Low-level, highly customizable.

• Seaborn: Statistical visualizations; built on Matplotlib; default themes, easier syntax for complex plots.

[!TIP]

Exam Focus: Know when to use each traditional chart. Explain t-SNE with an example (e.g., visualizing text clusters). Contrast Matplotlib vs. Seaborn.

4. Data Manipulation and Analysis Tools

4.1 Microsoft Excel

4.1.1 Data Validation

• Techniques:

  • Dropdown lists (from range/table).

  • Whole number/decimal/date restrictions.

  • Custom formulas (e.g., =AND(A1>0, A1<100)).

• Role: Ensures data accuracy, reduces entry errors, maintains consistency.

4.1.2 Lookup and Reference Functions

4.1.3 Conditional Functions

• SUMIFS: Sums cells meeting multiple criteria.

  • Syntax: =SUMIFS(sum_range, criteria_range1, criteria1, ...)

  • Customer Segmentation Example: Sum sales by region (criteria_range1=Region, criteria1="North") and product (criteria_range2=Product, criteria2="Laptop").

4.1.4 Pivot Tables and Scenario Manager

• Pivot Tables:

  • Drag fields to Rows, Columns, Values, Filters.

  • Summarize, aggregate, filter large datasets interactively.

• Scenario Manager:

  • Define scenarios (sets of input values) for "what-if" analysis.

  • Generates summary reports comparing scenarios.

4.1.5 Macros

• Definition: Recorded sequences of commands/actions to automate repetitive tasks.

• Steps to Create:

  1. Enable Developer tab.

  2. Record Macro → perform actions → Stop Recording.

  3. Assign to button/shortcut.

• Use Cases: Formatting reports, data cleaning steps, generating standard charts.

4.2 Python for Data Science

4.2.1 Pandas

• DataFrame: 2D labeled data structure with columns of potentially different types (like a spreadsheet/SQL table).

• Features:

  • Indexing (.loc, .iloc).

  • Handling missing data (.dropna(), .fillna()).

  • Merging/joining (.merge(), .concat()).

  • Grouping (.groupby()).

• vs. NumPy:

  | Aspect | NumPy | Pandas | |--------|-------|--------| | Data Type | Homogeneous numerical arrays | Heterogeneous tabular data | | Indexing | Integer-based | Label-based (rows/columns) | | Use Case | Mathematical operations | Data manipulation, analysis |

4.2.2 String Operations and Regular Expressions

• String Methods (via .str accessor):

  • .str.split(): Split strings.

  • .str.replace(): Replace patterns.

  • .str.extract(): Extract using regex.

• Regular Expressions: Pattern matching (e.g., extract emails: r'[\w\.-]+@[\w\.-]+').

4.2.3 Integration with Visualization

• df.plot() uses Matplotlib.

• Seaborn works directly with DataFrames (e.g., sns.scatterplot(data=df, x='col1', y='col2')).

4.3 R Programming (for Big Data Analytics)

• Features: Vectorized operations, extensive statistical packages, powerful graphics.

• Major Components:

  • R console (interactive).

  • Workspace (saved objects).

  • Packages (collections of functions).

  • Graphics device (plots).

• Vector Operations:

  • Indexing: x[1:5], x[x>10].

  • Arithmetic: x + y, log(x).

  • Functions: sum(x), mean(x).

• factor(): Encodes vector as categorical variable (e.g., factor(c("low","medium","high"))).

[!TIP]

Exam Focus: Contrast VLOOKUP vs XLOOKUP vs INDEX+MATCH. Explain Pandas DataFrame structure and its difference from NumPy. Write regex pattern for email extraction.

5. Statistical Analysis

5.1 Descriptive Statistics

• Measures of Central Tendency:

  • Mean: $$\displaystyle \bar{x} = \frac{\sum x_i}{n} $$; sensitive to outliers.

  • Median: Middle value; robust to outliers.

  • Mode: Most frequent value.

• Measures of Variability:

  • Range: $max - min$.

  • Variance: $$\displaystyle s^2 = \frac{\sum (x_i - \bar{x})^2}{n-1} $$.

  • Standard Deviation: $$\displaystyle s = \sqrt{s^2} $$.

  • IQR: $Q3 - Q1$ (robust to outliers).

5.2 Inferential Statistics

• Correlation Analysis:

  • Pearson correlation coefficient: $$\displaystyle r = \frac{\sum (x_i - \bar{x})(y_i - \bar{y})}{\sqrt{\sum (x_i - \bar{x})^2 \sum (y_i - \bar{y})^2}} $$.

  • Interpretation: $r \in [-1,1]$; magnitude = strength, sign = direction.

• Conditional Probability: $$\displaystyle P(A|B) = \frac{P(A \cap B)}{P(B)} $$; probability of A given B occurred.

• Hypothesis Testing Errors:

  • Type I Error: Reject true $$\displaystyle H_0 $$ (false positive); significance level $\alpha$.

  • Type II Error: Fail to reject false $$\displaystyle H_0 $$ (false negative); probability $\beta$.

  • Example (Medical Trial): Type I = approving ineffective drug; Type II = rejecting effective drug.

• Maximum Likelihood Estimation (MLE):

  • Concept: Find parameters $\theta$ that maximize likelihood $$\displaystyle L(\theta | data) = P(data | \theta) $$.

  • Application: Used in logistic regression to estimate coefficients by maximizing log-likelihood.

[!TIP]

Exam Focus: Calculate mean, median, mode for small datasets. Differentiate Type I vs Type II errors with a concrete example. State MLE’s objective function.

6. Machine Learning Algorithms

6.1 Supervised Learning

6.1.1 Regression

• Linear Regression:

  • Model: $$\displaystyle y = \beta_0 + \beta_1 x + \varepsilon $$.

  • Assumptions: Linearity, independence, homoscedasticity, normality of residuals.

  • Use Case: Predicting house prices, sales forecasting.

• Logistic Regression:

  • Binary classification; predicts probability via sigmoid: $$\displaystyle p = \frac{1}{1 + e^{-(\beta_0 + \beta_1 x)}} $$.

  • Example: Loan approval (Yes/No) based on income, credit score.

  • Coefficient Interpretation: Odds ratio $$\displaystyle e^{\beta} $$; change in odds per unit change in predictor.

6.1.2 Classification

• Decision Trees:

  • Structure: Root → internal nodes (splits) → leaves (class labels).

  • Splitting Criteria:

    • Gini Impurity: $$\displaystyle G = 1 - \sum p_i^2 $$; lower is better.

    • Information Gain (Entropy): $$\displaystyle IG = H(parent) - \sum \frac{n_{child}}{n_{parent}} H(child) $$; $$\displaystyle H = -\sum p_i \log_2 p_i $$.

  • Classification: CART (binary splits), C4.5 (multi-way, entropy).

  • Example: Classify loan approval based on income, employment length.

• Random Forest:

  • Ensemble of decision trees via bagging (bootstrap aggregating).

  • Feature Importance: Mean decrease in impurity/accuracy.

  • Out-of-Bag (OOB) Error: Internal cross-validation using samples not in bootstrap.

  • Example: Credit scoring with many financial features.

• Naive Bayes Classifier:

  • Based on Bayes' Theorem: $P(y|x) \propto P(x|y)P(y)$.

  • Assumption: Features conditionally independent given class.

  • Types:

    • Gaussian: Continuous features (assumes normal distribution).

    • Multinomial: Discrete counts (e.g., word frequencies in text).

    • Bernoulli: Binary features.

  • Example: Spam detection (words as features).

6.2 Unsupervised Learning

6.2.1 Clustering

• K-means Clustering:

  • Algorithm:

    1. Initialize $k$ centroids randomly.

    2. Assign each point to nearest centroid.

    3. Update centroids as mean of assigned points.

    4. Repeat 2–3 until convergence.

  • Choosing $k$: Elbow method (plot inertia vs. $k$; look for "elbow").

  • Example: Cluster customers into $$\displaystyle k=2 $$ groups (high/low spenders).

6.2.2 Association Rule Mining

• Apriori Algorithm:

  • Key Concepts:

    • Support: $P(\text{itemset})$; frequency in dataset.

    • Confidence: $$\displaystyle P(B|A) = \frac{\text{support}(A \cup B)}{\text{support}(A)} $$.

    • Lift: $$\displaystyle \frac{\text{confidence}(A \rightarrow B)}{\text{support}(B)} $$; >1 indicates positive correlation.

  • Steps:

    1. Generate frequent itemsets (minimum support).

    2. Derive rules with minimum confidence.

  • Advantages: Simple, easy to implement.

  • Applications: Market basket analysis (e.g., "customers who buy X also buy Y").

  • Interesting vs. Coincidental Rules: Use lift (>1) and conviction to filter non-random associations.

[!TIP]

Exam Focus: Write pseudo-code for k-means and Apriori. Explain Gini vs. entropy. Differentiate Naive Bayes types. Define support, confidence, lift with formulas.

7. Business Intelligence (BI)

7.1 Definition and Types

• Definition: Technologies, applications, and practices for collecting, integrating, analyzing, and presenting business information to support decision-making.

• Types:

  • Strategic BI: Long-term planning (e.g., market expansion).

  • Tactical BI: Medium-term decisions (e.g., quarterly sales strategy).

  • Operational BI: Day-to-day operations (e.g., real-time dashboards for call centers).

7.2 BI Tools and Comparison

7.3 Role in Strategic Decision-Making

• Process:

  1. Data Integration: Combine data from multiple sources.

  2. Analysis: OLAP, data mining, ad-hoc queries.

  3. Visualization: Dashboards, reports.

  4. Action: Insights drive strategy (e.g., adjust marketing based on sales dashboard).

• Example: A sales performance dashboard with regional trends guides resource allocation and promotional campaigns.

[!TIP]

Exam Focus: Compare Tableau vs Power BI on at least three criteria. Describe BI’s role in strategic decisions with a concrete example.

8. Data Wrangling and Analysis Approaches

8.1 Data Wrangling

• Process:

  1. Collection: Gather data from sources.

  2. Cleaning: Handle missing values (impute/remove), outliers (cap/remove).

  3. Transformation: Normalize, encode categorical variables.

  4. Integration: Merge datasets.

• Importance: "Garbage in, garbage out"; ensures quality for accurate models.

• Sampling Techniques:

  • Random Sampling: Every item equal chance.

  • Stratified Sampling: Preserve class proportions; useful for imbalanced data.

8.2 Data Analysis Methods

[!TIP]

Exam Focus: List steps in data wrangling. Differentiate descriptive, predictive, prescriptive with examples.

9. Ethics, Security, and Social Implications

9.1 Ethical Issues

• Unfair Discrimination:

  • Definition: Biased outcomes against protected groups (race, gender).

  • Example: Loan denial algorithm using zip code as proxy for race.

• Reinforcing Human Biases:

  • How: Biased training data → biased model predictions (e.g., hiring algorithm favoring male candidates from historical data).

  • Mitigation: Fairness-aware algorithms (reweighting, adversarial debiasing), diverse data, audits.

• Ethical Issues in BI:

  • Privacy violations (excessive tracking).

  • Misuse of data (e.g., selling customer data without consent).

  • Lack of transparency ("black box" models).

9.2 Security Issues

• Challenges: Data breaches, unauthorized access, insecure APIs.

• Best Practices:

  • Encryption (at rest, in transit).

  • Access controls (RBAC, MFA).

  • Regular security audits.

[!TIP]

Exam Focus: Explain unfair discrimination with an example. Describe how biases propagate in ML pipelines. List security best practices.

10. Big Data Technologies

10.1 Big Data Analytics Lifecycle

1. Data Acquisition: Collect from sensors, logs, APIs.

2. Data Storage: Distributed storage (HDFS, NoSQL).

3. Data Processing: MapReduce, Spark.

4. Data Analysis: ML, statistical models.

5. Data Visualization: BI tools, dashboards.

DiagramCANVAS: A flowchart with 5 boxes in sequence, each labeled as above, showing data flow from acquisition to visualization.

10.2 Hadoop Ecosystem

10.2.1 HDFS (Hadoop Distributed File System)

• Architecture:

  • NameNode (Master): Manages metadata (file names, permissions, block locations).

  • DataNodes (Slaves): Store actual data blocks (default 128 MB); handle read/write.

  • Secondary NameNode: Periodically merges edit logs with fsimage; not a backup for NameNode.

• Functions:

  • NameNode: Namespace management, block mapping.

  • DataNode: Block creation/deletion, replication.

  • Secondary NameNode: Checkpointing to prevent edit log overflow.

10.2.2 MapReduce

• Programming Model:

  • Map: Processes input key-value pairs → intermediate key-value pairs.

  • Reduce: Aggregates intermediate values by key.

• Job Scheduling (Fair Scheduler): Shares resources among multiple jobs; ensures fairness via resource pools.

• Word Count Example:

  
  Map: (line) → (word, 1) for each word in line.
  
  Reduce: (word, [1,1,...]) → (word, sum)
  
  

• Operations: Sorting, joining, filtering.

10.2.3 HBase

• Storage Mechanism: Column-oriented NoSQL database on top of HDFS.

• Architecture:

  • Region Servers: Serve read/write requests for regions (data ranges).

  • Master: Manages region assignment, metadata.

  • ZooKeeper: Coordination service (election, configuration).

• Use Cases: Real-time random read/write on big data (e.g., user profile store).

10.2.4 Apache Pig

• Architecture: Pig Latin script → compiler → MapReduce jobs.

• Pig Latin: High-level scripting for data transformation (LOAD, FILTER, GROUP, STORE).

• Application Flow:

  1. Load data (e.g., from HDFS).

  2. Transform (filter rows, group by key).

  3. Store results.

10.2.5 Apache Hive

• Data Warehouse on Hadoop: HiveQL (SQL-like queries) → MapReduce/Tez/Spark jobs.

• User-Defined Functions (UDFs): Custom functions in Java/Python for Hive queries.

  • Example: UDF to encrypt PII data: SELECT encrypt(ssn) FROM users;.

10.3 Analytic Sandbox

• Definition: Isolated environment (separate from production) for data experimentation, model prototyping, and validation.

• Importance: Safe testing without affecting live systems; fosters innovation.

• Use in Big Data Projects: Prototype ML models, test ETL pipelines, validate hypotheses.

[!TIP]

Exam Focus: Draw and explain HDFS architecture (NameNode, DataNode, Secondary NameNode). Write Word Count pseudo-code for MapReduce. Explain HBase storage mechanism. Describe Pig Latin flow.

11. Specialized and Emerging Topics

• Gradient Boosting:

  • Ensemble method using boosting; builds trees sequentially, each correcting errors of previous.

  • Example: XGBoost, LightGBM; used in competitions (Kaggle).

• Rug Plot: Marginal ticks along axis in density plot showing individual data points; helps visualize data distribution density.

• Data Scientist vs. Data Analyst:

  | Aspect | Data Scientist | Data Analyst | |--------|----------------|--------------| | Scope | Predictive modeling, ML, big data | Descriptive analysis, reporting | | Skills | Stats, ML, programming, domain | SQL, visualization, business acumen | | Tools | Python/R, Spark, ML libraries | Excel, BI tools, SQL |

• Role of Data Scientist:

  • Skills: Statistics, programming (Python/R), ML, data visualization, domain knowledge.

  • Tasks: Data collection/cleaning, exploratory analysis, model building, deployment, communication.

[!TIP]

Exam Focus: Explain gradient boosting briefly. Define rug plot. Contrast data scientist vs. analyst roles.

Final Note: This summary aligns strictly with the approved blueprint and past exam patterns. Focus on definitions, formulas, comparisons, and step-by-step algorithms (e.g., k-means, Apriori, MapReduce word count). Use diagrams where specified (HDFS, Big Data lifecycle). Practice calculations for descriptive stats and correlation.