Data Science (IT-702 (A)) - Unit 3 Short Notes

UNIT 3: Data Science - Short Notes

1. Fundamentals of Data

1.1 Data Types

[!TIP] Exam Focus: Be prepared to compare and contrast all three types with specific examples. A common pitfall is calling JSON/XML "structured"—they are semi-structured.

1.2 Importance and Applications of Unstructured Data

• Healthcare: Analysis of medical notes (NLP), medical imaging (MRI/X-ray via Computer Vision), genomic data.

• Finance: Sentiment analysis on news/financial reports, fraud detection from transaction logs, analyzing earnings call transcripts.

• Social Media: Sentiment analysis on posts/tweets, image/video content moderation, understanding user engagement through interaction logs.

2. Data Wrangling and Preprocessing

2.1 Data Wrangling

Definition: The process of cleaning, structuring, and enriching raw data into a desired format for analysis. Importance: Often consumes 60-80% of a data scientist's time. Garbage in, garbage out—quality data is foundational for accurate models.

2.2 Data Cleaning

• Missing Values: Identify (isna()), handle via deletion (if few) or imputation (mean/median/mode, predictive models).

• Outliers: Detect using IQR method or Z-score. Decide to cap, remove, or transform based on domain knowledge.

• Inconsistencies: Fix formatting (dates, units), correct typos, resolve duplicates.

2.3 Data Transformation

• Normalization (Min-Max): Scales features to [0,1]. Sensitive to outliers.

$$X_{norm} = \frac{X - X_{min}}{X_{max} - X_{min}}$$

• Standardization (Z-score): Transforms to mean=0, SD=1. Robust to outliers.

$$X_{std} = \frac{X - \mu}{\sigma}$$

• Encoding Categorical Variables: Label Encoding (ordinal), One-Hot Encoding (nominal).

2.4 Data Integration

Merging datasets from multiple sources using keys (e.g., pd.merge() in Pandas, SQL JOINs). Handle schema conflicts.

2.5 Data Reduction

• Sampling: Simple random, stratified (preserves class distribution).

• Dimensionality Reduction: t-SNE (t-Distributed Stochastic Neighbor Embedding) is primarily for visualization of high-dimensional data. It reduces to 2D/3D while preserving local structure. Not for feature engineering in pipelines due to computational cost and non-determinism.

3. Data Visualization

3.1 Principles of Effective Visualization

• Clarity & Simplicity: Avoid chartjunk.

• Accuracy: Represent data truthfully.

• Efficiency: Communicate message quickly.

• Appropriate Chart Type: Match data and question (comparison, distribution, relationship, composition).

3.2 Fundamental Graphical Representations

3.3 Advanced Visualization Techniques

• t-SNE: Algorithm Steps: 1) Compute pairwise similarities in high-D space. 2) Map to low-D space (2D/3D) minimizing KL-divergence between distributions. Interpretation: Clusters indicate similar data points. Example: Visualizing word embeddings (e.g., Word2Vec) or image feature vectors.

• Density Plot: Smoothed version of histogram (KDE). Shows probability density.

• Rug Plot: Small ticks along axis showing actual data point locations. Often added to density/histogram to show data sparsity.

3.4 Python Visualization Libraries

• Matplotlib: Foundation library. plt.plot(), plt.bar(), plt.hist(), plt.scatter(). Highly customizable (labels, titles, subplots via plt.subplot()).

• Seaborn: Built on Matplotlib. Statistical plots with better aesthetics. sns.histplot(), sns.boxplot(), sns.scatterplot(), sns.heatmap(). Integrates with Pandas DataFrames (data= parameter).

4. Descriptive Statistics and Probability

4.1 Measures of Central Tendency

• Mean ($\bar{x}$): $$\displaystyle \bar{x} = \frac{\sum x_i}{n} $$. Sensitive to outliers.

• Median: Middle value (ordered data). Robust to outliers.

• Mode: Most frequent value. Can be used for categorical data.

4.2 Measures of Variability

• Range: $Max - Min$. Sensitive to outliers.

• Variance ($$\displaystyle s^2 $$): $$\displaystyle s^2 = \frac{\sum (x_i - \bar{x})^2}{n-1} $$. Average squared deviation.

• Standard Deviation ($s$): $$\displaystyle s = \sqrt{s^2} $$. Same units as data.

• Interquartile Range (IQR): $$\displaystyle IQR = Q3 - Q1 $$. Measures spread of middle 50%, robust to outliers.

4.3 Probability Concepts

• Conditional Probability: Probability of A given B.

$$P(A|B) = \frac{P(A \cap B)}{P(B)}$$

• Correlation Analysis (Pearson): Measures linear relationship strength/direction.

$$r = \frac{\sum (x_i - \bar{x})(y_i - \bar{y})}{\sqrt{\sum (x_i - \bar{x})^2 \sum (y_i - \bar{y})^2}}$$

**Range:** [-1, 1]. **0** implies no linear correlation.

4.4 Statistical Inference Basics

• Type I Error ($\alpha$): False positive. Rejecting true null hypothesis.

• Type II Error ($\beta$): False negative. Failing to reject false null hypothesis.

  Trade-off: Decreasing $\alpha$ increases $\beta$.

• Maximum Likelihood Estimation (MLE): Method to estimate model parameters by finding parameters that maximize the likelihood function (probability of observed data). For i.i.d. data: $$\displaystyle L(\theta) = \prod P(x_i|\theta) $$. Often work with log-likelihood: $$\displaystyle \ell(\theta) = \log L(\theta) $$.

5. Data Analysis Tools

5.1 Microsoft Excel for Data Analysis

5.2 Python for Data Analysis

• Pandas DataFrame:

  • Structure: 2D labeled data structure with index (rows), columns, data. Axes: axis=0 (rows), axis=1 (columns). Heterogeneous data types per column.

  • Features: df.isna(), df.fillna(), pd.merge(), pd.concat(), df.groupby().

  • vs NumPy Array: DataFrame has labeled axes, supports heterogeneous data types, operations align on labels. NumPy arrays are homogeneous, use integer indexing, operations are element-wise without alignment.

• String Operations in Pandas:

  • Access via .str accessor: df['col'].str.contains('pattern'), .str.startswith(), .str.split(), .str.replace().

  • Regex: df['col'].str.extract(r'(pattern)') (capture group), df['col'].str.findall(r'pattern'). Example: Extract email: r'([\w\.-]+@[\w\.-]+)'.

• Python Visualization (Code Logic):

  
  # Matplotlib
  
  plt.pie(data, labels=labels)  # Pie
  
  plt.bar(categories, values)   # Bar
  
  plt.hist(data, bins=30)       # Histogram
  
  plt.scatter(x, y)             # Scatter
  
  # Seaborn (statistical, nicer defaults)
  
  sns.barplot(x='cat', y='val', data=df)
  
  sns.histplot(data=df, x='col', kde=True) # Histogram + Density
  
  sns.boxplot(x='cat', y='val', data=df)
  
  sns.scatterplot(x='x', y='y', hue='category', data=df)
  
  

5.3 R Programming for Data Analysis

• Features: Open-source, extensive statistical/graphical packages (CRAN), vectorized operations (fast), strong community.

• Major Components: Console (interactive), Workspace (.RData), Packages (library()), Help (?func).

• Vector Operations: Arithmetic (+,-,*,/), logical (>, ==, %in%), subsetting (x[condition] or x[indices]). Recycling Rule: Shorter vector repeated to match longer in operations.

• Categorical Data: factor(vector) creates a factor with levels. Essential for modeling categorical predictors.

6. Data Analysis Approaches

Role of Regression Analysis: Predicts continuous outcomes. Provides interpretable coefficients (effect size). Used for inference (understanding relationships) and prediction.

7. Business Intelligence (BI)

7.1 Definition & Evolution

BI: Technologies, applications, and practices for collecting, integrating, analyzing, and presenting business information to support better decision-making. Evolution: From static, IT-generated reports (1.0) to interactive, self-service dashboards (2.0) to AI-driven, embedded analytics (3.0).

7.2 Types of BI

7.3 BI Tools Comparison (Tableau vs. Power BI)

7.4 BI in Strategic Decision-Making

Logic of BI Systems:

1. ETL: Extract from sources, Transform (clean, integrate), Load into Data Warehouse.

2. Data Warehouse: Centralized repository of integrated, subject-oriented, time-variant data.

3. OLAP Cubes: Multidimensional structures for fast slicing/dicing of aggregated data.

4. Front-End: Dashboards, reports, visualizations for end-users.

How BI Supports Decisions:

• Data Aggregation: Summarizes vast data into KPIs.

• Interactive Visualization: Allows drill-down, slice-and-dice to explore "why."

• Predictive Insights: Integration with ML models for forecasting (e.g., sales prediction).

7.5 Ethical Issues in BI

• Privacy: Tracking employee/customer performance/behavior without consent.

• Bias: Reports/dashboards reflecting historical biases (e.g., in hiring, lending).

• Misuse: Using BI for manipulation (e.g., deceptive visualizations), lack of transparency in metrics.

8. Machine Learning and Predictive Modeling

8.1 Regression Models

• Linear Regression: $$\displaystyle y = \beta_0 + \beta_1 x_1 + ... + \beta_p x_p + \epsilon $$

  • Assumptions: Linearity, independence, homoscedasticity, normality of errors.

  • Interpretation: $$\displaystyle \beta_j $$ = change in $y$ for 1-unit change in $$\displaystyle x_j $$, holding others constant.

• Logistic Regression: For binary classification ($y \in \{0,1\}$). Models $$\displaystyle \log(\frac{p}{1-p}) = \beta_0 + \beta_1 x_1 + ... $$

  • Odds Ratio: $$\displaystyle e^{\beta_j} $$ = change in odds of $$\displaystyle y=1 $$ for 1-unit change in $$\displaystyle x_j $$.

  • Example: Predict loan approval (Yes=1) using income, credit score, debt-to-income ratio.

8.2 Classification Models

8.3 Clustering

• K-means: Algorithm:

  1. Initialization: Choose K centroids randomly.

  2. Assignment: Assign each point to nearest centroid (Euclidean distance).

  3. Update: Recalculate centroids as mean of assigned points.

  4. Repeat 2-3 until convergence (centroids stable).

  • Elbow Method: Plot SSE (inertia) vs. K. Choose K at "elbow" where SSE decrease slows.

  • Example: Segment customers into K groups based on spending/frequency.

8.4 Association Rule Mining (Apriori)

• Goal: Find items that frequently co-occur in transactions (e.g., market basket analysis).

• Key Metrics:

  • Support: $$\displaystyle P(\text{itemset}) = \frac{\text{freq( itemset )}}{\text{total transactions}} $$. Measures popularity.

  • Confidence: $$\displaystyle P(Y|X) = \frac{\text{support}(X \cup Y)}{\text{support}(X)} $$. Measures conditional probability.

  • Lift: $$\displaystyle \frac{\text{confidence}(X \rightarrow Y)}{\text{support}(Y)} = \frac{P(X \cap Y)}{P(X)P(Y)} $$. Lift > 1 implies positive correlation (rule is interesting). Lift = 1 = independence.

• Apriori Principle: Any subset of a frequent itemset must be frequent. Pruning: Eliminate candidate itemsets with any infrequent subset.

• Advantages: Simple, easy to implement, well-suited for sparse datasets (many items, few per transaction).

9. Big Data Analytics

9.1 Characteristics (5Vs)

• Volume: Size of data (TB, PB, EB).

• Variety: Different types (structured, unstructured, semi-structured).

• Velocity: Speed of data generation/processing (real-time, streaming).

• Veracity: Uncertainty, quality, trustworthiness of data.

• Value: Potential insights and benefit derived from data.

9.2 Processing Considerations

• Storage: Distributed File Systems (HDFS) for scalable, fault-tolerant storage.

• Processing: Parallel/distributed frameworks (MapReduce, Spark). Scalability: Horizontal (add more nodes). Fault Tolerance: Data replication, task re-execution on failure.

9.3 Phases of Big Data Analytics

[[DIAGRAM: CANVAS: A circular or linear flow diagram with 5 labeled stages in order:

1. Data Acquisition (sources: IoT, social media, logs)

2. Data Storage (HDFS, NoSQL)

3. Data Processing (MapReduce, Spark)

4. Data Analysis (ML, stats, mining)

5. Data Visualization (dashboards, reports)

Arrows connecting each stage in sequence.]]

9.4 Hadoop Ecosystem

• HDFS Architecture:

  • NameNode: Master server. Manages metadata (file system tree, block locations). Single Point of Failure (HA solutions exist).

  • DataNode: Slave server. Stores data blocks (default 128MB/256MB). Performs block operations (read/write, replication). Sends heartbeats & blockreports to NameNode.

  • Secondary NameNode: Not a backup NameNode. Performs checkpointing: Merges fsimage (metadata snapshot) and edit logs to prevent edit log from becoming too large. Does NOT provide failover.

  • Block Replication: Default factor = 3. One copy on local node, two on different racks (for rack awareness).

• MapReduce Programming Model:

  • Mapper: Processes input key-value pairs (e.g., <line_num, line_text>) → intermediate key-value pairs (e.g., <word, 1>). Shuffling & Sorting happens automatically between map and reduce phases, grouping values by key.

  • Reducer: Processes grouped intermediate keys (e.g., <word, [1,1,1...]>) → final output (e.g., <word, total_count>).

  • Job Scheduling (Fair Scheduler): Allocates cluster resources (containers) to jobs/pools to ensure fair share over time, not just FIFO. Supports preemption.

  • Word Count Example:

    
    // Mapper
    
    public void map(Object key, Text value, Context context) {
    
        StringTokenizer itr = new StringTokenizer(value.toString());
    
        while (itr.hasMoreTokens()) {
    
            word.set(itr.nextToken());
    
            context.write(word, one);
    
        }
    
    }
    
    // Reducer
    
    public void reduce(Text key, Iterable<IntWritable> values, Context context) {
    
        int sum = 0;
    
        for (IntWritable val : values) { sum += val.get(); }
    
        context.write(key, new IntWritable(sum));
    
    }
    
    

• HBase Storage Mechanism:

  • Column-family store (NoSQL). Data stored as rows with column families (groups of columns). Within a column family, data stored lexicographically.

  • Schema-less: Columns can be added dynamically per row.

  • Scalability: Automatic sharding (region splitting), built on HDFS.

  • Real-time: Low-latency read/write (random access), unlike HDFS (write-once, read-many).

• Hadoop File System Commands:

  
  # Copy file/dir from local FS to HDFS
  
  hadoop fs -copyFromLocal /local/path /hdfs/path
  
  # Show content of file (text files)
  
  hadoop fs -cat /hdfs/path/file.txt
  
  # For sequence/other binary files, use -text to attempt conversion
  
  hadoop fs -text /hdfs/path/file.seq
  
  

9.5 Higher-Level Tools

• Apache Pig:

  • Architecture: Pig Latin script → Pig Latin Compiler → Logical Plan → MapReduce Plan → Execution Engine (on Hadoop).

  • Pig Latin Script Flow: LOAD → FILTER/FOREACH/GROUP (TRANSFORM) → STORE.

  • Example: A = LOAD 'data' AS (user:chararray, item:chararray); B = GROUP A BY user; C = FOREACH B GENERATE group, COUNT(A); STORE C INTO 'output';

• Apache Hive:

  • Architecture: HiveServer2 (JDBC/ODBC server for clients), Metastore (RDBMS storing table schemas, partitions, locations).

  • HiveQL: SQL-like query language. Compiles to MapReduce/Tez/Spark jobs.

  • UDFs (User Defined Functions): Write Java class implementing UDF or GenericUDF interface. Register: CREATE TEMPORARY FUNCTION myfunc AS 'com.example.MyUDF'; Use: SELECT myfunc(col) FROM table;

9.6 Analytic Sandbox

Definition: An isolated, flexible environment (often cloud-based) where data scientists can experiment, prototype, and test analytics models and processes without affecting production systems. Importance: Enables rapid iteration, testing of new tools/algorithms, safe exploration of sensitive data, and proof-of-concept development before deployment.

10. Ethical, Legal, and Social Implications

10.1 Bias and Discrimination

• Reinforcing Human Biases: Algorithms learn from historical data that may contain societal biases (e.g., gender, racial). Example: Hiring algorithm downgrading resumes with "women's" in name if trained on historically male-hired data.

• Unfair Discrimination: Using protected attributes (race, gender, age) or proxies (zip code) to make adverse decisions, even if algorithmically "accurate." Example: Loan denial model using zip code as proxy for race.

10.2 Privacy and Security Issues

• Data Privacy: Collection: Transparency, informed consent. Anonymization/Pseudonymization: Removing direct identifiers (hard for re-identification). GDPR: EU regulation granting individuals rights (access, erasure, portability).

• Security Issues: Data Breaches: Unauthorized access. Access Control: Role-based permissions. Secure Storage: Encryption at rest and in transit.

10.3 Ethical Issues in Business Intelligence

• Misuse of Reports: Creating biased dashboards to mislead stakeholders.

• Lack of Transparency: "Black box" metrics or KPIs not clearly defined.

• Data Integrity: Manipulating source data or ETL processes for desired outcomes.

10.4 Transparency and Accountability

• Explainability (XAI): Ability to explain model predictions (e.g., SHAP, LIME).

• Audit Trails: Logging data lineage, model versions, and decision points.

• Responsibility: Clear ownership of models and their outcomes (data scientist, business owner).

11. Advanced and Specialized Topics

11.1 Dimensionality Reduction: t-SNE (Revisit)

• Goal: Visualization of high-D data in 2D/3D.

• Key Idea: Preserve local structure (similar points stay close). Uses t-distribution in low-D to alleviate crowding problem.

• Parameters: perplexity (roughly # of nearest neighbors), learning_rate.

• Limitation: Non-linear, non-parametric, results change with different runs/params. Not for feature extraction in supervised learning.

11.2 Regular Expressions (Regex)

• Syntax: . (any char), * (0+), + (1+), ? (0-1), ^ (start), $ (end), \d (digit), \w (word char), [] (character set), | (or).

• Pandas Examples:

  
  df['phone'].str.extract(r'(\d{3}-\d{3}-\d{4})')  # Extract (xxx-xxx-xxxx)
  
  df['email'].str.findall(r'[\w\.-]+@[\w\.-]+')     # Find all emails in a string
  
  

11.3 Exploratory Data Analysis (EDA)

• Role: Understand data, discover patterns, spot anomalies, test hypotheses before formal modeling.

• Techniques: Summary statistics (mean, SD, IQR), visualization (histograms, box plots, scatter matrix), correlation analysis.

• vs. Confirmatory Data Analysis (CDA): EDA is open-ended, inductive. CDA is hypothesis-driven, deductive (e.g., significance testing).

11.4 Role of Data Scientist vs. Data Analyst

11.5 Maximum Likelihood Estimation (MLE) (Revisit)

• Concept: Find parameter values ($\hat{\theta}$) that make the observed data most probable.

• Steps:

  1. Write down likelihood function $$\displaystyle L(\theta) = P(X_1, X_2, ..., X_n | \theta) = \prod_{i=1}^n P(X_i|\theta) $$ (assuming i.i.d.).

  2. Take log: $$\displaystyle \ell(\theta) = \log L(\theta) = \sum \log P(X_i|\theta) $$.

  3. Differentiate $\ell(\theta)$ w.r.t. $\theta$, set to zero, solve for $\hat{\theta}$.

• Example: For Normal($$\displaystyle \mu, \sigma^2 $$), MLE for $\mu$ is sample mean $\bar{x}$.