Data Science (IT-702 (A)) - Unit 1 Short Notes

UNIT 1: DATA SCIENCE - CORE FOUNDATIONS & TOOLS

1. Introduction to Data Science

Definition: An interdisciplinary field that uses scientific methods, processes, algorithms, and systems to extract knowledge and insights from structured and unstructured data.

• Scope & Evolution: From statistics and data mining to a unified field combining computer science, domain expertise, and communication.

• Applications:

  • Healthcare: Predictive diagnostics, personalized medicine.

  • Finance: Fraud detection, algorithmic trading.

  • Social Media: Sentiment analysis, recommendation systems.

• Role of Data Scientist: Combines skills in statistics, programming (Python/R), machine learning, and data visualization to solve business problems.

• Data Analysis vs. Data Science:

  • Data Analysis: Focuses on examining past data to identify trends and patterns (descriptive/predictive).

  • Data Science: Broader scope; includes building predictive models, machine learning, and creating data-driven products.

[!TIP]

Exam Focus: Distinguish between analysis (answering specific questions from existing data) and science (building models for future predictions and productization).

2. Data Types and Sources

[!TIP]

Exam Focus: Be prepared to compare all three types in a table. Emphasize that unstructured data (text, images) dominates modern data volume and requires specialized processing (e.g., t-SNE for images).

3. Data Analysis Methods

• Descriptive Analysis: Summarizes historical data to understand what happened.

  Tools: Mean, median, standard deviation, dashboards.

• Predictive Analysis: Uses statistical models and ML to forecast future outcomes.

  Example: Predicting customer churn, stock prices.

• Prescriptive Analysis: Recommends actions based on predictions.

  Example: Recommending optimal pricing or treatment plans.

• Exploratory Data Analysis (EDA): Initial investigation of data to discover patterns, spot anomalies, test hypotheses.

  Techniques: Visualization, summary statistics, correlation matrices.

4. Data Preprocessing and Wrangling

Data Cleaning:

• Missing Values: Remove, impute (mean/median), or flag.

• Outliers: Detect using IQR or Z-score; decide to cap/remove.

• Inconsistencies: Fix formatting (dates, units), correct errors.

Data Validation in Excel:

• Techniques: Data Validation rules (whole number, list, date), custom formulas, error alerts.

• Role: Prevents invalid entries at source, improves accuracy, reduces downstream cleaning effort.

Data Transformation:

• Normalization (Min-Max): Scales features to [0, 1].

$$X_{\text{norm}} = \frac{X - X_{\min}}{X_{\max} - X_{\min}}$$

• Standardization (Z-score): Centers to mean 0, std dev 1.

$$X_{\text{std}} = \frac{X - \mu}{\sigma}$$

String Operations in Pandas:

df['column'].str.lower()          # Lowercase

df['column'].str.strip()          # Remove whitespace

df['column'].str.contains('pattern') # Boolean mask

df['column'].str.split('_')       # Split on delimiter

Regular Expressions (Regex) in Pandas:

• Pattern extraction: df['text'].str.extract(r'(\d+)') (extract digits).

• Text cleaning: df['text'].str.replace(r'[^\w\s]', '', regex=True) (remove punctuation).

Importance: Garbage in → garbage out. Wrangling consumes ~60-80% of data science time but is critical for model accuracy.

5. Data Visualization

Principles: Clarity, accuracy, efficiency. Choose chart based on data type and message.

t-SNE (t-Distributed Stochastic Neighbor Embedding):

• Purpose: Visualize high-dimensional data (e.g., 100-D) in 2D/3D while preserving local structure.

• Example: Visualizing image datasets (MNIST digits) where each image is a high-dimensional pixel vector.

• Logic: Reduces dimensions by minimizing divergence between probability distributions of similarities in high-D and low-D space.

Python Visualization:

• matplotlib: Foundation library.

  
  plt.pie(data); plt.bar(categories, values); plt.hist(data); plt.scatter(x, y)
  
  

• seaborn: Statistical layer on matplotlib.

  
  sns.boxplot(x='category', y='value', data=df); sns.heatmap(corr_matrix)
  
  

[!TIP]

Exam Focus: Know which chart for which data (categorical vs quantitative). For t-SNE, stress it's for high-dimensional unstructured data (like images/text vectors).

6. Statistical Foundations

A. Descriptive Statistics

• Central Tendency:

  • Mean ($\mu$): $$\displaystyle \mu = \frac{\sum x_i}{n} $$ (sensitive to outliers).

  • Median: Middle value (robust to outliers).

  • Mode: Most frequent value (for categorical/nominal).

• Variability:

  • Range: $\max - \min$.

  • Variance ($$\displaystyle \sigma^2 $$): $$\displaystyle \sigma^2 = \frac{\sum (x_i - \mu)^2}{n} $$.

  • Standard Deviation ($\sigma$): $$\displaystyle \sigma = \sqrt{\sigma^2} $$ (same units as data).

  • IQR: $Q3 - Q1$ (spread of middle 50%, robust).

B. Probability

• Conditional Probability: $$\displaystyle P(A|B) = \frac{P(A \cap B)}{P(B)} $$.

  Example: Probability of loan default given low income.

C. Inferential Statistics

• Correlation Analysis: Measures linear relationship strength/direction.

  Pearson's $r$: $$\displaystyle r = \frac{\sum (x_i - \bar{x})(y_i - \bar{y})}{\sqrt{\sum (x_i - \bar{x})^2 \sum (y_i - \bar{y})^2}} $$, $-1 \le r \le 1$.

• Regression Analysis:

  • Linear Regression: $$\displaystyle y = \beta_0 + \beta_1 x + \epsilon $$ (simple); extends to multiple.

  • Logistic Regression (Binary Classification):

$$P(Y=1|X) = \frac{1}{1 + e^{-(\beta_0 + \beta_1 X)}}$$

*Example: Loan approval (Yes=1, No=0) based on income, credit score.*

• Hypothesis Testing:

  • Type I Error: Reject true $$\displaystyle H_0 $$ (false positive). Significance level $\alpha$.

  • Type II Error: Fail to reject false $$\displaystyle H_0 $$ (false negative). Power = $1 - \beta$.

  • Multiple Testing Example: Testing 100 drug effects at $$\displaystyle \alpha=0.05 $$ → ~5 false positives expected.

• Maximum Likelihood Estimation (MLE): Estimates parameters by maximizing likelihood function $L(\theta|data)$.

  Application: Estimating coefficients in logistic regression.

7. Introduction to Machine Learning

• Supervised vs. Unsupervised:

  • Supervised: Labeled data (classification, regression).

  • Unsupervised: No labels (clustering, association).

Classification Algorithms:

• Logistic Regression: (See above). Outputs probability; uses sigmoid.

• Random Forest: Ensemble of decision trees (bagging).

  Logic: Build many trees on bootstrapped samples, average predictions. Reduces overfitting.

• Decision Trees: Tree structure (root → splits → leaves).

  Splits based on Gini impurity/entropy. Example: Classify loan risk using income, debt.

Clustering:

• K-means:

  1. Choose $k$.

  2. Initialize $k$ centroids randomly.

  3. Assign points to nearest centroid.

  4. Recompute centroids.

  5. Repeat 3-4 until convergence.

Association Rule Mining:

• Apriori Algorithm:

  • Support: $\text{freq}(X \cup Y) / N$ (how often itemset appears).

  • Confidence: $\text{support}(X \cup Y) / \text{support}(X)$ (if X then Y).

  • Advantages: Simple, uses frequent itemset property (subset of frequent is frequent).

8. Tools for Data Science

A. Microsoft Excel

• Lookup Functions:

  • VLOOKUP(lookup_value, table, col_index, [exact]): Vertical search.

  • XLOOKUP(lookup, lookup_array, return_array, [if_not_found], [match_mode]): More flexible, horizontal/vertical.

• Conditional Functions:

  • SUMIFS(sum_range, criteria_range1, criteria1, ...): Multi-condition sum.

  • INDEX(array, row_num, [col_num]) + MATCH(lookup, lookup_array, 0): Two-way lookup, no column limit.

• Dynamic Array Formulas (Excel 365):

  • FILTER(), SORT(), UNIQUE(): Spill results automatically.

  • Customer Segmentation: FILTER(data, (age>30)*(income>50000)).

• Pivot Tables:

  • Create: Insert → PivotTable.

  • Group: Dates (months), numbers (bins).

  • Aggregate: Sum, Count, Average.

• Scenario Manager (What-If Analysis):

  • Data → What-If Analysis → Scenario Manager.

  • Save sets of input values (e.g., best/worst case revenue) and compare outputs.

• Macros:

  • Purpose: Automate repetitive tasks.

  • Steps: View → Macros → Record Macro → Perform actions → Stop Recording.

B. Python for Data Science

• Pandas:

  • DataFrame: 2D labeled table (rows: observations, columns: features).

    Creation: pd.DataFrame(data, columns=..., index=...).

  • vs. NumPy Array:

    | Pandas DataFrame | NumPy Array | | :--- | :--- | | Heterogeneous columns (dtypes). | Homogeneous (single dtype). | | Labeled axes (row/col names). | Integer-indexed axes. | | Built-in handling of missing data (NaN). | Requires manual handling. | | Rich I/O (CSV, Excel, SQL). | Limited I/O. |

  • Operations:

    • Filtering: df[df['age'] > 30].

    • Grouping: df.groupby('category')['sales'].mean().

    • Merging: pd.merge(df1, df2, on='key').

  • String & Regex: As in Section 4 (df['col'].str accessor).

• Visualization Libraries:

  • matplotlib: Basic plots (plt.plot, plt.bar, plt.scatter).

  • seaborn: Statistical plots (sns.distplot, sns.pairplot, sns.heatmap).

9. Business Intelligence (BI)

Definition: Technologies, applications, and practices for collecting, integrating, analyzing, and presenting business information to support decision-making.

Types of BI:

BI Tools Comparison (e.g., Tableau vs. Power BI):

Role in Strategic Decision-Making:

1. Data Aggregation: Integrates data from multiple sources (ERP, CRM).

2. Analysis: Identifies trends, opportunities, threats.

3. Visualization: Dashboards/scorecards for executives.

4. Support Logic: Provides evidence-based insights, reduces intuition-based decisions.

Ethical Issues in BI:

• Reinforcing human biases: Historical data may encode discrimination (e.g., hiring, lending).

• Unfair discrimination: Models may unfairly disadvantage protected groups (e.g., race, gender).

• Privacy: Tracking employee/customer behavior.

• Transparency: "Black box" models lack explainability.

10. Ethics, Security, and Social Issues

• Reinforcing Human Biases:

  • Cause: Biased training data (e.g., historical hiring favoring men).

  • Impact: AI perpetuates inequality (e.g., facial recognition errors for darker skin tones).

• Unfair Discrimination:

  • Definition: Systematic disadvantage to a group based on protected attributes.

  • Example: Loan approval algorithm denying loans disproportionately to minority neighborhoods due to zip-code proxy.

• Ethical Issues in Data Science & BI:

  • Lack of consent in data collection.

  • Opacity in algorithmic decisions.

  • Misuse of predictive analytics (e.g., predictive policing).

• Security Issues:

  • Data Privacy: Compliance (GDPR, HIPAA); anonymization techniques.

  • Breaches: Unauthorized access to sensitive data (PII, health records).

  • Mitigation:

    • Encryption (at rest, in transit).

    • Access controls (RBAC).

    • Regular audits and penetration testing.

[!TIP]

Exam Focus: Link ethics to real examples (bias in hiring, facial recognition). For security, know mitigation strategies (encryption, access control).