Data Mining (IT-603 (B)) Unit 2 Notes - RGPV

UNIT 2: DATA PREPROCESSING & DATA WAREHOUSING

1.0 Data Preprocessing

Data preprocessing prepares raw data for mining by cleaning, transforming, and integrating it. It is often the most time-consuming step (60-80% of effort) but is critical for quality results.

1.1 Data Transformation Strategies

Transformation converts data into appropriate forms for mining algorithms.

Strategy	Description	Key Formula/Concept
Normalization	Scales attribute values to a small, specified range (e.g., 0-1).	Min-Max: $$\displaystyle x' = \frac{x - \min(A)}{\max(A) - \min(A)} $$ <br> Z-Score: $$\displaystyle x' = \frac{x - \mu_A}{\sigma_A} $$ <br> Decimal Scaling: $$\displaystyle x' = \frac{x}{10^j} $$ (j smallest integer s.t. max\|x'\| < 1)
Aggregation & Summarization	Combines multiple values into a single value (e.g., daily sales → monthly sales). Reduces data volume.	Sum, Count, Avg, Max, Min over a defined granularity.
Generalization	Replaces low-level values with higher-level concepts using concept hierarchies.	e.g., `street` → `city` → `state` → `country`. Uses predefined or automatically generated hierarchies.
Feature/Attribute Construction	Creates new, more informative attributes from existing ones.	e.g., `BMI = weight(kg) / [height(m)]²` from `weight` and `height`. Can use domain knowledge or automated methods (e.g., polynomial features).

[!TIP] Exam Focus: Be ready to define and differentiate all normalization techniques with simple numerical examples. Know that generalization relies on concept hierarchies.

1.2 Handling Missing Values

Missing data can bias models. Common strategies:

Method	Description	Pros & Cons
Tuple Deletion	Remove entire record if any attribute value is missing.	Pro: Simple. Con: Can lose significant data if missingness is high.
Imputation	Fill missing values with estimated ones.	Mean/Median/Mode: Fast, but distorts distribution. <br> Regression Imputation: Uses other attributes to predict missing value (more accurate, computationally heavier).
Robust Algorithms	Use algorithms inherently tolerant to missing values.	e.g., Some decision tree variants (like C4.5) can handle missing values by fractional instance splitting or surrogate splits.

[!TIP] Common Pitfall: Do not blindly use mean imputation for skewed data—median is often better. Know when regression imputation is preferred (when attributes are correlated).

2.0 Data Warehousing Fundamentals

A Data Warehouse (DW) is a subject-oriented, integrated, time-variant, non-volatile collection of data for supporting management decisions.

Characteristic	Description
Subject-Oriented	Organized around key subjects (e.g., Customer, Product), not applications.
Integrated	Combines data from multiple, heterogeneous sources (resolves naming/format inconsistencies).
Time-Variant	Data is stored with a time dimension; historical data is preserved.
Non-Volatile	Data is stable (mostly read-only); not updated/deleted in real-time like OLTP.

ETL Process (Extract, Transform, Load):

Extract: Pull data from source systems.
Transform: Clean, integrate, and aggregate data (applies many preprocessing steps).
Load: Populate the data warehouse schema.

[!TIP] Key Distinction: DW is for analysis (OLAP), while operational databases are for transactions (OLTP).

3.0 OLAP (Online Analytical Processing)

Enables interactive analysis of multidimensional data.

Core OLAP Operations:

Roll-up (Drill-up): Summarize data by climbing up a concept hierarchy (e.g., city → country).
Drill-down: Reverse of roll-up; view finer-grained details (e.g., country → city).
Slice: Select a single dimension to create a sub-cube (e.g., time = "2023").
Dice: Select on two or more dimensions to create a sub-cube (e.g., time = "2023" AND product = "Electronics").
Pivot (Rotate): Reorient the cube to view data from a different perspective (swap rows/columns).

Types of OLAP:

Type	Storage	Performance	Scalability	Best For
MOLAP<br>(Multidimensional)	Pre-computed, proprietary multidimensional arrays.	Very Fast (aggregates pre-calculated).	Poor (limited by array size).	Fast query response on stable, dense data.
ROLAP<br>(Relational)	Standard relational DBMS (tables).	Slower (aggregates computed on-demand via SQL).	Excellent (handles large, sparse data).	Large-scale, detailed, volatile data.
HOLAP<br>(Hybrid)	Combines MOLAP (aggregates) & ROLAP (detailed data).	Balanced.	Balanced.	Balanced needs for speed and detail.

[!TIP] Exam Favorite: Compare ROLAP vs MOLAP in a table format (Storage, Performance, Scalability). Remember MOLAP = Fast but less scalable, ROLAP = Scalable but slower.

4.0 Data Warehouse Schemas

Design patterns for organizing tables in a DW.

Schema	Description	Advantages	Disadvantages
Star Schema	Central fact table connected to denormalized dimension tables. Simple, single join path.	Simple, high query performance.	Data redundancy (e.g., repeated city/state in customer dim).
Snowflake Schema	Normalized dimension tables (split into related tables).	Reduces redundancy, easier to maintain.	More complex joins, potentially slower queries.
Fact Constellation Schema	Multiple fact tables sharing dimension tables (a galaxy of stars).	Models complex business processes.	Most complex design, hardest to navigate.

[!TIP] Visualization: Always sketch a simple star vs. snowflake for the same scenario (e.g., Sales) to show normalization in dimensions.

5.0 Snowflake Schema Design (Case Study)

University Data Warehouse Example

Objective: Analyze student performance across courses, semesters, and instructors.

Dimensions & Hierarchies:

Student: student_id → major → college
Course: course_id → department → college
Semester: semester_id → term (Fall/Spring) → year
Instructor: instructor_id → rank → department

Fact Table: University_Fact

Foreign Keys: student_id, course_id, semester_id, instructor_id
Measures: count (number of students), avg_grade (average grade for the specific student-course-semester-instructor combination at the lowest level).

Schema Diagram Construction:


          [Student Dim]     [Course Dim]     [Semester Dim]   [Instructor Dim]

                 \               |               /                 /

                  \              |              /                 /

                   \             |             /                 /

                    \            |            /                 /

                     \           |           /                 /

                      \          |          /                 /

                       \         |         /                 /

                        \        |        /                 /

                         \       |       /                 /

                          \      |      /                 /

                           \     |     /                 /

                            \    |    /                 /

                             \   |   /                 /

                              \  |  /                 /

                               \ | /                 /

                                \|/                 /

                            [University_Fact Table]

Each dimension table is normalized (snowflaked). e.g., Student Dim links to a separate College table via college_id.
Granularity: The fact table's grain is "one student's enrollment in one course with one instructor in one semester."
Hierarchies Enable Roll-up/Drill-down: e.g., Drill-down from college → major → student in the Student dimension to see average grades at different levels.

[!TIP] Exam Question Pattern: You will likely be asked to draw this exact snowflake schema. Remember:

Fact table in the center.

Four dimension tables (Student, Course, Semester, Instructor) connected to it.

Each dimension table is normalized (has foreign keys to other tables like College, Department).

Clearly label measures (count, avg_grade) in the fact table.

Show hierarchies within dimensions (e.g., college → department → course).

Account

Data Mining (IT-603 (B)) - Unit 2 Short Notes