IT Business & Disaster Recovery Planning (CY-604 (A)) - Unit 3 Short Notes

UNIT 3: IT Business & Disaster Recovery Planning

(Focus: Data Warehousing & Mining based on RGPV Past Papers)

1.0 Fundamentals of Data Warehousing

1.1 Definition and Core Characteristics of Data Warehouse

Definition: A subject-oriented, integrated, time-variant, non-volatile collection of data supporting management’s decision-making process. (Bill Inmon)

Core Characteristics:

• Subject-oriented: Organized around key business subjects (e.g., Customer, Product), not applications.

• Integrated: Consolidated from multiple heterogeneous sources; consistent naming, encoding, and formats.

• Time-variant: Data stored with time stamps; historical data preserved for trend analysis.

• Non-volatile: Data is not updated/deleted in real-time; only appended and periodically refreshed.

[!TIP] Common Pitfall: Do not confuse with OLTP databases. DW is for analysis, not transaction processing.

1.2 Data Warehouse Architecture and Components

Typical Architecture:

1. Data Sources: Operational databases, external data, legacy systems.

2. Staging Area: Temporary storage for data cleaning, integration, and transformation.

3. Data Warehouse: Central repository storing integrated, historical data.

4. Data Marts: Department-specific subsets (e.g., Sales, Finance).

5. OLAP Server: Enables multidimensional analysis.

6. Client Tools: Query, reporting, data mining, dashboard tools.

Diagram:

1.3 Data Warehouse Implementation Techniques

• Top-down (Inmon): Build enterprise-wide DW first, then derive data marts. Comprehensive but costly and slow.

• Bottom-up (Kimball): Build data marts first, then integrate into DW. Quick initial benefits but risks integration issues.

• Hybrid: Combine both; start with critical data marts, gradually build integrated DW.

[!TIP] Exam Focus: Compare top-down vs. bottom-up in terms of cost, time, and integration.

1.4 Need for Vertical Partitioning in Data Warehousing

Definition: Splitting a table vertically into multiple tables with fewer columns but same rows.

Purpose:

• Improve query performance by reducing I/O (access only needed columns).

• Separate frequently accessed attributes from infrequent ones.

• Enhance security by isolating sensitive columns.

Example: A Customer table with 50 columns split into Customer_Basic (ID, Name, Region) and Customer_Detailed (ID, Income, Preferences).

2.0 Data Warehouse Schemas and Multidimensional Databases

2.1 Star Schema

• Structure: Central fact table (foreign keys to dimensions, measures) connected to denormalized dimension tables.

• Advantages: Simple, optimized for query performance (fewer joins).

• Disadvantages: Data redundancy in dimensions.

Diagram:

2.2 Snowflake Schema (Detailed)

• Structure: Normalized version of star schema; dimension tables decomposed into multiple related tables.

• Advantages: Reduces redundancy, saves storage, easier maintenance.

• Disadvantages: More complex queries (increased joins), slower performance.

Example: Product dimension normalized into Product (Prod_ID, Name), Category (Cat_ID, Cat_Name), Brand (Brand_ID, Brand_Name).

[!TIP] Exam Question: Draw and explain snowflake schema with example.

2.3 Galaxy Schema for Multidimensional Databases (Detailed)

• Also called Fact Constellation.

• Structure: Multiple fact tables sharing dimension tables.

• Use Case: Complex business processes with multiple facts (e.g., Sales and Inventory facts sharing Product and Time dimensions).

• Advantages: Avoids redundancy, supports multiple business processes.

• Disadvantages: Complex design, harder to maintain.

Diagram:

2.4 Multidimensional Database Fundamentals

• Data Organization: Multi-dimensional arrays (cubes).

• Dimensions: Perspectives for analysis (e.g., Time, Product, Location).

• Measures: Numerical values at dimension intersections (e.g., Sales amount).

• Operations:

  • Slice: Fix one dimension, view sub-cube.

  • Dice: Select specific values on multiple dimensions.

  • Roll-up: Aggregate up a hierarchy (e.g., day → month → year).

  • Drill-down: Reverse of roll-up.

2.5 Data Cubes and Computation Process

• Data Cube: N-dimensional array of data.

• Computation Challenge: Pre-computing all aggregations is expensive (exponential growth).

• Methods:

  • Full Cube: Compute all group-by aggregations.

  • Partial Cube: Compute only relevant aggregations (e.g., using iceberg cubes with min support threshold).

  • Star Cube Optimization: Compute only shared sub-cubes.

3.0 OLAP (Online Analytical Processing)

3.1 OLAP vs. OLTP: Concepts and Examples

3.2 Types of OLAP

• MOLAP (Multidimensional OLAP): Data stored in proprietary multidimensional arrays. Fast query, but limited scalability and data volume.

• ROLAP (Relational OLAP): Data stored in relational DB; OLAP server translates MD queries to SQL. Scalable, but slower for complex queries.

• HOLAP (Hybrid OLAP): Combines MOLAP and ROLAP; detailed data in relational, aggregates in multidimensional.

3.3 Need and Role of MOLAP Server

• Need: For fast, interactive analysis on pre-aggregated data.

• Role:

  • Stores data in compressed multidimensional arrays (cubes).

  • Pre-computes and stores aggregates for all dimension combinations.

  • Provides instant response to slice-dice, roll-up, drill-down operations.

• Limitation: Not suitable for very large datasets due to storage and processing constraints.

3.4 OLAP Server Architectures

• MOLAP Server: Directly manages multidimensional data storage and processing.

• ROLAP Server: Acts as middleware; translates MD queries to SQL for relational DB.

• HOLAP Server: Routes queries to relational DB for detailed data and to MOLAP store for aggregates.

3.5 ROLAP Concept with Diagram

• Concept: OLAP functionality built on top of relational DBMS. Data remains in relational tables (star schema). OLAP server generates optimized SQL for MD queries.

• Advantages: Leverages scalable RDBMS, handles large data volumes.

• Disadvantages: Slower than MOLAP for complex aggregations; relies on RDBMS optimization.

Diagram:

3.6 HOLAP and Hybrid Approaches

• HOLAP: Stores detailed data in relational tables, aggregates in MOLAP cubes. Balances storage efficiency and query speed.

• Hybrid Approaches: May also use other combinations (e.g., some dimensions in MOLAP, others in ROLAP).

4.0 Introduction to Data Mining

4.1 Definition and Scope of Data Mining

Definition: The process of discovering patterns, correlations, trends, and knowledge from large amounts of data using machine learning, statistics, and database systems.

Scope:

• Predictive modeling (classification, regression).

• Descriptive modeling (clustering, association).

• Anomaly detection.

• Pattern discovery in diverse data types (text, web, spatial).

4.2 Pros and Cons of Data Mining

Pros:

• Uncovers hidden patterns and knowledge.

• Predicts future trends and behaviors.

• Automates data analysis, reducing manual effort.

• Improves decision-making (e.g., marketing, risk management).

Cons:

• Privacy and ethical concerns (misuse of personal data).

• Data quality issues (garbage in, garbage out).

• High computational cost for large datasets.

• Overfitting or spurious patterns if not validated.

4.3 KDD Process and Role of Data Mining Engine

KDD (Knowledge Discovery in Databases) Steps:

1. Selection: Choose target dataset from sources.

2. Preprocessing: Clean, integrate, handle missing/noisy data.

3. Transformation: Normalize, aggregate, construct features.

4. Data Mining: Apply algorithms (core role of data mining engine).

5. Interpretation/Evaluation: Evaluate patterns for validity, usefulness.

6. Knowledge: Deploy discovered knowledge.

Role of Data Mining Engine: Executes mining algorithms (e.g., Apriori, k-means, decision trees) on preprocessed data to extract patterns.

4.4 Data Mining Task Primitives

• Task-relevant data: Specifies data subset (e.g., attributes, records).

• Background knowledge: Domain knowledge (e.g., taxonomies, constraints).

• Interestingness measures: Metrics like support, confidence, accuracy.

• Pattern types: Classification, clustering, association, etc.

• Constraints: Time, space, or algorithmic constraints.

4.5 Types of Data for Data Mining Applications

• Relational databases: Structured tables.

• Data warehouses: Integrated, historical data.

• Transactional databases: Purchase records, logs.

• Advanced types:

  • Text: Documents, emails.

  • Web: Hypertext, usage logs.

  • Spatial: Geographic data.

  • Time-series: Sequential data (stock prices).

  • Multimedia: Images, video, audio.

5.0 Data Preprocessing

5.1 Data Cleaning: Techniques and Methods

Common Issues: Missing data, noisy data, inconsistent data.

Techniques:

• Missing Data:

  • Ignore tuples (if few).

  • Fill with mean/median/mode.

  • Predict using regression or machine learning.

• Noisy Data:

  • Binning: Sort values, partition into bins, smooth by mean/median.

  • Regression: Fit regression function, smooth values.

  • Clustering: Group similar values, treat outliers.

  • Manual: Human inspection (costly).

• Inconsistent Data:

  • Use domain knowledge (e.g., age cannot be negative).

  • Check constraints (e.g., functional dependencies).

5.2 Data Transformation Methods

• Normalization:

  • Min-max: $$\displaystyle x' = \frac{x - \min}{\max - \min} $$

  • Z-score: $$\displaystyle x' = \frac{x - \mu}{\sigma} $$

  • Decimal scaling: $$\displaystyle x' = \frac{x}{10^j} $$

• Aggregation: Summarize data (e.g., daily sales → monthly sales).

• Generalization: Replace low-level values with higher-level concepts (e.g., age → age group).

• Feature Construction: Create new attributes from existing ones (e.g., BMI from height/weight).

• Attribute Selection: Remove irrelevant or redundant attributes (e.g., using correlation analysis).

6.0 Association Rule Mining

6.1 Basic Concepts and Boolean Association Rule Mining

Association Rule: Implication of the form $$\displaystyle X \rightarrow Y $$, where $$\displaystyle X \cap Y = \emptyset $$.

Key Measures:

• Support: Fraction of transactions containing $X \cup Y$.

$$ \boxed{\text{support}(X \rightarrow Y) = \frac{|\{t \in D : X \cup Y \subseteq t\}|}{|D|}} $$

• Confidence: Conditional probability of $Y$ given $X$.

$$ \boxed{\text{confidence}(X \rightarrow Y) = \frac{\text{support}(X \cup Y)}{\text{support}(X)}} $$

• Lift: $$\displaystyle \text{lift}(X \rightarrow Y) = \frac{\text{confidence}}{\text{support}(Y)} $$ (measures correlation).

Boolean Association Rule Mining:

• Items are binary (present/absent in transactions).

• Finds rules where items co-occur frequently.

• Uses support and confidence thresholds.

6.2 Apriori Algorithm (With Example)

Key Idea: All subsets of a frequent itemset must be frequent (anti-monotone property).

Steps:

1. Find frequent 1-itemsets by scanning DB, comparing with min_support.

2. Generate candidate $k$-itemsets from frequent $(k-1)$-itemsets ($$\displaystyle C_k $$).

3. Prune candidates with any infrequent $(k-1)$-subset.

4. Scan DB to count candidates, keep frequent $k$-itemsets ($$\displaystyle L_k $$).

5. Repeat until $$\displaystyle L_k $$ is empty.

6. Generate association rules from frequent itemsets using min_confidence.

Example:

Transactions:

T1: {A,B,C}

T2: {A,B}

T3: {A,C}

T4: {B,C}

T5: {A,B,C}

min_support = 0.4 (2 transactions), min_confidence = 0.7.

• Frequent 1-itemsets: A(4), B(3), C(3).

• Candidate 2-itemsets: {A,B}, {A,C}, {B,C} → all frequent (support 3,3,3).

• Candidate 3-itemset: {A,B,C} → support 2 (frequent).

• Rules from {A,B,C}:

  • A,B → C: conf = 2/3 ≈ 0.67 (<0.7, reject).

  • A,C → B: conf = 2/3 ≈ 0.67 (reject).

  • B,C → A: conf = 2/3 ≈ 0.67 (reject).

  • A → B,C: conf = 2/4 = 0.5 (reject).

6.3 FP-growth Algorithm (With Example)

Key Idea: Avoid candidate generation by using FP-tree (compressed representation of frequent patterns).

Steps:

1. Scan DB once to find frequent items (above min_support).

2. Build FP-tree:

   • Sort items in each transaction by frequency.

   • Insert into tree, sharing common prefixes.

   • Store node counts and links.

3. Mine FP-tree recursively:

   • For each frequent item, construct conditional pattern base.

   • Build conditional FP-tree.

   • Repeat until tree is empty.

Example:

Transactions (same as above). Frequent items: A(4), B(3), C(3).

FP-tree construction (order: A, B, C):

• T1: A:3 → B:2 → C:1

• T2: A:3 → B:2

• T3: A:3 → C:1

• T4: B:3 → C:1

• T5: A:3 → B:2 → C:1

Mining: Start from C (lowest freq), pattern {C} with support 3; conditional base for C: {A,B:2}, {B:1}, {A:1} → frequent 2-itemset {A,B} with support 2 → pattern {A,B,C}.

6.4 Techniques to Improve Efficiency of Apriori

• Reduce number of database scans:

  • Partitioning: Split DB into partitions, find local frequent itemsets, then global.

  • Sampling: Use random sample, then verify on full DB.

• Reduce candidate size:

  • Hash-based technique: Hash itemsets to reduce candidates.

  • Transaction reduction: Remove transactions that cannot contain frequent itemsets.

  • Pruning: Use advanced properties (e.g., contiguous patterns).

• Alternative structures: Use FP-tree (FP-growth) to avoid candidate generation entirely.

6.5 Time Series Mining for Association Rules

• Time Series Data: Sequence of values over time (e.g., stock prices, sensor readings).

• Transformation to Transactions:

  • Discretization: Convert continuous values to intervals (e.g., price → {low, medium, high}).

  • Sliding windows: Extract subsequences as "items".

• Apply association mining on transformed data.
  Example: Discretize daily stock changes into "up" or "down", then find rules like "up on Monday → up on Tuesday".

7.0 Clustering Analysis

7.1 Definition and Important Clustering Methods

Definition: Grouping objects into clusters such that intra-cluster similarity is high, inter-cluster similarity is low.

Important Methods:

• Partitioning: k-means, k-medoids (PAM).

• Hierarchical: Agglomerative (bottom-up), Divisive (top-down).

• Density-based: DBSCAN, OPTICS.

• Grid-based: STING, CLIQUE.

• Model-based: Expectation-Maximization (EM) for Gaussian mixtures.

7.2 Hierarchical Clustering: Top-down vs. Bottom-up Algorithms

• Bottom-up (Agglomerative):

  1. Start with each object as a cluster.

  2. Merge two closest clusters (using linkage: single, complete, average).

  3. Repeat until one cluster or stopping criterion.

• Top-down (Divisive):

  1. Start with all objects in one cluster.

  2. Split clusters recursively (using clustering method at each step).

  3. Stop when each object is a cluster or criterion met.

Comparison:

• Agglomerative is more common, simpler.

• Divisive is computationally expensive (exponential splits).

[!TIP] Exam Focus: Differentiate with examples; linkage methods in agglomerative.

7.3 Other Clustering Techniques

• k-means:

  • Choose k centroids.

  • Assign each point to nearest centroid.

  • Update centroids as mean of assigned points.

  • Repeat until convergence.

  • Sensitive to outliers and initial centroids.

• DBSCAN (Density-based):

  • Core point: At least min_samples points within eps distance.

  • Border point: Reachable from core point but not core itself.

  • Noise: Neither core nor border.

  • Forms clusters of arbitrary shape, robust to noise.

• STING (Grid-based):

  • Divide space into grid cells (multiple resolutions).

  • Store statistical info (mean, max, etc.) in cells.

  • Cluster by grouping adjacent high-density cells.

8.0 Classification Techniques

8.1 Learning Phase in Classification

Process:

1. Model Construction:

   • Input: Labeled training dataset (features + class labels).

   • Algorithm learns mapping from features to class.

2. Model Testing:

   • Use separate test dataset to evaluate accuracy.

3. Model Application:

   • Deploy model to classify new, unseen instances.

8.2 Decision Tree Induction Algorithm (Detailed)

Goal: Build a tree where internal nodes are tests on attributes, branches are outcomes, leaves are class labels.

Popular Algorithms:

• ID3: Uses information gain (based on entropy).

• C4.5: Extension of ID3; uses gain ratio to handle multi-valued attributes.

• CART: Uses Gini index for binary splits.

ID3 Steps:

1. If all instances in current node belong to same class, create leaf with that class.

2. If no attributes left, create leaf with majority class.

3. Else, select attribute $A$ with highest information gain.

4. Create branch for each value of $A$.

5. Recur on subsets for each branch.

Entropy (measure of impurity):

$$ \boxed{H(S) = -\sum_{i=1}^{c} p_i \log_2 p_i} $$

where $$\displaystyle p_i $$ is proportion of class $i$ in set $S$, $c$ is number of classes.

Information Gain:

$$ \boxed{Gain(S, A) = H(S) - \sum_{v \in Values(A)} \frac{|S_v|}{|S|} H(S_v)} $$

where $$\displaystyle S_v $$ is subset of $S$ for which attribute $A$ has value $v$.

[!TIP] Information gain biased toward many-valued attributes; use gain ratio (C4.5) to correct.

8.3 Rule-based Classification Algorithms

• Learn IF-THEN rules from training data.

• Methods:

  • RIPPER: Sequential covering algorithm; grows and prunes rules.

  • CN2: Uses beam search to find rules.

• Steps (RIPPER example):

  1. Start with empty rule.

  2. Add conditions to cover positive examples.

  3. Prune rule to avoid overfitting.

  4. Repeat until all positives covered or stopping criterion.

8.4 Bayesian Classification

• Based on Bayes’ theorem:

$$ P(C|X) = \frac{P(X|C) P(C)}{P(X)} $$

where $C$ is class, $X$ is feature vector.

• Naive Bayes: Assumes conditional independence of features given class.

$$ P(X|C) = \prod_{i=1}^{n} P(x_i|C) $$

• Compute posterior for each class, predict class with max $P(C|X)$.

• Example: Spam filtering (words as features).

8.5 Probabilistic Classifiers

• Output probability distribution over classes, not just class label.

• Examples:

  • Naive Bayes (as above).

  • Logistic Regression: Models $$\displaystyle P(C=1|X) = \frac{1}{1 + e^{-(\beta_0 + \beta^T X)}} $$.

  • Bayesian Networks: Graphical models representing conditional dependencies.

• Advantage: Provides confidence estimates for predictions.

8.6 Process of Verifying Classifier Accuracy

Steps:

1. Holdout Method: Split data into training set (e.g., 70%) and test set (e.g., 30%).

2. Metrics:

   • Accuracy: $$\displaystyle \frac{\text{correct predictions}}{\text{total predictions}} $$

   • Precision: $$\displaystyle \frac{TP}{TP + FP} $$

   • Recall: $$\displaystyle \frac{TP}{TP + FN} $$

   • F1-score: $$\displaystyle 2 \times \frac{\text{precision} \times \text{recall}}{\text{precision} + \text{recall}} $$

   • Confusion Matrix: Tabulate TP, TN, FP, FN.

3. Cross-validation:

   • k-fold: Split data into $k$ folds, train on $k-1$, test on 1, repeat $k$ times.

   • Leave-one-out: Special case of $k$-fold where $$\displaystyle k = n $$.

4. Compare with baseline (e.g., majority class classifier).

9.0 Advanced Data Mining Applications

9.1 Web Usage Mining

• Goal: Analyze web logs to discover user browsing patterns, preferences, and behaviors.

• Steps:

  1. Data Collection: Server logs, client-side cookies, proxy logs.

  2. Preprocessing: User identification, session segmentation, data cleaning.

  3. Pattern Discovery:

     • Association Rules: Pages visited together.

     • Sequential Patterns: Order of page visits.

     • Clustering: User groups or page groups.

  4. Analysis and Visualization: Interpret patterns for personalization, site redesign, marketing.

9.2 Text Mining

• Goal: Extract information and knowledge from unstructured text documents.

• Key Techniques:

  • Text Preprocessing: Tokenization, stemming, stop-word removal.

  • Vector Space Model: Represent documents as term vectors (TF-IDF weighting).

  • Tasks:

    • Text Classification: Categorize documents (e.g., spam detection).

    • Text Clustering: Group similar documents.

    • Sentiment Analysis: Determine opinion polarity.

    • Summarization: Generate concise summaries.

9.3 Spatial Mining

• Goal: Discover patterns in spatial data (objects with geographic or spatial locations).

• Data Types: Points, lines, polygons, raster images.

• Challenges: Spatial autocorrelation, complex data types, large volumes.

• Tasks:

  • Spatial Association Rules: Co-location patterns (e.g., “restaurants near malls”).

  • Spatial Clustering: Group spatial objects (e.g., DBSCAN for geographic data).

  • Spatial Classification: Predict class labels using spatial features.

  • Spatial Trend Detection: How attributes change over space.

9.4 Data Prediction Techniques

• Goal: Forecast future values based on historical data.

• Methods:

  • Regression:

    • Linear Regression: $$\displaystyle y = \beta_0 + \beta_1 x_1 + ... + \epsilon $$

    • Polynomial Regression: Non-linear relationships.

  • Time Series Forecasting:

    • ARIMA (AutoRegressive Integrated Moving Average): For stationary/non-stationary series.

    • Exponential Smoothing: Weighted averages of past observations.

  • Neural Networks: Multi-layer perceptrons for complex non-linear patterns.

• Applications: Sales forecasting, stock price prediction, demand planning.

10.0 Special Topics and Integration

10.1 Multidimensional Data Modeling for Business Continuity

• Use star/snowflake schemas to model critical business processes (e.g., finance, supply chain).

• Ensure data marts for key functions are available during disruptions.

• Leverage historical data for trend analysis post-disaster to improve recovery strategies.

10.2 Data Mining in Disaster Recovery Planning Context

• Risk Prediction: Spatial mining to identify disaster-prone areas (floods, earthquakes).

• Failure Analysis: Association rules to find common causes of system failures (e.g., “server crash + high load → data corruption”).

• Clustering: Group similar incident types for proactive maintenance.

• Pattern Discovery: From past disaster logs to optimize recovery time objectives (RTO).

10.3 Efficiency Optimization in Large-scale Data Mining

• Parallel/Distributed Mining: Use MapReduce, Spark for big data.

• Sampling: Work on representative subsets when full data is too large.

• Dimensionality Reduction: PCA, feature selection to reduce attributes.

• Incremental Mining: Update models without reprocessing entire dataset (e.g., streaming data).

10.4 Evaluation Metrics for Data Mining Models in IT Business

• Classification: Accuracy, Precision, Recall, F1-score, AUC-ROC.

• Clustering: Silhouette coefficient, Davies-Bouldin index, cluster purity.

• Association: Support, Confidence, Lift, Conviction.

• Regression: MSE, RMSE, MAE, R-squared.

• Business Impact: Cost-benefit analysis, ROI, reduction in operational downtime.

[!TIP] Always align metrics with business objectives (e.g., in fraud detection, prioritize recall over accuracy).

END OF UNIT 3 NOTES
Based on RGPV past papers (Jun 2025, May 2023, Dec 2024). Focus on definitions, algorithms with examples, diagrams, and comparative analysis.