Dataware Housing & Mining (CY-604 (C)) - Unit 5 Short Notes

UNIT 5: DATA WAREHOUSING & MINING

I. DATA WAREHOUSE FUNDAMENTALS & ARCHITECTURE

A. Core Definition & Characteristics

• Definition: A Data Warehouse (DWH) is a subject-oriented, integrated, time-variant, and non-volatile collection of data in support of management's decision-making process.

• Characteristics (IN-ORDER):

  1. Subject-Oriented: Organized around key subjects (e.g., Customer, Product, Sales) rather than applications. Provides a unified view.

  2. Integrated: Constructed by integrating data from multiple, heterogeneous sources (e.g., relational DBs, flat files). Resolves naming inconsistencies, data type conflicts.

  3. Time-Variant: Data is stored with a time stamp or time period. Historical data is preserved and not updated/deleted. Enables trend analysis.

  4. Non-Volatile: Data is not routinely updated or deleted by users. Only two operations: initial loading and access. Stable, read-only environment.

B. DWH Architecture & Components

• Three-Tier Architecture:

  1. Bottom Tier (Data Source Tier): Heterogeneous data sources (operational DBs, legacy systems, external data).

  2. Middle Tier (Data Warehouse Server Tier): ETL (Extract, Transform, Load) tools and the Warehouse Database itself. Performs data consolidation, cleaning, integration.

  3. Top Tier (Front-End Tier): Query and Analysis Tools (OLAP tools, data mining tools, reporting tools, dashboards).

• Key Components:

  • Data Sources: Operational databases, flat files.

  • Staging Area: Temporary storage for raw data during ETL. Used for data cleaning and integration.

  • Data Warehouse: Central repository of integrated, subject-oriented, time-variant data.

  • Data Marts: Subsets of the DWH tailored for a specific business line or team (e.g., Sales Data Mart). Can be dependent (from DWH) or independent.

  • Metadata Repository: "Data about the data." Defines source, structure, transformations, usage. Crucial for understanding and managing the DWH.

  • Access Tools: Tools for querying, reporting, analyzing, and mining the warehouse data.

[!TIP] Exam Focus: Be prepared to draw and label the three-tier architecture, clearly showing data flow from sources → staging → DWH → data marts → access tools.

C. Schema Designs for Multidimensional Databases

• 1. Star Schema

  • Structure: One large central Fact Table connected to multiple denormalized Dimension Tables. Fact table contains foreign keys to dimensions and measures (quantitative data, e.g., sales amount). Dimensions contain descriptive attributes.

  • Advantages: Simple to understand and design; High query performance (fewer joins).

  • Disadvantages: Data redundancy in dimensions (denormalization); Difficult to maintain consistency.

  • Example: Sales_Fact (Prod_Key, Time_Key, Store_Key, Cust_Key, Units_Sold, Revenue) linked to Product_Dim, Time_Dim, Store_Dim, Customer_Dim.

• 2. Snowflake Schema

  • Structure: Normalized version of the Star Schema. Dimensions are decomposed into multiple related tables, forming a snowflake pattern.

  • Advantages: Reduces data redundancy; Easier to maintain (update in one place).

  • Disadvantages: More complex queries (more joins); Potential performance degradation.

  • Comparison with Star Schema:

    | Feature | Star Schema | Snowflake Schema | | :--- | :--- | :--- | | Dimension Normalization | Denormalized | Normalized | | Query Complexity | Simpler (fewer joins) | Complex (more joins) | | Data Redundancy | High | Low | | Maintenance | Harder | Easier | | Performance | Generally faster | Can be slower |

• 3. Galaxy Schema (Fact Constellation Schema)

  • Structure: Multiple Fact Tables that share Dimension Tables. It's a collection of star schemas.

  • Use Case: When a business has multiple, interconnected decision processes (e.g., Sales and Inventory facts sharing Product and Time dimensions).

  • Implications: More complex design but avoids redundant dimension tables. Can model complex business scenarios.

II. DATA WAREHOUSE IMPLEMENTATION & OPERATIONAL TECHNOLOGIES

A. Implementation Techniques

1. Separate DWH: Build a single, enterprise-wide DWH. Costly, time-consuming, but provides unified view.

2. Data Marts First (Bottom-Up): Build individual data marts for specific departments. Faster, cheaper. Risk of inconsistent data ("islands of information"). Often later integrated into a DWH.

3. Hybrid Approach: Combine both. Start with a core DWH infrastructure and build dependent data marts from it.

B. OLAP (Online Analytical Processing)

• Definition: Technology that enables analysts and managers to interactively analyze multidimensional data from multiple perspectives.

• OLAP vs. OLTP:

  | Feature | OLTP (Online Transaction Processing) | OLAP (Online Analytical Processing) | | :--- | :--- | :--- | | Purpose | Support day-to-day operations (insert, update, delete) | Support decision-making (complex queries, analysis) | | Data | Current, detailed, volatile | Historical, summarized, integrated, non-volatile | | Schema | Highly normalized (3NF/BCNF) | Denormalized (Star/Snowflake) | | Operations | Short, fast transactions | Long, complex queries (aggregations) | | Users | Clerks, operational staff | Managers, analysts, executives |

• OLAP Operations (on a multidimensional cube):

  • Roll-up (Drill-up): Aggregation up the hierarchy (e.g., City → Country → Region).

  • Drill-down (Roll-down): Navigation down the hierarchy (e.g., Year → Quarter → Month).

  • Slice & Dice: Slice: Fix a dimension to view a sub-cube (e.g., Sales in 2023). Dice: Select a sub-cube by specifying ranges on multiple dimensions.

  • Pivot (Rotate): Reorient the cube to view data from a different perspective (swap rows and columns).

• OLAP Types:

  • MOLAP (Multidimensional OLAP):

    • Need: Fastest query performance for dense, aggregated data.

    • Architecture: Uses proprietary multidimensional databases (MDDBs). Data is stored in optimized multidimensional array format (cubes). Pre-computes aggregates.

    • Storage: Compact, efficient for dense data. Can have sparse storage issues.

    • Diagram:

      DiagramSEARCH: MOLAP architecture diagram showing client, OLAP server, multidimensional storage

  • ROLAP (Relational OLAP):

    • Concept: Uses relational DBMS (RDBMS) as backend. Maps multidimensional operations to SQL queries on relational tables (star/snowflake schema). No pre-computation of all aggregates.

    • Architecture with Diagram:

      DiagramCANVAS: ROLAP Architecture. Show: 1) Client/Query Tool, 2) OLAP Server (middleware that translates MD queries to SQL), 3) Relational Database (storing fact/dimension tables). Arrows show query flow.

    • Advantage: Handles large, sparse data well. Scalable. Uses standard RDBMS.

    • Disadvantage: Query performance can be slower than MOLAP for complex aggregations.

  • HOLAP (Hybrid OLAP): Combines MOLAP and ROLAP. Stores detailed data in RDBMS, aggregated data in MOLAP store.

C. OLAP Servers

• Role: Middleware layer between the front-end tools and the backend database. It understands multidimensional data models and operations, translating them into appropriate queries for the underlying storage (relational or multidimensional).

• Types: Correspond to OLAP types: MOLAP Servers, ROLAP Servers, HOLAP Servers.

D. Data Cube Computation

• Process: Pre-computing all possible aggregate queries (group-bys) for a given set of dimensions to enable fast OLAP querying.

• Lattice of Cuboids: A lattice is a directed acyclic graph where nodes represent cuboids (subsets of dimensions) and edges represent "roll-up" or "drill-down" operations. Computing the lattice means computing all cuboids.

• Aggregate Computation Methods:

  • Full Computation: Compute all cuboids. Simple but expensive (2^n for n dimensions).

  • Efficient Methods: Use the lattice structure to avoid recomputation.

    • Bottom-up (Compute from smallest to largest): e.g., Compute (A,B,C) from (A,B) and (C).

    • Top-down (Compute from largest to smallest): e.g., Compute (A,B) from (A,B,C).

    • Shared Computation: Compute multiple aggregates simultaneously.

  • Indexing: Use indexes (bitmap, join) on the computed cube to speed up queries.

III. DATA PREPROCESSING FOR DATA WAREHOUSING & MINING

A. Data Cleaning

• Definition: The process of detecting and correcting (or removing) corrupt, inaccurate, or irrelevant data from a dataset.

• Techniques in Brief:

  • Missing Data: Ignore tuple (if few), fill manually, use mean/median/mode, use global constant, predict using ML.

  • Noisy Data: Smoothing (binning, regression, clustering), outlier detection (boxplot, clustering), binning.

  • Inconsistent Data: Check constraints (range, uniqueness), use domain knowledge, resolve duplicates.

B. Data Transformation

• Methods:

  • Smoothing: Remove noise (see above).

  • Attribute Construction: Create new attributes from existing ones (e.g., Area = Length * Width).

  • Aggregation: Summarize data (e.g., daily sales → monthly sales).

  • Normalization: Scale numeric values to a small range (e.g., Min-Max to [0,1], Z-score).

  • Discretization: Convert continuous attributes to categorical (e.g., age → youth/adult/senior). Methods: binning, histogram analysis, clustering, decision trees.

  • Concept Hierarchy Generation: Map low-level values to higher-level, more general concepts (e.g., city → state → country). Can be manual or automatic (e.g., via clustering).

C. Vertical Partitioning

• Definition: Splitting a table vertically (by columns/attributes) into smaller tables. Each new table contains a subset of the original columns but all rows.

• How it's done: Identify groups of attributes that are often accessed together. Create separate tables for these groups, linked by the primary key.

• Purpose: Improves query performance for queries accessing only a subset of columns (reduces I/O). Used in columnar databases and some DWH designs.

IV. INTRODUCTION TO DATA MINING & KDD PROCESS

A. Data Mining Definition & Task Primitives

• Definition: The process of discovering interesting knowledge (patterns, models, associations) from large amounts of data stored in databases, data warehouses, or other repositories.

• Task Primitives (Specify a DM task):

  1. Task-relevant data: Subset of database (attributes, tuples) to be mined.

  2. Background knowledge: Domain knowledge, concept hierarchies, constraints.

  3. Interestingness measures: Thresholds for measures like support, confidence, accuracy to filter patterns.

  4. Representation of patterns/output: Type of pattern to be mined (classification rule, association rule, cluster).

  5. Visualization of discovered patterns: How results should be presented (graphs, tables).

B. KDD (Knowledge Discovery in Databases) Process

• Steps:

  1. Selection: Define the problem, select target data from DB.

  2. Preprocessing: Clean, handle missing data, remove noise.

  3. Transformation: Convert data to appropriate format (normalization, feature construction).

  4. Data Mining: Apply DM algorithms to find patterns.

  5. Interpretation/Evaluation: Interpret mined patterns, evaluate for interestingness and usefulness.

• Role of Data Mining Engine: It is the core component of the KDD process (Step 4). It executes the selected DM algorithms (e.g., Apriori, decision tree builder) on the preprocessed and transformed data to generate the raw patterns/models.

C. Pros and Cons of Data Mining

• Pros/Advantages:

  • Automated prediction of trends/behaviors.

  • Discovery of previously unknown patterns.

  • Improved decision-making (customer segmentation, risk analysis).

  • Increased revenue, reduced costs.

• Cons/Disadvantages:

  • Privacy concerns: Mining personal data.

  • Security risks: Misuse of discovered knowledge.

  • Information misuse: Discrimination, unfair practices.

  • Misinterpretation: Patterns may be spurious or not causally related.

  • Cost: Tools, expertise, infrastructure.

D. Types of Data for Data Mining

1. Relational Data: Traditional tables with rows and columns.

2. Transactional Data: Each record is a transaction (e.g., market basket). Contains itemsets.

3. Data Warehouse Data: Integrated, subject-oriented, historical data (multidimensional).

4. Flat-file Data: Simple text files, CSV.

5. Streaming Data: Continuous, rapid, time-sensitive (e.g., sensor data, stock ticks). Requires online/incremental mining.

6. Spatial/Temporal Data: Data with geographic or time references.

7. Text Data: Unstructured or semi-structured documents.

8. Web Data: Hypertext, web logs, structure.

V. ASSOCIATION RULE MINING (FREQUENT PATTERN MINING)

A. Core Concepts

• Association Rule: An implication of the form X → Y (X and Y are itemsets, X ∩ Y = ∅).

• Support (σ): P(X ∪ Y). Proportion of transactions containing both X and Y. Measures frequency.

$$supp(X \rightarrow Y) = \frac{| \{t \in D : X \cup Y \subseteq t\} |}{|D|}$$

• Confidence (c): P(Y|X). Proportion of transactions with X that also contain Y. Measures strength.

$$conf(X \rightarrow Y) = \frac{supp(X \cup Y)}{supp(X)}$$

• Frequent Itemset: An itemset whose support ≥ min_sup threshold.

• Strong Association Rule: A rule that satisfies both min_sup and min_conf thresholds.

B. Apriori Algorithm

• Core Principle (Apriori Property): All non-empty subsets of a frequent itemset must also be frequent. (If {A,B,C} is frequent, then {A,B}, {A,C}, {B,C} must be frequent).

• Steps:

  1. Find frequent itemsets (L_k): Use iterative, level-wise approach.

     • k=1: Scan DB, count single items, find L₁ (items with sup ≥ min_sup).

     • k>1: Generate candidate k-itemsets C_k from L_{k-1} (join step). Prune C_k using Apriori property (any subset not in L_{k-1} is removed). Scan DB to count support of candidates in C_k, form L_k.

     • Stop when L_k is empty.

  2. Generate strong rules: For each frequent itemset l (size ≥ 2), generate all non-empty subsets. For each subset A, form rule A → (l - A) if conf ≥ min_conf.

• Example: Given DB, min_sup=50%, min_conf=70%. Find L₁, L₂, L₃. Generate rules from L₃ (e.g., {Bread, Milk} → {Butter}).

C. Techniques to Improve Apriori Efficiency

1. Hash-based Technique: Use hash tables during candidate generation to prune candidates early based on hash bucket counts.

2. Transaction Reduction (Reduction of DB size): Remove transactions that do not contain any candidate k-itemsets before the next scan.

3. Partitioning: Partition DB into disjoint chunks. Find local frequent itemsets in each partition (must be globally frequent). Candidates are union of local frequent itemsets. Final scan verifies.

4. Sampling: Mine a random sample of DB. Use lower min_sup to find local patterns, then verify on full DB.

5. Dynamic Item Counting: Add new candidates during a single DB scan based on counts of subsets seen so far.

D. FP-Growth Algorithm

• Idea: Avoid candidate generation and multiple DB scans. Uses a compact data structure: FP-Tree.

• Steps:

  1. Scan DB once: Find frequent items (L₁), order by descending support.

  2. Build FP-Tree:

     • Create root null.

     • For each transaction, sort frequent items by L₁ order, insert into tree. Share common prefixes. Nodes store item name and count.

     • Also build header table linking all nodes of the same item.

  3. Mine FP-Tree (Pattern Growth):

     • For each item in header table (starting from least frequent), construct conditional pattern base (all paths from root to that item).

     • Build conditional FP-Tree from conditional pattern base.

     • Recursively mine conditional FP-Trees. Patterns are formed by suffixing the item to patterns from conditional tree.

• Example: Build FP-Tree for a small DB. Show mining process for item milk.

E. Specialized Association Mining

• Boolean Association Rule Mining: Rules on binary/categorical attributes. Can be converted from transactional data (each item is a binary attribute).

• Multilevel Association Rules: Rules involving items at different abstraction levels in a concept hierarchy. Use **support* and confidence* thresholds at each level to avoid too many rules at low levels.

• Time Series Mining Association Rules: Find associations between events over time (e.g., price_up → demand_down after 2 days). Requires handling temporal ordering and intervals.

VI. CLASSIFICATION (PREDICTIVE MODELING)

A. General Process

• Learning Phase (Training):

  1. Build model from training dataset (tuples with known class labels).

  2. Model learns patterns/relationships between attributes and the class.

• Verification/Accuracy Assessment:

  • Holdout Method: Split data into training set (e.g., 2/3) and test set (1/3). Train on train, evaluate on test.

  • Cross-Validation (k-fold): Partition data into k equal folds. Use k-1 folds for training, 1 for testing. Repeat k times. Average accuracy.

  • Bootstrap: Sample with replacement to create training sets (same size as original). Evaluate on unseen samples.

• Metrics:

  • Accuracy: (TP + TN) / Total

  • Precision (Positive Predictive Value): TP / (TP + FP) (How many selected are relevant?)

  • Recall (Sensitivity): TP / (TP + FN) (How many relevant are selected?)

  • F1-Score: Harmonic mean: 2 * (Precision * Recall) / (Precision + Recall)

  • Confusion Matrix: Table showing TP, TN, FP, FN.

B. Decision Tree Induction

• Algorithm (e.g., ID3, C4.5, CART):

  1. Start with all training examples at root.

  2. Select best attribute to split on using an attribute selection measure.

  3. Create branches for each value of the attribute.

  4. Partition examples among branches.

  5. Recurse on each branch with remaining attributes until:

     • All examples in a branch belong to the same class.

     • No more attributes.

     • No examples left (use majority class of parent).

• Attribute Selection Measures:

  • Information Gain (ID3/C4.5): Based on Entropy.

$$Entropy(t) = -\sum_{i=1}^{c} p_i(t) \log_2 p_i(t)$$

$$Gain(D, A) = Entropy(D) - \sum_{v \in Values(A)} \frac{|D_v|}{|D|} Entropy(D_v)$$

*   **Gain Ratio (C4.5):** Corrects Information Gain's bias towards many-valued attributes.

$$Gain\_Ratio(D, A) = \frac{Gain(D, A)}{SplitInfo_A(D)}$$

    where `SplitInfo_A(D) = -\sum_{v} \frac{|D_v|}{|D|} \log_2 \frac{|D_v|}{|D|}`

*   **Gini Index (CART):** Measures impurity.

$$Gini(D) = 1 - \sum_{i=1}^{c} (p_i)^2$$

    For split: `Gini_A(D) = \sum_{v} \frac{|D_v|}{|D|} Gini(D_v)`

• Tree Pruning:

  • Pre-pruning (Early stopping): Stop tree growth early (e.g., min samples per leaf, max depth, min gain).

  • Post-pruning: Grow full tree, then remove branches. Methods: Error rate pruning, cost-complexity pruning.

C. Rule-Based Classification

• Algorithms: RIPPER (Repeated Incremental Pruning to Produce Error Reduction), CN2.

• Idea: Learn a set of IF-THEN rules instead of a tree.

• Process (RIPPER):

  1. Growing Phase: Start with empty rule, add conditions (conjunctions) to maximize information gain (like FOIL). Stop when rule is "perfect" (covers only positive examples).

  2. Pruning Phase: Remove conditions from the rule if pruning reduces error on validation set.

  3. Repeat until all positive examples are covered or error rate too high.

• Advantage: Rules can be more understandable and independent of attribute ordering than trees.

D. Bayesian Classification

• Bayesian Theorem:

$$P(A|B) = \frac{P(B|A) \cdot P(A)}{P(B)}$$

• Naïve Bayes Classifier:

  • Assumption: Conditional Independence. Given the class, attribute values are conditionally independent.

$$P(x|C) = \prod_{i=1}^{n} P(x_i | C)$$

where `x = (x₁, x₂, ..., xₙ)` is a tuple, `C` is class.

*   **Classification:** Assign class `c` that maximizes posterior probability:

$$c_{NB} = \arg\max_c P(c) \prod_{i=1}^{n} P(x_i | c)$$

*   **Working:** For each class, compute prior `P(c)` from training data. For each attribute value `x_i` in test tuple, compute likelihood `P(x_i | c)` (using Gaussian for continuous, frequency for discrete). Multiply and compare.

*   **Example:** Classify weather (play tennis?) given outlook, temperature, humidity, wind.

E. Probabilistic Classifiers

• Definition: Classifiers that predict a probability distribution over classes, not just a single class label. Outputs P(C|x) for all classes C.

• Explanation: Naïve Bayes is a probabilistic classifier. Others include Logistic Regression (models log(P(C=1|x)/P(C=0|x)) as linear function), Bayesian Networks (graphical models representing conditional dependencies).

• Advantage: Provides confidence measure for predictions.

F. Other Classifiers (Brief)

• k-Nearest Neighbors (k-NN): Instance-based. Classify a point based on majority class among its k nearest neighbors (using Euclidean distance). Lazy learner.

• Support Vector Machines (SVM): Finds optimal hyperplane that maximizes margin between classes. Uses kernel trick for non-linear data.

VII. CLUSTERING (UNSUPERVISED LEARNING)

A. Definition & Types of Clusters

• Definition: Grouping a set of objects such that objects in the same group (cluster) are more similar to each other than to those in other groups.

• Types of Clusters:

  • Well-separated: Clusters are far from each other.

  • Center-based: Clusters are represented by a centroid (e.g., k-means).

  • Contiguity-based: Clusters are connected regions (e.g., DBSCAN).

  • Density-based: Clusters are dense regions separated by sparse regions.

B. Important Clustering Methods

• 1. Partitioning Methods:

  • k-Means:

    • Input: k (number of clusters).

    • Steps:

      1. Initialize k centroids (randomly).

      2. Assign each point to nearest centroid.

      3. Recompute centroids as mean of assigned points.

      4. Repeat 2-3 until convergence (centroids stable).

    • Complexity: O(n * k * I * d) (n=points, k=clusters, I=iterations, d=dimensions).

    • Pros: Simple, efficient for large datasets.

    • Cons: Sensitive to initial centroids, outliers, spherical clusters, requires k.

  • k-Medoids (PAM - Partitioning Around Medoids): Uses actual data points as cluster centers (medoids). More robust to noise/outliers. More expensive.

• 2. Hierarchical Methods:

  • Agglomerative (Bottom-up): Start with each point as its own cluster. Repeatedly merge closest pairs of clusters until one cluster or k clusters.

  • Divisive (Top-down): Start with all points in one cluster. Repeatedly split clusters until each point is its own cluster or k clusters.

  • Differentiation:

    | Aspect | Agglomerative (Bottom-up) | Divisive (Top-down) | | :--- | :--- | :--- | | Starting Point | n clusters (each point) | 1 cluster (all points) | | Approach | Merge | Split | | Computational Cost | Generally lower | Generally higher (needs to evaluate many splits) | | Common Algorithms | AGNES, Single/Complete/Average Linkage | DIANA, Bisecting K-Means |

  • Linkage Criteria (for Agglomerative):

    • Single Link (Min): Distance between closest points in two clusters. Chaining effect.

    • Complete Link (Max): Distance between farthest points. Compact clusters.

    • Average Link: Average pairwise distance. Balanced.

• 3. Density-Based Methods:

  • DBSCAN (Density-Based Spatial Clustering of Applications with Noise):

    • Core Concepts:

      • ε-neighborhood: Region within radius ε of a point.

      • MinPts: Minimum number of points in ε-neighborhood to be a core point.

      • Directly density-reachable: Point p is directly reachable from q if q is core and p is in q's ε-neighborhood.

      • Density-connected: Two points are connected if there is a path of core points between them.

    • Clusters: Maximal set of density-connected points.

    • Noise: Points not reachable from any core point.

    • Advantages: Discovers clusters of arbitrary shape, robust to outliers, no need to specify k.

• 4. Grid-Based Methods:

  • STING (STatistical INformation Grid): Divides space into grid cells at multiple resolutions. Computes statistical info (mean, max, etc.) for each cell. Clustering is performed on grid cells, not individual points. Very fast.

C. Clustering Evaluation

• Internal Evaluation (No ground truth): Based on the data and clustering result itself.

  • Cohesion (Compactness): How close points within a cluster are. (e.g., SSE - Sum of Squared Errors).

  • Separation: How well-separated clusters are. (e.g., Silhouette Coefficient combines cohesion and separation).

• External Evaluation (With ground truth/class labels): Compare clustering to predefined classes.

  • Entropy: Measures purity of clusters. Low entropy = good.

  • Purity: Fraction of total objects classified correctly. High purity = good.

  • Normalized Mutual Information (NMI), Rand Index.

VIII. SPECIALIZED DATA MINING APPLICATIONS & AREAS

A. Web Usage Mining

• Definition: Application of DM techniques to discover usage patterns from web data (server logs, browser logs, user profiles) to understand user behavior.

• Process:

  1. Data Collection: Server logs, client-side tracking, proxies.

  2. Data Preprocessing: Cleaning, user identification (from IPs), session identification, path completion.

  3. Pattern Discovery: Association rules (pages visited together), sequential patterns, clustering (user groups), classification (predicting next page).

• Applications: Personalization, site redesign, marketing, adaptive websites.

B. Text Mining

• Definition: Application of DM techniques to large collections of text documents to discover patterns, topics, and knowledge.

• Key Techniques:

  1. Text Preprocessing: Tokenization, stop-word removal, stemming/lemmatization.

  2. Text Representation: Convert text to numeric vectors. Bag-of-Words, TF-IDF (Term Frequency-Inverse Document Frequency).

  3. Text Categorization (Classification): Assign documents to predefined categories (e.g., spam detection, news topic classification). Uses Naïve Bayes, SVM.

  4. Text Clustering: Group similar documents (e.g., news grouping). Uses k-means, hierarchical.

  5. Topic Modeling: Discover latent topics (e.g., LDA - Latent Dirichlet Allocation).

  6. Sentiment Analysis: Determine opinion/emotion in text.

C. Spatial Data Mining

• Definition: Extraction of knowledge, spatial relationships, and patterns from large spatial databases.

• Spatial Patterns:

  • Spatial Association Rules: Rules with spatial predicates (e.g., near, intersects). A ∧ B → C where A, B, C have spatial relationships.

  • Spatial Clusters: Clusters based on spatial proximity and non-spatial attributes (e.g., GDBSCAN).

  • Spatial Classification: Use spatial attributes (location, distance to landmarks) as features.

  • Spatial Trend Detection: How non-spatial attribute changes over space.

D. Multidimensional Data Mining

• Definition: Mining in multidimensional databases (like Data Warehouses) where data is organized in cubes (dimensions and measures).

• Characteristics: Data is pre-aggregated, integrated, and subject-oriented. Mining can be done directly on the cube.

• Techniques:

  • Multidimensional Association Rules: Rules involving dimensions and measures. Can be mined by rolling up/down the cube.

  • Multidimensional Classification: Use OLAP operations to select relevant subspaces for classification.

  • Multidimensional Clustering: Cluster on multiple dimensions/measures.

  • Prediction in Multidimensional Space: Use data cube slices for prediction models.

E. Data Prediction

• Concept: The outcome of supervised learning tasks, primarily classification (predicting categorical class labels) and regression (predicting continuous values).

• Process: A model is built from labeled training data. This model is then applied to new, unseen data to predict the unknown target value.

• Algorithms: Decision trees, neural networks, SVM, linear regression, etc. Evaluation uses metrics like Mean Absolute Error (MAE), Mean Squared Error (MSE) for regression.