Ethical Hacking (CY-703 (A)) - Unit 3 Short Notes

UNIT 3: ADVANCED DATA ENGINEERING CONCEPTS & ARCHITECTURES

1.0 Foundations & Lifecycle

1.1 Definition and Scope of Data Engineering

Data Engineering is the discipline of designing, building, and maintaining the infrastructure and systems that enable the collection, storage, processing, and serving of data at scale. It focuses on creating reliable, efficient, and scalable data pipelines to support analytics, machine learning, and business intelligence.

1.2 The Data Engineering Lifecycle (End-to-End)

A continuous cycle with the following core components:

[!TIP] Exam Focus: Be prepared to draw and explain this lifecycle diagram. Link each stage to real-world tools and scenarios.

1.3 Gartner Data Maturity Model

A framework with 5 stages to assess an organization's data capabilities:

2.0 Data Architecture Patterns & Models

2.1 Types of Data Architectures

2.2 Data Lake Patterns

• Definition: A centralized repository that stores raw, unstructured, semi-structured, and structured data at any scale, without upfront schema definition.

• Core Principle: "Schema-on-Read" vs. Data Warehouse's "Schema-on-Write".

• Common Zones:

  • Raw Zone (Bronze): Immutable, original data copy.

  • Curated Zone (Silver/Gold): Cleaned, enriched, business-ready datasets.

  • Sandbox: For data scientists to explore raw data.

• Merits: Flexibility, cost-effective storage (object stores), scalability, supports ML/advanced analytics.

• Applications: Storing IoT streams, log files, clickstream data, archival data, ML training datasets.

2.3 The Zachman Framework

A 6x6 matrix for enterprise architecture, providing a schema for organizing descriptive representations (models) of an enterprise.

• Purpose: To ensure all aspects of a system are considered from multiple perspectives.

• Rows (Perspectives - Who?):

  1. Planner (Executive) - Why? (Scope, Context)

  2. Owner (Business Manager) - What? (Business Concepts)

  3. Designer (Architect) - How? (System Logic)

  4. Builder (Engineer) - Where? (Technology)

  5. Subcontractor (Implementer) - Who? (Detailed Components)

  6. Enterprise (Functioning System) - When? (Work Processes)

• Columns (Aspects - What?):

  1. What (Data) - Inventory

  2. How (Function) - Process

  3. Where (Network) - Distribution

  4. Who (People) - Roles

  5. When (Time) - Schedule

  6. Why (Motivation) - Intent

• Application: Each cell (intersection of row & column) defines a distinct, irreducible model (e.g., "Owner's view of What (Data)"). Ensures completeness by addressing all 36 cells.

3.0 Modern Processing Architectures (Lambda & Kappa)

3.1 Lambda Architecture

Core Philosophy: Separate paths for batch and real-time (speed) processing to balance latency, throughput, and fault-tolerance. Results are merged in a serving layer. Data Flow:

1. Batch Layer (Master Dataset): Processes all historical data from immutable storage (Data Lake). Computes accurate, comprehensive views (e.g., daily aggregates). Tech: Hadoop MapReduce, Spark.

2. Speed Layer (Real-time): Processes only new data (stream) to provide low-latency, approximate views. Compensates for batch layer's latency. Tech: Kafka, Storm, Flink.

3. Serving Layer: Indexes batch views for fast querying and merges them with speed layer views to serve final results. Advantages: Fault-tolerant (batch layer is source of truth), balances accuracy & speed. Complexities: Maintaining two separate codebases (batch & speed), complex merging logic, operational overhead.

3.2 Kappa Architecture

Core Philosophy: Simplification of Lambda. Achieve both batch and real-time processing using a single streaming pipeline. Historical data is re-processed by replaying the stream. Key Components & Data Flow:

1. All data is treated as a stream.

2. Incoming data is written to a distributed, immutable log (e.g., Kafka).

3. A single stream processing engine (e.g., Apache Flink, Spark Streaming) reads the log.

   • For real-time: processes recent data with short windows.

   • For batch: reprocesses the entire log from the beginning to recompute views.

4. Results are served from a serving database (e.g., Cassandra, Druid). Comparison with Lambda:

4.0 Data Warehousing & ETL/ELT

4.1 Building a Data Warehouse (University Case Study - Step-by-Step)

1. Requirement Gathering: Identify KPIs (student pass rate, course enrollment, faculty workload). Stakeholders: Admin, Exams, Finance.

2. Source System Analysis: Student Info DB (OLTP), Course DB, Finance DB, HR DB.

3. Dimensional Modeling: Design Star Schema.

   • Fact Tables: Fact_Enrollment (student_id, course_id, date_id, marks, fee_paid).

   • Dimension Tables: Dim_Student, Dim_Course, Dim_Date, Dim_Faculty.

4. ETL/ELT Pipeline Design:

   • Extract: Pull data from source DBs nightly (batch) via CDC or API.

   • Load: Initial full load to staging area, then incremental loads.

   • Transform: Clean (null handling), conform dimensions (standardize course codes), aggregate to fact tables.

5. Tool Selection: Cloud DWH (Snowflake), Orchestration (Airflow), Transformation (dbt).

6. Implementation & Testing: Build, validate data, create BI reports (Power BI).

4.2 ETL vs. ELT

4.3 Detailed ETL Process: Retail Database to DWH (Example)

• Source OLTP: Orders (order_id, cust_id, store_id, order_date), Order_Items (item_id, order_id, prod_id, qty, price), Products (prod_id, prod_name, category).

• Target DWH Star Schema:

  • Fact_Sales (order_id, date_id, prod_id, store_id, cust_id, qty, amount).

  • Dim_Date, Dim_Product, Dim_Store, Dim_Customer.

• Process:

  1. Extract: Nightly full/incremental dump of Orders, Order_Items, Products to staging tables.

  2. Transform:

     • Clean: Remove duplicate orders, handle null cust_id.

     • Join: Order_Items x Products to get amount = qty * price.

     • Aggregate: Daily sales per product/store.

     • Conform: Map source category to standardized dimension values.

  3. Load: Insert transformed records into Fact_Sales and update dimensions (Dim_Product, etc.).

4.4 Schema Migration

• Definition: The process of changing the structure (schema) of a database or data warehouse table to accommodate new business requirements (e.g., adding a column, changing data type).

• Need: Source system changes (new discount column in Orders table) must be reflected in downstream DWH.

• Strategies for Adding a discount Column:

  1. Single-step (Downtime): Stop pipeline, alter DWH table, backfill historical data, restart. Simple but causes downtime.

  2. Dual-write (Zero-downtime):

     • Phase 1: Deploy new pipeline version that writes to both old and new schema (new column gets NULL for old data).

     • Phase 2: Backfill historical discount values (e.g., from another system or default 0).

     • Phase 3: Switch all reads to new schema, decommission old column.

  3. Backfill-first: Add nullable column to DWH, backfill historical data in background, then update pipeline to populate new column for new records.

5.0 Data Lineage & Governance

5.1 Data Lineage Tracking in DataOps

• Definition: The lifecycle of data—its origins, movements, transformations, and destinations. Answers: "Where did this data come from? How was it transformed? Where is it used?"

• Importance: Impact analysis (change a source? see affected reports), debugging pipeline errors, compliance (GDPR, CCPA), data trust.

• Types:

  • Technical/Physical Lineage: Column/table-level flow. Shows exact transformations (SQL joins, filters). Tool: OpenLineage, Marquez.

  • Business Lineage: Maps business terms (e.g., "Monthly Revenue") to technical assets (tables, columns). Bridges business and tech.

• Implementation Approaches:

  • Pattern-Based Lineage (Parsing): Automatically extracts lineage by parsing code/configs (SQL scripts, Airflow DAGs, dbt manifests).

    • Example: Parse a Spark job's .py file to identify source_table -> transform -> target_table.

  • Lineage by Data Tagging (Manual/Semi-auto): Users tag datasets with provenance metadata (source system, PII flag). Lineage inferred from tag propagation.

    • Example: Tag raw_orders as "source: Shopify". Any downstream table derived from it inherits this tag via ETL job metadata.

  [!TIP] Exam Tip: Contrast the two approaches. Pattern-based is automated but complex; tagging is simpler but manual and less granular.

5.2 Data Governance

• Core Pillars:

  • Policies & Standards: Rules (data retention), formats (naming conventions).

  • Processes: How to request data, approve changes, handle incidents.

  • Roles & Responsibilities: Data Owners, Stewards, Custodians.

• Key Components:

  • Data Quality: Profiling, monitoring (null %, duplicates), validation rules.

  • Metadata Management: Technical (schema) & Business (definitions, owners) metadata. Data Catalog is a key tool.

  • Master Data Management (MDM): Single source of truth for critical entities (Customer, Product).

  • Data Security & Privacy: Access controls (RBAC), encryption, masking PII, compliance (GDPR).

• Implementation Strategy: Start with a pilot (one domain), define clear policies, implement a catalog tool, establish stewardship council, measure data quality scores, iterate.

6.0 Streaming & Time-Series Data

6.1 Real-Time Data Processing with Lambda Architecture (Recap & Deep Dive)

• Batch Layer: Computes authoritative results from all data (e.g., total sales per product ever). Outputs to a serving DB (e.g., HBase) for querying. High latency (hours).

• Speed Layer: Computes real-time delta (e.g., sales in last 5 minutes) from the stream. Outputs to a real-time DB (e.g., Redis). Low latency (seconds).

• Serving Layer: Merges results: Final_View = Batch_View + Speed_View. Query hits this merged view.

• Example: Website clickstream.

  • Batch: Hourly job computes unique users per page from all history.

  • Speed: Stream job counts clicks in last 1 min.

  • Serve: Dashboard shows "Total Unique Users (Batch) + Active Now (Speed)".

6.2 Detecting Patterns in Time-Series Data using Streaming Architecture

• Characteristics of Time-Series Data: Sequential, time-indexed, often high-frequency (metrics, logs), with trend, seasonality, noise, anomalies.

• Common Patterns:

  • Trend: Long-term increase/decrease.

  • Seasonality: Fixed-period cycles (daily, weekly).

  • Anomalies: Sudden spikes/dips (server failure, fraud).

• Streaming Techniques:

  • Windowed Aggregations: Compute metrics over sliding/tumbling/session windows.

    • Example: error_count = COUNT(*) WHERE status=500 OVER (LAST 5 MINUTES SLIDING EVERY 1 MIN).

  • Sessionization: Group events into sessions (e.g., user activity until 30-min inactivity).

• Tools: Apache Flink (stateful, exactly-once, event-time processing), Spark Structured Streaming.

• Example: Detecting Server Error Spike

  1. Stream: Server logs (timestamp, server_id, status_code).

  2. Flink Job: Key by server_id. Apply a tumbling window of 5 minutes. Filter status_code >= 500. Count errors.

  3. Alert: If error_count > 100 in window, trigger PagerDuty alert.

7.0 Data Acquisition & Integration Techniques

7.1 Webhooks

• Definition: An event-driven HTTP callback where a source application pushes data to a pre-configured URL (your endpoint) when an event occurs.

• How They Work:

  1. You provide a URL (endpoint) to the source app (e.g., GitHub, Stripe).

  2. Event happens (e.g., repo push, payment succeeded).

  3. Source app sends an HTTP POST request with a payload (JSON/XML) to your URL.

  4. Your endpoint receives and processes the data.

• Use Cases: Payment notifications (Stripe), CI/CD triggers (GitHub), chat alerts (Slack), IoT sensor events.

• Security Considerations:

  • Secret/Token Validation: Verify request signature (e.g., Stripe's sig_header) using a shared secret.

  • HTTPS: Always use TLS.

  • IP Whitelisting: Restrict source IPs if possible.

  • Idempotency: Handle duplicate events (check event ID).

7.2 Web Scraping

• Definition: Programmatically extracting data from websites by parsing HTML/XML.

• Legal/Ethical Considerations:

  • Check robots.txt (e.g., https://www.rgpvonline.com/robots.txt).

  • Review Terms of Service.

  • Respect rate limits (don't hammer servers).

  • Do not scrape personal/PII data without consent.

• Techniques:

  • HTML Parsing: Use DOM parsers (BeautifulSoup, lxml).

  • API Simulation: Reverse-engineer site's internal APIs (browser DevTools > Network tab).

  • Headless Browsers: Render JavaScript pages (Selenium, Puppeteer).

• Tools: Python: requests + BeautifulSoup (static), Scrapy (framework), Selenium (dynamic).

• Illustrative Scenario: rgpvonline.com

  
  # Pseudo-code for scraping press releases about "data"
  
  import requests
  
  from bs4 import BeautifulSoup
  
  url = "https://www.rgpvonline.com/press-releases"
  
  response = requests.get(url, headers={'User-Agent': 'MyBot'})
  
  soup = BeautifulSoup(response.content, 'html.parser')
  
  releases = soup.find_all('div', class_='press-release')
  
  for release in releases:
  
      title = release.find('h2').text
  
      if "data" in title.lower():
  
          rep_name = release.find('span', class_='representative').text
  
          print(f"Representative: {rep_name}, Title: {title}")
  
  

  • Steps: 1. Fetch page. 2. Parse HTML. 3. Find all release elements. 4. Filter titles containing "data". 5. Extract representative name.

8.0 Operational Excellence & Security

8.1 Logging, Monitoring, and Alerting

• Importance: Ensures reliability, debuggability, and performance of data pipelines. Detect failures early (data latency, quality drops).

• What to Log:

  • Pipeline Stages: Job start/end timestamps, row counts.

  • Data Metrics: Volume ingested, null %, duplicate count.

  • Errors & Exceptions: Stack traces, failed record IDs.

  • System Metrics: CPU, memory, disk I/O.

• Monitoring Tools & Dashboards:

  • Cloud-Native: AWS CloudWatch, GCP Operations Suite.

  • Open Source: Grafana (visualization) + Prometheus (metrics) + Loki (logs).

  • Data-Specific: Monte Carlo, Bigeye (data observability).

• Setting up Alerts:

  • Threshold-based: "Alert if ETL job duration > 2 hours".

  • Anomaly Detection: "Alert if daily row count drops >20% vs. 7-day avg".

  • Data Freshness: "Alert if last successful data load > 1 hour ago".

• Example Alert Rule (Grafana/Prometheus):

  
  alert: ETLJobFailed
  
  expr: job_status{job="daily_etl"} == 0
  
  for: 5m
  
  labels:
  
    severity: critical
  
  annotations:
  
    summary: "Daily ETL job failed"
  
    description: "The daily ETL pipeline has been failing for 5 minutes."
  
  

8.2 Secure Copy Protocol (SCP)

• Definition: A network protocol based on SSH for secure file transfer between a local and remote host or between two remote hosts.

• Basic Syntax:

  
  scp [options] source_file user@remote_host:/destination_path
  
  # Example: Copy local file to remote server
  
  scp report.csv [email protected]:/data/inbound/
  
  # Copy remote directory to local
  
  scp -r user@server:/data/archive ./local_backup/
  
  

• How it Works: Uses SSH for authentication and encryption. Establishes an encrypted tunnel, then copies the file over it. No separate daemon; uses SSH's port 22.

• Use Cases in Data Engineering: Securely move data files (CSV, Parquet) between on-prem servers and cloud VMs, backup config files, distribute scripts.

• Comparison:

  | Feature | SCP | SFTP | rsync | | :--- | :--- | :--- | :--- | | Protocol | SSH-based copy | SSH File Transfer Protocol | Differential sync | | Resume | No | Yes | Yes (delta transfer) | | Directory Recursion | -r flag | Built-in | -a (archive) | | Best For | Simple, one-time secure copy. | Interactive file management. | Efficient sync/backup (only changes). |

• Security Best Practices:

  • Use SSH Key-based Auth (disable password).

  • Limit User Permissions (chroot jail if possible).

  • Use -p flag to preserve timestamps (for audits).

  • Prefer rsync over SSH (rsync -e ssh) for large datasets to resume and minimize transfer.