Cyber Security Policies & Standards (CY-703 (B)) - Unit 2 Short Notes

UNIT 2: Data Engineering & Governance for Cyber Security Policies & Standards

1.0 Data Engineering Fundamentals & Lifecycle

Data Engineering is the discipline of designing, building, and maintaining systems for data collection, storage, processing, and serving to enable analytics and decision-making. It provides the foundational infrastructure for data-driven security policies.

1.2 The Data Engineering Lifecycle

A continuous cycle with key stages:

1. Data Generation & Acquisition: Sources (IoT, apps, logs, external APIs). Policy Focus: Data ownership, lawful basis for collection (GDPR).

2. Data Ingestion: Moving data into the storage system.

   • Batch Ingestion: Scheduled, large-volume transfers (e.g., nightly). Tool Example: Apache Sqoop.

   • Streaming Ingestion: Continuous, real-time flow of small records. Tool Example: Apache Kafka.

3. Data Storage: Choosing the right repository.

   • Data Warehouse: Structured, processed data (Schema-on-Write). Examples: Snowflake, Redshift.

   • Data Lake: Raw, unstructured/semi-structured data (Schema-on-Read). Storage: Object stores (AWS S3, ADLS).

   • Lakehouse: Hybrid model combining features of both. Example: Delta Lake on Databricks.

4. Data Processing & Transformation:

   • ETL (Extract, Transform, Load): Transform data before loading into target. Good for structured targets.

   • ELT (Extract, Load, Transform): Load raw data first, transform later. Leverages target system's power (common in cloud DWHs).

5. Data Serving & Consumption: Providing data to end-users/applications via BI tools, APIs, or ML pipelines.

6. Data Security & Privacy Integration: Embedding Policy Enforcement Points (PEPs) at each stage (e.g., encryption at rest/in transit, access controls, data masking).

[!TIP] Exam Focus: Be prepared to draw and explain this lifecycle, explicitly linking each stage to security policy considerations (e.g., "Ingestion requires validation against data quality policies").

1.3 Real-Time vs. Batch Processing

2.0 Data Architectures & Patterns

2.1 Traditional Data Warehouse: Inmon vs. Kimball

2.2 Data Lake Architecture & Patterns

Core Concept: A centralized repository storing raw, unprocessed data in its native format (object storage). Uses Zones to manage data:

• Raw / Landing Zone: Immutable copy of source data.

• Curated / Trusted Zone: Cleansed, validated, conformed data (ready for analysis).

• Consumption / Sandbox Zone: Data marts, feature stores for specific use cases.

Merits:

• Scalability & Cost: Cheap object storage, scale infinitely.

• Flexibility: Store any data type; schema applied on read.

• Future-Proofing: Preserve raw data for future, unforeseen questions.

Challenges & Best Practices:

• Zombie Data Lake: Becomes an unmanaged "data swamp." Best Practice: Enforce governance (metadata, catalog, access controls) from Day 1.

• Security: Requires robust IAM, encryption, and network controls on object storage.

• Performance: Query performance can be poor without proper indexing/format (Parquet/ORC).

2.3 Lambda Architecture

A unified architecture for both batch and real-time processing of large datasets.

Three Layers:

1. Batch Layer: Processes the entire historical dataset to create accurate, comprehensive views (e.g., daily). Stores output in a serving layer.

2. Speed Layer: Processes new data in real-time to provide low-latency, approximate views (compromises accuracy for speed).

3. Serving Layer: Indexes the batch views and merges them with real-time views to serve queries.

Step-by-Step Real-Time Processing:

1. New event arrives → sent to both Batch Layer (for historical completeness) and Speed Layer (for immediate insight).

2. Speed Layer updates a real-time view (e.g., in Redis).

3. Batch Layer periodically (e.g., nightly) recomputes the entire view from scratch, overwriting the serving layer.

4. Query merges results from the Serving Layer's batch view and the Speed Layer's real-time view.

Advantages: Fault-tolerant (batch layer corrects speed layer errors), balances latency & accuracy. Disadvantages: High complexity (maintain two codebases, merge logic).

2.4 Kappa Architecture

A simplification of Lambda that uses only a single stream processing path.

Core Idea: Treat all data as a stream. Historical data is replayed from a persistent log (e.g., Kafka) when needed.

Components & Workflow:

1. All incoming data is written to a central, immutable log (Kafka).

2. A stream processing engine (e.g., Apache Flink, Spark Streaming) continuously reads from this log.

3. The stream processor writes serving-ready views directly to a serving database (e.g., Elasticsearch, Cassandra).

4. To recompute a view (like batch layer), simply replay the log from the beginning with the same stream processing job.

Use Cases: Real-time dashboards, alerting, applications where eventual consistency is acceptable. Trade-offs: Simpler codebase, but replay can be expensive for very long histories; not ideal for extremely complex batch computations.

[!TIP] Comparison: Lambda = Accuracy + Latency (via two paths). Kappa = Simplicity + Latency (single path, replay for accuracy).

2.5 Modern Data Stack & Cloud-Native Architectures

• Cloud-Native: Built on cloud services (AWS, Azure, GCP) leveraging serverless, managed services (e.g., S3, BigQuery, Glue).

• Modern Data Stack: Integrated suite of cloud-based tools (Fivetran for ingestion, dbt for transformation, Snowflake for storage, Looker for BI).

• Policy Implication: Cloud responsibility model (Shared Responsibility) defines security controls: Provider secures infrastructure, Customer secures data, access, and configurations.

3.0 Data Governance, Maturity & Quality

3.1 Data Governance Framework

Definition: The overall management of data availability, usability, integrity, and security within an enterprise. It's the "how" of executing data strategy.

Objectives: Ensure data is trustworthy, secure, compliant, and valuable.

Three Pillars:

1. People: Roles (Data Owner, Steward, Custodian), accountability.

2. Process: Policies, standards, procedures, issue resolution.

3. Technology: Tools (catalogs, lineage, quality monitors, access control).

Key Components:

• Policies & Standards: High-level rules (e.g., "PII must be encrypted").

• Data Stewardship: Operational oversight by domain experts.

• Metadata Management: "Data about data" (technical, business, operational).

• Data Quality Management: Monitoring and improving accuracy, completeness, etc.

Role in Compliance: Provides audit trails, data lineage, and access logs required by regulations (GDPR's "right to be forgotten," HIPAA's data access logs).

3.2 Gartner Data Maturity Model

A 5-level model assessing an organization's data management sophistication.

Roadmap: Progress requires executive sponsorship, iterative pilots, and tool investment. Move from Level 2→3 by formalizing policies; 3→4 by automating metrics; 4→5 by linking data metrics to business outcomes.

3.3 Data Quality Dimensions

• Accuracy: Correctness (e.g., valid email format).

• Completeness: No missing values (e.g., all customers have a phone number).

• Consistency: Same data point matches across systems (e.g., customer address in CRM vs. ERP).

• Timeliness: Data is available when needed.

• Validity: Conforms to defined formats/ranges (e.g., date in YYYY-MM-DD).

• Uniqueness: No duplicate entities (e.g., one record per customer).

3.4 Master Data Management (MDM) & Reference Data

• MDM: Creates a single "golden record" for critical business entities (Customer, Product) by reconciling data from multiple source systems.

• Reference Data: A controlled subset of master data (e.g., country codes, currency codes). Used for classification and consistency. Governance: Must be centrally managed and versioned.

4.0 Data Lineage, Provenance & Metadata

Data Lineage: The complete lifecycle path of a data element: its origins, transformations, and movement across systems. Provenance is a subset, focusing on origin and ownership.

4.1 Importance

• Governance & Compliance: Trace data for audits (e.g., GDPR Article 30 records of processing).

• Impact Analysis: "What breaks if I change this source field?"

• Debugging: Root-cause analysis for erroneous reports.

• Trust: Demonstrates how a metric was calculated.

4.2 Types of Lineage Tracking

4.3 Implementation Techniques

1. Automated Parsing: Scans code (SQL, Python, Spark) to extract source -> transform -> target mappings. Scalable but may miss business logic.

2. Manual Annotation: Experts draw lineage in a tool. Accurate but not scalable.

3. Hybrid Approach: Auto-capture technical lineage, manually enrich with business context (tags, descriptions). Best Practice.

4.4 Data Cataloging & Metadata Repositories

A data catalog is a searchable inventory of data assets with their metadata.

• Technical Metadata: Schema, data type, file location.

• Business Metadata: Glossary terms, data owner, usage policy.

• Operational Metadata: Data freshness, row counts, last job run.

• Tool Examples: Apache Atlas, AWS Glue Data Catalog, Collibra.

5.0 Data Integration & Processing Techniques

5.1 ETL vs. ELT

5.2 Step-by-Step ETL Process (Retail Database → Data Warehouse)

1. Extract: Connect to OLTP retail database (orders, customers, products tables). Pull new/changed records since last run (using CDC or timestamp).

2. Transform:

   • Cleanse: Fix nulls, standardize formats (e.g., USA -> United States).

   • Integrate: Join orders with customers to get customer_name.

   • Aggregate: Compute daily_sales_per_product.

   • Conform: Ensure all date fields are in YYYY-MM-DD.

3. Load: Write transformed data into fact (fact_sales) and dimension (dim_product, dim_customer) tables in the data warehouse. Use UPSERT (update existing, insert new) for slowly changing dimensions (SCD Type 2).

5.3 Schema Migration & Evolution

Definition: The process of changing the structure (schema) of a database or data pipeline to accommodate new business requirements or source system changes.

Drivers: New business metric, source system adds a column, regulatory change.

Strategies:

1. Backward Compatibility: Add new columns as nullable. Old code continues to work.

2. Dual-Write: Run old and new schemas in parallel during transition.

3. Big Bang: Stop the system, change schema, restart. High risk.

Example: Adding a New Column to a Fact Table

• Scenario: Need to track discount_amount in fact_sales.

• Backward Compatible Migration:

  1. ALTER TABLE fact_sales ADD COLUMN discount_amount DECIMAL(10,2) NULL;

  2. Update ETL job to populate this column for new records.

  3. Backfill historical data (optional, in a separate batch job).

  4. Old reports ignoring the new column remain functional.

5.4 Change Data Capture (CDC) Patterns

Captures row-level changes (INSERT, UPDATE, DELETE) in source databases in real-time.

• Log-Based CDC: Reads database transaction logs (most efficient, non-intrusive). Tools: Debezium, AWS DMS.

• Trigger-Based CDC: Database triggers write changes to a "change table." Adds load to source.

• Timestamp/Version Column: Polls for rows where last_modified > last_run. Simple but can miss deletes.

5.5 Webhooks & Event-Driven Architectures

Webhook: An HTTP callback where one system sends an HTTP POST request to another system's URL when a specific event occurs.

How it Works:

1. System B (receiver) provides a URL endpoint to System A (sender).

2. System A registers this URL for a specific event (e.g., "payment_success").

3. When event occurs, System A immediately sends an HTTP POST with a payload (JSON/XML) to the URL.

4. System B processes the payload (e.g., updates order status, sends email).

Use Cases: Payment gateway notifications, CI/CD pipeline triggers, Slack/Teams alerts, syncing records between SaaS apps.

Security Considerations:

• Secret Management: Use shared secrets or HMAC signatures to verify payload authenticity.

• Validation: Validate payload schema and source IP.

• HTTPS: Always use TLS to encrypt the callback.

• Retry Logic: Handle transient failures; design for idempotency.

6.0 Security Protocols, Monitoring & Operational Practices

6.1 Secure Copy Protocol (SCP)

Working Principle: Uses SSH for authentication and encryption. It copies files between a local and a remote host or between two remote hosts. Encrypts both authentication and data transfer.

Commands & Usage:

# Local → Remote

scp /local/file.txt user@remotehost:/remote/dir/

# Remote → Local

scp user@remotehost:/remote/file.txt /local/dir/

# Recursive copy (directory)

scp -r /local/dir/ user@remotehost:/remote/dir/

Comparison:

• SCP vs SFTP: Both use SSH. SFTP is more feature-rich (resume, list dirs), but SCP is simpler and often faster for single file transfers.

• SCP vs rsync: rsync is superior for synchronization (transfers only differences, supports resume, checksums). SCP is a simple copy.

Security Best Practices:

1. Use Key-Based Auth: Disable password authentication in sshd_config.

2. Principle of Least Privilege: The SSH user should have only necessary filesystem permissions.

3. Limit Source IPs: Use firewall rules to restrict which IPs can connect via SSH/SCP.

4. Audit Logs: Monitor /var/log/auth.log for SCP/SSH activity.

6.2 Logging, Monitoring, and Alerting (LMA)

A triad for observability and incident response.

1. Logging: Collecting immutable records of events (application logs, system logs, audit trails).

   • Centralized Logging: Aggregate logs from all sources into a single system. Stacks: ELK (Elasticsearch, Logstash, Kibana), Splunk, CloudWatch Logs.

2. Monitoring: Measuring system health and performance using metrics (CPU, memory, latency, error rates).

   • Types: Infrastructure (host metrics), Application (JVM, request rates), Pipeline (job duration, record lag).

3. Alerting: Notifying humans when metrics/logs cross thresholds indicating a problem.

   • Strategies: Threshold-based (CPU > 90%), anomaly detection, absence of heartbeat.

   • Tools: Prometheus Alertmanager, PagerDuty, Opsgenie.

Integration with Incident Response: Alerts should automatically create tickets in IR systems (Jira, ServiceNow) with contextual logs, triggering the IR policy workflow.

6.3 Data Security in Transit & at Rest

• In Transit: Use TLS/SSL for network communication (HTTPS, FTPS, database connections). Never send credentials or data in plaintext.

• At Rest:

  • Encryption: Full-disk encryption (AWS EBS), database TDE (Transparent Data Encryption), file-level encryption (S3 SSE).

  • Tokenization: Replaces sensitive data (credit card) with non-sensitive tokens. Original data stored in a secure token vault. Better for compliance (PCI DSS) as tokens are not considered sensitive.

  • Masking: Dynamically hides data in non-prod environments (e.g., ****-****-****-1234).

7.0 Specialized Topics & Applications

7.1 Web Scraping & Data Acquisition

Techniques:

• HTML Parsing: Use libraries (BeautifulSoup, Scrapy) to parse DOM.

• APIs: Preferred method; use official, documented APIs.

• Headless Browsers: (Selenium, Puppeteer) for JavaScript-heavy sites.

Legal & Ethical Considerations (CRITICAL FOR POLICY):

• robots.txt: Respect the website's crawl directives. Not legally binding but a standard.

• Terms of Service (ToS): Violating ToS can lead to legal action (e.g., hiQ Labs v. LinkedIn). Scraping public data may be legal, but ToS can prohibit it.

• Rate Limiting: Do not overwhelm the server. Add delays between requests.

• Copyright: Data itself may be copyrighted; factual data often not.

• Policy Implication: Internal policy must mandate review of ToS and robots.txt before any scraping project.

Example Scenario (RGPV Press Releases):

1. Identify URL pattern: https://www.rgpvonline.com/press-releases?query=data.

2. Check robots.txt for /press-releases disallow rules.

3. Review ToS for scraping prohibitions.

4. If allowed, write a polite scraper with 1 request/sec delay.

5. Extract: representative_name, press_release_title, date, url.

6. Filter for titles containing "data".

7. Store in structured format (CSV/DB) with source URL and scrape timestamp for provenance.

7.2 Pattern Detection in Time-Series Data using Streaming

Stream Processing Engines: Apache Flink (stateful, exactly-once), Spark Streaming (micro-batch).

Windowing Operations: Define how to group infinite streams into finite sets for processing.

• Tumbling Window: Fixed-size, non-overlapping (e.g., 5-minute averages).

• Sliding Window: Fixed-size, overlapping (e.g., average every 1 min over last 5 min).

• Session Window: Gap-based (e.g., group events while user is active).

Anomaly & Trend Detection Algorithms (Streaming Context):

• Simple Threshold: Flag if value > 3σ from moving average.

• Exponential Smoothing (Holt-Winters): Forecast next value, compare actual.

• Machine Learning: Online clustering (K-Means), isolation forests for anomaly detection.

7.3 Enterprise Architecture Frameworks: The Zachman Framework

A schema for organizing enterprise architecture artifacts. A 6x6 matrix.

Six Interrogative Primitives (Columns):

Six Stakeholder Perspectives (Rows):

1. Planner (Executive): Scope, context.

2. Owner (Business Manager): Business concepts, rules.

3. Designer (Architect): System logic, models.

4. Builder (Developer): Technical components, code.

5. Subcontractor ( Technician): Detailed specs, tools.

6. Functioning Enterprise (Operating): Running system, metrics.

Application in Defining Security Policies & Standards:

• Provides a holistic view to ensure security policies are not just technical (Row 4) but also aligned with business rules (Row 2) and executive intent (Row 1).

• Example: A "Why" (Motivation) for Row 1 (Planner) might be "Comply with GDPR." This flows down to:

  • Row 2 (Owner): "Customer data must be retrievable for deletion."

  • Row 3 (Designer): "All PII stored in customer table with pii_flag."

  • Row 4 (Builder): "Implement DELETE /customers/{id} API with audit log."

  • Row 5 (Subcontractor): "Use AES-256 encryption for pii_flag column."

  • Row 6 (Enterprise): "Audit log shows 100% of deletion requests completed in <24h."

[!TIP] Exam Question: "Explain Zachman Framework" → Describe the 6x6 matrix and give one concrete example of how a security policy (e.g., access control) would be represented in each cell for a specific perspective.