Cyber Security Policies & Standards (CY-703 (B)) - Unit 1 Short Notes

UNIT 1: Data Management, Architectures, and Security Frameworks

1. Data Engineering Fundamentals

Data Engineering is the discipline of designing, building, and maintaining systems and infrastructure for collecting, storing, processing, and analyzing large volumes of data. It focuses on creating reliable, scalable, and efficient data pipelines to support analytics, machine learning, and business intelligence.

Data Engineering Lifecycle (Often depicted as a cyclical process):

[!TIP] Exam Focus: Be prepared to draw this lifecycle cycle and explain each stage with a security control example. The lifecycle is a frequent 7-mark question.

2. Enterprise and Data Frameworks

A. Zachman Framework

A schema for organizing enterprise architecture (EA) artifacts. It's a 6x6 matrix that classifies descriptive representations (What, How, Where, Who, When, Why) from different perspectives (Planner, Owner, Designer, Builder, Subcontractor, Enterprise).

• Purpose: Provides a comprehensive, logical structure for defining and documenting an enterprise's architecture, ensuring all stakeholder concerns are addressed.

• Security Relevance: The "Why" column (Motivation) is critical for documenting security policies, compliance requirements, and risk management rationale. The "Who" column identifies security roles and responsibilities.

• Visual:

  DiagramCANVAS: A 6x6 grid. Rows: Planner (Scope), Owner (Business), Designer (System), Builder (Technology), Subcontractor (Detailed), Enterprise (Functioning). Columns: What (Data), How (Function), Where (Network), Who (People), When (Time), Why (Motivation).

B. Gartner Data Maturity Model

A model with 5 levels to assess an organization's data management sophistication.

[!TIP] Exam Focus: Know the 5 levels in order and be able to give a concrete example for Level 3 (Defined) and Level 5 (Optimized). Link maturity to security posture (e.g., higher maturity implies better data classification & access controls).

3. Data Governance and Lineage

Data Governance is the overall management of data availability, usability, integrity, and security. It's a framework of policies, standards, processes, and roles to ensure data is a trusted, strategic asset.

• Core Pillars:

  1. Policies & Standards: Rules for data quality, privacy (GDPR), security (classification).

  2. Roles & Responsibilities: Data Owners (accountable), Data Stewards (manage), Data Custodians (implement).

  3. Processes: Data quality management, issue resolution, change management.

  4. Metrics & Monitoring: KPIs for data quality, policy compliance.

• Security Link: Governance mandates data classification (Public, Internal, Confidential, Restricted), which directly drives access control policies and encryption requirements.

Data Lineage Tracking in DataOps is the ability to track data's journey from source to consumption, showing transformations, movements, and dependencies. Critical for debugging, impact analysis, compliance, and trust.

• Pattern-Based Lineage: Automatically infers lineage by analyzing code/logic patterns in transformation scripts (SQL, Spark jobs).

  • Example: A tool scans all INSERT INTO ... SELECT ... SQL statements to map source tables to target tables.

  • Advantage: Automated, works with existing code.

  • Limitation: Can miss lineage from external sources or manual steps.

• Lineage by Data Tagging: Manually or semi-automatically attach metadata tags (e.g., source_system: CRM, PII: true) to datasets at each stage.

  • Example: Tag a table in the data warehouse as derived_from: sales_db.orders and contains: customer_email.

  • Advantage: Granular, captures business context.

  • Limitation: Requires discipline, prone to human error.

[!TIP] Exam Focus: Contrast the two lineage methods clearly. For the 7-mark question, use a single, consistent example (e.g., a customer data pipeline) to illustrate both techniques.

4. Data Architectures and Patterns

Types of Data Architectures (with Pros/Cons):

Data Lake Patterns:

• Merits: Single source of truth for all data, enables exploratory analytics/ML, cost-effective storage, flexible schema.

• Applications: Storing IoT sensor data, social media feeds, log files, raw backup data for future use cases.

• Key Pattern: "Landing Zone" pattern: Raw data ingested into a "bronze" zone, then refined to "silver" (cleaned) and "gold" (business-level) zones.

Lambda Architecture (for Real-Time Processing):

A hybrid architecture processing both batch and stream data to balance latency and accuracy.

1. Batch Layer (Speed Layer): Processes all historical data at rest (Hadoop/Spark). Outputs accurate, comprehensive views (slow, hours).

2. Speed Layer (Real-Time Layer): Processes incremental, real-time stream (Kafka/Flink). Outputs low-latency, approximate views (seconds).

3. Serving Layer: Merges batch and speed views to serve queries. Provides a single, up-to-date view.

Security Note: Securing both layers is complex. Batch layer needs secure storage (HDFS encryption). Stream layer needs secure brokers (TLS) and consumer auth.

Kappa Architecture:

A simplified alternative to Lambda. Only a stream processing layer is used.

• All data is treated as a stream. Historical data is re-processed by replaying the stream from a persistent log (Kafka).

• Advantage: Simpler codebase (one processing logic).

• Disadvantage: Requires powerful stream processing for large historical reprocessing.

• Security: Focus on securing the immutable log (Kafka) and stream processors.

Streaming Architecture for Time-Series Pattern Detection:

• Goal: Detect anomalies/patterns (e.g., fraud, sensor failure) in continuous data.

• Typical Stack: Sources (IoT sensors) → Message Broker (Kafka) → Stream Processor (Apache Flink, Spark Streaming) → Alerting/Storage (Elasticsearch, DB).

• Key Operations: Windowing (tumbling/sliding), stateful processing (tracking session), complex event processing (CEP).

• Example: Detect a temperature spike: (sensor_id, temp > 100) OVER (last 5 minutes) -> ALERT.

• Security: Secure sensor-to-broker communication, validate stream data, monitor processor resource usage (DoS risk).

[!TIP] Exam Focus: Lambda vs. Kappa is a classic 7-mark question. Draw a simple diagram for each. For streaming, know the components and a concrete pattern detection example (e.g., credit card fraud: 3 transactions in 2 mins, different locations).

5. Data Integration and Warehouse Design

Extract, Transform, Load (ETL) Process:

1. Extract: Pull data from source systems (DB, API, files). Security: Use secure connectors, credentials in vaults.

2. Transform: Clean, structure, enrich data (filter nulls, join, aggregate). Security: Sanitize to prevent SQLi, apply data masking for PII.

3. Load: Write transformed data to target (Data Warehouse). Security: Use bulk load with encrypted connections, validate row counts.

Example (Retail): Extract sales from POS DB & web logs → Transform (merge, calculate total, mask customer emails) → Load into fact_sales table in Redshift.

Schema Migration: The process of changing the structure (schema) of a database or data warehouse.

• Types:

  • Schema Evolution: Adding new columns/tables (backward compatible).

  • Schema Transformation: Renaming, changing data types, splitting tables (requires data migration).

• Example: Migrate a users table from (id, name, address) to (id, first_name, last_name, street, city, state, zip). This requires a transformation script to split name and address.

• Security Impact: Migration scripts must be reviewed to avoid accidental data exposure. Ensure new schema has appropriate constraints and permissions.

Step-by-Step Data Warehouse Design: University Case Study (3-Marker Outline):

1. Requirements Gathering: Identify key processes (Admissions, Academics, Finance). Key questions: "What is student pass rate by course?" "What is fee collection trend?"

2. Dimensional Modeling (Kimball): Design Star Schema.

   • Fact Tables: Fact_Enrollments (student_id, course_id, date_id, grade, credits).

   • Dimension Tables: Dim_Student (demographics), Dim_Course (details), Dim_Date (calendar).

3. Source System Analysis: Map source data (Student Info System, LMS, Finance DB) to target dimensions/facts.

4. ETL Design: Define how to extract, transform (e.g., calculate GPA), and load into the star schema.

5. Security & Governance: Implement row-level security (e.g., faculty see only their students), data classification (student data = Confidential), audit logs on fact table access.

6. Deploy & Test: Build, test with sample queries, validate results.

7. Maintain: Schedule ETL, monitor performance, manage slowly changing dimensions (SCD).

[!TIP] Exam Focus: For the 3-mark question, list the 5-6 key steps concisely. For a 7-mark ETL question, use a specific example (like Retail above) and walk through E, T, L with security touches.

6. Data Acquisition Techniques

Webhooks:

A user-defined HTTP callback (or small code snippet) that is triggered by a specific event in a source system. It's a push mechanism for real-time data delivery.

• How it works:

  1. Client (your app) registers a URL (webhook endpoint) with the source system (e.g., GitHub, Stripe).

  2. When an event occurs (e.g., "new commit", "payment succeeded"), the source system POSTs a JSON payload to your URL.

  3. Your endpoint receives and processes the data immediately.

• Example: A GitHub webhook sends a payload to your CI/CD server whenever code is pushed to the main branch, triggering an automated build.

• Security: Validate the request signature (HMAC) to ensure it's from the legitimate source. Use HTTPS. Implement rate limiting.

Web Scraping:

Automated programmatic extraction of data from websites. Used when no API exists.

• Hypothetical Scenario: https://www.rgpvonline.com - Find all representatives with press releases about "data".

• Step-by-Step Approach:

  1. Inspect: Analyze site structure (HTML). Find URLs listing representatives and their press release pages.

  2. Crawl: Use a library (Python: Scrapy, BeautifulSoup + Requests) to:

     • Crawl the main representatives listing page.

     • Extract links to individual representative pages.

     • For each rep, crawl their "Press Releases" section.

  3. Scrape & Filter: On each press release page, extract text. Filter for pages containing the keyword "data" (case-insensitive).

  4. Store: Save representative name, press release title, URL, and snippet containing "data" to a CSV/DB.

• Ethical & Legal Considerations:

  • Check robots.txt (guideline, not law).

  • Respect rate-limiting (don't hammer the server).

  • Review Terms of Service - scraping may be prohibited.

  • Copyright: Data facts may be free, but compilation/expression might be protected.

  • Security: Scraped data may contain PII - handle per data governance policy.

[!TIP] Exam Focus: For the 11-mark webhooks question, detail the step-by-step workflow and emphasize security (signature validation). For web scraping, present the algorithmic steps for the given scenario and discuss legality/ethics - a common follow-up.

7. Security Protocols and Operational Controls

Secure Copy Protocol (SCP):

A network protocol based on SSH for secure transfer of files between a local and a remote host (or between two remote hosts).

• How it works:

  1. Establishes an SSH connection (port 22) for authentication and encryption.

  2. Uses the SSH connection to execute the scp command on the remote host.

  3. Files are encrypted during transfer using the SSH session's symmetric cipher (e.g., AES).

• Command Example: scp -P 2222 file.txt user@remotehost:/path/

• Security Features: Relies on SSH's strong encryption, host key verification (prevents MITM), and authentication (password/public key).

• Limitations vs. SFTP: SCP is simpler but older. It has fewer features (no directory listing, resume, or permission control) and can be less secure in edge cases (e.g., filename expansion vulnerabilities). SFTP (SSH File Transfer Protocol) is generally preferred for its robustness and richer command set over the same SSH connection.

Logging, Monitoring, and Alerting (The LMA Triad):

A core operational security control for data systems.

• Integrated Workflow Example:

  1. Log: A user runs SELECT * FROM customer_ssn → Access log records (user, query, timestamp).

  2. Monitor: SIEM (Splunk, ELK) correlates this with the user's role (should not access SSN). Metric: "Unauthorized table access count" spikes.

  3. Alert: Rule triggers: IF unauthorized_table_access > 0 THEN alert SOC team.

• Critical Security Practices: Centralize logs (prevent tampering), enforce log retention (compliance), protect log integrity (write-once storage), and regularly review alert rules.

[!TIP] Exam Focus: For SCP, contrast it with SFTP and state why SFTP is often recommended. For LMA, describe the flow from logging to alerting with a concrete security incident example (like the unauthorized query above). This is a high-frequency 7-mark question.