CY-703-C Data Engineering - Nov 2023 Question Paper

Under each question, How to answer lists the points an answer should cover, outlined by CampusPrep from the syllabus. Check the details against your textbook.

1. 7 Marks
   aWhat do you understand by Data Engineering? Draw the data engineering lifecycle and explain each of its components.
   How to answer

   • Definition of Data Engineering and its role
   • Data engineering lifecycle stages: generation, ingestion, storage, transformation, serving
   • Diagram of lifecycle
   • Explanation of each component with example
2. 7 Marks
   bExplain data lineage tracking in DataOps with suitable examples.
   How to answer

   • Definition of data lineage and DataOps
   • Need for lineage tracking in DataOps
   • How tracking works: metadata, lineage graphs
   • Suitable example of source to transformation to consumption
3. 7 Marks
   aExplain Gartner Data Maturity Model in detail with help of examples.
   How to answer

   • Overview of Gartner Data Maturity Model
   • Levels/stages of maturity from unaware to transformative
   • Characteristics of each level
   • Examples for each level
4. 7 Marks
   bExplain different types of data architectures with the advantages and disadvantages.
   How to answer

   • Definition of data architecture
   • Types: data warehouse, data lake, lakehouse, Lambda, Kappa, etc.
   • Advantages of each type
   • Disadvantages/limitations of each type
5. 7 Marks
   aWhat do you understand by the data lake patterns. Mention its merits and applications.
   How to answer

   • Definition of data lake patterns
   • Common patterns: centralized, zoned, lambda-enabled lake
   • Merits: scalability, schema-on-read, cost
   • Applications in analytics and ML
6. 7 Marks
   bExplain in detail, how you will process Real-Time Data with Lambda Architecture.
   How to answer

   • Overview of Lambda Architecture layers: batch, speed, serving
   • Flow of real-time data through speed layer
   • Merging batch and real-time views
   • Example use case
7. 7 Marks
   aHow you will detect patterns in time-series data using streaming data architecture.
   How to answer

   • Concept of streaming data architecture
   • Steps: ingestion, windowing, pattern matching
   • Algorithms for time-series pattern detection
   • Example of anomaly/trend detection
8. 7 Marks
   bWhat do you understand by Schema Migration? Explain with the help of example.
   How to answer

   • Definition of Schema Migration
   • Need for migration: evolution, versioning
   • Steps/process with example
   • Challenges and handling
9. 7 Marks
   aExplain step-by-step approach to build a Data Warehouse for a university setup.
   How to answer

   • Requirements and planning for university data warehouse
   • Step-by-step: requirements, modeling, ETL, deployment
   • Dimensions/facts for university setup
   • Example schema and use
10. 7 Marks
    bWhat are Webhooks and explain how they work with help of examples.
    How to answer

    • Definition of Webhooks
    • How webhooks work: events, HTTP callbacks
    • Workflow steps with example
    • Use cases vs APIs/polling
11. 7 Marks
    aIllustrate web scraping by considering the below hypothetical scenario: https://www.rgpvonline.com The Manager at an IT company is worried about potential regulation of the data science industry and asks you to quantify what the government is saying on the topic. In particular, he wants you to find all the representatives who have press releases about "data".
    How to answer

    • Understanding the scraping task: press releases about "data"
    • Steps: inspect site, fetch, parse, filter, store
    • Handling pagination and search for representatives
    • Legal/ethical considerations
12. 7 Marks
    bExplain Pattern-Based Lineage and Lineage by Data Tagging with the help of suitable example.
    How to answer

    • Definition of Pattern-Based Lineage
    • Definition of Lineage by Data Tagging
    • How each method captures lineage
    • Suitable example comparing both
13. 7 Marks
    aExplain how you will Extract, Transform, and Load Data from a Retail Database to a Data Warehouse with the help of example.
    How to answer

    • Extract: select data from retail tables
    • Transform: cleaning, joining, aggregating, conforming dimensions
    • Load: into warehouse facts/dimensions
    • Example with retail sales data
14. 7 Marks
    bExplain the working of Logging, Monitoring, and Alerting with the help of examples.
    How to answer

    • Purpose of logging, monitoring, alerting
    • How logging collects events/metrics
    • Monitoring dashboards and alert rules
    • Example scenario
15. 14 Marks
    Write a short note on any two :

    1. i)The Zachman Framework
    2. ii)Kappa Architecture
    3. iii)Secure Copy Protocol (SCP)
    4. iv)Data Governance
    How to answer

    • Select any two topics from list
    • Definition and key components of each
    • Diagram/framework structure if applicable
    • Uses and examples