Data Analytics (CS-503 (A)) Unit 5 Notes

How unit 5 is examined

Pig and Hive on Hadoop: installing them, Pig Latin, HiveQL, UDFs and Oracle Big Data; the marks sit in Pig vs MapReduce, Pig Latin and its data model, Hive architecture, metastore and HiveQL.

Installing and Running Pig

<span style="display:inline-block;padding:.16em .6em;border:1.5px solid currentColor;border-radius:999px;font-size:.68em;font-weight:700;letter-spacing:.06em;text-transform:uppercase;opacity:.75">Medium weight</span>

Definition. <mark>Apache Pig is a high-level platform for analysing large data sets on Hadoop, in which programs are written in the dataflow language Pig Latin and are compiled automatically into MapReduce jobs.</mark>

Key points.

We need Pig because a plain MapReduce join or group-by takes dozens of lines of Java, while Pig does the same ETL work in a few lines of Pig Latin, so development is faster and easier to maintain.
To install, keep Java and Hadoop ready, download and extract the Pig tarball, set PIG_HOME, add $PIG_HOME/bin to PATH, and check with pig -version.
Local mode (pig -x local) runs on one machine over the local file system; MapReduce mode (pig, the default) runs on a Hadoop cluster over HDFS.
Pig runs interactively in the Grunt shell, as a batch script (pig script.pig), or embedded in a Java program.
Architecture: the Parser checks syntax and builds a logical plan (a DAG), the Optimizer improves it, the Compiler turns it into MapReduce jobs and the Execution Engine runs them on Hadoop.
Typical use case: log analysis, such as counting hits per URL from web-server logs.

Basis	Apache Pig	MapReduce
Language	Pig Latin, a dataflow scripting language	Java, low-level code
Abstraction	High level, hides mapper and reducer	Low level, programmer writes both
Effort	About 10 lines for a task	About 200 lines for the same task
Execution	Compiled automatically into MapReduce jobs	Runs directly as a job
Performance	Slightly slower, but the optimizer helps	Faster and fully tunable
Use	Ad-hoc ETL and analysis	Complex custom processing

Answer frame. Open with the definition; for "why needed" develop points 1, 6, 5 and 3 with a use case; for "differentiate" write the table with a one-line definition of each before it; close with "Pig trades a little speed for far less coding effort."

Asked: [7 marks] (Dec 2020) What is Apache Pig and why we need it? Asked: [7 marks] (Jun 2020) Differentiate between Apache Pig and Map Reduce.

Comparison with Databases

Definition. Pig differs from an RDBMS in being a procedural dataflow language over files in HDFS rather than a declarative SQL system over tables.

Key points.

Pig Latin is procedural (step by step), while SQL is declarative (state the result only).
Pig's schema is optional and applied at load time, while an RDBMS needs a fixed schema before data is stored.
Pig supports nested types (bag, tuple, map), while an RDBMS tables are flat.
Pig has no transactions or indexes and is built for batch scans of huge data, so latency is high compared with an RDBMS.

Pig Latin

Definition. <mark>Pig Latin is the dataflow scripting language of Pig, in which a program is a sequence of steps that each load, transform or store a relation.</mark>

Key points.

The Pig data model has four types: an Atom is a single value (int, chararray), a Tuple is an ordered set of fields, a Bag is a collection of tuples and a Map is a set of key#value pairs.
Example values: tuple (John,25), bag {(John,25),(Ann,30)}, map [name#John]; a relation is an outer bag of tuples.
The schema is optional and fields can be missing or nested, which lets Pig handle messy, semi-structured data and keeps the data flow flexible.
Main operators are LOAD, FILTER, FOREACH...GENERATE, GROUP, JOIN, ORDER, STORE and DUMP.
Each statement builds a new relation from the previous one, so the flow reads like an ETL pipeline and nothing runs until STORE or DUMP.
Compared with SQL, Pig Latin is procedural, needs no schema and handles nested data.

Example. Word count as a data flow:

lines = LOAD 'input.txt' AS (line:chararray);
words = FOREACH lines GENERATE FLATTEN(TOKENIZE(line)) AS word;
grp   = GROUP words BY word;
cnt   = FOREACH grp GENERATE group, COUNT(words);
STORE cnt INTO 'out';

Answer frame. Open with the definition; for the data model, list the four types with examples, then the flow and schema-flexibility points; for "explain Pig Latin", add the operators and the script; close with "Pig Latin makes ETL a readable chain of steps."

Asked: [7 marks] (Dec 2020) Explain the term Pig Latin in detail. Asked: [7 marks] (Nov 2023) Explain Pig Data Model in detail and discuss how it will help for effective data flow?

User-Defined Functions in Pig

Definition. A Pig UDF is a user-written function, usually in Java, that extends Pig Latin with custom logic.

Key points.

The Java class extends EvalFunc<T> and implements exec(Tuple input); Python and JavaScript UDFs are also allowed.
The jar is loaded with REGISTER myudfs.jar; and called as FOREACH a GENERATE myudfs.UPPER(name);.
Types are eval functions, filter functions (return boolean), and load or store functions.

Data Processing Operators

Definition. Pig's operators are the relational statements that load, transform and store data.

Key points.

LOAD reads data into a relation and STORE writes it out; DUMP prints it on screen.
FILTER keeps tuples meeting a condition; FOREACH...GENERATE projects or computes fields.
GROUP and COGROUP collect tuples by key; JOIN combines relations on a key.
ORDER sorts, DISTINCT removes duplicates, LIMIT takes the first n, UNION merges and SPLIT partitions a relation.

Installing and Running Hive

Definition. Hive is a data warehouse on Hadoop that is installed as a client over an existing Hadoop set-up.

Steps.

Prerequisites: Java and a working Hadoop (HDFS running), plus the Hive tarball, which is downloaded and extracted.
Set HIVE_HOME and add $HIVE_HOME/bin to PATH.
Configure hive-site.xml with the metastore; embedded Derby is the default, or point it to MySQL, and run schematool -initSchema -dbType derby.
Create /tmp and /user/hive/warehouse in HDFS and give them group write permission.
Start the Hive shell with hive (or hiveserver2 with beeline) and verify with SHOW DATABASES;.

Asked: [7 marks] (Jun 2020) Write down the process of installing and running Hive.

Hive QL

Definition. <mark>Hive is a data warehouse on Hadoop, and HiveQL is its SQL-like language whose queries are compiled into MapReduce, Tez or Spark jobs over data in HDFS.</mark>

Diagram. <figure class="ds-fig" style="margin:1.4rem 0;overflow-x:auto"><svg xmlns="http://www.w3.org/2000/svg" id="dsfig-u5-01" viewBox="0 0 637.4 252" width="637.4" height="252" role="img" aria-label="Hive architecture: Cli = client (CLI, Web UI, JDBC/ODBC via HiveServer), Drv = Driver, Cmp = Compiler, Meta = Metastore, Exe = Execution engine, HDFS = Hadoop storage"><style>#dsfig-u5-01 .e{stroke:#454C5A;stroke-width:1.4;fill:none}#dsfig-u5-01 .e.hi{stroke:#2340B8;stroke-width:2.6}#dsfig-u5-01 .n{fill:#FFFFFF;stroke:#16181D;stroke-width:1.4}#dsfig-u5-01 .n.hi{fill:#E3E9FC;stroke:#2340B8;stroke-width:2.2}#dsfig-u5-01 .n.rb-b{fill:#16181D;stroke:#16181D}#dsfig-u5-01 .n.rb-r{fill:#BD3227;stroke:#BD3227}#dsfig-u5-01 text{font-family:"JetBrains Mono",ui-monospace,Menlo,Consolas,monospace;font-size:13px}#dsfig-u5-01 .t{fill:#16181D;font-weight:500}#dsfig-u5-01 .t.inv{fill:#FFFFFF;font-weight:700}#dsfig-u5-01 .kd{stroke:#16181D;stroke-width:1.2}#dsfig-u5-01 .dot{fill:#16181D}#dsfig-u5-01 .ann{fill:#2340B8;font-size:11px;font-weight:700}#dsfig-u5-01 .lbl{fill:#6F7787;font-family:system-ui,-apple-system,sans-serif;font-size:12px;font-weight:700}#dsfig-u5-01 .ptr{fill:#2340B8;font-size:12px;font-weight:700}#dsfig-u5-01 .ah{fill:#454C5A}#dsfig-u5-01 .ah.hi{fill:#2340B8}#dsfig-u5-01 .wl rect{fill:#FFFFFF;stroke:#DCE0E7}#dsfig-u5-01 .wl .t{font-size:12px;font-weight:700}#dsfig-u5-01 .wl.hi rect{fill:#2340B8;stroke:#2340B8}#dsfig-u5-01 .wl.hi .t{fill:#FFFFFF}html.dark #dsfig-u5-01 .e{stroke:#B1B7C3}html.dark #dsfig-u5-01 .e.hi{stroke:#8FA3FF}html.dark #dsfig-u5-01 .n{fill:#161920;stroke:#E6E8ED}html.dark #dsfig-u5-01 .n.hi{fill:#1E2748;stroke:#8FA3FF}html.dark #dsfig-u5-01 .n.rb-b{fill:#E6E8ED;stroke:#E6E8ED}html.dark #dsfig-u5-01 .n.rb-r{fill:#FF7E71;stroke:#FF7E71}html.dark #dsfig-u5-01 .t{fill:#E6E8ED}html.dark #dsfig-u5-01 .t.inv{fill:#0F1115}html.dark #dsfig-u5-01 .kd{stroke:#E6E8ED}html.dark #dsfig-u5-01 .dot{fill:#E6E8ED}html.dark #dsfig-u5-01 .ann{fill:#8FA3FF}html.dark #dsfig-u5-01 .lbl{fill:#858D9C}html.dark #dsfig-u5-01 .ptr{fill:#8FA3FF}html.dark #dsfig-u5-01 .ah{fill:#B1B7C3}html.dark #dsfig-u5-01 .ah.hi{fill:#8FA3FF}html.dark #dsfig-u5-01 .wl rect{fill:#161920;stroke:#2A2E37}html.dark #dsfig-u5-01 .wl.hi rect{fill:#8FA3FF;stroke:#8FA3FF}html.dark #dsfig-u5-01 .wl.hi .t{fill:#0F1115}</style><defs><marker id="ah6" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="ah" d="M0,1 L9,5 L0,9 z"/></marker><marker id="ahh6" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="ah hi" d="M0,1 L9,5 L0,9 z"/></marker></defs><path class="e" d="M59,126 L165.2,126" marker-end="url(#ah6)"/><path class="e" d="M202.6,116.4 L314.3,50.6" marker-end="url(#ah6)"/><path class="e" d="M332.4,61 L332.4,184" marker-end="url(#ah6)" marker-start="url(#ah6)"/><path class="e" d="M348.5,50.1 L452.2,114.9" marker-end="url(#ah6)"/><path class="e" d="M489,126 L562.4,126" marker-end="url(#ah6)"/><circle class="n" cx="40" cy="126" r="18"/><text class="t" x="40" y="126" dy=".35em" text-anchor="middle">Cli</text><circle class="n" cx="186.2" cy="126" r="18"/><text class="t" x="186.2" y="126" dy=".35em" text-anchor="middle">Drv</text><circle class="n" cx="332.4" cy="40" r="18"/><text class="t" x="332.4" y="40" dy=".35em" text-anchor="middle">Cmp</text><rect class="n" x="307.4" y="197" width="50" height="30" rx="15"/><text class="t" x="332.4" y="212" dy=".35em" text-anchor="middle">Meta</text><circle class="n" cx="470" cy="126" r="18"/><text class="t" x="470" y="126" dy=".35em" text-anchor="middle">Exe</text><rect class="n" x="565.4" y="111" width="50" height="30" rx="15"/><text class="t" x="590.4" y="126" dy=".35em" text-anchor="middle">HDFS</text></svg><figcaption style="font-size:.82em;opacity:.72;margin-top:.45rem">Hive architecture: Cli = client (CLI, Web UI, JDBC/ODBC via HiveServer), Drv = Driver, Cmp = Compiler, Meta = Metastore, Exe = Execution engine, HDFS = Hadoop storage</figcaption></figure>

Key points.

Working: the client submits a HiveQL query to the Driver, the Compiler parses it and fetches table metadata from the Metastore, the plan is optimised, the Execution engine runs it as MapReduce jobs reading HDFS, and the result returns to the client.
Features: SQL-like HiveQL, schema on read, partitioning and bucketing, scalability over HDFS, and support for UDFs; it suits batch queries on structured data, not real-time transactions.
The Metastore is the central repository of Hive metadata: table and column schemas, partitions and HDFS locations; it is used in every query compilation.
Metastore types: embedded (Derby in the same JVM, one user), local (same JVM, external database such as MySQL) and remote (a separate metastore service, used in production).
DDL: CREATE TABLE emp (id INT, name STRING) ROW FORMAT DELIMITED FIELDS TERMINATED BY ','; creates the table; ALTER TABLE emp ADD COLUMNS (salary FLOAT); changes its schema; DROP TABLE emp; deletes it (and its data if it is managed).
Effect: SHOW TABLES; then lists emp after CREATE, DESCRIBE emp; shows the new column after ALTER, and the table disappears after DROP.
Insertion: LOAD DATA [LOCAL] INPATH 'f.csv' INTO TABLE emp; moves a file into the table; INSERT INTO TABLE emp SELECT ... ; fills it from a query; INSERT INTO emp VALUES (1,'Ravi'); adds a row; LOAD DATA ... INTO TABLE emp PARTITION (year=2023); loads into a partition.

Answer frame. For architecture, open with the definition, draw the diagram, explain working (point 1) and features (point 2), and close with a use case such as querying sales logs; for the metastore, use points 3 and 4; for the DDL question, give three commands with syntax, example and effect (points 5 and 6); for insertion, write point 7 with one example each.

Asked: [7 marks] (Jun 2020) Explain any three Hive QL DDL command with its syntax and example. Asked: [7 marks] (Nov 2023) Draw and explain architecture of APACHE HIVE. Explain various data insertion techniques in HIVE with example. Asked: [7 marks] (Dec 2020) Explain the architecture and features of Hive. Explain working of Hive with proper steps and diagram. Asked: [7 marks] (Jun 2020) Explain the concept of metastore in Hive.

Querying Data

Definition. Hive queries data with SELECT statements in HiveQL, which are translated into MapReduce jobs.

Key points.

SELECT name FROM emp WHERE id > 10; filters and projects, and LIMIT n restricts rows.
GROUP BY with COUNT or SUM aggregates, and ORDER BY sorts the result.
JOIN combines tables on a key, and partition pruning in WHERE reads only the needed partitions.

User-Defined Functions in Hive

Definition. A Hive UDF is a custom function, written in Java, that extends HiveQL.

Key points.

A UDF extends the UDF class and implements evaluate(), returning one value per row.
The jar is added with ADD JAR f.jar; and the function is created with CREATE TEMPORARY FUNCTION name AS 'pkg.Class';.
A UDAF aggregates many rows into one value and a UDTF turns one row into many.

Oracle Big Data

Definition. Oracle Big Data is Oracle's integrated offering for Hadoop and NoSQL, built around the Oracle Big Data Appliance.

Key points.

The Big Data Appliance is an engineered rack that ships with Cloudera Hadoop and Oracle NoSQL Database.
Oracle Big Data Connectors, such as Loader for Hadoop and SQL Connector for HDFS, move data between Hadoop and Oracle Database.
Oracle Big Data SQL lets one SQL query span Hadoop, NoSQL and Oracle Database.

Last-minute revision

Pig is a high-level platform over Hadoop; Pig Latin is its dataflow language and every script compiles to MapReduce.
Pig modes: local (pig -x local) and MapReduce (default); run via Grunt, script or embedded.
Pig architecture: Parser, Optimizer, Compiler, Execution engine.
Pig data model: Atom, Tuple, Bag, Map.
Pig vs MapReduce: about 10 lines against 200, high-level against low-level, slightly slower.
Pig UDF: EvalFunc, REGISTER; Hive UDF: UDF.evaluate, ADD JAR, CREATE TEMPORARY FUNCTION.
Hive is a data warehouse on Hadoop with SQL-like HiveQL and schema on read.
Hive architecture: Client, Driver, Compiler, Metastore, Execution engine, HDFS.
Metastore types: embedded, local, remote; default database is Derby.
Hive insertion: LOAD DATA, INSERT...SELECT, INSERT...VALUES, partition load.
Hive DDL: CREATE, ALTER, DROP; warehouse directory is /user/hive/warehouse.

Memory hooks

Pig data model "ATBM": Atom, Tuple, Bag, Map.
Pig architecture "POCE": Parser, Optimizer, Compiler, Execution.
Metastore types "ELR": Embedded, Local, Remote.
Hive insertion "LIV": Load, Insert-select, Values.

Coverage checklist

Installing and Running Pig: Apache Pig definition and need (Dec 2020), Pig vs MapReduce (Jun 2020).
Comparison with Databases: Pig vs RDBMS, no past question.
Pig Latin: Pig Latin (Dec 2020), Pig data model (Nov 2023).
User- Define Functions: Pig UDF, no past question.
Data Processing Operators: Pig operators, no past question.
Installing and Running Hive: install process (Jun 2020).
Hive QL: DDL commands (Jun 2020), Hive architecture and insertion (Nov 2023), architecture, features and working (Dec 2020), metastore (Jun 2020).
Querying Data: Hive SELECT queries, no past question.
User-Defined Functions: Hive UDF, no past question.
Oracle Big Data: Oracle Big Data Appliance, no past question.

Account

Data Analytics (CS-503 (A)) - Unit 5 Short Notes

How unit 5 is examined

Installing and Running Pig

Comparison with Databases

Pig Latin

User-Defined Functions in Pig

Data Processing Operators

Installing and Running Hive

Hive QL

Querying Data

User-Defined Functions in Hive

Oracle Big Data

Last-minute revision

Memory hooks

Coverage checklist