Big Data Analytics (AL-802 (C)) - Unit 5 Short Notes

UNIT 5: BLOCKCHAIN TECHNOLOGY & DISTRIBUTED LEDGERS

1.0 Foundational Concepts & Core Architecture

1.1 Public Ledger

A public ledger is a decentralized, immutable, and transparent digital record of transactions or data entries, maintained by a distributed network of nodes rather than a central authority.

Core Properties:

• Decentralized: No single entity controls the ledger; copies are distributed across many nodes.

• Immutable: Once data is recorded, it is extremely difficult to alter or delete due to cryptographic hashing and consensus.

• Transparent: All participants can independently verify the ledger's history (though transaction details may be pseudonymous).

Functioning in Financial Transactions:

1. Recording: A transaction (e.g., "Alice pays Bob 5 BTC") is broadcast to the network.

2. Verification: Nodes validate the transaction against consensus rules (e.g., sufficient funds, valid digital signature).

3. Audit Trail: Valid transactions are grouped into a block. This block is cryptographically linked to the previous block, creating an unbroken, timestamped chain. Anyone can audit the entire history from the genesis block to the present.

[!TIP] Exam Focus: Distinguish between a public ledger (the overall concept) and a blockchain (a specific implementation of a public ledger using chained blocks).

1.2 Block Structure & Blockchain

A block is the fundamental data structure that batches transactions together for inclusion in the blockchain.

Standard Block Composition:

"Block within a Block" Concept: This typically refers to the block header itself being a structured data object. The header contains a Merkle Root (a single hash representing all transactions in the block). Effectively, the hash of the entire block header is what gets chained to the next block. So, the chain links headers, and each header contains a commitment (the Merkle root) to the block's transaction data.

Cryptographic Chaining:

Each block header includes the hash of the previous block's header. This creates a backward-linked chain.

$$ \text{Block}_n.\text{header}.\text{prev\_hash} = \text{Hash}(\text{Block}_{n-1}.\text{header}) $$

Any alteration to a past block changes its header hash, breaking all subsequent links, making the ledger immutable.

1.3 Merkle Trees (Hash Trees)

A Merkle Tree is a binary tree of cryptographic hashes used to efficiently and securely verify the integrity of large datasets (like transactions in a block).

Structure & Construction:

1. Each transaction's ID (TXID) is hashed to form the leaf nodes.

2. Leaf nodes are paired and hashed together to form parent nodes.

3. This pairing and hashing continues recursively upward until a single root hash remains: the Merkle Root.

4. The Merkle Root is stored in the block header.

Role in Efficient & Secure Verification (Merkle Proofs):

• A Merkle Proof is a short list of hashes (the "path") from a specific transaction leaf to the Merkle Root.

• A lightweight client (e.g., mobile wallet) can verify a transaction is in a block without downloading the entire blockchain.

• Process: The client obtains the transaction hash, the necessary sibling hashes from the proof, and the Merkle Root from the block header. By recursively hashing upwards, it computes the root. If the computed root matches the header's root, the transaction is proven to be in that block.

[!TIP] Common Pitfall: A Merkle Tree verifies inclusion of a specific transaction in a specific block. It does not verify the transaction's validity (e.g., double-spend) or the block's position in the main chain—that requires full validation.

2.0 Consensus Mechanisms & Distributed Agreement

2.1 The Byzantine General Problem & Byzantine Fault Tolerance (BFT)

• Problem: A group of Byzantine generals must agree on a common plan of attack (attack or retreat). Some generals may be traitors who send conflicting messages. The challenge is to achieve consensus despite the presence of these Byzantine faults (arbitrary/malicious behavior).

• Relevance to Blockchain: In a decentralized network, nodes are the "generals." Byzantine faults represent hacked, malicious, or arbitrarily failing nodes. A consensus algorithm must reach agreement correctly even if some nodes are faulty.

• BFT Requirement: To tolerate t Byzantine nodes, the system must have at least n > 3t total nodes.

2.2 Classic Consensus Algorithms

Paxos Algorithm:

• Roles:

  • Proposer: Suggests a value (e.g., a transaction batch).

  • Acceptor: Votes to accept or reject proposals. A majority (quorum) must accept.

  • Learner: Learns the chosen value once consensus is reached.

• Process (Simplified): A proposer sends a Prepare request with a proposal number. Acceptors promise not to accept lower-numbered proposals and reply with any previously accepted proposal. The proposer then sends an Accept request for its value (or the highest-numbered accepted value). If a majority accepts, consensus is achieved.

• Application: Foundational for many fault-tolerant distributed systems (e.g., Google Chubby, some database clusters). Assumes non-Byzantine faults (crash-only).

Lamport-Shostak-Pease (Byzantine Generals' Solution - Oral Messages - OM):

• A recursive, message-passing algorithm for the Byzantine Generals' Problem.

• Core Idea (OM(m)): The commander sends an order to each lieutenant. Each lieutenant, upon receiving an order, acts as the commander for a new OM(m-1) instance with the other lieutenants. The final decision is based on the majority of values received after m levels of recursion.

• Limitation: Requires n > 3t (where t is max traitors). Communication overhead is O(n^m), making it impractical for large n or m.

2.3 Proof-of-Work (PoW)

HashCash Mechanism (Adapted by Bitcoin):

1. Puzzle: Miners must find a nonce such that:

$$ \text{Hash}(\text{block\_header} \parallel \text{nonce}) < \text{Target} $$

The `Target` is a very small number, meaning the hash output must have a certain number of leading zeros.

2. Brute-Force: The only way to find a valid nonce is through massive trial-and-error (computational work).

3. Difficulty Adjustment: The Target is automatically adjusted every 2016 blocks (~2 weeks) to maintain an average block time of 10 minutes, regardless of total network hash power.

Contribution to Security:

• Sybil Attack Resistance: Creating mining power requires real-world computational resources (electricity, hardware), making it costly to spawn many identities.

• Chain Immutability: To alter a past block, an attacker must re-mine that block and all subsequent blocks, requiring >50% of the current network's hash power (51% attack)—prohibitively expensive.

• Miner Incentivization: The first miner to solve the puzzle for a block receives the block reward (newly minted coins + transaction fees), aligning miner incentives with network security.

2.4 Design Considerations for Permissioned Blockchains

Permissioned (private/consortium) blockchains restrict participation. Key considerations differ from permissionless (like Bitcoin):

3.0 Bitcoin: The Pioneer Cryptocurrency Network

3.1 Bitcoin Peer-to-Peer (P2P) Network

A decentralized network of nodes (computers running Bitcoin Core) that communicate directly without central servers.

Topology & Node Types:

• Full Nodes: Store the entire blockchain (~500+ GB), validate all transactions/blocks, and relay them. The backbone of security.

• Lightweight/SPV Clients: Store only block headers. Use Merkle Proofs to verify their own transactions without full download.

• Mining Nodes: Full nodes that also compete to solve the PoW puzzle.

Facilitation of Bitcoin Transfer (Gossip Protocol):

1. Transaction Propagation: User A creates a signed transaction and broadcasts it to connected peers.

2. Validation & Relay: Each node validates the transaction (signature, UTXO existence, no double-spend). If valid, it relays it to its other peers (except the one it received from).

3. Mempool: Valid but unconfirmed transactions sit in the memory pool ("mempool") waiting to be included in a block.

4. Block Propagation: A miner who solves PoW broadcasts the new block. Nodes validate the block (PoW, transactions, Merkle root) and, if valid, add it to their chain and relay it.

5. Chain Selection: Nodes always adopt the longest valid chain (more precisely, the chain with the most cumulative PoW). This resolves temporary forks.

3.2 The Bitcoin Miner's Role & Lifecycle

Daily Routines & Lifecycle:

1. Transaction Selection: Miners construct a candidate block. They select high-fee transactions from their mempool, often prioritizing those with higher fee/byte ratios.

2. Block Template Creation: The miner creates the block header:

   • Sets prev_hash to the hash of the latest block.

   • Calculates the Merkle Root of the selected transactions.

   • Sets timestamp (updated roughly every few seconds).

   • Sets difficulty target (from network consensus).

   • Initializes nonce = 0.

3. PoW Puzzle Solving: The miner repeatedly:

   • Computes Hash(block_header).

   • Checks if hash < Target.

   • If not, increments nonce (or modifies the extraNonce in the coinbase transaction) and repeats.

   • This is a pure brute-force search.

4. Block Validation & Broadcast: Upon finding a valid nonce:

   • The miner broadcasts the complete block to its P2P network.

   • Other nodes validate it. If valid, they add it to their chain and start mining the next block on top of it.

5. Orphaned Blocks: If two miners solve a block near-simultaneously, a temporary fork occurs. The network eventually converges on one chain. Miners of the orphaned block lose the block reward but keep the fees from transactions included in it (if those transactions are re-included in the main chain).

Contribution to Network:

• Security: PoW makes altering history economically infeasible.

• Transaction Confirmation: Each new block "confirms" transactions in previous blocks. 6 confirmations (~60 mins) is considered secure.

• Coin Issuance: The block reward (currently 3.125 BTC) is the only way new bitcoins are created, enforcing a fixed, disinflationary supply (halving every 210,000 blocks).

4.0 Enterprise & Permissioned Blockchain Platforms

4.1 Hyperledger Fabric

A modular, permissioned blockchain framework for enterprise solutions.

Modular Architecture (Separation of Concerns):

Execute-Order-Validate Paradigm:

1. Execute: Client sends a transaction proposal to endorsing peers. Peers simulate execution (using chaincode) against their world state copy, producing a read-write set and an endorsement signature. No ledger update yet.

2. Order: Client submits endorsed transaction to the Ordering Service. The service sequences transactions into blocks (cryptographically orders them) and broadcasts blocks to all committing peers.

3. Validate: Committing peers validate each transaction in the block:

   • Check endorsement policy (sufficient valid endorsements?).

   • Check read-write set consistency (no concurrent modification).

   • If valid, update the world state and ledger. If invalid, the transaction is accepted into the block but marked invalid (no state change).

Scalability & Channels:

• Channels: Private subnets of communication between a specific set of peers and an ordering service. Ledger data is isolated per channel. Enables multi-lateral business networks with data confidentiality.

• Performance: Execute-order-validate allows parallel execution of non-conflicting transactions. No global state bottleneck. Achieves high TPS (100s-1000s) and low latency.

4.2 Ripple & Corda (Short Note Focus)

Ripple (XRP Ledger):

• Purpose: Real-time gross settlement system and cross-border payments network.

• Consensus: Ripple Protocol Consensus Algorithm (RPCA). A federated consensus model where trusted validators (known entities like banks) vote on transaction order. No mining/PoW. Finality is fast (~3-5 seconds).

• Key Mechanism: Uses a Unique Node List (UNL)—each validator chooses a set of other validators it trusts. Consensus requires supermajority agreement within the overlap of UNLs.

• Cryptocurrency: XRP is the native digital asset used for bridging currencies and paying transaction fees.

Corda:

• Purpose: Designed specifically for regulated financial institutions (e.g., for trade finance, syndicated loans).

• Not a Blockchain: No global broadcast. Data is shared point-to-point only between parties with a legitimate need to know (the "shared fact" model). There is no common, globally replicated ledger.

• Consensus: Achieved via Notary Pools. A notary is a service (single node or cluster) that validates a transaction's uniqueness (prevents double-spend) and timestamps it. Notaries do not see transaction content.

• Privacy by Design: Only transaction participants and required regulators see the transaction details. Uses confidential identities and transaction tear-offs (only the necessary hash is shared broadly).

• Development: Applications are called CorDapps (Corda Distributed Applications).

5.0 Applications, Processes & Industry Impact

5.1 Know Your Customer (KYC) Process

• Key Components:

  1. Customer Identification (CIP): Collecting foundational data (name, DOB, address, ID number).

  2. Customer Verification: Authenticating provided documents (e.g., passport, utility bill) via digital means or third-party services.

  3. Due Diligence (CDD/EDD): Assessing the customer's risk profile (PEPs, high-risk jurisdictions). Enhanced Due Diligence for high-risk customers.

  4. Ongoing Monitoring: Tracking transactions for suspicious activity, updating customer information periodically.

• Importance for Financial Institutions:

  • AML/CFT Compliance: Mandatory to prevent money laundering and terrorist financing.

  • Fraud Prevention: Detects identity theft and synthetic fraud.

  • Risk Management: Understands customer risk to manage exposure.

  • Regulatory Requirement: Heavy fines for non-compliance (e.g., under the Bank Secrecy Act, GDPR).

Potential Blockchain Application:

A consortium blockchain/KYC utility where banks, after receiving customer consent, can securely share verified KYC data. A customer's identity proof (hash) and verification status could be stored on-chain, allowing new banks to instantly verify a customer with the customer's permission, reducing duplication, cost, and onboarding time from weeks to hours.

5.2 Supply Chain & Trade Finance

Impact on International Trade & Global Supply Chains:

• Transparency & Traceability: Every step (origin, processing, shipping, customs) is recorded on an immutable ledger. Enables provenance tracking (e.g., organic cotton, conflict-free minerals).

• Fraud & Counterfeit Reduction: Immutable records prevent document tampering (e.g., bills of lading, certificates of origin).

• Process Automation: Smart contracts automate payments, releases, and compliance checks upon fulfillment of conditions (e.g., IoT sensor confirms temperature maintained -> payment released).

• Dispute Reduction: Single source of truth minimizes disagreements over shipment status, quality, or timing.

• Paperwork Reduction: Digitizes and standardizes trade documents (letters of credit, invoices), cutting processing time from days to hours.

Supply Chain Financing:

• Traditional Challenges: SMEs face late payments (60-90+ days), lack of trust from financiers, opaque invoice verification, high interest rates.

• Blockchain Enablement:

  • Invoice Financing: An SME can tokenize a verified, buyer-approved invoice on a blockchain. A financier can instantly purchase it at a discount, with the blockchain providing irrefutable proof of the underlying trade.

  • Dynamic Discounting: Smart contracts can automatically offer early payment discounts based on invoice age and buyer's credit.

  • Trust & Transparency: Financiers see the entire, verified supply chain transaction history, reducing risk assessment costs and enabling lower financing rates.

5.3 Smart Contracts

Definition: A smart contract is self-executing code stored on a blockchain that automatically enforces the terms of an agreement when predefined conditions are met. "If X happens, then Y occurs."

Functioning:

1. Code is deployed to the blockchain (e.g., as Ethereum smart contracts or Hyperledger Fabric chaincode).

2. It has a digital address and holds assets (cryptocurrency or tokenized real-world assets).

3. It listens for transactions or events (oracles provide external data).

4. Upon condition satisfaction, it executes its internal logic—transferring assets, updating states, triggering other contracts—without any intermediary.

Potential Applications Across Industries:

[!TIP] Critical Distinction: A smart contract is not a legally binding "contract" in the traditional sense in most jurisdictions (yet). It is a tool to automate performance. The legal enforceability of the underlying agreement still depends on traditional law.