OS Internals for Security Support (CY-501) Unit 3 Notes

UNIT 3: OS INTERNALS FOR SECURITY SUPPORT

1. CORE OPERATING SYSTEM STRUCTURE & INTERNALS

Kernel & System Architecture

Role of Kernel: The kernel is the core of the OS. It acts as a bridge between applications and hardware. Its primary functions include:
- Process Management: Scheduling, creation, termination.
- Memory Management: Allocation, paging, segmentation.
- Device Management: Via device drivers.
- I/O System Management: Handling system calls for I/O.
- Security & Protection: Enforcing access control, managing privileges.
- Resource Allocation: Mediating access to shared resources.
- Error Handling: Detecting and recovering from errors.
Monolithic vs. Microkernel Architectures:

| Feature | Monolithic Kernel | Microkernel | | :--- | :--- | :--- | | Structure | All OS services (scheduling, file system, drivers) run in kernel space as a single program. | Minimal kernel; only essential services (IPC, memory management, scheduling) in kernel space. Other services (drivers, file systems) run in user space as servers. | | Performance | High (direct function calls within kernel). | Lower (IPC overhead for cross-server communication). | | Reliability/Security | Lower. A bug in one service can crash the whole system. Larger attack surface. | Higher. Fault isolation; a crashed driver/server doesn't necessarily crash the kernel. Smaller trusted code base. | | Examples | Traditional UNIX, Linux, older Windows (9x/ME). | QNX, MINIX, macOS (XNU hybrid), modern Windows (NT kernel has microkernel influences). |
Windows System Architecture (NT Kernel Model):

[[DIAGRAM: CANVAS: A layered diagram showing:
1. User Mode: Applications & Subsystems (Win32, POSIX, etc.). They call via System Services.
2. Kernel Mode: The NT Kernel (ntoskrnl.exe). Contains:
  - Executive: Kernel-mode subsystems (Object Manager, I/O Manager, Process/Thread Manager, Security Reference Monitor, etc.).
  - Kernel: Lower-level services (scheduler, dispatcher, memory manager, kernel debugger).
  - Hardware Abstraction Layer (HAL): Isolates kernel from hardware differences.
3. Hardware: Physical devices.
Arrows show system calls from User Mode down through Executive & Kernel to HAL/Hardware. The Security Reference Monitor is a key component within the Executive enforcing security policies.]]

UNIX/Linux OS Components

Detailed Components & Features:
- Kernel: Heart of the OS; manages resources, processes, I/O.
- Shell: Command interpreter (e.g., bash, sh). Provides CLI.
- File System: Hierarchical, tree-structured. Everything is a file (devices, processes).
- Utilities/Commands: Standard tools (ls, cp, grep).
- System Libraries: Provide APIs for applications (e.g., libc).
- Processes & Daemons: Running programs; background services.
- Key Features: Multi-user, multitasking, portability, open source (Linux), security via permissions/SELinux.
Super Block:
- Definition: A critical data structure stored on disk (typically at a fixed location in the file system) that contains metadata about the mounted file system.
- Contents:
  - File system type (e.g., ext4, XFS).
  - Size of file system (total blocks/inodes).
  - Free block/inode counts and pointers to free lists.
  - Block size, inode size.
  - Mount status, timestamp of last mount/write.
  - Pointer to the inode table.
- Role: Read into memory during mount(). The in-memory copy (superblock structure) is used by the kernel for all subsequent file system operations to avoid repeated disk I/O for metadata.
File System Structure & Management:
- Structure: Boot Block -> Super Block -> Inode Table -> Data Blocks.
- Inode: Data structure per file/directory containing metadata (owner, permissions, size, timestamps, pointers to data blocks).
- Management: Kernal uses VFS (Virtual File System) layer to provide a unified interface to different file systems (ext4, NTFS, NFS). System calls (open, read, write, close) interact with VFS, which dispatches to specific file system drivers.

Buffer Management

Buffer Cache: A region in main memory used to temporarily store disk blocks (file system data) to reduce frequent, slow disk access. It acts as a cache for block I/O. Goal: Improve I/O performance via temporal locality (recently used blocks likely reused) and spatial locality (blocks near a recently used block likely needed).

Buffer Header:

[[DIAGRAM: CANVAS: A structure diagram showing fields of a buffer header:

`struct buf {

int dev;           // Device ID

int blocknum;      // Disk block number

struct buf *next;  // Pointer for free/busy/driver queues

struct buf *prev;

char *data;        // Pointer to actual 1KB (or block size) data in buffer pool

int flags;         // Status bits: B_BUSY, B_VALID, B_DIRTY, B_WANTED

semaphore lock;    // For synchronization

... // Other fields

};`

]]

Purpose: Each buffer in the pool has a header to describe and manage it. Tracks which disk block it holds, its state (valid, dirty, busy), and links it to queues.

Buffer Pool: The physical memory area containing all the actual data buffers. It's organized as a circular list (or multiple queues) of buffers.

[[DIAGRAM: CANVAS: Three linked lists (queues) emanating from a central "Buffer Pool" (array of buffers):
1. Free Queue: Buffers available for use. Head & Tail pointers shown.
2. Device Queues (per disk): Buffers holding blocks for a specific device, ordered by block number (for efficient read-ahead).
3. Driver Queue: Buffers currently being processed by the disk driver.
Arrows show movement: Buffer taken from Free Queue -> marked B_BUSY -> placed on Device Queue -> Driver takes from Device Queue -> after I/O, placed on Free Queue (if clean) or kept for write-back.

]]
Relationship:
- Buffer Pool is the memory storage for block data.
- Buffer Header is the metadata control block associated with each buffer in the pool.
- Buffer Cache is the logical concept of using the buffer pool as a cache. The cache is managed via the headers and the queue structures (free, device, driver).

2. PROCESS MANAGEMENT & CONCURRENCY

Process Fundamentals

Process Life Cycle (States & Transitions):
```
NEW -> READY -> RUNNING -> WAITING/BLOCKED -> READY -> TERMINATED

       ^                               |

       |_______________________________|
```
- New: Process created.
- Ready: In main memory, waiting for CPU.
- Running: Executing on CPU.
- Waiting/Blocked: Waiting for an event (I/O, signal).
- Terminated: Process completed.
Process Control Block (PCB): The OS's internal data structure that is the "process image" or "task control block." It's the only data structure the OS maintains for each process.
- Components:
  - Process State: (Ready, Running, etc.)
  - Process ID (PID): Unique identifier.
  - Program Counter (PC): Next instruction address.
  - CPU Registers: Saved context (accumulator, stack pointer, etc.).
  - CPU Scheduling Info: Priority, scheduling queue pointers.
  - Memory Management Info: Base/limit registers, page tables.
  - Accounting Info: CPU time used, limits.
  - I/O Status Info: Open files, allocated I/O devices.
- Role: Enables context switching. The OS saves the state of the current process in its PCB and loads the state of the next process from its PCB.
Process Scheduling Basics: The activity of selecting a process from the Ready Queue to run on the CPU. Goals: Max CPU utilization, throughput, minimize turnaround/waiting/response time, fairness. Types: Long-term (job), Mid-term (swapping), Short-term (CPU scheduler).

Concurrency & Synchronization

Principles of Concurrency: Multiple tasks (processes/threads) make progress simultaneously. Challenges: Race Conditions, Deadlocks, Starvation. Requires coordination and synchronization.
Requirements for Concurrent Execution:
1. Critical Section: Code segment accessing shared resource (variable, file, device).
2. Mutual Exclusion: Only one process in its critical section at a time.
3. Progress: If no process is in critical section and others wish to enter, the decision must be made in finite time.
4. Bounded Waiting: There exists a bound on the number of times other processes can enter their critical section after a process requests entry and before it is granted.
[!TIP] Common Pitfall: Confusing Progress (someone must eventually enter) with Bounded Waiting (fairness, no indefinite postponement).

Semaphores

Definition: A synchronization variable with an integer value and two atomic (indivisible) operations: wait() (P) and signal() (V). Used to control access to shared resources.
Types:
1. Binary Semaphore (Mutex): Value is 0 or 1. Used for mutual exclusion of a single resource. Initialized to 1.
  - wait(S) { while(S<=0); S--; } (Busy-wait, not practical)
  - Practical implementation uses blocking/waking.
2. Counting Semaphore: Value can be any non-negative integer. Used to control access to a pool of identical resources (e.g., 3 printers). Initialized to the number of resources.
3. Named/Unnamed: Unnamed semaphores reside in memory (for threads/related processes). Named semaphores have a filesystem pathname and can be used for unrelated processes.

Inter-Process Communication (IPC)

Definition & Need: Mechanisms that allow processes to communicate and synchronize with each other. Needed for data sharing, speed, modularity, and computation partitioning.
Methods of IPC:
- Shared Memory: Fastest. Processes share a region of physical memory. Requires synchronization (semaphores) to avoid race conditions.
- Message Queues: Messages passed via kernel-maintained queues. Can be named. Supports client/server.
- Pipes: Unidirectional byte stream. Anonymous pipes for related processes (parent-child). Named pipes (FIFOs) for unrelated processes.
- Sockets: For network communication between processes on different machines. Can also be used for local IPC (Unix domain sockets).
Detailed Explanation: Message Queues (Client/Server)
1. Server:
  - Creates/opens a message queue (msgget).
  - Waits for a message (msgrcv).
  - Processes request.
  - Sends reply (msgsnd) to client's queue (if separate) or same queue.
2. Client:
  - Creates/opens the same message queue.
  - Constructs request message (type = server's queue ID).
  - Sends request (msgsnd).
  - Waits for reply (msgrcv on its own queue or with specific type).
[!TIP] Exam Focus: Be ready to draw the sequence diagram for client-server using message queues.

3. SECURITY & PROTECTION FUNDAMENTALS

OS-Level Security Implementation

Goals & Mechanisms:
- Goals: Confidentiality, Integrity, Availability (CIA), Accountability.
- Mechanisms: Authentication (who are you?), Authorization/Access Control (what can you do?), Auditing (log activities).
Authentication & Access Control:
- Authentication: Verifying identity. Methods: Passwords, biometrics, tokens, multi-factor.
- Access Control: Policy deciding what an authenticated subject can do to an object.
  - Models: Discretionary Access Control (DAC - owner decides), Mandatory Access Control (MAC - system-enforced labels, e.g., SELinux), Role-Based Access Control (RBAC - roles).
  - Implementation: Access Control Matrix (conceptual), Access Control Lists (ACLs) (per object), Capability Lists (per subject).
Protection Rings & Privilege Levels:
- Concept: Hardware-supported privilege levels (e.g., x86 has Ring 0 to Ring 3).
- Ring 0 (Kernel/Supervisor): Highest privilege. Can execute any instruction, access any hardware/memory.
- Ring 3 (User): Lowest privilege. Restricted access. System calls trap to Ring 0.
- Purpose: Isolate the kernel from user applications. A user process fault in Ring 3 doesn't crash the system. Security: Malicious code in Ring 3 cannot directly access hardware or kernel memory.

Malware & Threats

Definitions:
- Malware: Malicious software. Any software designed to harm, steal, or gain unauthorized access.
- Trojan Horse: Disguised as legitimate software. Does not self-replicate. Creates backdoors, steals data.
- Root Kit: Software that hides its existence and the existence of other malware. Gains root/admin privileges and subverts the OS (e.g., by modifying kernel, system calls).
Types of Malicious Code:

| Type | Key Characteristic | Propagation | Sub-types | | :--- | :--- | :--- | :--- | | Virus | Requires host program to replicate. Attaches to executable/file. | User action (run infected program). | File infector, Macro, Boot sector, Polymorphic, Stealth. | | Worm | Standalone, self-replicating. Exploits vulnerabilities to spread over network without user action. | Network (email, vulnerabilities). | Network worm, Email worm, Internet worm. | | Ransomware | Encrypts victim's files/drives. Demands ransom for decryption key. | Phishing, exploit kits, RDP brute-force. | Crypto-ransomware, Locker ransomware, Scareware. |
Ransomware: Risks, Impact, Protection:
- Risks/Impact: Data loss (if no backup), financial loss (ransom + downtime), reputation damage, operational disruption.
- Protection Mechanisms:
  - Prevention: Regular offline/air-gapped backups, patch management, email filtering, user training, least privilege, disable macros.
  - Detection: Anomaly detection (sudden file encryption activity).
  - Response: Isolate infected systems, do not pay ransom (no guarantee), restore from backups, report to authorities.

Vulnerabilities & Defense

Common Vulnerabilities and Exposures (CVEs): A list/database of publicly known cybersecurity vulnerabilities and exposures. Each CVE has a unique ID (CVE-YYYY-XXXXX), description, and references. Used as a standard reference for vulnerability management.
Honeypot: A decoy system (data, server, network) deliberately made attractive to attackers.
- Use: Early detection of attack attempts, tactical intelligence (study attacker TTPs - Tactics, Techniques, Procedures), diverting attackers from real assets, improving defenses based on collected intelligence.
Virtualization for Security:
- Sandboxing: Running untrusted code in an isolated VM.
- Live Forensics & Analysis: Examining malware in a controlled VM.
- Disaster Recovery: Quick VM restore.
- Security Hardening: Using VMs with minimal OS for specific services.
- Limitations: VM escape vulnerabilities, performance overhead.

4. PLATFORM-SECIFIC SECURITY (MOBILE & WINDOWS)

Mobile OS Security (Android Focus)

Vulnerable Components: Baseband processor (radio), kernel drivers, system services (SMS, telephony), application framework, user-installed apps, hardware interfaces (camera, GPS).
Android Security Model (Layered):

| Layer | Security Mechanisms | | :--- | :--- | | Hardware | Trusted Execution Environment (TEE), Secure Boot, Hardware-backed keystore. | | Kernel | SELinux (enforcing mode), seccomp, address space layout randomization (ASLR), memory protection. | | OS / System | Application Sandboxing (each app runs as unique UID/GID), Permissions (install-time & runtime), verified boot, dm-verity. | | Application | Code signing, app sandbox, permission model, Google Play Protect. |

Windows Internals & Security

Windows System Architecture: (Recap from Section 1 - NT Kernel Model). Key security component: Security Reference Monitor enforces security policy (ACLs, privileges).
Windows-Specific Concepts:
- System Worker Threads: Kernel-mode threads created by the System process (PID 4). They perform background tasks like memory management, cache management, and handling system worker items from other drivers. Security Relevance: Can be exploited if a malicious driver queues work items.
- Windows Global Flags (GFlags): A tool (and registry settings HKLM\System\CurrentControlSet\Control\Session Manager\Memory Management) to enable system-wide debugging and diagnostic features.
  - Security Use: Enable PageHeap (detect heap corruption), Show Loader Snaps (debug DLL loading), Disable heap tail checking (can be abused). Attackers use them to bypass security checks or for persistence.
Winsock (Windows Sockets API):
- Purpose: API for network programming on Windows. Provides socket interface (BSD-like) over underlying TCP/IP stack.
- Key Functions:
  - WSAStartup(): Initialize Winsock.
  - socket(): Create socket (specify address family, type, protocol).
  - bind(): Bind socket to local address/port (server).
  - listen(): Mark socket as passive (server).
  - accept(): Accept incoming connection (server) -> returns new socket for client.
  - connect(): Initiate connection (client).
  - send()/recv(): TCP (stream) data transfer.
  - sendto()/recvfrom(): UDP (datagram) data transfer.
  - closesocket(): Close socket.
- Socket Types:
  - SOCK_STREAM (TCP): Reliable, connection-oriented, byte-stream. Guarantees delivery order.
  - SOCK_DGRAM (UDP): Unreliable, connectionless, message-oriented. No guarantee of delivery/order. Use: DNS, VoIP, video streaming where speed > reliability.
  - SOCK_RAW: Access to lower-layer protocols (IP, ICMP). Used for network diagnostics (ping, traceroute) and security tools (port scanners, packet sniffers). Requires admin privileges.
- Socket Programming (Client-Server TCP Model):
  1. Server: socket() -> bind() -> listen() -> accept() -> recv()/send() -> closesocket().
  2. Client: socket() -> connect() -> send()/recv() -> closesocket().

5. SPECIALIZED SECURITY HARDWARE & CONCEPTS

Secure Hardware Components

Secure Coprocessor (e.g., TPM - Trusted Platform Module):
- Definition: A dedicated, tamper-resistant hardware chip (or firmware) that provides cryptographic functions and secure storage for keys, passwords, and digital certificates.
- Architecture/Role:
  - Secure Storage: Stores attestation identity keys (AIKs), storage root key (SRK).
  - Cryptographic Engine: RSA, ECC, SHA, HMAC.
  - Attestation: Generates cryptographic proofs (quotes) about the platform's state (measured boot).
  - Sealing: Binds data to specific platform state (only unsealed if state matches).
  - Measured Boot: Extends hashes of boot components into TPM Platform Configuration Registers (PCRs).
- Security Uses: Full disk encryption (BitLocker), secure key storage, platform integrity verification, DRM.
Secure Wallet (Cryptocurrency/ Digital Wallet):
- Definition: A hardware device or software application that securely stores cryptographic keys (private keys) used to authorize transactions (e.g., cryptocurrency, digital signatures).
- Purpose & Importance:
  - Key Security: Private keys never leave the secure element of the hardware wallet.
  - Transaction Signing: Signs transactions internally; only signed transaction is exported.
  - Protection from Malware: Immune to keyloggers and malware on the host computer.
  - User Verification: Requires physical button press/touch for transaction authorization (2FA at hardware level).
  - Seed Phrase Backup: Generates and secures the recovery seed.

File System Operations & Security

Mounting & Unmounting:
- Mounting: The process of making a file system (on a partition/disk) accessible to the OS at a specific mount point (directory) in the existing directory tree.
  - Process: mount(device, mount_point, filesystem_type, options). Kernel reads super block from device, validates file system, integrates it into VFS.
  - Advantages: Seamless integration, centralized access control (permissions on mount point), supports multiple file systems.
  - Disadvantages: Security risk if mounted with wrong options (e.g., noexec, nosuid). Single point of failure (if mount point corrupted).
- Unmounting: Detaches a mounted file system. Flushes all cached data to disk (sync), updates super block, removes from VFS. Must not have open files or active processes in that file system.
  - Advantages: Safely removes media, ensures data integrity, allows filesystem check (fsck).
  - Disadvantages: Requires no active use; can fail if processes are using files.
Role of System Calls in File Security:
- open(): Checks permissions (read/write/execute) of the file against the calling process's UID/GID and ACLs. Returns file descriptor on success.
- read()/write(): Enforce mandatory locks if set. Checks if file descriptor is valid and has appropriate access mode.
- create(): Checks write permission on the containing directory. Creates inode, sets initial permissions (from umask).
- chmod(): Changes access permissions (rwx for owner/group/others). Requires appropriate privilege (owner or root). This is a direct security control mechanism.
- Underlying: All these calls are mediated by the kernel's VFS and specific file system driver, which consult the inode's mode bits and ACLs to enforce security policy.