OS Internals for Security Support (CY-501) - Unit 1 Short Notes

UNIT 1: OS INTERNALS FOR SECURITY SUPPORT

1.0 OS Fundamentals & Architecture

1.1 Kernel Role and Types

• Kernel Role: The core of the OS, operating in privileged mode. It is the sole interface between hardware and user processes.

  • Responsibilities:

    1. Resource Management: Allocates CPU, memory, I/O devices.

    2. Abstraction: Provides simplified, uniform interfaces (e.g., system calls) to hide hardware complexity.

    3. Protection & Security: Enforces access control, isolates processes, prevents unauthorized access.

• Architectural Types:

  • Monolithic Kernel: All core services (scheduling, file systems, device drivers) run in a single address space in kernel mode.

    • Advantage: High performance (direct function calls).

    • Disadvantage: Less secure/reliable; a bug in any service can crash the entire system.

    • Examples: Traditional UNIX, Linux.

  • Microkernel: Minimal kernel providing only fundamental mechanisms (IPC, basic scheduling, memory management). Other services (drivers, file systems) run as user-space servers.

    • Advantage: More secure/reliable (fault isolation), extensible.

    • Disadvantage: Performance overhead due to frequent IPC for service requests.

    • Examples: QNX, MINIX, macOS (XNU hybrid).

[!TIP] Exam Focus: Compare monolithic vs. microkernel based on performance, security, and reliability. Know which services run where.

1.2 UNIX Operating System Components & Features

• Components:

  1. Kernel: Core, manages hardware/resources.

  2. Shell: Command interpreter (e.g., bash, sh), user interface.

  3. System Libraries: Collections of reusable functions (e.g., libc), wrap system calls.

  4. Utilities/Commands: Standalone programs (e.g., ls, grep) for specific tasks.

• Key Design Principles:

  • Everything is a File: Uniform I/O model—devices, sockets, pipes all accessed via file descriptors and standard syscalls (read, write).

  • Small, Composable Programs: Utilities do one job well; combined via shell pipelines (|) for complex tasks.

  • Configuration via Text Files: Human-readable, editable configuration (e.g., /etc/passwd).

1.3 Super Block (UNIX)

• Role: The on-disk metadata structure for a mounted file system. It is loaded into memory at mount time and kept in memory as the superblock structure. It is the single source of truth for file system state.

• Contents:

  • File system magic number (type identifier).

  • Size: Total number of blocks/inodes.

  • Free block/inode counts & lists (bitmaps or linked lists).

  • File system state (clean/dirty).

  • Root inode pointer.

• Security Relevance: Corruption of the super block can make the entire file system inaccessible. Backups (e.g., fsck backups) are critical.

1.4 Buffer Cache & Buffer Management

• Purpose: Caching disk blocks in main memory to reduce physical I/O. Acts as a buffer pool between kernel's block I/O and disk device.

• Buffer Header Structure: (Per-buffer metadata in memory)

  
  struct buf {
  
      int dev;           // Device number
  
      int blocknum;      // Disk block number
  
      struct buf *next;  // Link for hash queue/free list
  
      char *data;        // Pointer to actual data (512/4K bytes)
  
      int flags;         // Status: B_BUSY, B_VALID, B_DIRTY, B_AGE
  
  };
  
  

• Buffer Pool Organization: Three linked lists managed by the kernel:

  1. Hash Queue: Hash table of lists (key: (dev, blocknum)). For fast lookup of a specific block.

  2. Free List: List of available (unlocked) buffers. Used when a new buffer is needed.

  3. Device Queue: List of buffers waiting for I/O on a specific device.

• Operation Flow:

  1. Search Hash Queue for (dev, blocknum).

  2. If found and B_VALID, use it (cache hit).

  3. If not found, take a buffer from Free List.

  4. If Free List empty, may need to write back a dirty buffer (from Device Queue) to free one.

  5. Mark buffer B_BUSY, issue I/O, place on Device Queue.

  6. On I/O completion, mark B_VALID, move to Hash Queue, wake waiters.

[!TIP] Exam Focus: Be able to draw and explain the three lists (Hash, Free, Device) and buffer header fields. Understand the cache hit/miss and buffer allocation flow.

1.5 File System Operations: Mounting & Unmounting

• Mounting: Attaching a file system (on a device/partition) to the directory tree at a mount point (existing directory).

  • Process: Kernel reads the device's super block, validates it, adds a mount table entry linking the mount point to the file system's root inode.

  • Purpose: Makes file system accessible; integrates multiple storage devices into a single namespace.

• Unmounting: Detaching a mounted file system.

  • Process: Kernel ensures no open files/active references, writes back all dirty buffers (sync), invalidates cached inodes/data for that FS, removes mount entry.

  • Advantages of Mounting: Unified namespace, access control via directory permissions, resource sharing.

  • Disadvantages/Risks: Single point of failure (if root FS corrupt), security risk if mount points are misconfigured (e.g., noexec, nosuid flags missing).

1.6 System Calls Interface

• Purpose: The controlled gateway for user processes to request services from the kernel (hardware access, process control). Provides abstraction and protection.

• Categories: Process Control (fork, exit), File Management (open, read, write, close), Device Management, Information Maintenance, Communication.

• Detailed System Calls:

  • open(path, flags, mode): Opens a file/device. Returns a file descriptor (small integer). flags: O_RDONLY, O_WRONLY, O_CREAT, O_TRUNC. mode: permissions (e.g., 0644).

  • read(fd, buffer, count): Reads count bytes from fd into buffer. Returns bytes read or -1 on error.

  • create(path, mode): Synonym for open with O_CREAT|O_EXCL|O_WRONLY. Creates a new regular file with mode permissions.

  • chmod(path, mode): Changes file permission bits (owner/group/other: rwx). Requires ownership or CAP_FOWNER (root).

[!TIP] Exam Focus: Know parameters, return values, and error conditions for these syscalls. open vs create is a common distinction.

1.7 Data Structures in OS

• Queue (FIFO):

  • Structure: Linear list with enqueue (add to tail) and dequeue (remove from head).

  • Applications:

    • Process Scheduling: Ready queue (FCFS), I/O device queues.

    • Bounded Buffer Problem: Producer-Consumer synchronization.

    • Spooling: Print job queues.

• Tree (Hierarchical):

  • Structure: Nodes with parent/child relationships. Typically multi-way trees (B-trees for file systems).

  • Applications:

    • Directory Structures: File system hierarchy (tree of inodes/dentries).

    • Memory Management: Page tables (multi-level), segment trees.

    • Process Management: Process tree (parent-child relationships).

2.0 Process Management

2.1 Process Concept & Life Cycle

• Process: An instance of a program in execution. It is a resource container (memory, files, I/O) and a schedulable entity.

• States & Transitions:

  
  New → Ready → Running → Waiting → Ready → Running → Terminated
  
         ↑________|___________________________|
  
  

  1. New: Process created (PCB allocated).

  2. Ready: Waiting for CPU (in Ready Queue).

  3. Running: Executing on CPU.

  4. Waiting/Blocked: Waiting for an event (I/O, signal, resource).

  5. Terminated: Process completed, PCB/resources reclaimed.

• State Transition Triggers:

  • Admit (New→Ready): Scheduler.

  • Dispatch (Ready→Running): CPU scheduler.

  • Timeout/Interrupt (Running→Ready): Scheduler.

  • I/O Request (Running→Waiting): Process.

  • I/O Complete (Waiting→Ready): OS.

  • Exit (Running→Terminated): Process.

2.2 Process Control Block (PCB)

• Definition: The kernel's internal data structure that completely represents a process. It is the "process's identity card".

• Components:

  • Process State: (New, Ready, etc.)

  • Process ID (PID): Unique integer identifier.

  • Program Counter (PC): Address of next instruction.

  • CPU Registers: General, status, stack pointer (saved on context switch).

  • Memory Management Info: Page tables, segment tables, base/limit registers.

  • I/O Status Info: Open file descriptors, I/O devices allocated.

  • Accounting Info: CPU time used, limits, priority.

  • Scheduling Info: Priority, scheduling queue pointers.

[!TIP] Exam Focus: PCB is kernel-resident. Context switch involves saving/restoring CPU registers and PC from/to PCB.

2.3 Process Scheduling

• Objectives: Maximize CPU utilization and throughput, minimize turnaround time, waiting time, response time.

• Scheduling Criteria:

  • CPU Utilization: % time CPU busy.

  • Throughput: # processes completed per unit time.

  • Turnaround Time: Total time from submission to completion.

  • Waiting Time: Total time spent in Ready Queue.

  • Response Time: Time from submission to first response (for interactive).

• Common Algorithms:

  | Algorithm | Principle | Preemptive? | Pros | Cons | | :--- | :--- | :--- | :--- | :--- | | FCFS/FIFO | Processes in arrival order. | No | Simple, fair? | Convoy effect, high avg. wait/turnaround. | | SJF/SRTF | Shortest burst time first. | SRTF: Yes | Minimizes avg. wait time. | Starves long jobs; burst time prediction hard. | | Priority Scheduling| Higher priority first. | Yes/No | Flexible. | Starvation of low-priority. Solution: Aging. | | Round Robin (RR)| Time quantum (q) per process. | Yes | Good response time for interactive. | High context switch overhead if q too small. | | Multilevel Queue| Separate queues for classes (e.g., system, interactive, batch). | Yes (between queues) | Policy separation. | Scheduling between queues fixed; starvation possible. | | Multilevel Feedback Queue (MLFQ)| Multiple RR queues with different q. Move process between queues based on behavior. | Yes | Balances response & throughput; favors I/O-bound. | Complex parameter tuning. |

[!TIP] Exam Focus: Calculate avg. waiting/turnaround time for FCFS, SJF, RR. Know convoy effect (FCFS), aging (priority), time quantum effect (RR).

3.0 Concurrency & Synchronization

3.1 Principles of Concurrency

• Need: Multiprogramming, multi-processors, distributed systems. Improves resource utilization and throughput.

• Core Problem: Race Condition: Multiple processes/threads concurrently access shared data, and final outcome depends on execution timing.

• Critical Section Problem: Code segment accessing shared resource (variable, data structure, file) that must execute atomically.

• Requirements for Solution:

  1. Mutual Exclusion: Only one process in critical section at a time.

  2. Progress: If no process in CS and others wish to enter, decision cannot be postponed indefinitely.

  3. Bounded Wait: Finite wait time for a process to enter CS after request.

3.2 Semaphores

• Definition: An integer variable (protected by atomic operations) used for signaling and coordination between processes/threads. Not a counter of resources itself, but a control variable.

• Operations (must be atomic):

  • wait(S) / P(S):

    
    while (S <= 0) ; // busy wait (software)
    
    S--;
    
    

    Or with hardware support (atomic test-and-set):

    
    if (S <= 0) block(); // block process, add to semaphore list
    
    else S--;
    
    

  • signal(S) / V(S):

    
    S++;
    
    if (processes blocked on S) wakeup(a process);
    
    

• Types:

  • Counting Semaphore: Integer value can be any non-negative number. Used to control access to a pool of identical resources (e.g., 3 printers).

  • Binary Semaphore (Mutex): Value is only 0 or 1. Used for mutual exclusion (protecting a critical section). Not the same as a mutex lock (which often includes ownership).

• Implementation: Kernel maintains a wait queue per semaphore. wait atomically checks value; if <=0, process is blocked and placed on queue. signal increments value and wakes one blocked process (FIFO).

[!TIP] Exam Focus: Distinguish counting vs binary semaphore. wait/signal must be atomic; busy-wait vs block/wakeup. Use semaphores for ordering (e.g., producer-consumer) and mutual exclusion.

4.0 Interprocess Communication (IPC)

4.1 IPC Overview

• Need: Processes need to exchange data and synchronize beyond parent-child (e.g., unrelated processes, network).

• Mechanisms Comparison:

  | Mechanism | Speed | Synchronization | Kernel Involvement | Use Case | | :--- | :--- | :--- | :--- | :--- | | Shared Memory | Fastest | Explicit (semaphores) | Setup/attach only | High-throughput, local | | Message Queues | Medium | Implicit (msg blocking) | Full (copy via kernel) | Reliable, queued, local/network | | Pipes | Medium | Implicit (blocking) | Full | Simple parent-child | | Sockets | Slower (network) | Implicit/Explicit | Full (network stack) | Network communication |

4.2 Shared Memory

• Implementation Model:

  1. Create/Attach: One process (shmget, shmat) creates a shared memory segment (identified by key). Others attach to same physical pages mapped into their virtual address spaces.

  2. Access: Processes read/write directly to the shared memory area as if it were local memory.

• Advantages: Extremely fast (no kernel copy on each access).

• Challenges: No implicit synchronization. Processes must use semaphores or mutexes placed in shared memory to coordinate access. Risk of race conditions and corruption.

4.3 Message Queues

• Implementation Model: Kernel maintains a linked list of messages (each with type, size, data) in a fixed-size queue. Processes send/receive via kernel-mediated syscalls (msgsnd, msgrcv).

  • Client/Server Example:

    • Server: Creates queue (msgget), loops msgrcv (blocks if empty).

    • Client: msgsnd to queue, optionally blocks if full.

    • Messages can be typed; client can receive specific type.

• Advantages: Kernel-managed buffering, reliable delivery, message typing, natural synchronization (blocking on empty/full).

• Disadvantages: Slower (copy between user and kernel space twice), limited by kernel memory.

4.4 Sockets

• Definition: An endpoint for bidirectional communication between two processes, potentially on different machines. Identified by an IP address + port number.

• Types:

  • Stream Sockets (SOCK_STREAM): Use TCP. Reliable, connection-oriented, in-order, no message boundaries. send()/recv() byte-stream.

  • Datagram Sockets (SOCK_DGRAM): Use UDP. Unreliable, connectionless, preserves message boundaries. sendto()/recvfrom() specify address each time.

• Socket Programming Steps (TCP):

  1. socket(): Create socket descriptor.

  2. bind() (Server): Attach to local IP:port.

  3. listen() (Server): Mark socket as passive, set backlog.

  4. accept() (Server): Block until connection request; returns new connected socket.

  5. connect() (Client): Actively connect to server.

  6. send()/recv() or write()/read(): Communicate on connected socket.

  7. close(): Release socket.

[!TIP] Exam Focus: Contrast stream (TCP) vs datagram (UDP). Know the state diagram for TCP socket server (socket→bind→listen→accept). bind is optional for clients (OS picks ephemeral port).

5.0 Security Fundamentals

5.1 Authentication & Access Control

• Authentication Methods:

  • Something you know: Passwords, PINs.

  • Something you have: Tokens, smart cards, OTP apps.

  • Something you are: Biometrics (fingerprint, face).

  • Somewhere you are: Location-based.

• Access Control Models:

  • DAC (Discretionary Access Control): Owner decides permissions. Implementation: User ID (UID), Group ID (GID), Permission Bits (rwx for owner/group/other). chmod changes bits.

  • MAC (Mandatory Access Control): System-enforced policy based on security labels (e.g., military levels: Top Secret, Secret). Implementation: SELinux, AppArmor.

  • RBAC (Role-Based Access Control): Permissions assigned to roles, users assigned to roles. Common in enterprises.

• OS Implementation: UID/GID in PCB, file inode stores st_uid, st_gid, st_mode. Kernel checks credentials on every access.

5.2 Malware & Threats

5.3 Common Vulnerabilities & Exposures (CVEs)

• Concept: Publicly known cybersecurity vulnerabilities assigned CVE IDs (e.g., CVE-2021-44228 - Log4Shell). Database: cve.mitre.org.

• Common OS-Level Vulnerabilities:

  • Buffer Overflow: Overwriting adjacent memory by writing beyond buffer bounds. Leads to code execution (stack/heap overflow).

  • Privilege Escalation: Exploit allowing unprivileged user/process to gain higher privileges (root/Administrator). Often via kernel bugs or misconfigurations.

  • Misconfigurations: Default passwords, unnecessary services running, improper file permissions (777), weak firewall rules.

  • Race Conditions: TOCTOU (Time-of-Check-Time-of-Use) in file access, temp file creation.

5.4 Security Mechanisms & Techniques

• Honeypots:

  • Definition: Decoy system or resource designed to attract, detect, and analyze attacks.

  • Types:

    • Production Honeypot: Inside production network, low-interaction (emulated services), early warning.

    • Research Honeypot: Isolated, high-interaction (real OS), detailed attack analysis.

  • Deployment: Placed in DMZ or isolated network. Logs all interaction to study attacker TTPs (Tactics, Techniques, Procedures).

• Virtualization for Security:

  • Isolation/Sandboxing: Run untrusted code/apps in a VM or container. Compromise contained.

  • Live Forensic Analysis: Snapshot/suspend a compromised VM for offline analysis without affecting host.

  • Hardware-Assisted Virtualization (Intel VT-x, AMD-V): Enforces strong isolation between VMs and hypervisor.

6.0 Mobile & Specialized Security

6.1 Mobile OS Security

• Unique Vulnerabilities:

  • App Permissions: Over-privileged apps, permission creep.

  • SMS/Telephony Exploits: USSD codes, SS7 protocol flaws, SIM attacks.

  • Sensor Access: Microphone, camera, GPS, accelerometer misuse.

  • OS Fragmentation: Delayed patches, outdated versions in use.

  • Side-Channel Attacks: Power analysis, EM emanations.

  • Physical Access: Theft, loss, JTAG/debug port access.

6.2 Android Security

• Security Levels (Defense-in-Depth):

  1. Linux Kernel Security: SELinux (enforcing mode) for MAC; user/group isolation.

  2. Application Sandbox: Each app runs as unique Linux UID/GID. Isolated by Linux namespaces and seccomp filters.

  3. Permission Model: Declarative permissions (install-time or runtime for dangerous). AndroidManifest.xml.

  4. Verified Boot: Chain of trust from bootloader to system partitions. Detects tampering.

  5. Keystore: Hardware-backed (TEE/TrustZone) storage for cryptographic keys. Keys never leave secure hardware.

6.3 Secure Hardware & Elements

• Secure Coprocessor:

  • Definition: Isolated, tamper-resistant processor (often separate chip) with its own secure memory and OS.

  • Features: Physical tamper detection (epoxy, sensors), secure key storage (non-exportable), Trusted Execution Environment (TEE).

  • Use Cases: DRM, secure boot, payment (Apple Secure Enclave, TPM).

• Secure Wallet:

  • Definition: Digital credential storage (cryptocurrency keys, credit cards, IDs) in a hardware-backed, encrypted store.

  • Implementation: Uses Secure Element (SE) or TEE (TrustZone). Keys are non-extractable; cryptographic operations happen inside secure hardware.

  • Use Cases: Mobile payments (Apple Pay, Google Pay), cryptocurrency wallets (Ledger, Trezor), digital IDs.

7.0 Windows Internals (Security Perspective)

7.1 Windows System Architecture

• Layered Architecture:

  
  User Mode:
  
    Win32 Subsystem (CSRSS.EXE) - Console, windows
  
    Other Subsystems (POSIX)
  
    NTDLL.DLL - Syscall stub (Nt* functions)
  
  Kernel Mode:
  
    Executive (NTOSKRNL.EXE) - Object Manager, I/O Manager, Process Manager, Security Reference Monitor
  
    Kernel - Thread scheduler, interrupt/dispatch, exceptions
  
    Hardware Abstraction Layer (HAL) - Hardware-specific code
  
  

• Key Components:

  • Object Manager: Central namespace (\, \Device, \BaseNamedObjects) for all kernel objects (processes, threads, files, events). Security descriptor attached to each object.

  • I/O Manager: Handles I/O requests via IRPs (I/O Request Packets), routes to drivers.

  • Process Manager: Creates/kills processes/threads, manages PEPROCESS and PETHREAD objects.

7.2 Windows Sockets (Winsock)

• Winsock API Overview: Microsoft's implementation of Berkeley sockets for Windows.

  • Initialization: WSAStartup() before any other call.

  • Core Functions: socket(), bind(), listen(), accept(), connect(), send(), recv(), closesocket(), WSACleanup().

• Datagram Sockets (UDP):

  • Use: Connectionless, unreliable but low-latency. No listen/accept needed.

  • Functions: sendto() (specify destination address), recvfrom() (gets source address).

  • Security Note: Spoofable, no guaranteed delivery. Used for DNS, VoIP, gaming.

7.3 Windows Kernel Objects & Threads

• System Worker Threads:

  • Definition: Kernel-mode threads created by the Executive (not by any user process) to perform asynchronous work on behalf of the system.

  • Purpose:

    • Delayed Work: Execute work items after a delay (KeDelayExecutionThread).

    • Asynchronous I/O Completion: Handle completed I/O requests.

    • System Services: Memory manager, cache manager worker threads.

  • Management: Created via PsCreateSystemThread(). Run in System Process (PID 4). No user-mode context.

7.4 Windows Debugging & Forensics Flags

• Global Flags (gflags):

  • Definition: Registry settings (HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Image File Execution Options\<exe>) or PE header flags that alter system/process behavior for debugging.

  • Tool: gflags.exe (from Debugging Tools for Windows).

  • Common Uses:

    • Loader Snaps: Break on DLL load/unload (FLG_LOADER_BREAK_ON_LOAD).

    • Heap Debugging: Enable stack traces, page heap (FLG_HEAP_ENABLE_TAIL_CHECK, FLG_HEAP_ENABLE_FREE_CHECK).

    • Stop on Exceptions: First-chance exceptions (FLG_STOP_ON_EXCEPTION).

    • Disable Protected Mode (IE): For exploit development.

  • Forensics: Presence of global flags can indicate malware debugging or defensive monitoring.

[!TIP] Exam Focus: Know Windows architecture layers (Executive, Kernel, HAL). System process (PID 4) owns system worker threads. gflags modifies Image File Execution Options in registry for per-executable debugging.