IT-407: Open Source Software Lab (Linux and R) - UNIT 2 Short Notes

I. ADVANCED LINUX SYSTEM ADMINISTRATION & OPERATIONS

A. File System Management & Permissions

• Inodes & Links:

  • Inode: Data structure storing file metadata (permissions, size, timestamps, block pointers). Each file has a unique inode number. ls -i displays inode numbers.

  • Hard Link (ln): Additional directory entry pointing to the same inode. Cannot cross filesystems. Cannot link directories (prevents loops). ls -l shows link count in the second column.

  • Symbolic (Soft) Link (ln -s): Special file containing path to target. Can cross filesystems, link directories. Becomes broken (dangling) if target is moved/deleted.

• Key Directories:

  • /etc: Configuration files.

  • /var: Variable data (logs, spool, caches).

  • /usr: User programs and read-only data.

  • /tmp & /var/tmp: Temporary files (often mounted with sticky bit).

• Permission Management:

  • chmod: Changes permissions.

    • Symbolic: u/g/o/a +/- rwx (e.g., chmod u+x file, chmod o-w file).

    • Numeric: 3 octal digits (owner, group, others). r=4, w=2, x=1. chmod 755 file = rwxr-xr-x.

    [!TIP] Calculate numeric: r=4, w=2, x=1. Sum for each class. chmod 640 file = rw-r-----.

  • chown: Change owner (chown user:group file).

  • chgrp: Change group (chgrp group file).

  • umask: Mask subtracted from default permissions (usually 022 for files 666, dirs 777). Sets default for new files. umask 002 gives 664 for files.

• Special Permissions:

  • SUID (Set User ID): 4xxx on executable. Runs with file owner's privileges (not executor's). Example: /usr/bin/passwd.

  • SGID (Set Group ID): 2xxx on executable or directory.

    • On file: Runs with file group's privileges.

    • On directory: New files inherit the directory's group.

  • Sticky Bit: 1xxx on directory (e.g., /tmp). Only file owner can delete/rename their files.

• Mounting:

  • mount /dev/sdXN /mnt/point: Mount temporarily.

  • umount /mnt/point: Unmount.

  • /etc/fstab: Persistent mounts. Format: <device> <mount_point> <type> <options> <dump> <pass>.

    
    # /etc/fstab: static file system information
    
    /dev/sda1  /  ext4  defaults,noatime  0  1
    
    

• Disk Usage & Quotas:

  • df -h: Disk free (filesystem usage).

  • du -sh <dir>: Disk usage (summary for directory).

  • Quotas: Per-user/group disk limits.

    1. Enable in /etc/fstab with usrquota,grpquota.

    2. quotacheck -cug / (create quota files).

    3. quotaon /.

    4. edquota -u username: Edit limits (blocks & inodes).

B. Process Management & System Monitoring

• Process Basics:

  • PID: Process ID.

  • PPID: Parent Process ID.

  • Process States: R (Running), S (Sleeping), D (Uninterruptible), Z (Zombie), T (Stopped).

  • Process Tree: pstree or ps -ef --forest.

• Job Control:

  • command &: Run in background.

  • jobs: List background jobs.

  • bg %n: Resume job n in background.

  • fg %n: Bring job n to foreground.

  • kill -SIGNAL PID:

    • -15 (SIGTERM): Graceful termination (default).

    • -9 (SIGKILL): Force kill (cannot be caught/ignored).

• Monitoring Tools:

  • top / htop: Interactive, real-time system view (CPU, MEM, load).

  • ps:

    • ps aux: BSD style (USER, %CPU, %MEM, VSZ, RSS, START, TIME, COMMAND).

    • ps -ef: System V style (UID, PID, PPID, C, STIME, TTY, TIME, CMD).

  • free -h: Memory usage (total, used, free, cached).

  • vmstat 1: Virtual memory stats (procs, memory, swap, io, system, cpu).

  • iostat 1: CPU & I/O statistics.

• System Load:

  • Load Average: Average number of runnable + uninterruptible processes over 1, 5, 15 mins.

  • uptime & w: Show load averages.

  • [!TIP] Rule of thumb: Load avg should not exceed number of CPU cores for sustained periods.

• systemd Service Management:

  • systemctl start/stop/restart <service>: Control service now.

  • systemctl enable/disable <service>: Control auto-start on boot.

  • systemctl status <service>: Detailed status (active, logs).

  • systemctl list-units --type=service --all: List all services.

  • Unit Files: Located in /lib/systemd/system/ or /etc/systemd/system/. Use systemctl edit <service> to create override.

C. Package Management & Software Maintenance

• apt (Debian/Ubuntu):

  • apt update: Update package lists.

  • apt upgrade: Upgrade all upgradable packages.

  • apt install <pkg>: Install.

  • apt remove <pkg>: Remove (keep configs).

  • apt purge <pkg>: Remove (delete configs).

  • apt search <pattern>: Search.

  • apt list --installed: List installed.

  • apt show <pkg>: Show package details.

  • apt autoremove: Remove unused dependencies.

  • apt clean: Clear local repo cache.

• yum/dnf (RHEL/CentOS/Fedora):

  • yum/dnf install/remove/update/search.

  • yum/dnf list installed.

  • yum/dnf clean all.

• Repository Configuration:

  • APT: /etc/apt/sources.list & /etc/apt/sources.list.d/*.list.

  • YUM/DNF: /etc/yum.repos.d/*.repo.

• Compiling from Source:

  1. ./configure: Checks system, generates Makefile. Use --prefix=/custom/path.

  2. make: Compiles source code.

  3. sudo make install: Installs to system.

  [!TIP] Dependencies: Install -dev or -devel packages for header files.

• Low-Level Package Tools:

  • dpkg -i <file.deb>: Install local .deb.

  • dpkg -r <pkg>: Remove.

  • dpkg -l: List installed.

  • rpm -i <file.rpm>: Install local .rpm.

  • rpm -e <pkg>: Remove.

  • rpm -qa: Query all installed.

D. User & Group Administration

• Key Files:

  • /etc/passwd: User info (username:password:UID:GID:GECOS:home:shell). Password field is x (shadowed).

  • /etc/shadow: Encrypted passwords & aging info.

  • /etc/group: Group definitions (groupname:password:GID:members).

  • /etc/gshadow: Secure group passwords.

• Commands:

  • useradd [options] username: Create user (-m create home, -s /bin/bash, -G group1,group2).

  • usermod [options] username: Modify (-l newname, -G newgroups, -s newshell).

  • userdel -r username: Delete user & home/mail.

  • groupadd groupname: Create group.

  • groupmod -n newname oldname: Rename group.

  • passwd username: Set/change password.

• Password Policies & Aging (chage):

  • chage -l username: List aging info.

  • chage -M 90 username: Max days (90).

  • chage -m 7 username: Min days.

  • chage -E 2025-12-31 username: Account expiry.

  • /etc/login.defs: Defaults for new users (PASS_MAX_DAYS, UID_MIN).

• Privilege Escalation:

  • su - username: Switch user (load target's environment).

  • sudo command: Execute as root (or other user).

  • /etc/sudoers: Configured via visudo.

    
    %admin  ALL=(ALL) ALL      # Group 'admin' can sudo anywhere
    
    alice   ALL=(root) /usr/bin/systemctl restart nginx
    
    

  • sudo -i: Get root shell.

  • sudo -u otheruser command: Run as otheruser.

II. NETWORKING, SECURITY & TROUBLESHOOTING

A. Network Configuration & Diagnostics

• Interface Configuration:

  • ip addr show / ip a: Modern, detailed IP info.

  • ip link set <iface> up/down: Bring interface up/down.

  • ifconfig: Legacy (still used). Often requires sudo.

  • nmcli: NetworkManager CLI (common on desktops).

• Connectivity Testing:

  • ping <host/IP>: ICMP echo test.

  • traceroute <host> / tracepath <host>: Path & hop latency.

  • mtr <host>: Combines ping + traceroute (continuous).

  • dig <domain> / nslookup <domain>: DNS queries.

  • /etc/resolv.conf: DNS resolver config (nameserver).

  • /etc/hosts: Static hostname-IP mapping.

• Port & Service Checking:

  • ss -tuln: Modern replacement for netstat. Show listening (-l) TCP/UDP (-tuln).

  • netstat -tulpn: Show process name/PID (-p).

  • telnet <host> <port>: Test TCP connectivity (legacy).

  • nc -zv <host> <port>: Netcat (modern, scriptable).

• Network Config Files:

  • Netplan (Ubuntu): /etc/netplan/*.yaml. Apply with netplan apply.

  • RHEL/CentOS (legacy): /etc/sysconfig/network-scripts/ifcfg-<iface>.

B. Basic System Security

• Firewall (ufw - Ubuntu / firewalld - RHEL):

  • UFW:

    • ufw status verbose: Show rules.

    • ufw allow 22/tcp: Allow SSH.

    • ufw deny from 192.168.1.100: Block IP.

    • ufw enable/disable.

  • firewalld:

    • firewall-cmd --state: Status.

    • firewall-cmd --zone=public --add-service=ssh --permanent.

    • firewall-cmd --reload.

• SSH Hardening (/etc/ssh/sshd_config):

  • PermitRootLogin no: Disable root login.

  • PasswordAuthentication no: Disable password auth (use keys only).

  • Port 2222: Change default port (22).

  • AllowUsers alice bob: Restrict users.

  • Restart: systemctl restart sshd.

• Key-Based Auth:

  1. ssh-keygen -t ed25519 (or rsa): Generate key pair.

  2. ssh-copy-id -i ~/.ssh/id_ed25519.pub user@host: Copy public key.

  3. Test: ssh user@host (no password).

• Auditing & Scanning:

  • Logs: /var/log/auth.log (Debian) or /var/log/secure (RHEL) for sudo/SSH attempts.

  • fail2ban: Monitors logs, bans IPs after repeated failures. Configured in /etc/fail2ban/jail.local.

  • File Integrity: aide or tripwire create baseline hashes; detect changes.

C. Log Analysis & System Troubleshooting

• Log Hierarchy (/var/log/):

  • syslog / messages: General system.

  • auth.log / secure: Authentication/SSH/sudo.

  • kern.log: Kernel messages.

  • dmesg: Kernel ring buffer (boot messages).

• journalctl (systemd journal):

  • journalctl -u nginx.service: Filter by unit.

  • journalctl -f: Follow (tail).

  • journalctl --since "1 hour ago": Time filter.

  • journalctl -p err: Priority filter (emerg, alert, crit, err).

  • journalctl -k: Kernel messages only.

• Diagnostic Workflow:

  1. systemctl status <service>: Is it running? What's the error?

  2. journalctl -u <service> -p err --since today: Check service logs.

  3. dmesg | tail: Check kernel ring for hardware/driver issues.

  4. Verify config: nginx -t (for web server), sshd -t.

  5. Check resource: free, df -h, top.

III. SHELL SCRIPTING & AUTOMATION (BASH)

A. Script Fundamentals

• Shebang: #!/bin/bash (or #!/usr/bin/env bash).

• Permissions & Execution: chmod +x script.sh then ./script.sh or bash script.sh.

• Variables:

  • var=value (no spaces). read var for input.

  • Special Variables:

    • $?: Last command's exit status.

    • $#: Number of arguments.

    • $@` / `$*: All arguments (preserves/doesn't preserve quoting).

    • $$: Current script's PID.

    • $!: Last background command's PID.

    • $0: Script name.

• I/O:

  • echo "text" / printf "format": Output.

  • read -p "Prompt: " var: Input.

  • Redirection:

    • >: Stdout to file (overwrite).

    • >>: Stdout to file (append).

    • 2>: Stderr to file.

    • &>: Both stdout & stderr.

    • <: Input from file.

  • Pipes (|): Connect stdout of left to stdin of right.

B. Control Structures

• Conditionals (test / [ ] / [[ ]]):

  
  if [ -f "$file" ]; then   # -f: exists and is regular file
  
      echo "Exists"
  
  elif [[ $var -eq 10 ]]; then  # [[ ]] supports &&, ||, pattern matching
  
      echo "Var is 10"
  
  else
  
      echo "Else"
  
  fi
  
  

  • String: -z (empty), -n (non-empty), = (equal), !=.

  • Numeric: -eq, -ne, -gt, -lt, -ge, -le.

  • File: -e (exists), -f, -d, -r, -w, -x.

  • (( )): Arithmetic evaluation (e.g., if (( a > b )); then).

• Loops:

  
  for i in 1 2 3; do echo $i; done           # List
  
  for file in *.txt; do echo $file; done     # Globbing
  
  for (( i=0; i<10; i++ )); do echo $i; done # C-style
  
  while read line; do echo "$line"; done < file
  
  until [ condition ]; do ...; done
  
  

  • break: Exit loop.

  • continue: Skip to next iteration.

• Case Statement:

  
  case $var in
  
      start|run) echo "Starting..." ;;
  
      stop) echo "Stopping..." ;;
  
      *) echo "Unknown" ;;
  
  esac
  
  

C. Functions & Advanced Scripting

• Functions:

  
  myfunc() {
  
      local local_var=$1   # Local variable
  
      echo "Arg1: $1"
  
      return 0            # Exit status (0-255)
  
  }
  
  myfunc arg1
  
  echo "Returned: $?"   # Capture return status
  
  

• Arrays:

  
  arr=(one two three)
  
  echo ${arr[0]}        # one
  
  echo ${#arr[@]}       # Length: 3
  
  for i in "${arr[@]}"; do echo $i; done
  
  

• String & Arithmetic:

  • Parameter Expansion: ${var#pattern}` (remove shortest prefix), `${var##pattern} (longest), ${var%pattern}`, `${var%%pattern}.

  • Arithmetic: $(( expression )), let "var = 10 + 5", expr.

• Error Handling:

  • set -e: Exit on any command failure.

  • set -u: Treat unset variables as error.

  • set -o pipefail: Pipeline fails if any command fails.

  • trap 'cleanup' EXIT: Run cleanup on script exit.

• Practical Examples:

  • Backup: tar -czf /backup/$(date +%F).tar.gz /home/user.

  • Log Rotation: logrotate config or custom script with find /var/log -mtime +30 -delete.

  • Health Check: Script checking disk (df -h), load (uptime), services (systemctl is-active).

IV. INTRODUCTION TO R FOR DATA ANALYSIS

A. R Environment & Basics

• RStudio Panes:

  1. Console: Interactive command line.

  2. Source/Editor: Write scripts (.R files).

  3. Environment/History: Variables, command history.

  4. Files/Plots/Help/Packages: Navigation, output, documentation.

• Basic Syntax:

  • Vectors: c(1, 2, 3), 1:10, seq(1, 10, 0.5).

  • Arithmetic: +, -, *, /, ^, %% (mod), %/% (integer div).

  • Logical: ==, !=, >, <, >=, <=, !, &, |.

• Objects & Workspace:

  • Assignment: x <- 10 (preferred) or x = 10.

  • ls(): List objects.

  • rm(x): Remove object.

  • save(list=ls(), file="workspace.RData"): Save all.

  • load("workspace.RData"): Load.

  • .RData is loaded automatically on startup.

• Getting Help:

  • help(lm) or ?lm: Help on function lm.

  • example(lm): Run examples.

  • help.search("regression") or ??regression: Search help pages.

  • help(package="ggplot2"): Package help.

B. Data Structures in R

• Atomic Vectors: Homogeneous type.

  • numeric, character, logical, complex.

  • c(TRUE, FALSE, TRUE) -> logical.

• Attributes: Metadata (names, dim, class, str).

  • names(x) <- c("a","b"): Name vector elements.

• Matrices & Arrays:

  • matrix(1:6, nrow=2, byrow=TRUE).

  • array(1:24, dim=c(2,3,4)): 3D array.

  • dim(x): Get/set dimensions.

  • rbind(), cbind(): Row/column bind.

• Lists: Heterogeneous. list(a=1, b="text", c=TRUE).

  • x[[1]]: Extract element (preserves type).

  • x[1]: Extract sub-list (still a list).

  • lapply(list, function): Apply function to each element.

• Data Frames: Primary tabular structure. List of equal-length vectors (columns).

  • df <- data.frame(id=1:3, name=c("A","B","C"), score=c(8.5, 9.0, 7.5)).

  • str(df): Structure (column names, types, preview).

  • summary(df): Descriptive stats for each column.

  • nrow(df), ncol(df): Dimensions.

C. Data Import/Export & Manipulation (Base R)

• Import:

  • read.csv("file.csv", header=TRUE, stringsAsFactors=FALSE).

  • read.table("file.txt", sep="\t", header=TRUE).

  • readRDS("file.rds"): Save/load R objects efficiently.

  • [!TIP] Always use stringsAsFactors=FALSE (R 4.0+ default) to avoid automatic factor conversion.

• Export:

  • write.csv(df, "out.csv", row.names=FALSE).

  • write.table(df, "out.txt", sep="\t").

  • saveRDS(df, "out.rds").

• Subsetting Data Frames:

  • df[rows, cols]: df[1:5, c("name","score")].

  • df$column: Extract column as vector.

  • subset(df, score > 8, select=c(name,score)): Logical subset.

• Factors:

  • factor(c("low","medium","high"), levels=c("low","medium","high"), ordered=TRUE).

  • Caution: as.numeric(factor) returns internal integer codes, not original numbers! Use as.numeric(as.character(factor)) or levels(factor)[factor].

• Basic Cleaning:

  • is.na(df), sum(is.na(df)): Identify NAs.

  • df[complete.cases(df), ]: Keep complete rows.

  • na.omit(df): Remove rows with any NA.

  • unique(df$column), duplicated(df$column).

V. DATA VISUALIZATION & STATISTICAL ANALYSIS IN R

A. Base R Graphics

• Graphics Device: Start with png("plot.png", width=800, height=600), plot, then dev.off().

• Core Functions:

  • plot(x, y): Scatter.

  • hist(x): Histogram.

  • boxplot(x ~ group): Box-and-whisker.

  • barplot(height): Bar chart.

  • pie(x): Pie chart.

• Customization:

  • title(main="Title", sub="Sub", xlab="X", ylab="Y").

  • legend("topright", legend=c("A","B"), col=c("red","blue"), pch=1).

  • grid(): Add grid.

  • par(mfrow=c(2,2)): 2x2 multi-panel. Reset with par(mfrow=c(1,1)).

  • layout(matrix(1:3, nrow=1)): Complex layouts.

B. ggplot2 (Grammar of Graphics)

• Philosophy: Plot = Data + Aesthetics (aes) + Geometries (geom_*) + Facets + Scales + Theme.

• Building Layer-by-Layer:

  
  library(ggplot2)
  
  ggplot(data=mtcars, aes(x=wt, y=mpg, color=cyl)) +  # Data & Aesthetics
  
    geom_point(size=3) +                              # Geometry 1
  
    geom_smooth(method="lm", se=FALSE) +              # Geometry 2
  
    labs(title="Car Weight vs MPG", x="Weight (1000 lbs)") +
  
    theme_minimal()
  
  

• Common Geoms: geom_point(), geom_line(), geom_bar(stat="identity"), geom_histogram(binwidth=), geom_boxplot().

• Faceting: facet_wrap(~cyl), facet_grid(rows ~ cols).

• Customization: labs(), scale_x_continuous(breaks=seq(0,10,2)), theme_bw().

C. Basic Statistical Analysis & Tests

• Descriptive Stats:

  • mean(x, na.rm=TRUE), median(x), sd(x), var(x).

  • min(x), max(x), range(x), quantile(x, probs=c(0.25,0.75)).

  • summary(df): Min, Q1, Median, Mean, Q3, Max.

  • cor(x, y, method="pearson"): Correlation.

• Distribution Functions (Normal):

  • rnorm(n, mean=0, sd=1): Random generation.

  • dnorm(x, mean, sd): Density.

  • pnorm(q, mean, sd): Cumulative distribution (P(X ≤ q)).

  • qnorm(p, mean, sd): Quantile (inverse CDF).

  • Similar for dunif, punif, qunif, dpois, ppois, etc.

• Hypothesis Tests:

  • t.test(x, y=NULL, paired=FALSE)

    • One-sample: t.test(x, mu=0).

    • Two-sample (unpaired): t.test(x, y).

    • Paired: t.test(x, y, paired=TRUE).

    • Output: t-statistic, df, p-value, confidence interval.

  • chisq.test(table): Chi-square test of independence on contingency table.

  • cor.test(x, y): Test correlation significance.

• Linear Regression:

  
  model <- lm(mpg ~ wt + cyl, data=mtcars)
  
  summary(model)      # Coefficients, p-values, R-squared, F-stat
  
  predict(model, newdata=data.frame(wt=3, cyl=6))
  
  abline(model)       # Add regression line to existing plot
  
  

• ANOVA:

  
  model <- aov(score ~ group, data=df)
  
  summary(model)      # Between-group & within-group SS, F, p-value
  
  

\boxed{\text{Key Exam Focus:}}

1. Linux: chmod numeric/symbolic, SUID/SGID/sticky, systemctl, fstab, ss/netstat, journalctl, sudoers syntax, shell script control structures.

2. R: Data frame subsetting ($, [ , ]), factor pitfalls, ggplot2 layer syntax (ggplot() + geom_*()), lm()/summary(), t.test() interpretation.

3. Common Pitfalls: Hard vs soft links; set -e in scripts; factor to numeric conversion; ggplot2 aesthetics mapping vs setting; interpreting p-values in statistical tests.