Open Source Software Lab (Linux and R) (IT-407) - Unit 3 Short Notes

UNIT 3: Advanced Linux Administration & R Programming Integration

1.0 Advanced Linux Shell Scripting & Automation

1.1 Shell Scripting Fundamentals Review (Bash)

• 1.1.1 Shebang, permissions, execution

  • Shebang (#!): First line #!/bin/bash (or #!/usr/bin/env bash) specifies the interpreter.

  • Permissions: Script must be executable. Use chmod +x script.sh.

  • Execution: Run via ./script.sh (if in PATH) or bash script.sh.

• 1.1.2 Variables, quoting, and special characters

  • Variables: var=value (no spaces). Access with $var` or `${var}.

  • Quoting:

    • Double quotes ("): Allows variable/command expansion.

    • Single quotes ('): Literal string, no expansion.

    • **Backticks (`) or $()`**: Command substitution (prefer `$()).

  • Special Characters: $, \, *, ?, [, ], |, &, ;, (, ), <, >, #, space, tab. Escape with \ or quote.

• 1.1.3 Command substitution and arithmetic

  • Command Substitution: output=$(command) or output=`command`.

  • Arithmetic: $(( expression )) for integers. let, expr, or bc for floats.

[!TIP] Always quote variables ("$var") to prevent word splitting and globbing.

1.2 Control Structures in Scripts

• 1.2.1 Conditional statements (if, elif, else, case)

  • Syntax:

    
    if [ condition ]; then
    
        # commands
    
    elif [ condition ]; then
    
        # commands
    
    else
    
        # commands
    
    fi
    
    

  • case: Multi-way branch for string matching.

    
    case "$var" in
    
        pattern1) commands ;;
    
        pattern2) commands ;;
    
        *) default_commands ;;
    
    esac
    
    

• 1.2.2 Loops (for, while, until)

  • for: Iterate over list or command output.

    
    for i in {1..5}; do echo $i; done
    
    for file in *.txt; do echo "$file"; done
    
    

  • while: Loop while condition true.

    
    while [ condition ]; do commands; done
    
    

  • until: Loop until condition true.

• 1.2.3 Functions and argument handling ($1`, `$@, $*)

  • Define: function_name() { commands; } or function function_name { commands; }.

  • Call: function_name arg1 arg2.

  • Arguments: $1 (first), $2` (second), ..., `$# (count), $@` (all as separate words), `$* (all as single word).

1.3 Text Processing & Data Extraction

• 1.3.1 grep: Regular expressions, pattern matching, options (-i, -v, -r, -n)**

  • Purpose: Search files for patterns.

  • Key Options:

    • -i: Case-insensitive.

    • -v: Invert match (non-matching lines).

    • -r or -R: Recursive search.

    • -n: Show line numbers.

    • -E: Extended regex (or use egrep).

  • Basic Regex: . (any char), * (zero or more), ^ (start), $ (end), [] (char class), [^] (negated class).

• 1.3.2 sed: Stream editing, substitution, deletion, insertion**

  • Syntax: sed 'script' input_file.

  • Common Commands:

    • Substitution: s/pattern/replacement/flags (flags: g global, i case-insensitive).

    • Delete: d.

    • Insert: i\text (before line), a\text (after line).

  • Example: sed -i 's/old/new/g' file (in-place global replace).

• 1.3.3 awk: Field processing, patterns and actions, built-in variables**

  • Syntax: awk 'pattern { action }' input_file.

  • Fields: $0 (whole line), $1, $2, ... (space/tab delimited by default). FS (field separator), OFS (output FS).

  • Built-in Variables: NR (record number), NF (field count).

  • Example: awk '{ sum += $2 } END { print sum }' file (sum column 2).

• 1.3.4 cut, sort, uniq, tr, wc

  • cut: Extract columns. -d (delimiter), -f (fields). Example: cut -d',' -f1,3 file.csv.

  • sort: Sort lines. -n (numeric), -r (reverse), -k (key column).

  • uniq: Remove adjacent duplicates. -c (count), -d (only duplicates). Often piped after sort.

  • tr: Translate or delete characters. tr '[:lower:]' '[:upper:]' (uppercase).

  • wc: Word/line/byte count. -l (lines), -w (words), -c (bytes).

1.4 Script Debugging & Error Handling

• 1.4.1 Using set -x, set -e, set -u

  • set -x: Trace execution (print commands and args).

  • set -e: Exit immediately on any command failure (non-zero exit status).

  • set -u: Treat unset variables as error.

  • Combined: set -euxo pipefail (also -o pipefail fails pipeline if any command fails).

• 1.4.2 Redirecting stderr and stdout

  • >: Redirect stdout (overwrite).

  • >>: Redirect stdout (append).

  • 2>: Redirect stderr.

  • &>: Redirect both stdout and stderr.

  • 2>&1: Redirect stderr to stdout.

• 1.4.3 Exit codes and trap command

  • Exit Codes: 0 = success, non-zero = error. exit N sets code.

  • trap: Execute commands on signal (e.g., exit, interrupt). trap 'cleanup' EXIT INT TERM.

[!TIP] Use set -euo pipefail at script start for robust error handling. Always check exit codes of critical commands.

2.0 Linux System Administration & Process Management

2.1 Process Management

• 2.1.1 Viewing processes (ps, top, htop)

  • ps: Snapshot. Common: ps aux (all users, full format), ps -ef (standard format).

  • top / htop: Dynamic real-time view. htop is interactive (color, scroll).

  • Key Columns: PID (process ID), PPID (parent PID), %CPU, %MEM, COMMAND.

• 2.1.2 Signals and process control (kill, pkill, killall)

  • Signals: SIGTERM (15, polite terminate), SIGKILL (9, forceful), SIGINT (2, interrupt Ctrl+C).

  • Commands:

    • kill [signal] PID: Send signal to specific PID.

    • pkill [options] pattern: Kill by name/pattern.

    • killall [signal] name: Kill all processes with exact name.

• 2.1.3 Job control (&, jobs, fg, bg)

  • &: Run command in background.

  • jobs: List current shell's background jobs.

  • fg %n: Bring job n to foreground.

  • bg %n: Resume job n in background.

2.2 System Monitoring & Performance

• 2.2.1 Memory usage (free, vmstat)

  • free -h: Human-readable memory (RAM + swap). total, used, free, shared, buff/cache, available.

  • vmstat [interval]: Virtual memory stats. Reports: procs (r, b), memory (swpd, free, buff, cache), swap (si, so), io (bi, bo), system (in, cs), cpu (us, sy, id, wa, st).

• 2.2.2 Disk I/O and space (iostat, df, du)

  • df -h: Disk free space per filesystem.

  • du -sh [dir]: Disk usage summary for directory.

  • iostat [interval]: CPU and I/O stats for devices. %util (device utilization), await (avg wait time).

• 2.2.3 Network monitoring (netstat, ss, ip)

  • ss -tuln: Modern replacement for netstat. Show listening TCP/UDP sockets.

  • netstat -tuln: Older, similar.

  • ip addr / ip link: Show network interfaces and addresses.

2.3 Task Automation

• 2.3.1 cron and crontab syntax and scheduling

  • Crontab Format: minute hour day_of_month month day_of_week command

  • Fields: Allowed values:

    • Minute: 0-59

    • Hour: 0-23

    • Day of month: 1-31

    • Month: 1-12

    • Day of week: 0-7 (0 or 7 = Sunday)

  • Special Strings: @reboot, @yearly, @monthly, @weekly, @daily, @hourly.

  • Example: 0 2 * * * /path/to/backup.sh (daily at 2 AM).

  • Edit: crontab -e (user), sudo crontab -e (root). List: crontab -l.

• 2.3.2 at command for one-time tasks

  • Schedule a command to run once at a specific time.

  • Usage: echo "command" | at 14:30 or at 14:30 then type commands, Ctrl+D.

  • View queue: atq. Remove: atrm job_id.

• 2.3.3 Systemd timers (basic concepts)

  • Timer Units: .timer files (like cron) trigger .service units.

  • Advantages over cron: Calendar events + monotonic time (e.g., after boot), better logging, dependency control.

  • Key Timer Options: OnCalendar= (cron-like), OnBootSec=, OnUnitActiveSec=.

[!TIP] Cron environment is minimal. Use full paths in commands or set PATH in crontab. For systemd timers, ensure the corresponding .service file is WantedBy=timers.target.

3.0 Advanced R Programming Concepts

3.1 Functional Programming in R

• 3.1.1 Apply family (apply, lapply, sapply, vapply, mapply)

  • apply(X, MARGIN, FUN, ...): Apply over rows/columns of matrix/array. MARGIN=1 (rows), 2 (columns). Returns vector/matrix.

  • lapply(X, FUN, ...): Apply over list/vector, always returns list.

  • sapply(X, FUN, ...): Simplify lapply result to vector/matrix if possible.

  • vapply(X, FUN, FUN.VALUE, ...): Safer sapply with explicit return type (FUN.VALUE).

  • mapply(FUN, ...): Multivariate version of sapply (parallel apply over multiple arguments).

  • Time Complexity: Generally O(n) over elements, but overhead varies. lapply often fastest for lists; vapply safest for vectors.

• 3.1.2 purrr package: map functions, functional patterns

  • Consistent naming: map() (list output), map_lgl() (logical), map_int() (integer), map_dbl() (double), map_chr() (character), map_df() (data frame).

  • Advantages over apply: Type-safe, better error messages, works with lists/data frames, integrates with pipes (%>%).

  • Example: mtcars %>% map_dbl(mean).

• 3.1.3 Anonymous functions and functionals

  • Anonymous: function(x) x^2. Used inline: lapply(1:5, function(x) x*2).

  • Functionals: Functions that take functions as arguments (e.g., lapply, map, Reduce).

3.2 Object-Oriented Programming in R

• 3.2.1 S3 system: generic functions, methods, UseMethod

  • Generic: generic <- function(x) UseMethod("generic").

  • Method: generic.classname <- function(x) { ... }.

  • Dispatch: Based on class attribute of first argument. Informal, no formal class definition.

• 3.2.2 S4 system: setClass, setMethod, validity

  • Formal: Define class with setClass("Class", slots = list(field1 = "type")).

  • Method: setMethod("generic", "Class", function(x) { ... }).

  • Validity: setValidity("Class", function(object) { ... }) for custom checks.

  • More rigorous than S3 (formal slots, multiple dispatch).

• 3.2.3 R6 classes: reference semantics, public/private

  • Reference-based: Objects modified in place (like Python/Java).

  • Definition: R6Class("Class", public = list(initialize = function() {...}, method = function() {...}), private = list(priv = function() {...})).

  • Public/Private: Methods/fields in public accessible; private hidden.

  • Use when: Need mutable state, encapsulation, or OOP with side effects.

[!TIP] S3 is most common in base R. S4 used in Bioconductor. R6 for mutable objects. Choose based on need for formality vs. simplicity.

3.3 Advanced Data Manipulation (tidyverse deep dive)

• 3.3.1 dplyr: multi-table verbs (join, union), window functions

  • Joins: inner_join(x, y), left_join(x, y), right_join(x, y), full_join(x, y), semi_join(x, y), anti_join(x, y).

  • Set Operations: union(x, y), intersect(x, y), setdiff(x, y).

  • Window Functions: group_by() + mutate(new = rank(desc(value))), slice_max(), slice_min(), lag(), lead().

• 3.3.2 tidyr: advanced pivoting (pivot_longer, pivot_wider), nest/unnest

  • pivot_longer(data, cols, names_to, values_to): Wide to long.

  • pivot_wider(data, names_from, values_from): Long to wide.

  • nest(data, ...): Create list-column of data frames. unnest(col) to expand.

• 3.3.3 stringr: advanced string manipulation and regex

  • String Detection: str_detect(string, pattern), str_which().

  • Extraction: str_extract(string, pattern), str_match() (capture groups).

  • Manipulation: str_replace(), str_c() (concatenate), str_sub() (substring).

  • Regex: Uses ICU regex engine (similar to Perl). regex(pattern, ignore_case = TRUE).

3.4 R Package Development Essentials

• 3.4.1 Package structure (DESCRIPTION, NAMESPACE, R/, man/)

  • DESCRIPTION: Metadata (Package, Version, Title, Author, Maintainer, Depends, Imports, Suggests).

  • NAMESPACE: Exported functions/imports (auto-generated by roxygen2).

  • R/: R source code (.R files).

  • man/: Documentation (.Rd files, auto-generated by roxygen2).

  • Other: data/ (datasets), tests/ (testthat), vignettes/.

• 3.4.2 Documenting functions with roxygen2

  • Format: Special comments #' before function.

  • Tags: @param, @return, @examples, @export (to export), @importFrom (import from other packages).

  • Workflow: Write roxygen comments → devtools::document() → generates NAMESPACE and man/ files.

• 3.4.3 Unit testing with testthat

  • Structure: Tests in tests/testthat/. Files test-*.R.

  • Expectations: expect_equal(), expect_true(), expect_error(), expect_output().

  • Run: devtools::test() or testthat::test_dir().

• 3.4.4 Building, checking, and installing packages (R CMD)

  • Check: R CMD check package.tar.gz (or devtools::check()). Must pass before CRAN submission.

  • Build: R CMD build package_dir → .tar.gz source.

  • Install: R CMD INSTALL package.tar.gz or devtools::install().

[!TIP] Use usethis::create_package() to set up structure. Always run devtools::check() before sharing. Use @export roxygen tag for user-facing functions.

4.0 Data Visualization & Reporting with R

4.1 ggplot2 Advanced Customization

• 4.1.1 Custom themes and scales

  • Themes: theme_minimal(), theme_bw(). Customize with theme(): axis.text, legend.position, plot.title.

  • Scales: scale_x_continuous(), scale_y_log10(), scale_color_manual(values = c("red","blue")).

• 4.1.2 Faceting (facet_grid, facet_wrap)

  • facet_grid(rows ~ cols): Rows and columns based on variables.

  • facet_wrap(~ variable, ncol = 2): Wrap facets into grid.

• 4.1.3 Complex geoms and statistical transformations

  • Geoms: geom_smooth() (regression), geom_bin2d(), geom_hex(), geom_density_2d().

  • Stats: stat_bin(), stat_ecdf(), stat_qq(). Often implicit in geoms.

• 4.1.4 Plot annotation and layout (patchwork, cowplot)

  • patchwork: Combine plots with +, /, |, plot_layout(ncol=2).

  • cowplot: plot_grid(p1, p2), ggdraw() for drawing, draw_plot() for inset.

  • Annotations: labs(title=, subtitle=, caption=), annotate("text", x=, y=, label=).

4.2 Reproducible Reporting

• 4.2.1 R Markdown: YAML header, code chunks, options

  • YAML Header:

    
    ---
    
    title: "Report"
    
    output: html_document
    
    ---
    
    

  • Code Chunks: ```{r chunk_name, echo=FALSE, warning=FALSE} code ```.

  • Chunk Options: echo (show code), results (hide/show output), fig.width, message, error.

• 4.2.2 Parameterized reports

  • YAML: params: region: "US".

  • Access in R: params$region.

  • Render: rmarkdown::render("report.Rmd", params = list(region = "EU")).

• 4.2.3 Output formats: HTML, PDF, Word, presentations

  • HTML: html_document, html_vignette, flexdashboard::flex_dashboard.

  • PDF: pdf_document (requires LaTeX).

  • Word: word_document.

  • Presentations: ioslides_presentation, slidy_presentation, beamer_presentation (PDF).

4.3 Interactive Visualizations & Dashboards

• 4.3.1 plotly for interactive graphs

  • Convert ggplot: ggplotly(p).

  • Direct: plot_ly(data, x=, y=, type="scatter", mode="markers").

  • Interactive features: hover info, zoom, legend toggle.

• 4.3.2 Introduction to Shiny: UI, server, reactivity

  • UI: fluidPage(), sidebarLayout(), selectInput(), plotOutput().

  • Server: function(input, output) { output$plot <- renderPlot({ ... }) }.

  • Reactivity: reactive({ ... }) for expressions; observe({ ... }) for side effects. Inputs trigger recomputation.

• 4.3.3 Deploying simple Shiny apps

  • Local: shiny::runApp("app_dir").

  • Deployment: shinyapps.io (RStudio), Shiny Server (open-source), ShinyProxy (Docker).

[!TIP] For Shiny, separate reactive expressions (reactive({})) to avoid redundant computation. Use req(input$x) to require input before proceeding.

5.0 Integration: Linux & R Workflows

5.1 Calling R from Shell Scripts

• 5.1.1 Using Rscript for command-line execution

  • Shebang: #!/usr/bin/env Rscript at top of .R file.

  • Run: Rscript script.R arg1 arg2.

  • Advantages: Clean, passes args via commandArgs(trailingOnly=TRUE).

• 5.1.2 Passing arguments and capturing output

  • Args in R: args <- commandArgs(trailingOnly = TRUE).

  • Capture output in shell: result=$$\displaystyle (Rscript -e "R code")` or `result= $$(Rscript script.R).

• 5.1.3 Batch processing with R CMD BATCH

  • Legacy: R CMD BATCH script.R [output_file].

  • Output: Redirects all output (including errors) to file. Less preferred over Rscript.

5.2 Calling Shell Commands from R

• 5.2.1 system(), system2(), processx package

  • system(command, intern = FALSE): If intern=TRUE, captures stdout as R character vector.

  • system2(command, args, stdout = TRUE, stderr = TRUE): More control over args and output streams.

  • processx: Advanced process management (async, timeouts, background processes).

• 5.2.2 Reading command output into R objects

  • Example: files <- system("ls *.csv", intern = TRUE).

  • Parse: strsplit(files, "\n") or read.table(text = files).

• 5.2.3 Error handling for external commands

  • Check system(..., intern=TRUE, ignore.stderr=FALSE, ignore.stdout=FALSE) return value (exit status).

  • Use tryCatch() or withCallingHandlers() for robust error trapping.

5.3 Workflow Automation & Reproducibility

• 5.3.1 Using make or drake/targets for pipeline management

  • make: Declarative build tool. target: dependencies \n\tcommand. Tracks file timestamps.

  • drake / targets: R-native pipeline management. Define targets in drake::drake_plan() or targets::tar_target(). Handles in-memory objects, not just files. targets is successor to drake.

• 5.3.2 Environment management with renv

  • Purpose: Snapshot and restore package libraries per project.

  • Workflow: renv::init() (creates renv.lock), renv::snapshot() (save state), renv::restore() (reinstall exact versions).

  • Isolates project dependencies from system library.

• 5.3.3 Containerization basics with Docker for R/Linux environments

  • Dockerfile: Define base image (e.g., rocker/r-ver:4.3.0), install system deps (apt-get), R packages (install.packages()), copy code.

  • Build: docker build -t my-r-app ..

  • Run: docker run -p 3838:3838 my-r-app.

  • Benefits: Reproducible OS + R environment across machines.

[!TIP] Use targets for complex R pipelines; it's faster than drake and has better debugging. For simple file-based pipelines, make is sufficient. Always use renv for project portability.

6.0 Open Source Collaboration & Best Practices

6.1 Version Control with Git (Advanced)

• 6.1.1 Branching strategies (Git Flow, GitHub Flow)

  • Git Flow: Long-lived master/main (releases), develop (integration), feature branches, release branches, hotfix branches. Complex.

  • GitHub Flow: Simpler. main always deployable. Feature branches → PR → merge → delete branch. Recommended for most projects.

• 6.1.2 Rebasing vs. merging, resolving complex conflicts

  • Merge: Creates merge commit. Preserves history exactly.

  • Rebase (git rebase): Rewrites commits onto new base. Linear history. Do not rebase shared/public branches.

  • Conflicts: Edit files marked with <<<<<<<, =======, >>>>>>>. Then git add and git rebase --continue or git merge --continue.

• 6.1.3 Using .gitignore for R and system files

  • R-specific: *.RData, .Rhistory, .Rproj.user/, renv/library/, packrat/, *.Rproj.

  • System: .DS_Store (macOS), Thumbs.db (Windows), *~ (backup files), node_modules/ (if using web components).

6.2 Collaborative Development on GitHub

• 6.2.1 Pull requests, code reviews, issues

  • PR: Propose changes from branch to base. Must pass CI checks.

  • Code Review: Reviewers comment, request changes. Approve → merge.

  • Issues: Track bugs, features. Use templates (.github/ISSUE_TEMPLATE/).

• 6.2.2 GitHub Actions for CI/CD (basic R checks)

  • Workflow File: .github/workflows/R-CMD-check.yaml.

  • Basic Steps:

    
    jobs:
    
      R-CMD-check:
    
        runs-on: ubuntu-latest
    
        steps:
    
          - uses: actions/checkout@v3
    
          - uses: r-lib/actions/setup-r@v2
    
          - uses: r-lib/actions/setup-r-dependencies@v2
    
          - uses: r-lib/actions/check-r-package@v2
    
    

  • Runs R CMD check on push/PR.

• 6.2.3 Managing package dependencies with DESCRIPTION/renv

  • DESCRIPTION: Lists Imports (required), Suggests (for tests/vignettes), Depends (rarely used now).

  • renv: Locks exact versions in renv.lock. Commit lockfile. CI restores with renv::restore().

6.3 Open Source Licensing & Documentation

• 6.3.1 Common licenses (MIT, GPL, Apache) for R packages and scripts

  • MIT: Permissive, minimal restrictions. Most common for R packages on CRAN.

  • GPL-2/3: Copyleft, requires derivatives to be open source.

  • Apache 2.0: Permissive, includes patent grant.

  • Choose: MIT for maximum adoption; GPL for strong copyleft.

• 6.3.2 Writing effective READMEs and vignettes

  • README.md (GitHub): What, why, how (install, basic usage), badge (CI, CRAN), license, contact.

  • Vignettes (R): Long-form guides. Use rmarkdown::draft("vignette.Rmd", template = "package", package = "devtools"). Explain package philosophy, complex workflows.

• 6.3.3 Contributing guidelines and code of conduct

  • CONTRIBUTING.md: How to submit PRs, coding style, test requirements.

  • CODE_OF_CONDUCT.md: Adopt Contributor Covenant. Ensures welcoming community.

  • Issue/PR Templates: Standardize submissions.

[!TIP] Always include a license file (LICENSE or LICENSE.md) and a DESCRIPTION file with License: MIT + file LICENSE if using MIT. Use usethis::use_mit_license().

\boxed{\text{Key Integration Command: } \texttt{Rscript \textless script.R \textgreater \textgreater output.log 2\textgreater\textless1}}