UNIT 1: Foundations of the Open Source Lab Environment

A. Introduction & Environment Setup

Course & Lab Objectives

Dual-Tool Approach:
- Linux: Provides the stable, open-source operating system environment for system-level tasks, file management, and running software.
- R: Dedicated open-source language/environment for statistical computing, data analysis, and visualization.
Lab Workflow Distinction:
- Linux Command Line: For system administration, file operations, package management, and automation via shell scripts.
- R Scripting/IDE: For data manipulation, statistical modeling, and generating reports/plots.

Setting Up the Development Environment

Linux Access Options:
- Virtual Machine (VM): Using VirtualBox/VMware. Isolates lab environment from host OS. Recommended for beginners.
- Live USB: Boot Linux from USB without installation. Good for temporary use, but changes are not persistent by default.
- Dual-boot: Install Linux alongside host OS (e.g., Windows). Requires partitioning disk and bootloader setup.
R Installation:
- System Package Manager (apt/yum): Installs system-wide R. Version may be older than CRAN.
```
sudo apt update && sudo apt install r-base
```
- CRAN Download: Gets latest stable version. Add CRAN repo to sources.list for apt to manage updates.
RStudio Installation:
- Why an IDE?: Provides integrated console, script editor, workspace viewer, plot/help panes, and project management. Dramatically improves productivity.
- Installation: Download .deb (Debian/Ubuntu) or .rpm (Fedora) from RStudio website, install via sudo dpkg -i or sudo rpm -i.

B. Linux Fundamentals for the Lab

The Linux Filesystem Hierarchy

Directory	Purpose
`/`	Root directory, top of hierarchy.
`/home/<user>`	User's personal files and directories.
`/usr`	User programs and data (usr = Unix System Resources).
`/etc`	System configuration files.
`/var`	Variable data (logs, caches, spool).
`/tmp`	Temporary files, cleared on reboot.

Absolute Path: Starts from root / (e.g., /home/user/data.csv).
Relative Path: Relative to current working directory (e.g., ./data/raw.csv, ../scripts).

Command Line Navigation & File Management

Command	Purpose	Common Options
`pwd`	Print Working Directory
`ls`	List directory contents	`-l` (long), `-a` (all incl. hidden), `-h` (human-readable sizes)
`cd`	Change Directory	`cd ~` (home), `cd -` (previous)
`mkdir`	Make Directory	`-p` (create parent dirs)
`cp`	Copy	`-r` (recursive for dirs)
`mv`	Move/Rename
`rm`	Remove	`-r` (recursive), `-f` (force) *Use with caution!
`rmdir`	Remove empty Directory

File Permissions & Ownership

ls -l Output Example: -rwxr-xr-- 1 user group 1024 Jan 01 12:00 file.txt
- File Type: - (file), d (directory).
- Permissions: rwx (user/owner), r-x (group), r-- (others). r=read(4), w=write(2), x=execute(1).
chmod (Change Mode):
- Numeric (Octal): chmod 755 file → rwxr-xr-x (7=4+2+1, 5=4+0+1).
- Symbolic: chmod u+x file (add execute for user), chmod o-w file (remove write for others), chmod a=r file (set all to read-only).
chown & chgrp: Change owner (sudo chown newuser file) and group (sudo chgrp newgroup file).

Text File Manipulation

Viewing: cat (concatenate, whole file), less (scrollable, q to quit), head/tail (first/last 10 lines, -n 20 for 20 lines).
Editing:
- nano: Simple, intuitive. Ctrl+O (save), Ctrl+X (exit).
- vim: Modal editor. Insert Mode (i) to type, Command Mode (Esc) to navigate/save (:wq), quit without save (:q!).
Redirection & Pipes:
- > : Redirect stdout to file (overwrite). echo "text" > file.txt
- >> : Redirect stdout to file (append). echo "more" >> file.txt
- | : Pipe stdout of one command as stdin to next. ls | grep ".csv"

Package Management (apt for Debian/Ubuntu)


sudo apt update          # Refresh package lists

sudo apt install <pkg>   # Install package

sudo apt remove <pkg>    # Remove package (configs may remain)

apt search <keyword>     # Search for packages

Installing R packages: Can install r-cran-<packagename> from system repo, but within R (install.packages()) is more common and gets latest version from CRAN.

C. R Fundamentals & Data Handling

R & RStudio Interface

Four Panes:
1. Console: Interactive R command line.
2. Source: Script editor (.R files). Primary workspace.
3. Environment/History: Shows loaded objects (variables, data frames) and command history.
4. Files/Plots/Help/Viewer: Navigate files, view plots, access documentation.
Working Directory:
- getwd() : Get current working directory.
- setwd("path/to/dir") : Set working directory. Avoid hardcoding absolute paths.
- RStudio Projects (.Rproj): Best Practice. Creates a project-specific working directory. Opening the .Rproj file sets the correct WD automatically. Ensures reproducibility.

Basic R Syntax & Operations

Assignment: <- (preferred, clear) vs = (can work, but ambiguous in function calls).
Data Types:
- numeric (e.g., 3.14), integer (e.g., 2L), character (e.g., "text"), logical (TRUE/FALSE), factor (categorical), NA (missing).
Core Data Structures:

| Structure | Dimension | Homogeneity | Key Use | | :--- | :--- | :--- | :--- | | Vector | 1D | All elements same type | Basic data container | | Matrix | 2D | All elements same type | Math operations | | List | 1D | Elements can be any type | Heterogeneous collection | | Data Frame | 2D | Columns can be different types | Tabular data (like spreadsheet) |

Importing & Exporting Data

Reading Text Files:


read.table("file.txt", header=TRUE, sep="\t", stringsAsFactors=FALSE)

read.csv("file.csv")  # sep=",", header=TRUE by default

read.delim("file.txt") # sep="\t", header=TRUE by default

[!TIP] Always use stringsAsFactors=FALSE (or options(stringsAsFactors=FALSE)) to prevent automatic conversion of character strings to factors unless explicitly needed.

Reading Excel: readxl::read_excel("file.xlsx") (from readxl package).

Writing Data:


write.csv(data, "output.csv", row.names=FALSE)

write.table(data, "output.txt", sep="\t", row.names=FALSE)

Linux Integration: Use shell commands to discover paths within R.


system("ls -l")  # List files in current WD

file_path <- paste0(getwd(), "/data/", "file.csv") # Manual construction

file_path <- file.path(getwd(), "data", "file.csv")  # **Preferred, OS-independent**

Basic Data Inspection & Manipulation

Inspection: str(df) (structure), summary(df) (stats), dim(df) (rows, cols), head(df)/tail(df), names(df).
Accessing Data Frame Elements:
- df$column_name : Extract a column as a vector.
- df[rows, cols] : Subset by row/column indices or names.
- df[["column_name"]] or df[[i]] : Extract a single column as a vector (like $ but for programmatic use).
Type Conversion: as.numeric(x), as.character(x), factor(x, levels=...).

D. Integrating Linux & R in a Lab Workflow

Executing R from the Linux Terminal

Non-interactive Execution: Rscript script.R runs an R script and exits. Ideal for automation.

Passing Arguments:


# In script.R

args <- commandArgs(trailingOnly = TRUE)

input_file <- args[1]

output_file <- args[2]


Rscript script.R input.csv output.csv

Using Shell Commands Within R

system("command", intern=TRUE) : Runs command, returns output as R character vector if intern=TRUE.
system2("command", args=c("arg1", "arg2")) : More robust handling of arguments.

Use Case: Check if a required data file exists before reading.


file_exists <- system2("test", args=c("-e", "data/raw.csv"), stdout=FALSE)

if (file_exists == 0) { data <- read.csv("data/raw.csv") }

File Path Best Practices

Always use forward slashes / in R paths, even on Windows (R accepts them).
Construct paths with file.path(): Automatically uses correct OS separator.
```
data_path <- file.path("data", "processed", "clean.csv")
```
Use relative paths within RStudio Projects. Makes project portable.
Avoid setwd() in shared scripts. Rely on project root or here::here() package.

E. Essential Lab Practices & Troubleshooting

Reproducibility from the Start

Set Seed: set.seed(123) ensures random number generation (e.g., sampling, model splits) is reproducible.
Commenting: Use # to explain why, not just what.
RStudio Projects: The single most important tool for managing paths and project state.

Common Errors & Debugging

Environment	Common Error	Likely Cause & Fix
Linux	`Permission denied`	File not executable (`chmod +x script.sh`) or trying to write to protected dir. Use `sudo` cautiously.
	`Command not found`	Package not installed or not in `$PATH`. Check spelling.
	`No such file or directory`	Incorrect path. Use `pwd` and `ls` to verify. Check for typos.
R	`Error: object 'x' not found`	Variable `x` doesn't exist in current environment. Check spelling, case, and if it's in a loaded package (`library()`).
	`Error: incorrect number of dimensions`	Trying to access a matrix/data frame with wrong number of indices (e.g., `df[1]` returns a data frame, `df[[1]]` a vector).
	`Warning: NAs introduced by coercion`	`as.numeric()` on non-numeric strings (e.g., `"12a"`). Check data import (`stringsAsFactors`).
	`Error in factor(...): invalid 'labels'`	Providing fewer/more labels than levels.

Getting Help

Linux:
- man <command> : Manual page (e.g., man ls).
- <command> --help : Quick reference.
- apropos "keyword" : Search man pages.
- Online: Stack Overflow, Arch Wiki (excellent for concepts).
R:
- ?function or help(function) : Documentation for a function.
- example(function) : Run examples.
- help.search("keyword") or ??keyword : Search all help pages.
- CRAN Task Views: Curated package lists for domains (e.g., Finance, TimeSeries).
- RStudio Help Pane: Search box integrates with R's help system.

Account

Open Source Software Lab (Linux and R) (IT-407) - Unit 1 Short Notes