SOFTWARE LAB (ME-406) - Unit 4 Short Notes

UNIT 4: ADVANCED SOFTWARE DEVELOPMENT, QUALITY & DEVOPS PRACTICES

⚠️ Critical Note: This unit covers modern software engineering practices essential for professional development and deployment. Since no official syllabus/past papers exist for ME-406 Unit 4, these notes are based on the speculative generic outline provided. Cross-reference with your actual course material.

4.1 Advanced Software Development Practices & Tooling

4.1.1 Continuous Integration/Continuous Deployment (CI/CD) Pipelines

• Definition: A method to frequently deliver apps to customers by introducing automation into the stages of app development (build, test, deploy).

• Core Concepts:

  • Continuous Integration (CI): Developers merge code changes frequently (multiple times a day). Each merge triggers an automated build and test sequence.

  • Continuous Deployment/Delivery (CD): Automatically deploys every change that passes the test suite to production (Deployment) or a staging environment (Delivery).

• Pipeline as Code: Defining the CI/CD pipeline in a configuration file (e.g., .gitlab-ci.yml, Jenkinsfile, GitHub Actions YAML). This makes pipelines versionable, reviewable, and repeatable.

• Typical Pipeline Stages:

  1. Source/Trigger: Code push/PR to repository.

  2. Build: Compile code, resolve dependencies.

  3. Test: Run automated unit/integration tests.

  4. Package: Create deployable artifact (JAR, Docker image).

  5. Deploy: To staging/production environments.

  6. Validate/Test: Post-deployment smoke tests, health checks.

• Common Tools:

  • Jenkins: Open-source, highly extensible automation server.

  • GitLab CI/CD: Integrated into GitLab, uses .gitlab-ci.yml.

  • GitHub Actions: Event-driven automation directly in GitHub repos.

  • CircleCI, Travis CI: Cloud-based CI/CD services.

[!TIP] Exam Focus: Know the difference between CI and CD. Be able to describe a basic pipeline stage sequence. Understand the benefits of "Pipeline as Code" (version control, collaboration).

4.1.2 Infrastructure as Code (IaC)

• Definition: Managing and provisioning infrastructure (servers, networks, databases) through machine-readable definition files, rather than manual processes.

• Key Principles: Idempotency (applying code multiple times yields same result), version control, repeatability.

• Tool Categories:

  • Configuration Management: Ansible (agentless, push-based), Puppet, Chef. Focuses on configuring existing servers.

  • Infrastructure Provisioning: Terraform (declarative, cloud-agnostic), AWS CloudFormation. Focuses on creating and managing cloud resources.

• Example (Terraform): Uses HCL (HashiCorp Configuration Language) to define resources like aws_instance, google_storage_bucket.

4.1.3 Containerization & Orchestration

• Docker Fundamentals:

  • Image: A read-only template with instructions to create a container. Built from a Dockerfile.

  • Container: A runnable instance of an image. Isolated process with its own filesystem.

  • Dockerfile: Text document with commands to assemble an image (e.g., FROM, RUN, COPY, CMD).

  • Registry: Repository for images (Docker Hub, AWS ECR, Google Container Registry).

• Kubernetes (K8s) Basics:

  • Pod: Smallest deployable unit; one or more containers with shared storage/network.

  • Deployment: Manages a set of identical Pods, handles updates and rollbacks.

  • Service: Abstraction defining a logical set of Pods and a policy to access them (load balancing).

  • Core Concept: Declarative management. You declare the desired state (e.g., "run 3 replicas of my app"), and K8s works to achieve/maintain it.

[!TIP] Common Pitfall: Confusing Docker image (blueprint) with Docker container (running instance). Know that Kubernetes orchestrates containers (often in Pods), not images directly.

4.1.4 Advanced Version Control Workflows

• Gitflow: Feature branches off develop, releases off develop to master, hotfixes off master. Complex, suited for versioned software.

• GitHub Flow: Simplified. main branch is always deployable. Feature branches created from main, PRs back to main after review. Suited for SaaS/web apps.

• Trunk-Based Development: All developers integrate to a single branch (trunk/main) frequently (multiple times/day). Uses short-lived feature branches (<2 days) and feature flags. Enables CI/CD.

• Rebase vs. Merge:

  • Merge (git merge): Creates a new merge commit that joins histories. Preserves history exactly as happened.

  • Rebase (git rebase): Moves/combines commits to a new base. Rewrites history to create a linear sequence. Never rebase shared/public history.

4.2 Software Quality Assurance & Advanced Testing

4.2.1 Test Automation Frameworks

• Unit Testing: Tests individual components/functions in isolation.

  • Tools: JUnit (Java), pytest (Python), Jest (JavaScript).

  • Key: Fast, uses mocks/stubs for dependencies.

• Integration Testing: Tests interaction between integrated components/modules (e.g., API + DB).

• End-to-End (E2E) Testing: Tests complete user workflows from UI to backend.

  • Tools: Selenium (browser automation), Cypress (modern, fast), Playwright (multi-browser).

  • Key: Slower, more brittle, simulates real user scenarios.

[!TIP] Test Pyramid: Advocate for many Unit tests, fewer Integration tests, and few E2E tests. This optimizes for speed and reliability.

4.2.2 Performance, Load, and Stress Testing

• Goal: Assess system behavior under load.

• Load Testing: Simulate expected user load to check performance metrics (response time, throughput).

• Stress Testing: Push system beyond normal capacity to find breaking points and how it recovers.

• Tools: Apache JMeter (GUI-based, protocol-level), k6 (code-centric, modern, for cloud-native).

• Key Metrics: Response Time (p95, p99), Throughput (req/sec), Error Rate, CPU/Memory Utilization.

4.2.3 Security Testing (DevSecOps)

• SAST (Static Application Security Testing): Analyzes source code/binary for vulnerabilities without running it. (e.g., SonarQube, Checkmarx, Semgrep).

• DAST (Dynamic Application Security Testing): Tests running application for vulnerabilities (e.g., OWASP ZAP, Burp Suite). Simulates attacks.

• SCA (Software Composition Analysis): Scans dependencies (open-source libraries) for known vulnerabilities (CVEs) and license issues. (e.g., OWASP Dependency-Check, Snyk, Dependabot).

4.3 Cloud-Native Application Development

4.3.1 Microservices Architecture Patterns

• Service Discovery: How services find each other (e.g., Netflix Eureka, Consul, K8s native DNS).

• API Gateway: Single entry point for clients, handles routing, auth, rate limiting, caching. (e.g., Spring Cloud Gateway, Kong, AWS API Gateway).

• Circuit Breaker Pattern: Prevents cascading failures by "tripping" when a service call fails repeatedly. (e.g., Resilience4j, Hystrix).

4.3.2 Serverless Computing (FaaS)

• Concept: Deploy individual functions (event-driven) without managing servers. Cloud provider manages scaling, patching.

• Platforms: AWS Lambda, Azure Functions, Google Cloud Functions.

• Triggers: HTTP requests, file uploads, DB events, message queues.

• Considerations: Cold start latency, execution time limits, statelessness.

4.3.3 Cloud Database & Storage Services

• Managed SQL: Amazon RDS, Google Cloud SQL, Azure Database for PostgreSQL. Handles backups, replication, patching.

• Managed NoSQL: Amazon DynamoDB (key-value), MongoDB Atlas, Google Firestore (document).

• Object Storage: Amazon S3, Google Cloud Storage. For unstructured data (images, backups). Highly durable, scalable.

4.4 Monitoring, Logging, and Observability

4.4.1 The Three Pillars

1. Metrics: Quantitative measurements over time (CPU, latency, error rate). Numerical, aggregatable.

2. Logs: Immutable, timestamped records of discrete events. Textual, detailed.

3. Traces: Track a request's path through distributed systems. Shows causality and latency between services.

4.4.2 Monitoring & Alerting

• Prometheus: Pull-based metrics collection and alerting toolkit. Uses a multi-dimensional data model.

• Grafana: Visualization platform for metrics, logs, traces. Connects to Prometheus, Loki, etc.

• Alerting: Defined on Prometheus rules (e.g., rate(http_requests_total[5m]) > 100). Sent to Alertmanager.

4.4.3 Centralized Logging

• ELK Stack: Elasticsearch (search/analytics), Logstash (ingestion/processing), Kibana (visualization). Now often Elastic Stack (Beats instead of Logstash).

• Loki: Log aggregation system inspired by Prometheus. Indexes labels, not full log content. Lightweight, cost-effective. Visualized with Grafana.

4.4.4 Distributed Tracing

• Purpose: Understand flow and latency in microservices.

• Tools: Jaeger (open-source, CNCF), Zipkin.

• Key Terms: Span (unit of work), Trace (tree of spans for a single request), Context Propagation (passing trace ID across service boundaries).

[!TIP] Observability vs. Monitoring: Monitoring tells you something is wrong (e.g., CPU > 90%). Observability lets you ask why it's wrong by exploring logs, traces, and metrics for that specific incident.

4.5 Modern Software Development Methodologies & Collaboration

4.5.1 Scaling Agile

• SAFe (Scaled Agile Framework): Structured, role-based framework for large enterprises. Multiple Agile teams coordinate at Program/Portfolio level.

• LeSS (Large-Scale Scrum): Extends Scrum principles to multiple teams working on one product. Simpler than SAFe, less prescriptive.

4.5.2 DevOps Culture & Metrics (DORA)

• Key Performance Indicators (KPIs):

  1. Deployment Frequency: How often do you deploy to production?

  2. Lead Time for Changes: Time from commit to running in production.

  3. Mean Time to Restore (MTTR): Time to recover from a failure.

  4. Change Fail Rate: % of deployments causing failures.

• Goal: High deployment frequency, short lead time, low MTTR, low change fail rate.

4.5.3 Collaborative Development & Code Review

• Best Practices:

  • Small, focused PRs/MRs (< 400 lines).

  • Clear description: What changed, Why changed, How to test.

  • Use checklist templates.

  • Review logic, not style (enforce style with linters/formatters).

  • Be constructive, ask questions.

  • Pair Programming: Real-time collaborative coding.

4.6 Capstone/Lab Project Integration (Typical Structure)

A typical end-to-end project lifecycle in this lab:

1. Project Specification & Design:

   • Define requirements (user stories), create architecture diagrams (e.g., microservices, data flow).

   • Choose tech stack (e.g., Spring Boot/Node.js backend, React frontend, PostgreSQL, Docker, K8s).

2. Implementation using Modern Stack:

   • Develop application code following modular design.

   • Write unit/integration tests (pytest/JUnit).

   • Containerize each service (Dockerfile).

3. Full CI/CD Pipeline Setup:

   • Pipeline Stages: test -> build (Docker image) -> push to registry -> deploy to staging -> smoke test -> approval -> deploy to prod.

   • Tools: GitHub Actions / GitLab CI.

   • Infrastructure as Code: Use Terraform to provision cloud resources (VPC, K8s cluster, DB).

4. Deployment to Cloud/Container Platform:

   • Deploy containers to Kubernetes cluster (EKS, GKE, AKS, or minikube/kind for local).

   • Define K8s manifests: Deployment, Service, Ingress, ConfigMap, Secret.

   • Use Helm charts for templating if complex.

5. Monitoring & Maintenance Setup:

   • Instrument application with metrics (Prometheus client library).

   • Configure structured logging (JSON) and ship to Loki/ELK.

   • Set up distributed tracing (Jaeger) with OpenTelemetry.

   • Create Grafana dashboards for key metrics.

   • Configure Prometheus alerts (e.g., high error rate, pod down).

6. Final Demonstration & Documentation:

   • Live demo of the running system, CI/CD pipeline execution, rollback.

   • Comprehensive README.md with:

     • Architecture diagram.

     • Setup instructions (local & cloud).

     • CI/CD pipeline explanation.

     • Monitoring dashboard links.

     • API documentation (Swagger/OpenAPI).

[!TIP] Exam Strategy: For project-based questions, think in terms of the entire lifecycle: Code -> Build -> Test -> Package -> Deploy -> Monitor. Be ready to map tools to each stage and justify choices (e.g., "We used Terraform for IaC because it's cloud-agnostic and stateful").