UNIT 4: Advanced Python Programming & Applications

1. Object-Oriented Programming (OOP) Deep Dive

Core Concepts:

• Class: Blueprint for objects. Defined with class keyword.

• Object: Instance of a class. Created via ClassName().

• __init__(): Constructor method. Initializes instance attributes. First parameter is self (reference to instance).

• self: Explicit reference to current instance (mandatory in method definitions).

Inheritance Types:

Polymorphism:

• Method Overriding: Child redefines parent's method.

• Duck Typing: "If it walks like a duck...". Object suitability determined by methods/attributes, not class type.

Encapsulation & Access Control:

• Public: No underscore (default). Accessible anywhere.

• Protected: Single underscore _attr. Convention: internal use.

• Private: Double underscore __attr. Triggers name mangling: __attr → _ClassName__attr.

Special (Dunder) Methods:

Properties & Decorators:

• @property: Turns method into read-only attribute.

  
  class Circle:
  
      def __init__(self, r): self._radius = r
  
      @property
  
      def radius(self): return self._radius
  
      @radius.setter
  
      def radius(self, value): self._radius = value
  
  

• @classmethod: Receives cls (class). Used for factory methods.

• @staticmethod: No self/cls. Utility function inside class.

Abstract Base Classes (ABCs):

from abc import ABC, abstractmethod

class Shape(ABC):

    @abstractmethod

    def area(self): pass  # Must be overridden

[!TIP] Abstract methods cannot have implementation. Subclass must override all @abstractmethods to instantiate.

2. File Handling & Data Persistence

File Modes:

Context Manager (with):

with open('file.txt', 'r') as f:

    data = f.read()  # Auto-closes file after block

[!TIP] Always use with for file operations. Prevents resource leaks if exception occurs.

Text File Operations:

• f.read(size): Read size bytes (or all if omitted).

• f.readline(): Read single line (includes \n).

• f.readlines(): Read all lines → list.

• f.write(str): Write string. Returns number of characters written.

• f.writelines(list): Write list of strings (no separators added).

CSV Handling (csv module):

import csv

# Reading

with open('data.csv') as f:

    reader = csv.DictReader(f)  # Returns dict per row

    for row in reader: print(row['column'])

# Writing

with open('out.csv', 'w', newline='') as f:

    writer = csv.writer(f)

    writer.writerow(['col1', 'col2'])

JSON Handling (json module):

import json

# Serialization (Python → JSON string/file)

json_str = json.dumps(obj, indent=4)  # Pretty print

json.dump(obj, file)  # Write to file

# Deserialization (JSON → Python)

obj = json.loads(json_str)

obj = json.load(file)

[!TIP] JSON supports: dict, list, str, int, float, bool, None. Custom objects need default hook.

Pickle Module (Object Serialization):

import pickle

with open('data.pkl', 'wb') as f:

    pickle.dump(obj, f)  # Serialize

with open('data.pkl', 'rb') as f:

    obj = pickle.load(f)  # Deserialize

[!WARNING] Security Risk: Unpickling untrusted data can execute arbitrary code. Never unpickle from unknown sources.

File Paths:

• os.path (legacy): os.path.join(), os.path.exists().

• pathlib (modern, OOP):

  
  from pathlib import Path
  
  p = Path('folder') / 'file.txt'
  
  p.exists(), p.read_text(), p.write_text()
  
  

3. Exception Handling & Custom Errors

Block Structure:

try:

    # Risky code

except SpecificError as e:

    # Handle specific error

except (Error1, Error2) as e:

    # Handle multiple errors

else:

    # Runs if NO exception

finally:

    # Always runs (cleanup)

Raising Exceptions:

raise ValueError("Invalid input")

# Exception chaining (preserves original context)

raise CustomError("msg") from original_exception

Built-in Hierarchy (Key Branches):

BaseException

 ├── SystemExit

 ├── KeyboardInterrupt

 └── Exception  ← Most errors inherit from here

      ├── StopIteration

      ├── ArithmeticError (ZeroDivisionError, OverflowError)

      ├── LookupError (IndexError, KeyError)

      ├── OSError (FileNotFoundError, PermissionError)

      └── TypeError, ValueError, RuntimeError, etc.

Custom Exceptions:

class InsufficientFundsError(Exception):

    def __init__(self, balance, amount):

        self.balance = balance

        self.amount = amount

        super().__init__(f"Balance {balance}, tried to withdraw {amount}")

Best Practices:

• Catch specific exceptions, not bare except:.

• Use finally for cleanup (close files, release locks).

• Log exceptions (logging.exception()).

• Avoid empty except: blocks.

4. Modules, Packages, and Distribution

Importing:

import module                # Access via module.func()

from module import func      # Direct access: func()

from module import *         # Avoids namespace pollution

# Relative imports (inside package)

from .sibling import func

from ..parent import func

Package Structure:

mypackage/

├── __init__.py        # Can be empty; marks directory as package

├── module1.py

├── subpackage/

│   ├── __init__.py

│   └── module2.py

[!TIP] __init__.py can initialize package-level variables or control from package import *.

Namespace Packages (PEP 420):

• No __init__.py required.

• Spread across multiple directories.

• Created automatically when multiple paths contribute to same package name.

Key Standard Library Modules:

• os: OS interactions (env, paths, processes).

• sys: System parameters (argv, path, exit).

• datetime: Date/time objects.

• collections: Specialized containers (deque, Counter, defaultdict).

• itertools: Iterator building blocks (chain, cycle, combinations).

Third-Party Packages:

pip install package_name

pip install -r requirements.txt  # Install from list

• requirements.txt format: package==version or package>=version.

Virtual Environments:

# venv (built-in)

python -m venv myenv

source myenv/bin/activate  # Linux/Mac

myenv\Scripts\activate     # Windows

# conda (cross-language)

conda create -n myenv python=3.9

conda activate myenv

[!TIP] Always use virtual environments to isolate project dependencies.

5. Advanced Language Features

Decorators:

def decorator(func):

    from functools import wraps

    @wraps(func)  # Preserves metadata

    def wrapper(*args, **kwargs):

        # Pre-processing

        result = func(*args, **kwargs)

        # Post-processing

        return result

    return wrapper

@decorator

def my_func(): pass

[!TIP] @functools.wraps is essential to maintain original function's name/docstring.

Generators:

• Function: Uses yield. Returns generator object (lazy evaluation).

  
  def count_up_to(n):
  
      i = 0
  
      while i < n:
  
          yield i
  
          i += 1
  
  

• Expression: (x**2 for x in range(10)).

• Memory efficient for large/infinite sequences.

Iterators:

class Counter:

    def __init__(self, low, high):

        self.current = low

        self.high = high

    def __iter__(self):

        return self

    def __next__(self):

        if self.current > self.high:

            raise StopIteration

        else:

            self.current += 1

            return self.current - 1

Custom Context Managers:

• Class-based:

  
  class ManagedFile:
  
      def __init__(self, name): self.name = name
  
      def __enter__(self): 
  
          self.file = open(self.name, 'w')
  
          return self.file
  
      def __exit__(self, exc_type, exc_val, exc_tb):
  
          if self.file: self.file.close()
  
  

• @contextmanager (simpler):

  
  from contextlib import contextmanager
  
  @contextmanager
  
  def managed_file(name):
  
      f = open(name, 'w')
  
      try:
  
          yield f
  
      finally:
  
          f.close()
  
  

Regular Expressions (re module):

Common Patterns:

• \d: digit, \w: word char, \s: whitespace.

• +: 1+, *: 0+, ?: 0-1.

• ^: start, $: end.

• (group): Capturing group. \1, \2 backreferences.

• (?:...): Non-capturing group.

Type Hints (typing module):

from typing import List, Dict, Callable, TypeVar, Generic, Optional

def process(items: List[int]) -> Dict[str, int]:

    ...

T = TypeVar('T')  # Generic type

class Box(Generic[T]):

    def __init__(self, item: T): self.item = item

    def get(self) -> T: return self.item

# Union types

def func(x: int | str) -> None: ...

# Optional (same as Union[T, None])

def func(x: Optional[int] = None) -> None: ...

[!TIP] Type hints are not enforced at runtime (unless using mypy). They improve readability and IDE support.

6. Concurrency and Parallelism

Threading (threading):

• GIL (Global Interpreter Lock): Only one thread executes Python bytecode at a time. Limits CPU-bound parallelism.

• Use for I/O-bound tasks (network, file ops).

import threading

lock = threading.Lock()

def thread_func():

    with lock:  # Synchronization

        # Critical section

t = threading.Thread(target=thread_func)

t.start()

t.join()

Multiprocessing (multiprocessing):

• Bypasses GIL. Separate memory space.

• Use for CPU-bound tasks.

from multiprocessing import Pool

def compute(x): return x*x

with Pool(4) as p:

    results = p.map(compute, [1,2,3,4])

• IPC: Queues, Pipes, shared memory.

Asyncio (Single-threaded concurrency):

• Event loop, coroutines (async def), await.

• Cooperative multitasking. I/O-bound.

import asyncio

async def fetch_data():

    await asyncio.sleep(1)  # Simulate I/O

    return "data"

async def main():

    task1 = asyncio.create_task(fetch_data())

    task2 = asyncio.create_task(fetch_data())

    await task1; await task2

asyncio.run(main())

concurrent.futures (High-level API):

from concurrent.futures import ThreadPoolExecutor, ProcessPoolExecutor

# Thread pool (I/O-bound)

with ThreadPoolExecutor(max_workers=4) as executor:

    future = executor.submit(func, arg)

    result = future.result()

# Process pool (CPU-bound)

with ProcessPoolExecutor() as executor:

    results = list(executor.map(func, iterable))

[!TIP] Choose threading for I/O, multiprocessing for CPU, asyncio for many network connections.

7. Testing and Debugging

Unit Testing (unittest):

import unittest

class TestMath(unittest.TestCase):

    def setUp(self):  # Runs before each test

        self.value = 5

    def test_add(self):

        self.assertEqual(1+1, 2)

    def test_raises(self):

        with self.assertRaises(ValueError):

            raise ValueError()

if __name__ == '__main__':

    unittest.main()

Pytest (Simpler):

• Test files: test_*.py or *_test.py.

• Functions: def test_func(): assert ....

• Fixtures: Setup/teardown via @pytest.fixture.

• Parametrization: @pytest.mark.parametrize("x,y", [(1,2),(3,4)]).

Debugging:

• pdb:

  
  import pdb; pdb.set_trace()  # Breakpoint
  
  # Commands: n(ext), s(tep), c(ontinue), l(ist), p(rint)
  
  

• Logging:

  
  import logging
  
  logging.basicConfig(level=logging.DEBUG)
  
  logging.debug("Value: %s", x)  # Lazy formatting
  
  

Code Coverage (coverage.py):

coverage run -m pytest

coverage report -m  # Show missing lines

coverage html       # Generate HTML report

[!TIP] Aim for high coverage but focus on meaningful tests (edge cases, error paths).

8. Performance Optimization & Profiling

Profiling:

• cProfile: Deterministic profiling.

  
  python -m cProfile -s cumtime script.py
  
  

• timeit: Small code snippets.

  
  import timeit
  
  timeit.timeit('"-".join(str(n) for n in range(100))', number=10000)
  
  

Memory Management:

• sys.getsizeof(obj): Memory in bytes (shallow).

• gc module: gc.collect(), gc.get_objects().

• Generators reduce memory vs lists.

Efficient Data Structures (collections):

Lazy Evaluation:

• Generators (yield) produce values on-demand.

• Iterator expressions (x for x in ...) vs list comprehensions [x for x in ...].

• Saves memory for large datasets.

9. Popular Libraries for Engineering/Scientific Computing

NumPy:

• ndarray: Homogeneous, fixed-size, multi-dimensional.

• Vectorization: Operations on entire arrays (no Python loops).

• Broadcasting: Implicit element-wise operations on arrays of different shapes.

  
  import numpy as np
  
  a = np.array([[1,2],[3,4]])
  
  b = np.array([10,20])
  
  result = a + b  # b broadcasted to [[10,20],[10,20]]
  
  

• Basic linear algebra: np.dot(), np.linalg.inv().

Pandas:

• Series: 1D labeled array.

• DataFrame: 2D table (columns of Series).

import pandas as pd

df = pd.read_csv('data.csv')  # Load

df[df['col'] > 5]             # Filter

df.groupby('category').mean() # Group

Matplotlib:

import matplotlib.pyplot as plt

x = np.linspace(0, 10, 100)

plt.plot(x, np.sin(x), label='sin')  # Line plot

plt.scatter(x, np.cos(x))           # Scatter

plt.hist(data, bins=30)             # Histogram

plt.subplot(2,1,1)                  # Subplots

plt.xlabel('X'); plt.legend()

plt.show()

SciPy (Intro):

• Builds on NumPy.

• Modules: scipy.optimize (minimization), scipy.integrate (numerical integration), scipy.interpolate.

10. Web Development Fundamentals (Optional/Introductory)

Flask (Microframework):

from flask import Flask, jsonify, render_template

app = Flask(__name__)

@app.route('/')  # Routing

def home():

    return render_template('index.html')  # Template

@app.route('/api/data')

def api():

    return jsonify({'key': 'value'})  # JSON response

if __name__ == '__main__':

    app.run(debug=True)

• MVC Pattern: Models (data), Views (templates), Controllers (routes).

REST APIs:

• HTTP Methods: GET (read), POST (create), PUT (update), DELETE.

• Stateless. JSON payloads.

• Flask-RESTful/FastAPI simplify creation.

Database (SQLite with sqlite3):

import sqlite3

conn = sqlite3.connect('app.db')

cursor = conn.cursor()

cursor.execute("CREATE TABLE IF NOT EXISTS users (id INTEGER PRIMARY KEY, name TEXT)")

cursor.execute("INSERT INTO users (name) VALUES (?)", ("Alice",))  # Parameterized!

conn.commit()

[!TIP] Always use parameterized queries to prevent SQL injection.

ORM (SQLAlchemy):

• Maps Python classes to database tables.

• Abstraction layer: session.add(obj), session.query(Model).filter_by(...).

11. Code Quality and Best Practices

PEP 8 Key Rules:

• Indentation: 4 spaces (no tabs).

• Line Length: ≤ 79 chars (code), ≤ 72 (comments).

• Naming:

  • snake_case for functions/variables.

  • CamelCase for classes.

  • UPPER_SNAKE_CASE for constants.

• Imports: Standard → third-party → local. One per line.

• Whitespace: Around operators, after commas, but not inside brackets.

Docstrings:

• Google Style:

  
  def func(arg1, arg2):
  
      """Short description.
  
      
  
      Args:
  
          arg1 (int): Description.
  
          arg2 (str): Description.
  
      
  
      Returns:
  
          bool: Description.
  
      """
  
  

• Access via help(func) or IDE tooltips.

Static Analysis:

• pylint: Checks for errors, style, refactoring.

• flake8: Combines PyFlakes (errors), pycodestyle (PEP 8), McCabe (complexity).

• mypy: Type checking using type hints.

  
  mypy script.py  # Reports type mismatches
  
  

Packaging:

• setup.py (legacy) or pyproject.toml (modern).

• Build: python setup.py sdist bdist_wheel.

• Upload to PyPI: twine upload dist/*.

12. Security Considerations

Common Vulnerabilities (Web Context):

• Injection: SQL, command, LDAP. Mitigation: Parameterized queries, input validation.

• XSS (Cross-Site Scripting): Inject scripts into web pages. Mitigation: Escape user input in templates.

• CSRF (Cross-Site Request Forgery): Unauthorized actions. Mitigation: CSRF tokens.

Secure Coding Practices:

• Input Validation: Validate type, length, format, range. Use pydantic for data validation.

• Avoid eval(): Arbitrary code execution. Use ast.literal_eval() for safe literals.

• Secrets Management: Never hardcode passwords/keys. Use environment variables or secret managers.

• Least Privilege: Run processes with minimal permissions.

Dependency Security:

• Regularly update packages.

• Scan for vulnerabilities:

  
  pip-audit        # Audits installed packages
  
  safety check     # Checks against known vulnerabilities
  
  

• Use requirements.txt with pinned versions (package==1.2.3) for reproducibility.

[!TIP] Security is layered. Validate inputs, use parameterized queries, keep dependencies updated, and never trust user data.