Advance Python Programming (CY-406) - Unit 2 Short Notes

2.1 Advanced Object-Oriented Programming (OOP)

Class and Instance Namespaces

• __dict__ Attribute:

  • Class Namespace: ClassName.__dict__ stores class attributes (methods, class variables). Shared across all instances.

  • Instance Namespace: instance.__dict__ stores instance-specific attributes. Unique per object.

• Attribute Lookup Order (MRO): When accessing obj.attr, Python searches:

  1. Instance namespace (obj.__dict__)

  2. Class namespace (Class.__dict__) and its base classes (following MRO)

  3. __getattr__ if defined

  [!TIP] Use ClassName.__mro__ or ClassName.mro() to view the Method Resolution Order. Crucial for understanding inheritance and super() calls.

Properties and Descriptors

• @property Decorator: Creates a data descriptor for managed attribute access.

  
  class Circle:
  
      def __init__(self, radius):
  
          self._radius = radius
  
      @property
  
      def radius(self):
  
          return self._radius
  
      @radius.setter
  
      def radius(self, value):
  
          if value <= 0: raise ValueError("Radius must be positive")
  
          self._radius = value
  
  

• Custom Descriptors: Implement __get__, __set__, __delete__.

  • Data Descriptor: Implements __set__. Takes precedence over instance dictionary.

  • Non-data Descriptor: Only __get__. Instance dictionary wins.

  
  class Descriptor:
  
      def __get__(self, obj, objtype=None):
  
          ...
  
      def __set__(self, obj, value):
  
          ...
  
  

__slots__ Mechanism

• Purpose: Memory optimization (prevents __dict__ creation) and restricts allowed attributes.

• Syntax: class MyClass: __slots__ = ('attr1', 'attr2')

• Implications:

  • Subclasses must also define __slots__ to add new slots.

  • Pickling: Requires extra handling; default pickling may fail.

  • Inheritance: If a parent has __slots__, child's __dict__ is not created unless __slots__ includes '__dict__'.

  [!WARNING] __slots__ prevents adding arbitrary new attributes (obj.new_attr = x raises AttributeError).

Metaclasses

• type Metaclass: The default metaclass. type(name, bases, dict) creates classes.

• Custom Metaclass: Inherit from type and override __new__ or __init__.

  
  class Meta(type):
  
      def __new__(cls, name, bases, dct):
  
          # Modify class before creation (e.g., add methods)
  
          return super().__new__(cls, name, bases, dct)
  
  class MyClass(metaclass=Meta):
  
      pass
  
  

• Use Cases:

  • API Enforcement: Ensure subclasses implement required methods.

  • Registration: Automatically register classes in a plugin system.

  • ORM Frameworks: Map class attributes to database columns.

Abstract Base Classes (ABCs)

• abc Module:

  • Inherit from ABC (or use metaclass=ABCMeta).

  • Decorate methods with @abstractmethod (or @abstractproperty for properties).

  • Cannot instantiate a class with unimplemented abstract methods.

• Virtual Subclasses: ABC.register(SubClass) allows issubclass(SubClass, ABC) to return True without actual inheritance (structural subtyping).

  [!TIP] ABCs define interfaces. Use them for isinstance() checks and to enforce contracts in large codebases or frameworks.

2.2 Functional Programming Tools & Techniques

First-Class Functions & Closures

• First-Class: Functions can be assigned to variables, passed as arguments, returned from other functions.

• Closure Mechanics:

  • A nested function that captures variables from its enclosing scope.

  • Captured variables are stored in cell objects accessible via func.__closure__.

  • The nonlocal keyword allows modification of captured variables.

  
  def outer(x):
  
      def inner(y):
  
          nonlocal x  # Required to modify x
  
          x += y
  
          return x
  
      return inner
  
  

Decorators (Deep Dive)

• Function Decorator: A callable that takes a function and returns a replacement.

  
  def decorator(func):
  
      def wrapper(*args, **kwargs):
  
          # Do something before
  
          result = func(*args, **kwargs)
  
          # Do something after
  
          return result
  
      return wrapper
  
  @decorator
  
  def my_func(): ...
  
  

• Decorator with Arguments (Factory):

  
  def repeat(times):
  
      def decorator(func):
  
          def wrapper(*args, **kwargs):
  
              for _ in range(times):
  
                  result = func(*args, **kwargs)
  
              return result
  
          return wrapper
  
      return decorator
  
  @repeat(times=3)
  
  def greet(): ...
  
  

• Class Decorator: Applied to a class; typically modifies or wraps the class object.

• functools.wraps: Decorator to copy metadata (__name__, __doc__) from func to wrapper. Always use it in custom decorators.

  [!TIP] Decorators are evaluated at import time, not runtime. Be mindful of side effects.

functools Module

• partial: Freezes some arguments/keywords of a function, creating a new callable.

  
  from functools import partial
  
  int_base2 = partial(int, base=2)
  
  int_base2('1010')  # Returns 10
  
  

• lru_cache: Memoization with Least Recently Used eviction.

  
  from functools import lru_cache
  
  @lru_cache(maxsize=128, typed=False)
  
  def fib(n):
  
      if n < 2: return n
  
      return fib(n-1) + fib(n-2)
  
  

  • typed=True treats 3 and 3.0 as distinct calls.

• reduce: Applies a binary function cumulatively to items of an iterable.

  
  from functools import reduce
  
  reduce(lambda x, y: x*y, [1,2,3,4])  # (((1*2)*3)*4) = 24
  
  

Iterators and Generators (Advanced)

• Iterator Protocol: Objects with __iter__() (returns iterator) and __next__() (returns next item or raises StopIteration).

• Generator Functions: Use yield to produce a sequence lazily. State is saved between calls.

  • yield from: Delegates to a subgenerator. Used for coroutines and complex generator chains.

• Generator Expressions: (x**2 for x in range(10)). Memory-efficient vs. list comprehensions [x**2 for x in range(10)].

• Context Managers for Generators: @contextlib.contextmanager decorator allows writing a generator that yields once, with setup before yield and teardown after.

  
  from contextlib import contextmanager
  
  @contextmanager
  
  def managed_file(filename):
  
      f = open(filename, 'w')
  
      try:
  
          yield f
  
      finally:
  
          f.close()
  
  

2.3 Concurrency and Parallelism

Threading (threading module)

• Thread Class: t = Thread(target=func, args=(...)); t.start(), t.join().

• Synchronization Primitives:

  • Lock: Basic mutual exclusion.

  • RLock: Reentrant lock (same thread can acquire multiple times).

  • Semaphore: Counter-based access for limited resources.

  • Event: Thread signaling (set/clear/wait).

  • Condition: Complex coordination (wait/notify).

• Daemon Threads: t.daemon = True (or Thread(..., daemon=True)). Dies when main program exits.

• Thread-Local Data: threading.local() creates an object whose attributes are local to the current thread.

• GIL (Global Interpreter Lock):

  • Only one thread executes Python bytecode at a time.

  • I/O-bound tasks: Threading effective (GIL released during I/O).

  • CPU-bound tasks: Threading ineffective; use multiprocessing.

Multiprocessing (multiprocessing module)

• Process Class: Similar API to Thread. Spawns separate Python interpreter.

• IPC (Inter-Process Communication):

  • Queue: Process-safe FIFO queue.

  • Pipe: Two-way communication between two processes.

• Process Synchronization: Lock, Semaphore (for shared memory coordination).

• Pool Classes:

  • Pool: Process pool for parallel execution (map, apply_async).

  • ThreadPool: (In multiprocessing.pool) Hybrid approach.

• Sharing State:

  • Value, Array: Shared memory primitives (ctypes).

  • Manager: Provides server process managing shared objects (dict, list, Namespace).

Asynchronous Programming (asyncio)

• Core Concepts:

  • Event Loop: Central scheduler.

  • Coroutine: Defined with async def. Must be awaited.

  • await: Pauses coroutine, yields control to event loop.

• Tasks: asyncio.create_task(coro) schedules coroutine. asyncio.gather(*tasks) waits for all. asyncio.wait(tasks) returns completed/pending.

• Async Context Managers/Iterators: async with, async for require __aenter__, __aiter__, etc.

• Running Async Code:

  
  asyncio.run(main())  # Top-level entry (Python 3.7+)
  
  loop = asyncio.get_event_loop()
  
  loop.run_until_complete(main())
  
  

• Async I/O: asyncio.open_connection() (streams), asyncio.create_subprocess_exec().

• Async Synchronization: asyncio.Lock, asyncio.Event, asyncio.Semaphore (non-blocking, awaitable).

• asyncio.to_thread(func, *args): Runs blocking func in a thread, returns awaitable.

[!COMPARISON] Concurrency Model Choice:

| Model | Best For | GIL Impact |

|-------|----------|------------|

| threading | I/O-bound (network, file) | Released during I/O |

| multiprocessing | CPU-bound (computation) | Bypassed (separate processes) |

| asyncio | High-concurrency I/O (thousands of connections) | Single-threaded, cooperative |

2.4 Performance Analysis and Optimization

Profiling Code

• time Module: Simple wall-clock timing (time.perf_counter()).

• cProfile: Deterministic profiler. Generates call statistics.

  
  python -m cProfile -s cumulative my_script.py
  
  

  Use pstats module to sort/filter output.

• line_profiler (Third-party): @profile decorator, kernprof -l -v script.py. Line-by-line timing.

• memory_profiler (Third-party): @profile decorator, python -m memory_profiler script.py. Tracks line-wise memory usage.

Optimization Strategies

1. Algorithmic Complexity (Big O): Primary optimization target. Reduce time complexity (e.g., O(n²) → O(n log n)).

2. Leverage Built-ins & C Libraries: Built-in functions (sum, map) and libraries (collections, itertools) are implemented in C → faster.

3. Local Variable Lookups: Accessing local variables (x) is faster than globals (global_x) or attributes (obj.x).

4. Data Structures:

   • list: Dynamic array, fast indexed access O(1), slow insert/delete O(n).

   • deque (collections): Double-ended queue, fast appends/pops at both ends O(1).

   • set: Hash table, fast membership test O(1), unordered.

5. String Concatenation: Avoid s += "a" in loops (creates new string each time). Use "".join(list_of_strings).

6. JIT Compilation: numba decorator (@njit) compiles Python to machine code (numeric code).

7. C Extensions/Cython: Write performance-critical sections in C or Cython (Python-like syntax compiled to C).

[!CRITICAL] Optimization Rule: Profile before optimizing. Focus on bottlenecks (often 20% of code causing 80% of slowdown). Algorithmic change > micro-optimization.

2.5 Packaging and Distribution

Project Structure

• Modern Standard: pyproject.toml (PEP 518, 621). Declares build system & metadata.

• Legacy: setup.py (executable script) + setup.cfg (declarative config).

• Layout:

  • Flat: Package dir at root.

  • src/: Package inside src/ directory (prevents import confusion).

setuptools & wheel

• Metadata (setup.cfg/pyproject.toml):

  
  [project]
  
  name = "mypackage"
  
  version = "0.1.0"
  
  dependencies = ["requests>=2.0"]
  
  [tool.setuptools.packages.find]
  
  where = ["src"]
  
  

• Distributions:

  • sdist (source): .tar.gz archive. Platform-independent.

  • bdist_wheel (binary): .whl file. Pre-compiled, faster install.

• Development Mode: pip install -e . → installs as symlink; changes reflect immediately.

pip and Dependencies

• requirements.txt (legacy): package==version (pinned). Generated by pip freeze.

• pyproject.toml (modern): [project] dependencies = ["package>=1.0"].

• Version Specifiers:

  • ==: Exact version.

  • >=: Minimum version.

  • ~=: Compatible release (PEP 440). ~=1.4 → >=1.4, ==1.*.

• Virtual Environments: Isolate dependencies (python -m venv .venv, conda create).

Creating and Publishing Packages

1. Build: python -m build (creates dist/ with .tar.gz and .whl).

2. Test Upload: twine upload --repository testpypi dist/*.

3. Production Upload: twine upload dist/* (to PyPI).

4. Versioning: Semantic Versioning (MAJOR.MINOR.PATCH). Pre-releases: 1.0.0a1, 1.0.0rc1.

2.6 Advanced Modules and Ecosystem

collections Module

itertools Module

• Infinite: count(start=0, step=1), cycle(iterable), repeat(elem, n=None).

• Finite:

  • chain(*iterables): Concatenate.

  • compress(data, selectors): Filter by boolean mask.

  • dropwhile(pred, seq), takewhile(pred, seq).

  • islice(iterable, start, stop, step).

• Combinatoric:

  • product(*iterables, repeat=1): Cartesian product.

  • permutations(iterable, r=None): r-length permutations, no repeats.

  • combinations(iterable, r): r-length combinations, no order, no repeats.

  • combinations_with_replacement(iterable, r).

contextlib Utilities

• @contextmanager: Decorator for generator-based context managers.

• closing(obj): Context manager calling obj.close() on exit.

• suppress(*exceptions): Ignores specified exceptions in block.

• redirect_stdout/redirect_stderr(new_target): Redirects output streams.

dataclasses Module (Python 3.7+)

from dataclasses import dataclass, field

@dataclass(order=True, frozen=False)

class Employee:

    name: str

    id: int = field(default=0, compare=False)

    skills: list[str] = field(default_factory=list)

    def __post_init__(self):

        # Custom validation/modification

        self.name = self.name.title()

• Auto-generates __init__, __repr__, __eq__.

• field(): Customize individual fields (default, default_factory, compare, metadata).

• __post_init__(): Hook for additional initialization.

• frozen=True: Makes instances immutable (like namedtuple).

Type Hints & typing Module (Advanced)

• Generics:

  
  from typing import TypeVar, Generic, List
  
  T = TypeVar('T')
  
  class Box(Generic[T]):
  
      def __init__(self, item: T): ...
  
      def get(self) -> T: ...
  
  

• Protocols (Structural Subtyping):

  
  from typing import Protocol
  
  class SupportsClose(Protocol):
  
      def close(self) -> None: ...
  
  def close_all(items: List[SupportsClose]): ...
  
  

• TypedDict: Dict with specific key types.

  
  from typing import TypedDict
  
  class User(TypedDict):
  
      name: str
  
      age: int
  
  

• Annotated: Attach metadata to type hints.

• @overload: Multiple type signatures for a function.

• mypy: Static type checker (mypy script.py).

2.7 Testing and Quality Assurance

unittest Framework

• Structure:

  
  import unittest
  
  class TestMath(unittest.TestCase):
  
      def setUp(self): ...  # Runs before each test
  
      def tearDown(self): ...  # Runs after each test
  
      def test_add(self):
  
          self.assertEqual(1+1, 2)
  
          with self.assertRaises(TypeError):
  
              "a" + 1
  
  

• Test Discovery: python -m unittest discover -s tests/.

• Assertions: assertEqual(a,b), assertTrue(x), assertRaises(exc, func, *args).

pytest Framework (Industry Standard)

• Simple Tests: Just functions with assert.

  
  def test_add():
  
      assert 1+1 == 2
  
  

• Fixtures:

  
  import pytest
  
  @pytest.fixture(scope="module")
  
  def db_connection():
  
      conn = connect_db()
  
      yield conn
  
      conn.close()
  
  def test_query(db_connection):
  
      db_connection.execute("SELECT ...")
  
  

• Parametrize:

  
  @pytest.mark.parametrize("a,b,expected", [(1,1,2), (2,3,5)])
  
  def test_add(a,b,expected):
  
      assert a+b == expected
  
  

• Markers: @pytest.mark.skip(reason="..."), @pytest.mark.xfail.

• conftest.py: Shared fixtures across test files.

• Capturing Output: capsys (stdout/stderr), capfd (file descriptors).

Test Doubles (unittest.mock)

• Mock/MagicMock: Replace objects, record calls.

  
  from unittest.mock import Mock, patch
  
  m = Mock()
  
  m.method(1,2,3)
  
  m.method.assert_called_once_with(1,2,3)
  
  

• patch: Temporarily replace an object.

  
  @patch('module.ClassName')
  
  def test_something(mock_class):
  
      mock_class.return_value.method.return_value = 42
  
      ...
  
  

• Stub: Predefined response.

• Spy: Wraps real object, records calls.

Code Coverage

• coverage.py:

  
  coverage run -m pytest
  
  coverage report -m  # Terminal report
  
  coverage html      # Generates htmlcov/
  
  

• Measures line coverage and branch coverage.

• Set target in .coveragerc: [run] branch = True.

2.8 Debugging and Development Tools

pdb - The Python Debugger

• Basic Commands:

  • l (list): Show current code.

  • n (next): Execute current line, step over.

  • s (step): Step into function.

  • c (continue): Resume execution.

  • p expr: Print expression value.

  • q (quit): Exit debugger.

• Breakpoints:

  • In code: breakpoint() (Python 3.7+ → calls sys.breakpointhook, default pdb.set_trace()).

  • In pdb: b filename:lineno or b function.

• Post-Mortem Debugging: import pdb; pdb.pm() after exception.

Logging (logging module)

• Levels: DEBUG < INFO < WARNING < ERROR < CRITICAL.

• Basic Config:

  
  import logging
  
  logging.basicConfig(level=logging.INFO,
  
                      format='%(asctime)s - %(levelname)s - %(message)s')
  
  

• Hierarchy: Logger → Handler → Formatter.

  
  logger = logging.getLogger(__name__)
  
  handler = logging.FileHandler('app.log')
  
  formatter = logging.Formatter('%(name)s - %(levelname)s - %(message)s')
  
  handler.setFormatter(formatter)
  
  logger.addHandler(handler)
  
  

• Dict Config: logging.config.dictConfig({...}) for complex setups.

rich Library (Modern Terminal Formatting)

• Pretty Print: from rich import print; print({"data": [1,2,3]}) → syntax-highlighted, aligned.

• Console Objects: console = Console(); console.print("[bold red]Error![/]").

• Panels/Tables: Panel("text", title="Log"), Table() for structured data.

• Traceback: from rich.traceback import install; install() → colorful tracebacks.

IPython / Jupyter Enhancements

• Magic Commands:

  • %timeit expr: Measure execution time of single statement.

  • %run script.py: Run external script in IPython namespace.

  • %debug: Launch debugger at last exception.

  • %load_ext module: Load extension.

• Shell Interaction:

  • !ls -la: Run shell command.

  • obj? / obj??: Show docstring / source code.

• %debug After Exception: Automatically enters pdb at point of failure.

[!ADVANCED] Combine tools: Use pdb/%debug for interactive debugging, logging for persistent logs, and rich for enhanced console output in development scripts.