Advance Python Programming (CY-406) - Unit 1 Short Notes

1.1 Function & Method Advanced Techniques

Decorators (Deep Dive)

A decorator is a function that takes another function (or class) as input and returns a modified version, extending its behavior without permanent modification.

Implementation Core (Closure):

def decorator(func):

    def wrapper(*args, **kwargs):

        # Pre-processing

        result = func(*args, **kwargs)

        # Post-processing

        return result

    return wrapper

Key Types & Syntax:

• Function Decorators: @decorator syntax above function definition.

• Class Decorators: Decorator that modifies or returns a new class.

• Parameterized Decorators: Decorators that accept arguments. Requires an extra nesting level:

  
  def repeat(n):
  
      def decorator(func):
  
          def wrapper(*args, **kwargs):
  
              return [func(*args, **kwargs) for _ in range(n)]
  
          return wrapper
  
      return decorator
  
  

• functools.wraps: Crucial for preserving original function's __name__, __doc__, etc. Always use it in wrapper functions.

  
  from functools import wraps
  
  def decorator(func):
  
      @wraps(func)
  
      def wrapper(*args, **kwargs): ...
  
  

Built-in Decorators:

• @staticmethod: Method receives no implicit first argument (neither self nor cls). Behaves like a plain function attached to a class.

• @classmethod: Receives the class (cls) as first argument. Often used for factory methods.

• @property: Turns a method into a read-only attribute. Enables getter/setter/deleter pattern.

Common Use Cases:

• Logging: Record function calls and arguments.

• Timing: Measure execution time.

• Access Control/Rate Limiting: Check permissions or limit calls.

• Memoization: Caching results (see functools.lru_cache).

[!TIP] Exam Focus: Be prepared to write a decorator from scratch, especially a parameterized one. Understand the difference between @staticmethod and @classmethod.

Generators & Iterator Protocol

Generator Function: Uses yield to produce a sequence of values lazily (on-demand), pausing and resuming execution.

def count_up_to(n):

    i = 0

    while i < n:

        yield i  # Pauses here, returns i

        i += 1

Generator Expression: Similar to list comprehension but with (); produces a generator.

• list_comp = [x**2 for x in range(10)] (list, eager)

• gen_exp = (x**2 for x in range(10)) (generator, lazy)

Iterator Protocol: Objects that implement:

• __iter__(): Returns the iterator object itself.

• __next__(): Returns next value or raises StopIteration.

Advanced Generator Methods:

• .send(value): Resumes generator, sending a value that becomes the result of the yield expression.

• .throw(exc_type): Raises exception at the yield point.

• .close():** Raises GeneratorExit inside generator to terminate it.

itertools Module (Key Functions):

[!TIP] Memory Efficiency: Generators are O(1) in memory for the sequence length, unlike lists which are O(n). Use for large or infinite streams.

Function Arguments Unpacking

• *args in definition: collects extra positional arguments into a tuple.

• **kwargs in definition: collects extra keyword arguments into a dict.

• *iterable in call: unpacks an iterable into positional arguments.

• **dict in call: unpacks a dict into keyword arguments.

• Keyword-only arguments: Place a bare * in the signature before the keyword-only parameters.

  
  def func(pos1, pos2, *, kw1, kw2):
  
      # pos1, pos2: positional-or-keyword
  
      # kw1, kw2: keyword-only
  
  

1.2 Context Managers & Resource Management

The with Statement

Purpose: Deterministic setup and teardown. Ensures resources (files, locks, network connections) are properly acquired and released, even if exceptions occur.

Protocol: An object is a context manager if it implements:

• __enter__(self): Sets up resource, returns value bound to as variable (or None).

• __exit__(self, exc_type, exc_val, exc_tb): Performs cleanup. If it returns True, it suppresses the exception.

Implementing Custom Context Managers

Class-Based:

class ManagedFile:

    def __init__(self, filename):

        self.filename = filename

    def __enter__(self):

        self.file = open(self.filename, 'r')

        return self.file

    def __exit__(self, exc_type, exc_val, exc_tb):

        if self.file:

            self.file.close()

        # Return False to propagate exceptions (typical)

        return False

Generator-Based (contextlib.contextmanager):

from contextlib import contextmanager

@contextmanager

def managed_file(filename):

    file = open(filename, 'r')

    try:

        yield file

    finally:

        file.close()

• Code before yield is __enter__.

• Code after yield is __exit__.

• The yield expression's value is what as receives.

contextlib.ExitStack: Manages a dynamic number of context managers. Useful for handling resources where the number isn't known until runtime.

[!TIP] Common Pitfall: In a generator-based context manager, always use a try/finally block around yield to guarantee cleanup, even if the block inside with raises an exception.

1.3 Metaprogramming & Object Model

Metaclasses

A metaclass is the class of a class. Just as a class defines how an instance behaves, a metaclass defines how a class behaves.

• The default metaclass is type.

• type is itself a class (a metaclass).

Custom Metaclass Creation:

class Meta(type):

    def __new__(cls, name, bases, dct):

        # Modify class attributes (dct) before creation

        return super().__new__(cls, name, bases, dct)

    def __init__(cls, name, bases, dct):

        # Initialize the class after creation

        super().__init__(name, bases, dct)

class MyClass(metaclass=Meta):

    pass

• __new__ creates the class object.

• __init__ initializes the class object.

• Use cases: API enforcement (e.g., require certain methods), automatic registration (e.g., plugin systems), singleton pattern, ORM field mapping.

Descriptors

An object that defines any of __get__, __set__, __delete__. Used to manage attribute access on other objects.

Protocol:

• __get__(self, instance, owner): Returns attribute value. instance is the object accessing the descriptor, owner is the class.

• __set__(self, instance, value): Sets attribute value.

• __delete__(self, instance): Deletes attribute.

Data vs. Non-Data Descriptors:

• Data Descriptor: Implements __set__ and/or __delete__. Takes precedence over instance dictionaries.

• Non-Data Descriptor: Only implements __get__. Lower priority; instance dictionary wins.

Relationship:

• @property is a non-data descriptor (getter only) unless a setter is defined, making it a data descriptor.

• Methods are non-data descriptors (their __get__ returns a bound method).

Use Case Example (Validation):

class Typed:

    def __init__(self, name, expected_type):

        self.name = name

        self.expected_type = expected_type

    def __get__(self, instance, owner):

        return instance.__dict__[self.name]

    def __set__(self, instance, value):

        if not isinstance(value, self.expected_type):

            raise TypeError(f"Expected {self.expected_type}")

        instance.__dict__[self.name] = value

class Person:

    age = Typed('age', int)  # Descriptor

    def __init__(self, age):

        self.age = age  # Calls descriptor's __set__

[!TIP] Descriptor Precedence: Data descriptor > instance dictionary > non-data descriptor. This is key to understanding how property works with a setter.

1.4 Concurrency Foundations (Threading & Multiprocessing)

The Global Interpreter Lock (GIL)

• Definition: A mutex that allows only one thread to execute Python bytecode at a time within a single process.

• Impact:

  • CPU-bound tasks (number crunching): No speedup with threading. Use multiprocessing to bypass GIL.

  • I/O-bound tasks (network, disk): Effective speedup with threading. Threads release GIL during I/O waits, allowing others to run.

• asyncio: Single-threaded, cooperative concurrency using async/await. Efficient for high-concurrency I/O tasks.

threading Module

• Thread class: t = Thread(target=func, args=(...)); t.start().

• Synchronization Primitives:

  • Lock: Basic mutual exclusion. acquire(), release().

  • RLock: Reentrant lock; same thread can acquire multiple times.

  • Condition: For complex waiting/notification patterns (wait(), notify()).

  • Event: Simple flag for signaling between threads (set(), clear(), wait()).

  • Semaphore: Counter-based lock; allows n threads simultaneously.

• threading.local(): Creates thread-local storage. Data is isolated per thread.

• Daemon Threads: t.daemon = True; program exits when only daemon threads remain. Use for background tasks.

multiprocessing Module

• Bypasses GIL by using separate processes (each with its own Python interpreter and memory space).

• Process class: Similar API to Thread. Requires picklable arguments/functions.

• Inter-Process Communication (IPC):

  • Queue: Process-safe, like queue.Queue.

  • Pipe: Two-way communication between two processes.

• Synchronization: Lock, Semaphore (separate from threading versions).

• Pool classes:

  • multiprocessing.Pool: Manages a pool of worker processes. pool.map(func, iterable).

  • multiprocessing.pool.ThreadPool: Thread-based pool (uses threading).

Choosing Between Threads and Processes

[!TIP] Rule of Thumb: If task spends time waiting (I/O, sleep) → threading/asyncio. If task spends time computing → multiprocessing.

1.5 Abstract Base Classes (ABCs) & Interfaces

abc Module

Purpose: Define interfaces (abstract methods) and enable virtual subclassing.

Core Components:

• class MyABC(ABC): Inherit from ABC to create an abstract base class.

• @abstractmethod: Decorator marks a method as abstract. Subclasses must implement it.

• @abstractclassmethod, @abstractstaticmethod: For abstract class/static methods.

• @abstractproperty: Deprecated since Python 3.3. Use @property + @abstractmethod.

• ABC.register(subclass): Registers a virtual subclass. issubclass(VirtualSub, MyABC) returns True even without inheritance, but isinstance checks will fail unless the subclass also inherits.

Example:

from abc import ABC, abstractmethod

class Shape(ABC):

    @abstractmethod

    def area(self):

        pass

    @abstractmethod

    def perimeter(self):

        pass

class Circle(Shape):  # Must implement both abstract methods

    def __init__(self, r): self.r = r

    def area(self): return 3.14 * self.r**2

    def perimeter(self): return 2 * 3.14 * self.r

Common ABCs in collections.abc

[!TIP] Exam Application: Use ABCs for isinstance(obj, collections.abc.Iterable) checks instead of hasattr(obj, '__iter__') for more robust, semantic type checking.

1.6 Data Structures & Collections Deep Dive

collections Module

array Module

• Provides space-efficient arrays of basic C numeric types ('i' for int, 'f' for float).

• Not for general objects; use list or array.array for homogeneous numeric data.

• Methods similar to list but constrained to one type.

[!TIP] When to Use: deque for queues/stack with O(1) ends operations. Counter for frequency counting. defaultdict for grouping/accumulating. namedtuple for lightweight, immutable data structures.

1.7 Performance & Memory Optimization

Profiling Basics

• timeit: Measures small code snippets with high precision, disables GC.

  
  python -m timeit "sum(range(1000))"
  
  

• cProfile: Deterministic profiler. Shows call counts, total time, per-call time.

  
  python -m cProfile -s cumtime my_script.py
  
  

  • Sort by cumtime (cumulative time) to find bottlenecks.

• profile: Pure Python version of cProfile (slower).

Memory Optimization

• sys.getsizeof(obj): Returns size in bytes. Note: For containers, only size of container, not contents. Use recursively for total.

• __slots__: Class-level declaration that reserves space for a fixed set of attributes, preventing __dict__ and __weakref__ creation.

  
  class Point:
  
      __slots__ = ('x', 'y')  # Saves memory, restricts attributes
  
      def __init__(self, x, y):
  
          self.x = x; self.y = y
  
  

  • Memory Saving: Significant for many instances (~40-50%).

  • Limitation: Cannot add new attributes not in __slots__. No __dict__.

• Generators for Lazy Evaluation: Avoid building large intermediate lists. Use generator expressions or yield.

Caching & Memoization

• Manual: cache = {}; check key before compute, store after.

• functools.lru_cache(maxsize=128, typed=False): Least Recently Used cache.

  • maxsize=None for unbounded cache.

  • typed=True treats 3 and 3.0 as distinct keys.

  • Cache stats: cache_info() (hits, misses, size, maxsize).

  • Clear cache: cache_clear().

• functools.cache (Python 3.9+): Simple unbounded cache (alias for lru_cache(maxsize=None)).

[!TIP] Profiling Workflow: 1) Use cProfile to find slow functions. 2) Use timeit on isolated snippets. 3) Use sys.getsizeof to compare memory of data structures (e.g., list vs generator). 4) Apply __slots__ or lru_cache where appropriate.

1.8 Pythonic Idioms & Best Practices

EAFP vs. LBYL

• EAFP (Easier to Ask for Forgiveness than Permission): Use try/except. Preferred in Python.

  
  try:
  
      value = my_dict['key']
  
  except KeyError:
  
      # handle missing
  
  

• LBYL (Look Before You Leap): Check conditions first.

  
  if 'key' in my_dict:
  
      value = my_dict['key']
  
  

• Why EAFP? Often faster when exceptions are rare, avoids race conditions in concurrent code.

Comprehensions & Unpacking

• Comprehensions: Concise creation of lists, dicts, sets.

  
  [x**2 for x in range(10) if x%2]  # List
  
  {x: x**2 for x in range(10)}      # Dict
  
  {x**2 for x in range(10)}         # Set
  
  

• Extended Iterable Unpacking: * collects rest.

  
  first, *middle, last = [1,2,3,4,5]  # first=1, middle=[2,3,4], last=5
  
  

• Nested Unpacking:

  
  (a, b), (c, d) = [(1,2), (3,4)]
  
  

Iteration Helpers

• enumerate(iterable, start=0): Yields (index, element) pairs. Better than for i in range(len(seq)).

• zip(*iterables): Aggregates elements from each iterable into tuples. Stops at shortest iterable.

  
  names = ['a','b']; scores = [90,85]; dict(zip(names, scores))  # {'a':90, 'b':85}
  
  

Resource Management & Script Idioms

• with statement: Always for files, locks, network connections.

• if __name__ == "__main__":: Code inside runs only when script is executed directly, not when imported as module.

Type Hinting Essentials (PEP 484)

• Purpose: Static type analysis (via mypy), IDE support, documentation.

• Syntax:

  • Function annotations: def func(a: int) -> str: ...

  • Variable annotations: x: int = 5

• Common typing Types:

  • List[int], Dict[str, float], Tuple[int, ...] (variable length), Tuple[int, str] (fixed).

  • Optional[T] = Union[T, None].

  • Union[int, str].

  • Callable[[int, str], bool] (function type).

  • Any (disables checking).

• mypy: Static type checker. Run mypy script.py to find type inconsistencies.

[!TIP] Pythonic Mindset: Favor readability and simplicity. Use built-in functions (enumerate, zip) over manual indexing. Use comprehensions over map/filter + lambda for clarity. Adopt EAFP. Use type hints for public APIs and complex codebases.