Advance Python Programming (CY-406) Unit 3 Notes

3.1 Advanced Functions & Closures

First-Class Objects: Functions can be assigned to variables, passed as arguments, and returned from other functions. They have intrinsic attributes:
- __name__: Function's name.
- __doc__: Documentation string.
- __annotations__: Dict of type hints.
Nested Functions & Lexical Scoping: An inner function can access variables from its enclosing (non-local) scope. Resolution follows the LEGB rule (Local, Enclosing, Global, Built-in).
Closures: A nested function that captures and remembers variables from its enclosing scope, even after the outer function has finished execution.
- Creation: Occurs when a nested function references a variable from an outer scope.
- nonlocal keyword: Used inside a nested function to modify a variable from the nearest enclosing (non-global) scope.
```
def outer():

    count = 0

    def inner():

        nonlocal count  # Required to modify 'count'

        count += 1

        return count

    return inner
```
- Function Factories: Functions that return other functions, often using closures to "freeze" parameters.
  
  [!TIP] Common Pitfall: In loops, closures capture variables by reference, not by value. Use default arguments (lambda i=i: i) to capture the current value.

3.2 Decorators (Deep Dive)

Function Decorators: A syntax sugar for applying a wrapper function to another function.
- Mechanism: @decorator is equivalent to func = decorator(func).
- Implementation: A decorator is a callable that takes a function and returns a replacement function.
- functools.wraps: A decorator used inside custom decorators to preserve the original function's metadata (__name__, __doc__, etc.).
```
import functools

def my_decorator(func):

    @functools.wraps(func)

    def wrapper(*args, **kwargs):

        # pre-processing

        result = func(*args, **kwargs)

        # post-processing

        return result

    return wrapper
```

Decorators with Arguments (Decorator Factories): A decorator that accepts parameters. Requires an extra layer of nesting.


def repeat(n):  # Outer function: accepts decorator arguments

    def decorator(func):  # Middle: accepts the function

        @functools.wraps(func)

        def wrapper(*args, **kwargs):

            for _ in range(n):

                result = func(*args, **kwargs)

            return result

        return wrapper

    return decorator

@repeat(n=3)

def greet(): ...

Class Decorators: A callable that takes a class and returns a modified class (e.g., adding methods or attributes).


def add_repr(cls):

    def __repr__(self):

        return f"{cls.__name__}({self.__dict__})"

    cls.__repr__ = __repr__

    return cls

@add_repr

class Person: ...

functools Module:
- @lru_cache(maxsize=None): Memoization decorator using a Least Recently Used cache. Provides O(1) lookup for repeated calls with same arguments. Time Complexity: O(1) average for cache hits.
- @partial(func, *args, **kwargs): Freezes some portion of a function's arguments, yielding a new function with fewer parameters.
- @total_ordering: Generates missing ordering methods (__lt__, __le__, __gt__, __ge__) based on __eq__ and one of the others.
- @singledispatch: Transforms a function into a generic function that can have different implementations based on the type of the first argument.

Property Decorators: Manage attribute access via methods.

@property: Defines a getter.
@x.setter: Defines a setter (must have same name as property).
@x.deleter: Defines a deleter.


class Circle:

    def __init__(self, radius):

        self._radius = radius

    @property

    def radius(self):

        return self._radius

    @radius.setter

    def radius(self, value):

        if value <= 0: raise ValueError

        self._radius = value

Practical Applications: Logging, access control (permissions), memoization (lru_cache), registration (plugins), lazy evaluation.

3.3 Context Managers & The `with` Statement

Protocol: Objects used in a with statement must implement:
- __enter__(self): Called at the start of the with block. Returns value bound to as variable (if any).
- __exit__(self, exc_type, exc_val, exc_tb): Called at the end of the block. Can suppress exceptions by returning True.

Class-Based Implementation:


class ManagedFile:

    def __init__(self, filename):

        self.filename = filename

    def __enter__(self):

        self.file = open(self.filename, 'r')

        return self.file

    def __exit__(self, exc_type, exc_val, exc_tb):

        if self.file:

            self.file.close()

        # Return False to propagate exceptions

contextlib Module:
- @contextmanager: Decorator to create a context manager from a generator function. yield divides setup and cleanup.
```
from contextlib import contextmanager

@contextmanager

def managed_file(filename):

    file = open(filename, 'r')

    try:

        yield file

    finally:

        file.close()
```
- suppress(*exceptions): Context manager to ignore specified exceptions.
- redirect_stdout(new_target): Temporarily redirects sys.stdout.
Use Cases: File handling, database transactions, lock acquisition/release, temporary state changes.

3.4 Generators & Coroutines

Generator Functions: Defined with yield. Produce values lazily (on-demand), saving memory.
- Memory Efficiency: Only one value exists in memory at a time. Ideal for large/streaming datasets.
- Generator Expressions: Similar to list comprehensions but with () instead of []. E.g., (x*2 for x in range(10)).
Advanced Generator Methods:
- .send(value): Sends a value into the generator, becoming the result of the current yield expression. First call must be next() or send(None).
- .throw(type, value=None, traceback=None): Raises an exception inside the generator at the yield point.
- .close()``: Raises GeneratorExit` inside the generator to terminate it.
Coroutines: Generalized form of subroutines used for cooperative multitasking. They can pause (yield) and resume, often using .send() for two-way communication.

[!TIP] Key Distinction: All coroutines are generators (use yield), but not all generators are coroutines. Coroutines typically consume data sent via .send().
yield from (PEP 380): Delegates part of a generator's operations to a subgenerator. Handles:
- Value delegation (yield from subgen yields values from subgen).
- Exception propagation (.throw(), .close() passed through).
- Return value: The argument to StopIteration from the subgenerator becomes the value of the yield from expression.
```
def chain(*iterables):

    for it in iterables:

        yield from it  # Delegates to each iterator
```
Data Pipelines: Chain generators to process streams efficiently (e.g., reading a file, filtering, transforming).

3.5 Iterators & The Iterator Protocol

Protocol: An object is an iterator if it implements:
- __iter__(self): Returns the iterator object itself.
- __next__(self): Returns the next item. Raises StopIteration when exhausted.
Iterable vs Iterator:
- Iterable: Implements __iter__() (returns an iterator). Can be used in for loops. E.g., list, tuple, dict.
- Iterator: Implements __iter__() (returns self) and __next__(). Represents a stream of data.
```
it = iter([1,2,3])  # iter() calls __iter__ on list, returns list_iterator

next(it)  # 1
```

Creating Custom Iterators: Implement __iter__ and __next__. Maintain state (e.g., current index).


class CountUpTo:

    def __init__(self, max):

        self.max = max

        self.current = 0

    def __iter__(self):

        return self

    def __next__(self):

        if self.current >= self.max:

            raise StopIteration

        self.current += 1

        return self.current - 1

Built-in Functions: iter(), next(), enumerate(), zip().
collections.abc:
- Iterable: Abstract Base Class (ABC) requiring __iter__.
- Iterator: ABC requiring __iter__ and __next__.
- Use isinstance(obj, Iterable) to check.

3.6 Descriptors & Attribute Management

Descriptor Protocol: An object that defines any of __get__, __set__, __delete__. Used to manage attribute access on other objects.
- __get__(self, instance, owner): Returns attribute value. instance is the object instance, owner is the class.
- __set__(self, instance, value): Sets attribute value.
- __delete__(self, instance): Deletes attribute.
Types:
- Data Descriptor: Implements __set__ and/or __delete__. Takes precedence over instance dictionaries.
- Non-Data Descriptor: Only implements __get__. Lower precedence; instance dictionary wins.
property as a Descriptor: @property is a built-in non-data descriptor (getter only). Adding a setter makes it a data descriptor.
__getattribute__ vs __getattr__ vs __setattr__:
- __getattribute__: Called unconditionally for every attribute access. Must be used with extreme care to avoid infinite recursion.
- __getattr__: Called only when attribute is not found via normal lookup (i.e., not in instance __dict__ or class).
- __setattr__(self, name, value): Called for every attribute assignment. Must handle recursion carefully.
```
class Safe:

    def __setattr__(self, name, value):

        if name not in ['allowed']:

            raise AttributeError(f"{name} is read-only")

        super().__setattr__(name, value)  # Crucial!
```
Use Cases: Type checking, lazy attribute computation (load on first access), managed attributes (validation), ORM field mapping.

3.7 Metaclasses

type Metaclass: The default metaclass. In Python, classes are objects; their type is type.
```
class MyClass:  # Implicitly uses type as metaclass

    pass

type(MyClass)  # <class 'type'>
```
You can create classes dynamically: MyClass = type('MyClass', (Base,), {'attr': value}).
Custom Metaclasses: A class that inherits from type and overrides:
- __new__(mcs, name, bases, attrs): Creates the class object. mcs is the metaclass. Returns the new class.
- __init__(cls, name, bases, attrs): Initializes the class object after creation.
```
class Meta(type):

    def __new__(mcs, name, bases, attrs):

        attrs['added'] = True

        return super().__new__(mcs, name, bases, attrs)

class MyClass(metaclass=Meta):

    pass

MyClass.added  # True
```
Metaclass Hook: Specified via metaclass keyword in class definition (Python 3) or __metaclass__ in Python 2.

__prepare__ (PEP 3115): A special class method in a metaclass that returns the namespace dictionary (usually an OrderedDict) used to collect class body definitions. Allows customizing attribute order or behavior during class creation.


class OrderedMeta(type):

    @classmethod

    def __prepare__(mcs, name, bases, **kwargs):

        return collections.OrderedDict()

    def __new__(mcs, name, bases, attrs):

        # attrs is now an OrderedDict

        return super().__new__(mcs, name, bases, dict(attrs))

Common Use Cases:
- API Enforcement: Automatically add required methods or check signatures.
- Automatic Registration: Register subclasses in a registry (e.g., plugin systems).
- Singleton Pattern: Control instance creation.
- ORM Field Mapping: Convert class attributes to database columns.
Caution: "Metaclasses are deeper magic." Overuse leads to complex, hard-to-debug code. Often, class decorators or simple inheritance can achieve the same with less complexity.

3.8 Concurrency & Parallelism

Threading (threading module):
- GIL (Global Interpreter Lock): Allows only one thread to execute Python bytecode at a time. CPU-bound tasks do not benefit from threading; I/O-bound tasks do.
- Thread class: Create threads via subclassing or passing a target function.
- Synchronization Primitives:
  - Lock: Basic mutual exclusion.
  - RLock: Reentrant lock (same thread can acquire multiple times).
  - Semaphore: Limits concurrent access to a resource.
- Thread-Local Storage: threading.local() creates an object whose attributes are local to the current thread.
Multiprocessing (multiprocessing module):
- Bypasses GIL by using separate processes. Each process has its own Python interpreter and memory space.
- Pool: Manages a pool of worker processes (Pool.map, Pool.apply_async).
- IPC (Inter-Process Communication): Queue, Pipe, shared memory (Value, Array).
- Overhead: Higher than threading due to process creation and serialization (pickling) of data.
concurrent.futures (High-Level API):
- ThreadPoolExecutor: For I/O-bound tasks.
- ProcessPoolExecutor: For CPU-bound tasks.
- Unified interface: submit(fn, *args) returns a Future object; as_completed(futures) yields completed futures.
Asynchronous Programming (asyncio):
- Core Concepts: Single-threaded, event-driven concurrency using coroutines (async def).
- await: Pauses coroutine execution until the awaited awaitable (coroutine, Task, Future) completes.
- Event Loop: Central scheduler that runs coroutines and handles I/O events.
- Tasks & Futures: asyncio.create_task(coro) schedules a coroutine. Future represents a result that may not be available yet.
- Key Functions:
  - asyncio.gather(*coros): Runs coroutines concurrently and returns list of results in order.
  - asyncio.wait(futures): Low-level; waits for a set of futures to complete.
- Integrating Blocking Calls: Use loop.run_in_executor(None, blocking_func, *args) to run synchronous code in a thread pool without blocking the event loop.
[!TIP] Rule of Thumb: Use asyncio for high-concurrency I/O (web servers, APIs). Use multiprocessing for CPU-heavy parallel computation. Use threading for legacy I/O libraries that aren't async-compatible.

3.9 Performance Analysis & Optimization

Profiling:
- cProfile: Deterministic profiling (measures function call counts and time). Output can be analyzed with pstats.
```
python -m cProfile -o profile.prof my_script.py
```
- timeit: Measures execution time of small code snippets in a controlled environment (avoids common timing pitfalls).
```
import timeit

timeit.timeit('"-".join(str(n) for n in range(100))', number=10000)
```
- time.perf_counter(): High-resolution timer for measuring short durations.
Memory Profiling:
- sys.getsizeof(obj): Returns size of an object in bytes (does not track referenced objects).
- tracemalloc: Tracks memory allocations and provides statistics.
- pympler (third-party): Detailed object size and memory consumption analysis.
Optimization Strategies:
1. Algorithmic Complexity: First optimize Big O (e.g., replace O(n²) with O(n log n)).
2. Local Variable Lookup: Access to local variables is faster than global or built-in. Copy globals to locals in tight loops.
3. Use Built-in Functions & Libraries: Written in C, they are much faster (e.g., map(), sum(), collections.Counter).
4. functools.lru_cache: Memoize pure functions with expensive computations.
5. Avoid Dot Operations in Loops: Cache method lookups: local_func = obj.method before loop.
6. Data Structures: Choose the right one (list vs deque vs set).
Cython/Numba (Conceptual):
- Cython: Adds static type declarations to Python-like syntax, compiles to C extensions. Offers near-C speed for numerical code.
- Numba: Uses LLVM to JIT-compile Python functions (especially NumPy-aware) at runtime. Simple decorator (@njit).

3.10 Design Patterns in Python

Creational:
- Singleton: Simplest in Python via module (module-level variables are singleton). Metaclass or decorator alternatives exist.
- Factory Method: Use first-class functions or classes to return instances based on input.
- Builder: Chain methods to construct complex objects. Often replaced by flexible constructors or **kwargs.
Structural:
- Adapter: Wraps an object to provide a different interface. Simple wrapper class or functools.partial.
- Decorator (Pattern): Same as language feature (Section 3.2) but applied to add responsibilities dynamically.
- Facade: Provides a simplified interface to a complex subsystem. A class with coordinated method calls.
- Proxy: Controls access to another object (e.g., lazy loading, access control). Implement descriptor or __getattr__.
Behavioral:
- Strategy: Encapsulate algorithms. In Python, often just pass a function instead of a class.
- Observer (Publisher-Subscriber): Maintain a list of dependents (subscribers) and notify them of state changes. Simple list of callbacks.
- Command: Encapsulate a request as an object. Useful for undo/redo, queues. Simple callable objects or functions.
- Iterator: Built into Python via __iter__/__next__ protocol.
Pythonic Adaptations: Leverage duck typing, first-class functions, and closures to reduce boilerplate. Patterns are often implicit in the language's design.

3.11 Packaging & Distribution

Project Structure:


project/

├── src/                    # Source code (prevents import of local dir)

│   └── mypackage/

│       ├── __init__.py

│       └── module.py

├── tests/

├── pyproject.toml          # Modern metadata & build system

├── README.md

└── LICENSE

setuptools & pyproject.toml (PEP 517/518/621):
- pyproject.toml declares build system ([build-system]) and package metadata ([project]).
- Key fields: name, version, dependencies, authors, description.
- Entry points (console scripts): [project.scripts] mycli = "mypackage.module:main".
Building Distributions:
- sdist (source distribution): python -m build --sdist
- wheel (binary distribution): python -m build --wheel (preferred for pure Python).
Publishing to PyPI:
1. Build distributions.
2. Upload with twine upload dist/*.
3. Requires PyPI account and API token.
Virtual Environments: Isolate project dependencies.
- venv (built-in): python -m venv .venv
- pipenv / poetry: Higher-level tools with lock files (Pipfile.lock, poetry.lock) for deterministic builds.

3.12 Code Quality & Testing

Static Analysis:
- flake8: Combines PyFlakes (errors), pycodestyle (PEP 8), and McCabe (complexity).
- pylint: Deep analysis, checks for errors, enforces coding standards, provides refactoring hints.
- black: Uncompromising code formatter (opinionated).
- isort: Sorts imports.
- mypy: Static type checker for PEP 484 type hints.
Testing Frameworks:
- unittest: xUnit-style (JUnit inspired). Requires classes inheriting from TestCase.
- pytest (Preferred):
  - Simple functions with assert.
  - Fixtures (@pytest.fixture): Setup/teardown code.
  - Parametrization (@pytest.mark.parametrize): Run test with multiple inputs.
  - Fixtures scope: function, class, module, session.
Mocking & Patching:
- unittest.mock / pytest-mock:
  - Mock(): Generic mock object.
  - patch(): Temporarily replace a target (e.g., @patch('module.Class')).
  - MagicMock: Mock with default magic methods.
Test Coverage: coverage.py measures which lines are executed. coverage run -m pytest, coverage report -m.
Continuous Integration (CI): Automated pipeline (e.g., GitHub Actions, GitLab CI) that runs tests, linting, and coverage on every push/PR. Ensures code quality and prevents regressions.

3.13 Advanced Standard Library Modules

dataclasses (PEP 557):
- @dataclass: Automatically adds __init__, __repr__, __eq__ based on class attributes.
- field(): Customize fields (default, default_factory, init, repr, compare).
- Methods: asdict(), astuple(), replace(), fields().
- Inheritance: Fields from base classes are included.
```
from dataclasses import dataclass, field

@dataclass(order=True)

class User:

    name: str

    id: int = field(default=0, compare=False)
```
enum (PEP 435):
- Enum: Base class for enumerations.
- auto(): Automatically assign values.
- Flag: For bitwise-combinable enumerations.
- Unique enforcement: @unique.
typing Module (PEP 484):
- Type Hints: List[int], Dict[str, Any], Callable[[int], str].
- TypeVar: Generic type variable. T = TypeVar('T').
- Generic: Base class for generic types. class Box(Generic[T]): ....
- Protocol (PEP 544): Structural subtyping (duck typing). Define an interface by method signatures.
```
from typing import Protocol

class SupportsWrite(Protocol):

    def write(self, data: bytes) -> int: ...

def send_all(writer: SupportsWrite, data: bytes) -> None: ...
```
- Annotated: Attach metadata to type hints. Annotated[int, Range(0, 100)].
abc Module:
- ABC: Abstract Base Class helper.
- @abstractmethod: Decorator to mark abstract methods. Class cannot be instantiated until all abstract methods are overridden.
```
from abc import ABC, abstractmethod

class Shape(ABC):

    @abstractmethod

    def area(self) -> float: ...
```
weakref Module:
- WeakKeyDictionary, WeakValueDictionary: Dictionaries that hold weak references to keys/values. Entries are automatically removed when key/value is garbage collected.
- weakref.finalize(obj, func, *args): Registers a finalizer function to be called when obj is about to be destroyed.
array Module: Memory-efficient arrays of basic C types ('i' for signed int, 'f' for float). More efficient than list for homogeneous numeric data.
bisect Module: Provides bisect_left, bisect_right for binary search in sorted lists. Time Complexity: O(log n) for search.

3.14 Introduction to C Extensions & FFI

Conceptual Overview: Drop to C/C++/Rust when:
- Performance-critical inner loops (CPU-bound).
- Interfacing with existing C libraries.
- Need fine-grained memory control.

ctypes:

Loads shared libraries (.dll, .so, .dylib) at runtime.
Define function prototypes (argtypes, restype) for type safety.
Directly call C functions from Python.


from ctypes import cdll, c_int

lib = cdll.LoadLibrary("./mylib.so")

lib.my_func.argtypes = [c_int]

lib.my_func.restype = c_int

result = lib.my_func(42)

cffi (Conceptual): Alternative Foreign Function Interface. More Pythonic API than ctypes. Can operate in "in-line" mode (C code in Python strings) or "out-of-line" mode (separate C file). Often faster.
Cython:
- A superset of Python with static type declarations.
- Files: .pyx (Cython source), .pxd (Cython declaration files).
- Compiles to C extension modules. Can call C functions directly and use C data types (cdef int).
- Provides near-C speed for numerical loops while maintaining Python-like syntax.
- Workflow: .pyx → C code → compiled shared object (.so/.pyd) → importable in Python.

Final Exam Strategy: For RGPV CY-406, prioritize Sections 3.2 (Decorators), 3.4 (Generators/Coroutines), 3.8 (Concurrency), and 3.13 (typing/dataclasses). These are core language features frequently tested in coding and theory questions. Understand the protocols (iterator, context manager, descriptor) deeply—they are common exam topics for "write a class that..." questions. For concurrency, clearly distinguish threading vs multiprocessing vs asyncio use cases and GIL implications.

Account

Advance Python Programming (CY-406) - Unit 3 Short Notes

3.1 Advanced Functions & Closures

3.2 Decorators (Deep Dive)

3.3 Context Managers & The `with` Statement

3.4 Generators & Coroutines

3.5 Iterators & The Iterator Protocol

3.6 Descriptors & Attribute Management

3.7 Metaclasses

3.8 Concurrency & Parallelism

3.9 Performance Analysis & Optimization

3.10 Design Patterns in Python

3.11 Packaging & Distribution

3.12 Code Quality & Testing

3.13 Advanced Standard Library Modules

3.14 Introduction to C Extensions & FFI

3.1 Advanced Functions & Closures

3.2 Decorators (Deep Dive)

3.3 Context Managers & The with Statement

3.4 Generators & Coroutines

3.5 Iterators & The Iterator Protocol

3.6 Descriptors & Attribute Management

3.7 Metaclasses

3.8 Concurrency & Parallelism

3.9 Performance Analysis & Optimization

3.10 Design Patterns in Python

3.11 Packaging & Distribution

3.12 Code Quality & Testing

3.13 Advanced Standard Library Modules

3.14 Introduction to C Extensions & FFI

3.3 Context Managers & The `with` Statement