UNIT 5: ADVANCED PYTHON PROGRAMMING - SHORT NOTES

MODULE 5.1: METAPROGRAMMING & INTROSPECTION

The __dunder__ (double underscore) methods deep dive

Object construction:
- __new__(cls, ...): Static method, creates and returns instance. Called before __init__.
- __init__(self, ...): Initializes instance after creation.
Attribute access:
- __getattr__(self, name): Called when attribute not found via normal lookup.
- __getattribute__(self, name): Called for all attribute access. Must avoid infinite recursion.
- __setattr__(self, name, value): Called to set attribute.
- __delattr__(self, name): Called to delete attribute.
Representation:
- __repr__(self): Official string representation, ideally eval(repr(obj)) == obj.
- __str__(self): Informal string representation, used by print() and str().
Container/sequence emulation:
- __len__(self): Returns length for len(obj).
- __getitem__(self, key): Access obj[key]. Supports slicing if key is slice object.
- __setitem__(self, key, value): Assign obj[key] = value.
- __delitem__(self, key): Delete del obj[key].
- __iter__(self): Returns iterator object for iter(obj).
- __contains__(self, item): Implements item in obj.
Callable objects: __call__(self, ...) makes instance callable like a function.
Context managers: __enter__(self) returns resource, __exit__(self, exc_type, exc_val, exc_tb) handles cleanup.

Introspection & the inspect module

Examining live objects:
- inspect.getmembers(obj): Returns list of (name, value) tuples.
- inspect.isfunction(), inspect.ismethod(), inspect.isclass(): Type checks.
- inspect.getsource(obj): Retrieves source code (if available).
Inspecting call signatures:
- inspect.signature(func): Returns Signature object with parameters.
- inspect.getfullargspec(func): Detailed argspec (legacy).
Stack frames:
- inspect.stack(): Returns list of frame records for current call stack.

Metaclasses

What is a metaclass?: Class of a class. Default metaclass is type.

Custom metaclass creation:


class Meta(type):

    def __new__(mcs, name, bases, attrs):

        # Modify attrs before class creation

        return super().__new__(mcs, name, bases, attrs)

    def __init__(cls, name, bases, attrs):

        # Initialize class after creation

        super().__init__(name, bases, attrs)

class MyClass(metaclass=Meta):

    pass

Practical use cases:
- ORM: Automatically map class attributes to database columns.
- API registration: Auto-register classes in a registry.
- Enforcing patterns: Ensure all subclasses implement required methods.

Advanced Decorators

Decorator factories (decorators with arguments):


def repeat(times):

    def decorator(func):

        @functools.wraps(func)

        def wrapper(*args, **kwargs):

            for _ in range(times):

                result = func(*args, **kwargs)

            return result

        return wrapper

    return decorator

Preserving metadata: Use functools.wraps to copy __name__, __doc__, etc.

Class decorators:


def add_repr(cls):

    def __repr__(self):

        attrs = ', '.join(f"{k}={v!r}" for k, v in self.__dict__.items())

        return f"{cls.__name__}({attrs})"

    cls.__repr__ = __repr__

    return cls

@add_repr

class Point:

    def __init__(self, x, y): self.x, self.y = x, y

Decorating methods and descriptors: Decorators on methods receive function; use @property or custom descriptors for managed attributes.

[!TIP]

Common Pitfall: __getattribute__ must use object.__getattribute__ or super().__getattribute__ to avoid recursion. Metaclasses are inherited; use metaclass=Meta in base class to apply to all subclasses.

MODULE 5.2: ADVANCED ITERATION & GENERATORS

Generator Functions & Expressions

yield and generator state machines:
- yield pauses function, saves local state, returns value.
- Next call resumes after last yield.
- Generator is a state machine with states for each yield.
Generator methods:
- .send(value): Sends value to yield expression, resumes. First call must be None.
- .throw(exc_type, exc_val, exc_tb): Raises exception at yield point.
- .close(): Raises GeneratorExit at yield; performs cleanup.
Generator expressions vs. list comprehensions:
- (x for x in iterable): Lazy, memory-efficient, returns generator.
- [x for x in iterable]: Eager, builds full list in memory.

Coroutines & Asynchronous Generators (pre-asyncio/async-await)

Coroutines using yield:

Consumer-producer pattern: yield sends data, send() receives.


def consumer():

    while True:

        data = yield

        print(f"Consumed: {data}")

c = consumer()

next(c)  # Prime

c.send("item")  # Send data

@asyncio.coroutine and yield from:
- @asyncio.coroutine marks generator-based coroutine.
- yield from delegates to subcoroutine, enabling chaining.
- Historical context: Replaced by async def/await in Python 3.5+.

The itertools Module

Infinite iterators:
- count(start=0, step=1): Infinite arithmetic progression.
- cycle(iterable): Repeats elements of iterable forever.
- repeat(elem, times=None): Repeats elem indefinitely or times.
Finite iterators:
- chain(*iterables): Chains iterables sequentially.
- compress(data, selectors): Filters data where selectors is truthy.
- dropwhile(pred, seq): Drops elements while pred true, then returns rest.
- takewhile(pred, seq): Takes elements while pred true.
- islice(iterable, start, stop, step): Slices iterable like seq[start:stop:step].
Combinatoric iterators:
- product(*iterables, repeat=1): Cartesian product.
  - Number of tuples: $$\displaystyle \prod_{i=1}^{k} n_i $$ for $k$ iterables of lengths $$\displaystyle n_i $$.
- permutations(iterable, r=None): Length-$r$ permutations.
  - Count: $$\displaystyle P(n, r) = \frac{n!}{(n-r)!} $$.
- combinations(iterable, r): Length-$r$ combinations, no repetition.
  - Count: $$\displaystyle C(n, r) = \frac{n!}{r!(n-r)!} $$.
- combinations_with_replacement(iterable, r): Combinations with repetition.
  - Count: $C(n+r-1, r)$.

The functools Module

partial and partialmethod:
- partial(func, *args, **kwargs): Returns new function with fixed arguments.
- partialmethod(method, *args, **kwargs): For methods in class definition.

lru_cache for memoization:


@lru_cache(maxsize=128)

def fib(n):

    if n < 2: return n

    return fib(n-1) + fib(n-2)

Cache size bounded by maxsize (LRU eviction). Use maxsize=None for unbounded.

singledispatch and singledispatchmethod:

Implement generic functions based on type of first argument.


@singledispatch

def func(arg):

    return "default"

@func.register(int)

def _(arg):

    return "int"

reduce:
- reduce(func, iterable, initializer): Applies func cumulatively.
- Caution: Often less readable than explicit loops; prefer functools.reduce only when necessary.

[!TIP]

Exam Focus: Generator .send() can pass values into coroutines. itertools functions return iterators, not lists—use list() to materialize. lru_cache uses O(1) average lookup due to dict + linked list.

MODULE 5.3: CONCURRENCY & PARALLELISM

Threading

threading module:
- Thread(target=func, args=(), kwargs={}): Start with start(), join with join().
- Lock: Basic mutual exclusion. acquire(blocking=True), release().
- RLock: Reentrant lock; same thread can acquire multiple times.
- Condition: Wait/notify pattern. wait(), notify(), notify_all().
- Semaphore(value=1): Permits value concurrent acquisitions.
Thread-local storage: threading.local() creates namespace object with per-thread attributes.
The Global Interpreter Lock (GIL):
- Only one thread executes Python bytecode at a time.
- Implications: CPU-bound multi-threaded programs do not achieve true parallelism; I/O-bound programs benefit.

Thread pools: concurrent.futures.ThreadPoolExecutor(max_workers=None):


with ThreadPoolExecutor() as executor:

    futures = [executor.submit(func, arg) for arg in args]

    results = [f.result() for f in futures]

Multiprocessing

multiprocessing module:
- Process(target=func, args=()): Start with start(), join with join().
- Queue: Process-safe FIFO queue.
- Pipe(): Returns connection objects for duplex/simplex communication.
- Manager(): Returns manager object for shared data (list, dict, etc.) via proxies.
Process-based parallelism: Each process has its own Python interpreter and memory space → bypasses GIL.
Process pools: concurrent.futures.ProcessPoolExecutor(max_workers=None):
- Uses multiprocessing under the hood; pickles tasks/results.
Inter-process communication (IPC) challenges:
- Objects must be picklable.
- Overhead: Serialization, context switching, memory duplication.
- Prefer Queue/Pipe over shared memory for simplicity.

Asynchronous I/O (asyncio)

Core concepts:
- Event loop: Central controller; runs tasks, handles I/O events.
- Coroutines: Defined with async def. Paused with await.
- Tasks: Wrapped coroutines scheduled on event loop (asyncio.create_task()).
High-level APIs:
- asyncio.run(main()): Creates event loop, runs main until complete.
- asyncio.create_task(coro): Schedules coroutine as Task.
- asyncio.gather(*coros): Runs coroutines concurrently, returns results list.
- asyncio.wait(tasks): Waits for tasks to complete; returns (done, pending).
Streams and subprocesses:
- asyncio.open_connection(host, port): Returns (reader, writer).
- asyncio.start_server(handler, host, port): Returns Server.
- asyncio.create_subprocess_exec(*cmd): Returns Process.
Synchronization primitives (asyncio versions):
- asyncio.Lock(), asyncio.Event(), asyncio.Semaphore(value), asyncio.Queue(maxsize=0).

[!TIP]

GIL vs. Multiprocessing: Use multiprocessing for CPU-bound parallelism; use threading or asyncio for I/O-bound. asyncio is single-threaded but handles many concurrent I/O operations efficiently. Always join() threads/processes to avoid zombie processes.

MODULE 5.4: DESIGN PATTERNS IN PYTHON

Creational Patterns

Singleton:

Ensures one instance globally.

Thread-safe implementation:


import threading

class Singleton:

    _instance = None

    _lock = threading.Lock()

    def __new__(cls):

        if cls._instance is None:

            with cls._lock:

                if cls._instance is None:

                    cls._instance = super().__new__(cls)

        return cls._instance

Pitfalls: Global state, testing difficulty. Consider module-level variables instead.

Factory Method: Subclass decides which class to instantiate.


class ShapeFactory:

    def create_shape(self, shape_type):

        if shape_type == "circle":

            return Circle()

        elif shape_type == "square":

            return Square()

Abstract Factory: Creates families of related objects without specifying concrete classes.

Builder: Separates construction of complex object from representation.


class PizzaBuilder:

    def __init__(self): self.reset()

    def reset(self): self.pizza = Pizza()

    def add_topping(self, topping): self.pizza.toppings.append(topping)

    def build(self): return self.pizza

Structural Patterns

Adapter:

Inheritance: Subclass adaptee, override methods.
Composition: Hold adaptee instance, translate calls.


class Adapter:

    def __init__(self, adaptee): self.adaptee = adaptee

    def request(self): return self.adaptee.specific_request()

Facade: Provides simplified interface to complex subsystem.


class Facade:

    def __init__(self):

        self._subsystem1 = Subsystem1()

        self._subsystem2 = Subsystem2()

    def operation(self):

        self._subsystem1.method1()

        self._subsystem2.method2()

Proxy: Controls access to object (lazy loading, access control, logging).


class Proxy:

    def __init__(self, real_subject): self._real_subject = real_subject

    def request(self):

        if self.check_access():

            return self._real_subject.request()

Composite: Composes objects into tree structures; treats individual and composite uniformly.


class Component:

    def operation(self): pass

class Composite(Component):

    def __init__(self): self.children = []

    def add(self, child): self.children.append(child)

    def operation(self):

        for child in self.children: child.operation()

Behavioral Patterns

Observer (publish-subscribe):


class Subject:

    def __init__(self): self._observers = []

    def attach(self, observer): self._observers.append(observer)

    def notify(self, data):

        for observer in self._observers: observer.update(data)

Strategy: Encapsulates interchangeable algorithms.


class Context:

    def __init__(self, strategy): self._strategy = strategy

    def execute(self, data): return self._strategy.algorithm(data)

Template Method: Defines algorithm skeleton in base class; subclasses override steps.


class AbstractClass:

    def template_method(self):

        self.step1()

        self.step2()  # Hook or abstract

    def step1(self): pass

    def step2(self): raise NotImplementedError

Command: Encapsulates request as object; supports undo/redo.


class Command:

    def __init__(self, receiver): self.receiver = receiver

    def execute(self): self.receiver.action()

class Invoker:

    def __init__(self): self._commands = []

    def store_and_execute(self, cmd):

        self._commands.append(cmd)

        cmd.execute()

Pythonic "Patterns" & Idioms

Context Managers: Use __enter__/__exit__ or @contextlib.contextmanager.

Borg pattern (monostate):


class Borg:

    _shared_state = {}

    def __init__(self):

        self.__dict__ = self._shared_state

All instances share state via __dict__.

Flyweight pattern:
- Use __slots__ to reduce memory overhead per instance.
- Use weakref.WeakValueDictionary for caching shared objects.

[!TIP]

Singleton in Multithreading: Double-checked locking pattern shown above. Prefer Borg for shared state without strict singleton constraint. Composite pattern naturally fits Python's duck typing.

MODULE 5.5: ADVANCED OBJECT-ORIENTED PROGRAMMING

Descriptors

Descriptor protocol:
- __get__(self, instance, owner): Returns attribute value.
- __set__(self, instance, value): Sets attribute.
- __delete__(self, instance): Deletes attribute.
Data vs. non-data descriptors:
- Data descriptor: Defines __set__ or __delete__. Takes precedence over instance dictionary.
- Non-data descriptor: Only __get__. Instance dictionary overrides.

Building managed attributes (type validation):


class Typed:

    def __init__(self, name, expected_type):

        self.name = name

        self.expected_type = expected_type

    def __get__(self, instance, owner):

        return instance.__dict__[self.name]

    def __set__(self, instance, value):

        if not isinstance(value, self.expected_type):

            raise TypeError(f"Expected {self.expected_type}")

        instance.__dict__[self.name] = value

class Person:

    age = Typed('age', int)

Relationship with properties and class decorators: @property is a data descriptor; class decorators can inject descriptors.

Advanced Properties & @property

Cached properties: functools.cached_property (Python 3.8+):


from functools import cached_property

class Data:

    @cached_property

    def expensive(self):

        return compute()

Computed once, stored in instance __dict__.

Writable properties and setter/deleter:


class Circle:

    def __init__(self, radius): self._radius = radius

    @property

    def radius(self): return self._radius

    @radius.setter

    def radius(self, value): self._radius = max(0, value)

    @radius.deleter

    def radius(self): del self._radius

Multiple Inheritance & MRO

C3 Linearization algorithm:
- Merges MROs of parents with list of parents, preserving order and monotonicity.
- Goal: Consistent ordering respecting local precedence order (child before parents) and preserving parent order.
- Example: For class D(A, B, C), MRO = [D] + merge(MRO(A), MRO(B), MRO(C), [A, B, C]).
super() in complex hierarchies:
- super() with no args: Uses __class__ and self (or cls for classmethod) to compute MRO.
- super(CurrentClass, self): Explicitly specifies starting point in MRO.
- Rule: All cooperative classes must use super() in diamond hierarchies.

Mixins: Small classes providing specific functionality, designed for multiple inheritance.


class LogMixin:

    def log(self, msg): print(f"[LOG] {msg}")

class Service(LogMixin, BaseClass):

    def process(self): self.log("Processing")

Abstract Base Classes (ABCs)

abc module:


from abc import ABC, abstractmethod

class MyABC(ABC):

    @abstractmethod

    def do_something(self): pass

    @property

    @abstractmethod

    def read_only(self): pass

Cannot instantiate; subclasses must override abstract methods.

Registering virtual subclasses:


class MyABC(ABC):

    pass

MyABC.register(list)  # list is virtual subclass

issubclass(list, MyABC)  # True

Defining interfaces and structural subtyping: ABCs define protocols; isinstance(obj, ABC) checks if object implements required methods (duck typing with explicit registration).

[!TIP]

MRO Calculation: Use ClassName.__mro__ or help(ClassName) to view. super() follows MRO; ensure all classes in diamond call super() to avoid duplicate calls. Descriptors: data descriptors always win over instance dict.

MODULE 5.6: PERFORMANCE OPTIMIZATION & TOOLING

Profiling & Benchmarking

timeit module:


import timeit

timeit.timeit('"-".join(str(n) for n in range(100))', number=1000)

Best for micro-benchmarks; disables GC by default.

cProfile and profile:
```
import cProfile

cProfile.run('my_function()')
```
- Deterministic profiling; records function calls and times.

Analyzing output with pstats:


import pstats

stats = pstats.Stats('profile_stats')

stats.sort_stats('cumulative').print_stats(10)

Sort by 'cumulative', 'time', 'calls', etc.

Line-by-line profiling (line_profiler third-party):
```
@profile

def my_func():

    ...
```
Run with kernprof -l -v script.py.

Memory Profiling

tracemalloc:


import tracemalloc

tracemalloc.start()

# ... code ...

snapshot = tracemalloc.take_snapshot()

top_stats = snapshot.statistics('lineno')

Tracks memory allocations; compares snapshots.

memory_profiler (third-party):


from memory_profiler import profile

@profile

def my_func():

    ...

Run with python -m memory_profiler script.py.

Optimization Techniques

Algorithmic optimization: Reduce time complexity (e.g., $$\displaystyle O(n^2) $$ to $O(n \log n)$).
Built-in functions: sum(), map(), filter() implemented in C → faster than Python loops.
Local variable lookups: Local variables faster than globals; cache global lookups in local vars inside loops.
String concatenation: Use ''.join(list_of_strings) instead of += in loops (quadratic vs. linear time).
Generators for memory efficiency: Use generator expressions ((x for x in ...)) instead of list comprehensions when possible.

Introduction to C Extensions & Compilation

Cython: Write Python-like code (.pyx files) compiled to C extensions.
- Add static type declarations for speedups.
- Compile with setup.py or pyproject.toml.
C extension modules:
- Write C functions with PyObject* return types.
- Define PyMethodDef array, module initialization function.
- Build with setuptools.Extension and setup.py.
- Concept: Bypass Python interpreter for CPU-critical loops.

[!TIP]

Profiling First: Optimize only after profiling—avoid premature optimization. cProfile shows cumulative time; line_profiler shows per-line. Memory leaks often from circular references; use gc.collect() or weakref.

MODULE 5.7: PACKAGING, DISTRIBUTION & BEST PRACTICES

Modern Packaging (setuptools, wheel, pip)

pyproject.toml and PEP 517/518:
- Specifies build system (e.g., [build-system] requires = ["setuptools", "wheel"]).
- Replaces setup.py for build isolation.
setup.cfg vs. setup.py:
- setup.cfg: Declarative configuration (INI format). Preferred for simple packages.
- setup.py: Imperative, allows custom logic. Still needed for complex builds.
Building distributions:
- python -m build: Creates sdist (source) and bdist_wheel (binary) in dist/.
- sdist: .tar.gz with source; requires build on target.
- wheel (.whl): Pre-built; installs without build step.
Publishing to PyPI:
- Test on TestPyPI: twine upload --repository testpypi dist/*.
- Production: twine upload dist/*.

Dependency Management

requirements.txt:
```
package==1.2.3

git+https://github.com/user/repo.git@tag
```
- Simple but no lock file; reproducibility issues.
Modern tools:
- Pipenv: Pipfile + Pipfile.lock.
- Poetry: pyproject.toml + poetry.lock; handles dependencies and publishing.
- uv: Fast resolver/installer; compatible with requirements.txt and pyproject.toml.
Pinning versions: Use == for production; >= for development flexibility.

Code Quality & Static Analysis

Linters:
- flake8: Combines pyflakes (errors), pycodestyle (PEP 8), mccabe (complexity).
- pylint: More checks (design, conventions), configurable.
Formatters:
- black: Uncompromising, PEP 8 compliant; no configuration.
- isort: Sorts imports by section, by module, by alphabetical.
Type checkers: mypy checks type hints; integrates with CI.
Pre-commit hooks: .pre-commit-config.yaml runs linters/formatters on git commit.

Testing Advanced Features

Mocking and patching: unittest.mock.patch, Mock, MagicMock.


with patch('module.function') as mock_func:

    mock_func.return_value = 42

    result = module.function()

Testing context managers, decorators, generators:
- Context managers: Use with in test; assert __enter__/__exit__ calls.
- Generators: Iterate and collect results; test .send()/.throw().

Parametrized tests (pytest):


@pytest.mark.parametrize("input,expected", [(1,2), (3,4)])

def test_func(input, expected):

    assert func(input) == expected

Fixtures and test scopes:
- @pytest.fixture(scope="function"): Default; runs per test.
- scope="module", "session" for reuse.

[!TIP]

Packaging: Always include __init__.py in packages (even empty). Use include_package_data=True and MANIFEST.in for non-code files. wheel is preferred over sdist for binary packages.

MODULE 5.8: ADVANCED STANDARD LIBRARY & THIRD-PARTY ECOSYSTEM (SELECTED)

Advanced collections

namedtuple: Immutable tuple with named fields.
```
Point = namedtuple('Point', ['x', 'y'])

p = Point(1, 2)
```
- Limitations vs. dataclasses: No default values easily, immutable, no post-init.
OrderedDict: Maintains insertion order (guaranteed in Python 3.7+ for dict). Still useful for move_to_end() and popitem(last=False).

defaultdict with complex default factories:


dd = defaultdict(list)

dd['key'].append(1)  # Auto-initializes to []

Counter: Multiset operations.


c = Counter('abracadabra')

c.most_common(3)  # [('a', 5), ('b', 2), ('r', 2)]

deque: Thread-safe, fast appends/pops on both ends ($O(1)$).
```
dq = deque([1,2,3])

dq.appendleft(0); dq.pop()
```

Data Classes (dataclasses)

@dataclass decorator:


from dataclasses import dataclass, field

@dataclass(order=True, frozen=True)

class Point:

    x: float

    y: float = field(default=0.0, compare=False)

Parameters: init, repr, eq, order, unsafe_hash, frozen.

Field metadata and default factories:
```
my_list: list[int] = field(default_factory=list)
```
- default_factory called for each instance.

Post-init processing:


def __post_init__(self):

    self.x = abs(self.x)  # Validation/transformation

Inheritance: Fields from base classes included; @dataclass on subclass adds new fields.

Type Hints Deep Dive

Generic types:


from typing import TypeVar, Generic, List

T = TypeVar('T')

class Box(Generic[T]):

    def __init__(self, item: T): self.item = item

    def get(self) -> T: return self.item

Protocols (structural subtyping):


from typing import Protocol

class SupportsClose(Protocol):

    def close(self) -> None: ...

def close_all(objs: List[SupportsClose]):

    for obj in objs: obj.close()

TypedDict for dictionary type hints:


class Movie(TypedDict):

    name: str

    year: int

m: Movie = {'name': 'Inception', 'year': 2010}

Annotated for metadata:


from typing import Annotated

def func(x: Annotated[int, Positive()]): ...

Using mypy: Run mypy script.py; integrates with editors/CI.

Pathlib & Modern File I/O

Path object:


from pathlib import Path

p = Path('data/file.txt')

text = p.read_text()

p.write_text('new content')

for f in p.parent.glob('*.py'): print(f)

/ operator for path joining: p / 'subdir' / 'file.txt'.

contextlib Utilities

@contextmanager:


from contextlib import contextmanager

@contextmanager

def managed_file(name):

    f = open(name, 'w')

    try:

        yield f

    finally:

        f.close()

contextlib.suppress: Suppress specified exceptions.


with suppress(FileNotFoundError):

    os.remove('missing.txt')

contextlib.redirect_stdout: Redirect stdout to file or other stream.

ExitStack: Manage multiple context managers dynamically.


with ExitStack() as stack:

    files = [stack.enter_context(open(f)) for f in filenames]

    # All files closed at exit

[!TIP]

dataclasses vs. namedtuple: Use dataclasses for mutable objects with defaults/validation. Counter arithmetic: c1 + c2 sums counts; c1 & c2 intersection (min). Path objects are immutable; methods return new Path.

END OF UNIT 5 NOTES
Always refer to official Python documentation (docs.python.org) for latest details and edge cases.

Account

Advance Python Programming (CY-406) - Unit 5 Short Notes

UNIT 5: ADVANCED PYTHON PROGRAMMING - SHORT NOTES

MODULE 5.1: METAPROGRAMMING & INTROSPECTION

MODULE 5.2: ADVANCED ITERATION & GENERATORS

MODULE 5.3: CONCURRENCY & PARALLELISM

MODULE 5.4: DESIGN PATTERNS IN PYTHON

MODULE 5.5: ADVANCED OBJECT-ORIENTED PROGRAMMING

MODULE 5.6: PERFORMANCE OPTIMIZATION & TOOLING

MODULE 5.7: PACKAGING, DISTRIBUTION & BEST PRACTICES

MODULE 5.8: ADVANCED STANDARD LIBRARY & THIRD-PARTY ECOSYSTEM (SELECTED)