Advance Python Programming (CY-406) - Unit 4 Short Notes

UNIT 4: ADVANCED PYTHON PROGRAMMING - CORE CONCEPTS & APPLICATIONS

4.1 Concurrency and Parallelism

Core Concept: Concurrency is about dealing with many tasks at once (structure), while Parallelism is about doing many tasks at once (execution on multiple cores).

The Global Interpreter Lock (GIL):

• A mutex that protects access to Python objects, preventing multiple native threads from executing Python bytecodes at once.

• Impact: CPU-bound multithreaded programs do not achieve speedup. Use multiprocessing for CPU work.

• I/O-bound threads release the GIL during wait, allowing concurrency.

threading Module:

• Creation: threading.Thread(target=func, args=(...)), start(), join().

• Synchronization Primitives:

  • Lock: Basic mutual exclusion. acquire(), release().

  • RLock: Reentrant lock (same thread can acquire multiple times).

  • Semaphore: Counter-based lock for limiting concurrent access (e.g., to a pool of connections).

  • Condition: For complex thread coordination (producer-consumer).

• Daemon Threads: thread.daemon = True. Threads that die when the main program exits.

• Thread-Local Data: threading.local(). Creates variables that are local to a specific thread.

multiprocessing Module:

• Process Creation: multiprocessing.Process(target=func).

• Inter-Process Communication (IPC):

  • Pipe(): Bidirectional or unidirectional connection between two processes.

  • Queue(): Process-safe, FIFO queue (multiple producers/consumers).

  • Manager().list()/dict(): Shared objects via a server process (slower).

  • Array, Value: Shared memory for simple C types.

• Process Pools: multiprocessing.Pool(processes=n). Manages a pool of worker processes. Use pool.map() or pool.apply_async().

asyncio Framework:

• Core: async def defines a coroutine. await suspends execution until the awaited task completes, yielding control to the event loop.

• Event Loop: Central scheduler. asyncio.run(main()) creates and manages the loop.

• Tasks: Wrap coroutines for concurrent execution. task = asyncio.create_task(coro()).

• Concurrent Patterns:

  • Gather: await asyncio.gather(coro1(), coro2()) runs coroutines concurrently and returns results.

  • Semaphore: asyncio.Semaphore(value) limits concurrent access (e.g., to an API).

• Asynchronous Context Managers/Iterators: Use async with and async for with objects defining __aenter__, __aexit__, __aiter__, __anext__.

Common Concurrency Patterns & Pitfalls:

• Producer-Consumer: Use Queue (threading/multiprocessing) or asyncio.Queue.

• Race Condition: Outcome depends on unpredictable thread/process timing. Fix: Use locks/semaphores.

• Deadlock: Two or more threads/processes wait for each other indefinitely (e.g., Thread1 holds LockA, needs LockB; Thread2 holds LockB, needs LockA). Fix: Acquire locks in a global order, use timeouts (lock.acquire(timeout=1)).

• Starvation: A thread/process is perpetually denied resource access.

• Debugging: Use threading.enumerate(), multiprocessing.active_children(). Logging is crucial. Tools: vscode debugger, pdb (limited for threads).

[!TIP] Exam Key: GIL only affects CPU-bound threading. For I/O, threading is fine. For CPU, use multiprocessing. For massive I/O concurrency (e.g., web server), use asyncio.

4.2 Networking and Web Programming

Low-Level Socket Programming (socket module):

• Socket Types: SOCK_STREAM (TCP - reliable, ordered), SOCK_DGRAM (UDP - fast, unreliable).

• Client-Server Model:

  
  # Server (TCP)
  
  s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
  
  s.bind(('0.0.0.0', port))
  
  s.listen()
  
  conn, addr = s.accept()  # Blocking
  
  # Client
  
  s = socket.socket()
  
  s.connect(('server_ip', port))
  
  

• Key Methods: send(), recv() (TCP); sendto(), recvfrom() (UDP). Always handle partial sends/receives in production.

• Socket Options: setsockopt() (e.g., SO_REUSEADDR to reuse address).

High-Level Libraries:

• requests (HTTP): Synchronous, user-friendly. requests.get(url), response.json(), response.status_code.

• paramiko (SSH): For secure remote execution, file transfer (SFTP).

Building Web Services (http.server, socketserver):

• Simple HTTP server: http.server.HTTPServer. Subclass BaseHTTPRequestHandler and override do_GET(), do_POST().

• socketserver provides threading/multiprocessing mixins (ThreadingMixIn, ForkingMixIn) for concurrent handling.

Web Frameworks (Flask/Django Core):

• Routing: Map URLs to view functions (@app.route('/path') in Flask, urlpatterns in Django).

• Views: Functions/classes that process requests and return Response objects (HTML, JSON, etc.).

• Templates: Server-side rendering (Jinja2 in Flask, Django Templates). Use {{ variable }} and {% tag %}.

• ORM Basics: Object-Relational Mapper (Django ORM, SQLAlchemy). Define models as Python classes; ORM handles SQL queries. Model.objects.filter(field=value), Model.objects.all().

RESTful API Design Principles:

• Resources: Nouns (e.g., /users, /users/123).

• HTTP Methods: GET (read), POST (create), PUT (replace), PATCH (update), DELETE (remove).

• Status Codes:

  • 2xx Success: 200 OK, 201 Created, 204 No Content.

  • 4xx Client Error: 400 Bad Request, 401 Unauthorized, 404 Not Found.

  • 5xx Server Error: 500 Internal Server Error.

• JSON: Standard data format. Use json.dumps()/json.loads() or framework serializers.

WebSockets & Real-Time:

• Full-duplex, persistent connection over a single TCP socket. Server can push data to client without request.

• Libraries: websockets (asyncio-based), django-channels.

[!TIP] Exam Key: Know the difference between HTTP methods and status code classes. REST is stateless; resources are nouns.

4.3 Data Science and Numerical Computing Stack

NumPy Fundamentals:

• ndarray: N-dimensional, homogeneous-typed array. Fast vectorized operations.

• Creation: np.array(), np.zeros(shape), np.ones(), np.arange(), np.linspace().

• Indexing/Slicing: arr[1:5, ::2]. Boolean indexing: arr[arr > 0].

• Broadcasting: Implicit element-wise operation on arrays of different shapes. Rules: trailing dimensions must be 1 or equal.

  
  # Example: (3,1) + (1,4) -> (3,4)
  
  a = np.array([[1],[2],[3]])  # (3,1)
  
  b = np.array([10,20,30,40])  # (4,)
  
  c = a + b  # Result is (3,4)
  
  

• Universal Functions (ufuncs): Element-wise operations (np.sqrt, np.sin, np.add). reduce(), accumulate().

• Linear Algebra: np.dot(), np.linalg.inv(), np.linalg.eig().

pandas Core Structures:

• Series: 1D labeled array (index + values).

• DataFrame: 2D table (collection of Series with shared index). Columns are Series.

• Data Loading: pd.read_csv(), read_excel(), read_sql().

• Data Cleaning: df.dropna(), df.fillna(), df.drop_duplicates(), df.astype().

• Selection:

  • df['col'] or df.col (Series).

  • df[['col1','col2']] (DataFrame).

  • .loc[] (label-based), .iloc[] (integer position-based).

• Grouping & Aggregation: df.groupby('col')['value'].agg(['mean','sum']).

• Merging/Joining: pd.merge(df1, df2, on='key'), df1.join(df2).

Basic Visualization (matplotlib, seaborn):

• matplotlib: Low-level. plt.plot(x,y), plt.scatter(), plt.hist(), plt.bar(). plt.subplot(nrows, ncols, index).

• seaborn: High-level, statistical. sns.histplot(), sns.boxplot(), sns.heatmap(). Uses matplotlib under the hood.

scipy Intro:

• scipy.stats: Distributions, statistical tests (ttest_ind, norm).

• scipy.optimize: Minimization (minimize), root finding (root).

[!TIP] Exam Key: Broadcasting in NumPy and groupby in pandas are extremely high-yield. Know the difference between .loc and .iloc.

4.4 Testing, Debugging, and Profiling

Advanced Testing (pytest):

• Fixtures: Setup/teardown code. @pytest.fixture provides reusable context.

  
  @pytest.fixture
  
  def db_connection():
  
      conn = connect()
  
      yield conn  # Setup before test, teardown after
  
      conn.close()
  
  def test_query(db_connection):
  
      db_connection.execute(...)
  
  

• Parametrization: @pytest.mark.parametrize("input,expected", [(1,2), (3,4)]) runs test multiple times.

• Mocking/Patching: unittest.mock.patch() replaces objects with mocks. mock.assert_called_once_with().

• Test Discovery: pytest automatically finds test_*.py files.

• Coverage: pytest --cov=my_module. Use coverage.py for detailed reports.

Debugging Techniques:

• pdb/ipdb: import pdb; pdb.set_trace(). Commands: n (next), s (step), c (continue), l (list), p expr (print).

• logging Module:

  • Levels: DEBUG, INFO, WARNING, ERROR, CRITICAL.

  • Configuration: logging.basicConfig(level=logging.INFO, format='...').

  • Handlers: FileHandler, StreamHandler, RotatingFileHandler.

• IDE Debugging: Breakpoints, step-through, watch variables (VSCode, PyCharm).

Performance Profiling:

• cProfile: Deterministic, measures function call times. python -m cProfile -o output.prof script.py. Analyze with pstats.

• line_profiler: Line-by-line timing. Decorate function with @profile. Run with kernprof -l -v script.py.

• memory_profiler: Line-by-line memory usage. Decorate with @profile. Run with python -m memory_profiler script.py.

• Identifying Bottlenecks: Look for:

  1. Hotspots (functions with high tottime in cProfile).

  2. Unnecessary computations in loops.

  3. Excessive object creation.

  4. Poor algorithmic complexity (e.g., O(n²) where O(n) possible).

[!TIP] Exam Key: Know pytest fixtures and mock.patch. For profiling, cProfile gives function-level, line_profiler gives line-level.

4.5 Packaging, Distribution, and Environment Management

Creating Reusable Packages:

• Directory Structure:

  
  mypackage/
  
  ├── src/ or mypackage/          # Source code
  
  │   ├── __init__.py
  
  │   └── module.py
  
  ├── tests/
  
  ├── README.md
  
  ├── LICENSE
  
  └── pyproject.toml   # Modern standard (replaces setup.py)
  
  

• pyproject.toml (PEP 518/621): Declares build system ([build-system]) and project metadata ([project]). Tools: setuptools, poetry, flit.

  
  [project]
  
  name = "mypackage"
  
  version = "0.1.0"
  
  dependencies = ["requests>=2.0"]
  
  

• setup.py (Legacy): from setuptools import setup; setup(name='...', packages=...). Still supported but pyproject.toml preferred.

Dependency & Environment Management:

• pip + requirements.txt: pip freeze > requirements.txt. pip install -r requirements.txt.

• pipenv: Combines pip + virtualenv. Pipfile + Pipfile.lock. pipenv install.

• poetry: Modern tool. pyproject.toml manages deps & packaging. poetry add requests, poetry build, poetry publish.

• Virtual Environments:

  • python -m venv .venv (standard library).

  • conda create -n myenv python=3.9 (manages non-Python deps too).

Building & Distributing:

• Source Distribution (sdist): python -m build --sdist. .tar.gz file.

• Binary Wheel: python -m build --wheel. .whl file (pre-compiled, faster install).

• Publishing: twine upload dist/* to PyPI (after poetry build or python -m build).

[!TIP] Exam Key: pyproject.toml is the modern standard. Know the difference between sdist (source) and wheel (binary). Use poetry or pipenv over raw pip + venv for reproducibility.

4.6 Design Patterns and Best Practices in Python

Pythonic Design Patterns:

• Creational:

  • Factory: Function/class returns instance of different classes based on input. Use @classmethod as alternative constructors.

  • Singleton:

    1. Module-level: Python modules are singletons by default (imported once).

    2. Borg (Monostate): All instances share state. self.__dict__ = shared_state.

• Structural:

  • Adapter: Wraps an object to provide a different interface. Simple wrapper class.

  • Decorator: Wraps a function/class to add behavior. Use @decorator syntax. Function decorator pattern.

  • Proxy: Controls access to another object (e.g., lazy loading, access control).

• Behavioral:

  • Observer: Maintain list of dependents (subscribers) and notify them of state changes.

  • Strategy: Encapsulate algorithms, make them interchangeable. Pass strategy object to context.

  • State: Object changes behavior when internal state changes. Use classes for each state.

  • Context Manager (with statement): Manages resources (acquire/release). Implement __enter__, __exit__.

Writing Clean Code:

• PEP 8: Style guide (indentation, naming, imports). Use black auto-formatter.

• Type Hinting (typing): def func(name: str, age: int) -> bool:. Use List, Dict, Optional, Union, Callable.

• Docstrings: Google/NumPy/Sphinx styles. Describe args, returns, raises.

  
  def add(a: int, b: int) -> int:
  
      """Adds two integers.
  
      Args:
  
          a: First integer.
  
          b: Second integer.
  
      Returns:
  
          Sum of a and b.
  
      """
  
      return a + b
  
  

• Principles:

  • DRY: Don't Repeat Yourself.

  • KISS: Keep It Simple, Stupid.

  • YAGNI: You Ain't Gonna Need It (don't over-engineer).

  • Composition over Inheritance: Favor has-a (composition) over is-a (inheritance) for flexibility.

Metaprogramming Basics:

• @property: Turn method into read-only attribute. @x.setter for write.

• __getattr__, __setattr__: Intercept attribute access/missing.

• __call__: Make instance callable like a function.

• Simple Metaclass: class MyMeta(type): def __new__(mcs, name, bases, dct): .... class MyClass(metaclass=MyMeta):.

[!TIP] Exam Key: Singleton via module is most Pythonic. Context Manager is a key pattern. Type hints and docstrings are mandatory for professional code.

4.7 Advanced Language Features and Idioms

Descriptors:

• Objects defining __get__(self, obj, type=None), __set__, __delete__.

• Used to manage attribute access. @property is a data descriptor.

  
  class Descriptor:
  
      def __get__(self, obj, type=None):
  
          return obj._value
  
      def __set__(self, obj, value):
  
          obj._value = value * 2  # Example logic
  
  class MyClass:
  
      value = Descriptor()
  
  

Context Managers (contextlib):

• Manual: __enter__, __exit__.

• @contextlib.contextmanager: Decorator for generator-based CM.

  
  @contextmanager
  
  def managed_file(name):
  
      f = open(name, 'w')
  
      try:
  
          yield f
  
      finally:
  
          f.close()
  
  

• contextlib.ExitStack: Manage multiple context managers dynamically.

Iterators & Generators:

• Generator: Function with yield. Lazy, stateful.

• Advanced Methods:

  • .send(value): Send value into generator (becomes result of yield expression).

  • .throw(type, value, traceback): Raise exception inside generator.

  • .close():** Raise GeneratorExit inside.

• yield from: Delegates to a subgenerator. Used for coroutines and complex generator chains.

  
  def subgen():
  
      yield 1
  
      yield 2
  
  def gen():
  
      yield from subgen()  # Delegates
  
      yield 3
  
  

Function Decorators:

• Building: Decorator is a callable that takes a function and returns a wrapper.

  
  def decorator(func):
  
      @functools.wraps(func)  # Preserves metadata
  
      def wrapper(*args, **kwargs):
  
          # pre
  
          result = func(*args, **kwargs)
  
          # post
  
          return result
  
      return wrapper
  
  

• Parameterized Decorators: Decorator factory (returns decorator).

  
  def repeat(n):
  
      def decorator(func):
  
          def wrapper(*args, **kwargs):
  
              for _ in range(n):
  
                  func(*args, **kwargs)
  
          return wrapper
  
      return decorator
  
  @repeat(3)
  
  def say_hello(): ...
  
  

functools Module:

• partial(func, *args, **kwargs): Freeze some arguments.

• lru_cache(maxsize=None): Memoization (cache recent calls). Boxed Result: \boxed{O(1) lookup for cached calls}.

• total_ordering: Fill in missing ordering methods (__lt__, __le__, etc.) from __eq__ and one other.

• singledispatch: Generic function (different implementations based on type of first argument).

collections Module:

• namedtuple('Point', ['x','y']): Immutable tuple with named fields.

• defaultdict(func): Dict with default value for missing keys (list, int).

• Counter(iterable): Dict subclass for counting hashable objects.

• OrderedDict: Maintains insertion order (less needed in Python 3.7+ as dict is ordered).

• deque: Double-ended queue. O(1) append/pop from both ends.

• ChainMap(*dicts): Search multiple dicts as one.

[!TIP] Exam Key: yield from for generator delegation. lru_cache for memoization. defaultdict/Counter for counting. @wraps is essential in decorators.

4.8 Interoperability and Extending Python

Calling C/C++ with ctypes:

• Load shared library: lib = ctypes.CDLL('./mylib.so') or ctypes.WinDLL.

• Define argument/return types: lib.myfunc.argtypes = (ctypes.c_int, ctypes.c_double), lib.myfunc.restype = ctypes.c_char_p.

• Call directly: result = lib.myfunc(10, 3.14).

• Pros: No recompilation of Python. Cons: No direct access to Python C API; manual type conversion.

Creating C Extensions (Python C API):

• Core Type: PyObject*. All Python objects are PyObject or subtypes.

• Reference Counting: Py_INCREF(obj), Py_DECREF(obj). Critical: Avoid leaks and crashes.

• Module Definition: PyModuleDef struct. Define functions with signature PyObject* func(PyObject* self, PyObject* args). Parse args with PyArg_ParseTuple(args, "i", &my_int).

• Building: setup.py with Extension('module', sources=['module.c']). Uses distutils/setuptools.

Cython:

• Write .pyx files (Python-like with C type declarations).

• Add static types: cdef int i for C integers, cdef double[:] arr for typed memoryviews.

• Compile to C extension: cythonize() in setup.py or pyproject.toml.

• Performance Gains: Often 2-100x for CPU-bound loops by avoiding Python interpreter overhead.

• Use Case: Speed up numerical kernels, wrap C libraries with Python-like syntax.

System Interaction (subprocess, os, sys):

• subprocess: Run external commands.

  • subprocess.run(['ls', '-l']): Simple, waits.

  • subprocess.Popen(): More control (async I/O, pipes). proc.communicate().

  • Security: Avoid shell=True with untrusted input.

• os: OS-level: os.getenv(), os.path.join(), os.listdir(), os.fork() (Unix).

• sys: Python interpreter: sys.argv, sys.path, sys.exit(), sys.modules.

[!TIP] Exam Key: ctypes for quick C calls, Cython for performance. Reference counting is the #1 pitfall in C extensions. Use subprocess.run() for simple commands, Popen for complex pipelines.