Efficient Data Processing on Kubernetes with PySpark, Pandas & Polars Training Course

Module 1: Big Data & Spark Foundations

• Overview of the Big Data ecosystem and Spark’s role in contemporary data platforms
• Deep dive into Spark architecture: drivers, executors, cluster managers, lazy evaluation, DAGs, and execution planning
• Distinguishing between RDD and DataFrame APIs and determining the appropriate use case for each
• Initialization and configuration of SparkSession, along with the fundamentals of application-level settings

Module 2: PySpark DataFrames

• Reading and writing data from enterprise sources and formats, including CSV, JSON, Parquet, and Delta
• Mastering PySpark DataFrames: transformations, actions, column expressions, filtering, joins, and aggregations
• Implementing advanced operations such as window functions, timestamp handling, and processing nested data
• Applying data quality checks and developing reusable, maintainable PySpark code

Module 3: Kubernetes & Docker Fundamentals for Spark Users

• Conceptual understanding of how Docker images and Kubernetes pods interact with Spark driver and executor processes, tailored for cluster consumers rather than administrators
• Exploring the mechanics of Spark jobs on Kubernetes: pod scheduling, resource requests/limits, and the mapping of session settings
• Distinguishing between controls available via Spark session parameters and those managed by the platform team through cluster provisioning, and why this boundary is critical for tuning
• Hands-on exercise: Examine a live Spark-on-Kubernetes job in a sandbox cluster to identify driver and executor pods

Module 4: Spark Session Configuration on Kubernetes — Deep Dive

• Analyzing executor count versus executor size: navigating memory-versus-cores trade-offs and logical reasoning for optimization
• Managing driver and executor memory allocation, overhead, and their translation into pod resources
• Dynamic allocation: understanding its behavior on Kubernetes, identifying when it conserves resources and when it does not
• Hands-on exercise: Execute identical jobs under varying executor and core configurations to compare runtime and resource consumption

Module 5: Scaling Behaviour & Cost Optimisation

• Understanding how node scaling (adding or removing) affects job behavior, completion time, and resource usage
• Comparing the performance and cost implications of using many small executors versus fewer large ones
• Shuffle behavior and partition tuning, including estimating the financial impact of specific configuration choices
• Hands-on exercise: Scale the sandbox cluster from five to ten nodes to observe effects on job completion time and resource consumption

Module 6: Efficient Data Ingestion & Partitioning

• The small-files problem: analyzing why sources split into numerous 1–5 MB Parquet files degrade performance and distort partitioning
• Strategies for repartitioning and coalescing data
• Controlling partition sizes during both read and write operations
• Writing Parquet efficiently to prevent the recurrence of small-file issues downstream
• Hands-on exercise: Load a dataset consisting of many small Parquet partitions, apply various repartitioning strategies, and benchmark performance improvements

Module 7: Pandas Memory Management & Failure Diagnosis

• Identifying root causes of out-of-memory errors in Pandas and recognizing key symptoms
• Implementing memory-efficient conversion patterns between Spark and Pandas
• Preventing memory spikes when writing large datasets to CSV
• Leveraging chunked processing and data type (dtype) optimization for resource-constrained environments
• Hands-on exercise: Replicate a typical Pandas out-of-memory scenario and resolve it using chunking and dtype optimization

Module 8: Polars as a Complementary Tool

• Positioning Polars relative to Pandas: analyzing performance characteristics, lazy evaluation, and memory behavior
• Understanding where Polars integrates with PySpark and Pandas in modern cloud data stacks and migration roadmaps, including AWS environments
• Hands-on exercise: Rewrite a Pandas-centric transformation using Polars to compare memory usage and processing speed

Module 9: Applying Optimisation to ETL & ML Workloads

• Applying configuration, partitioning, and memory management principles across a realistic ETL pipeline
• Understanding optimization considerations specific to machine learning workloads running on the same cluster
• Implementing a practical tuning workflow to systematically diagnose cost and performance issues
• Hands-on exercise: Complete an end-to-end mini-project combining data loading, transformation, and a simple model training step, with participants independently tuning the Spark configuration