If you are interviewing for a data engineering role at a US company in 2025, Apache Spark is almost certainly going to come up. These are the actual questions that companies like Amazon, Microsoft, Uber, Lyft and Goldman Sachs ask — drawn from our experience supporting hundreds of data engineers through their interview preparation.
Core PySpark Concepts
1. What is the difference between a transformation and an action in Spark?
Transformations are lazy operations that define a new RDD or DataFrame — they do not trigger computation. Examples: filter(), map(), select(), groupBy(). Actions trigger actual computation and return results to the driver or write to storage. Examples: collect(), count(), show(), write(). This lazy evaluation is key to Spark's efficiency — it builds an execution plan first and optimises it before running.
2. Explain the difference between narrow and wide transformations.
Narrow transformations do not require data to move between partitions — each input partition maps to exactly one output partition. Examples: map(), filter(), union(). Wide transformations require a shuffle — data moves across the network between partitions. Examples: groupByKey(), reduceByKey(), join(). Wide transformations are expensive and are where most Spark performance problems originate.
3. What is a shuffle in Spark and why is it expensive?
A shuffle is the process of redistributing data across partitions — it involves serialising data, writing it to disk and transferring it over the network. It is expensive because it involves disk I/O, network I/O and deserialisation — the three slowest operations in distributed computing. Minimising shuffles is the most important performance optimisation in Spark.
Performance Tuning — The Most Common Interview Topic
4. How would you optimise a slow Spark job?
A structured approach works best in interviews. Start by identifying where the job is slow using the Spark UI — look at stage timelines, task durations and shuffle read/write sizes. Common fixes include: repartitioning data to avoid data skew, using broadcast joins for small DataFrames, caching intermediate results that are reused, replacing groupByKey with reduceByKey, and tuning executor memory and cores.
5. What is data skew and how do you handle it?
Data skew occurs when data is unevenly distributed across partitions — some tasks process much more data than others, creating bottlenecks. Solutions include: salting the skewed key (adding a random prefix to distribute the data), using adaptive query execution (AQE) in Spark 3.x which handles skew automatically, broadcasting the smaller side of a join, or repartitioning before the join using a more evenly distributed key.
6. When would you use cache() vs persist()?
cache() stores the DataFrame in memory using the default storage level (MEMORY_AND_DISK). persist() lets you specify the storage level explicitly — MEMORY_ONLY, DISK_ONLY, MEMORY_AND_DISK_SER, etc. Use cache() for frequently accessed DataFrames that fit in memory. Use persist() with DISK_ONLY for very large DataFrames. Always unpersist() when you no longer need the cached data.
Databricks-Specific Questions
7. What is Delta Lake and what problems does it solve?
Delta Lake is an open-source storage layer that adds ACID transactions to data lakes. It solves: inconsistent reads during writes (via optimistic concurrency control), inability to update or delete records efficiently, lack of schema enforcement, and no support for time travel or data versioning. It uses transaction logs (the Delta log) to track all changes, enabling features like time travel, rollback and schema evolution.
8. Explain the medallion architecture.
The medallion architecture is a data design pattern that organises data into three layers. The Bronze layer contains raw, unprocessed data exactly as it arrived — no transformations. The Silver layer contains cleaned, filtered and joined data — business logic is applied here. The Gold layer contains aggregated, business-ready data optimised for reporting and analytics. Each layer refines the previous one, improving data quality progressively.
Spark Streaming
9. What is the difference between Spark Streaming and Structured Streaming?
Spark Streaming (legacy) uses a micro-batch model based on DStreams — it processes data in fixed time intervals as small RDD batches. Structured Streaming (current standard) treats streaming data as an unbounded table and uses the DataFrame/Dataset API, making it far easier to use and maintain. Structured Streaming supports event-time processing, watermarking, exactly-once semantics and integrates with the Spark SQL engine for optimisation.
10. What is watermarking in Spark Structured Streaming?
Watermarking handles late-arriving data in event-time stream processing. You define a watermark threshold — for example, "accept data up to 10 minutes late." Spark maintains state for each time window until the watermark passes, then discards the state. This prevents memory from growing unboundedly while still handling reasonable delays in data arrival.
Need job support right now?
Book a free 30-minute demo session. We assign a dedicated developer within 24 hours.