PySpark Interview Questions

🟢 Basic Level (1–20)

1. What is PySpark?
2. What is Apache Spark?
3. Why do we use PySpark?
4. Difference between Hadoop and Spark?
5. What is RDD in PySpark?
6. What is DataFrame in PySpark?
7. Difference between RDD and DataFrame?
8. What is SparkSession?
9. What is SparkContext?
10. What are transformations in PySpark?
11. What are actions in PySpark?
12. What is lazy evaluation?
13. What is immutable in Spark?
14. What is partition in Spark?
15. What is cluster computing?
16. What is driver program?
17. What is executor?
18. What is DAG in Spark?
19. What is PySpark API?
20. What languages does Spark support?

---

⚙️ Core Concepts (21–40)

21. What is narrow transformation?
22. What is wide transformation?
23. Difference between narrow and wide transformations?
24. What is shuffle in Spark?
25. What is caching in PySpark?
26. What is persistence?
27. What is broadcast variable?
28. What is accumulator?
29. What is map() function?
30. What is flatMap()?
31. What is filter() function?
32. What is reduceByKey()?
33. What is groupByKey()?
34. Difference between reduceByKey and groupByKey?
35. What is join operation?
36. Types of joins in PySpark?
37. What is select() function?
38. What is withColumn()?
39. What is drop() function?
40. What is schema in DataFrame?

---

📊 DataFrame & SQL (41–60)

41. What is Spark SQL?
42. What is temp view?
43. What is global temp view?
44. What is show() function?
45. What is collect() function?
46. What is count() function?
47. What is distinct()?
48. What is groupBy()?
49. What is aggregation?
50. What is orderBy()?
51. What is sort()?
52. What is alias() in PySpark?
53. What is SQL vs DataFrame API?
54. How to read CSV in PySpark?
55. How to read JSON file?
56. How to write DataFrame to file?
57. What is inferSchema?
58. What is null handling?
59. What is dropna()?
60. What is fillna()?

---

⚡ Advanced Level (61–80)

61. What is Spark architecture?
62. How does Spark execute job?
63. What is task in Spark?
64. What is stage in Spark?
65. What is job in Spark?
66. What is partitioning strategy?
67. What is bucketing?
68. What is checkpointing?
69. What is lineage graph?
70. What is fault tolerance?
71. How Spark handles memory?
72. What is Tungsten engine?
73. What is Catalyst optimizer?
74. What is data serialization?
75. What is Parquet file format?
76. Why Parquet is faster?
77. What is ORC format?
78. What is Avro format?
79. Difference between Parquet and CSV?
80. What is Spark streaming?

---

🚀 Scenario-Based (81–100)

81. How do you handle big data in PySpark?
82. How to optimize PySpark performance?
83. How to reduce shuffle in Spark?
84. How to handle skewed data?
85. How to improve join performance?
86. How to debug Spark jobs?
87. How to handle missing data?
88. How to process real-time data?
89. How to scale Spark application?
90. How to tune Spark memory?
91. How to handle large joins?
92. How to use caching effectively?
93. How to use broadcast joins?
94. How to write efficient Spark code?
95. How to handle failures in Spark?
96. How to monitor Spark jobs?
97. How to optimize DataFrame operations?
98. How to design ETL pipeline in Spark?
99. Why companies use PySpark?
100. Difference between batch and streaming processing?

PySpark Interview Answers (1–100)

🟢 Basic (1–20)

1. Python API for Apache Spark
2. Big data processing engine
3. For large-scale data processing
4. Hadoop is disk-based, Spark is in-memory
5. Resilient Distributed Dataset
6. Distributed table-like data structure
7. RDD = low-level, DataFrame = optimized
8. Entry point for Spark applications
9. Low-level Spark API (older)
10. Operations that create new RDD/DataFrame
11. Operations that trigger execution
12. Delayed execution until action is called
13. Data cannot be changed once created
14. Data split across cluster nodes
15. Computing across multiple machines
16. Program that runs Spark job
17. Worker node executing tasks
18. Execution plan of Spark job
19. Python interface for Spark
20. Python, Java, Scala, R

---

⚙️ Core (21–40)

21. Transformation with no shuffle
22. Requires data movement across cluster
23. Narrow = fast, Wide = slow due to shuffle
24. Data movement between nodes
25. Storing data in memory/disk
26. Storing intermediate data
27. Shared read-only variable
28. Global counter variable
29. Applies function to each element
30. Flattens nested structure
31. Filters data based on condition
32. Aggregation by key (faster)
33. Groups all values by key
34. reduceByKey is optimized
35. Combines two datasets
36. Inner, outer, left, right
37. Select columns
38. Add or modify column
39. Remove column
40. Structure of DataFrame

---

📊 DataFrame & SQL (41–60)

41. SQL queries on Spark data
42. Temporary DataFrame view
43. Global accessible view
44. Displays DataFrame
45. Returns all data (not recommended for big data)
46. Counts rows
47. Removes duplicates
48. Groups data
49. Mathematical operations on data
50. Sort in ascending/descending
51. Sorting function
52. Rename column
53. SQL is query-based, DataFrame is API-based
54. read.csv() function
55. read.json() function
56. write() method
57. Automatically detects schema
58. Handling missing values
59. Removes null rows
60. Fills null values

---

⚡ Advanced (61–80)

61. Driver + Executors architecture
62. Job → Stage → Task execution
63. Small unit of execution
64. Group of tasks
65. Entire computation request
66. Splitting data across nodes
67. Pre-partitioning data
68. Saves intermediate results
69. Tracks execution history
70. System recovery ability
71. Uses memory + disk storage
72. Optimizes memory usage
73. Query optimization engine
74. Conversion of data formats
75. Columnar storage format
76. Faster reading and compression
77. Columnar format like Parquet
78. Row-based format
79. Parquet is faster and compressed
80. Real-time data processing

---

🚀 Scenario (81–100)

81. Distributed processing with Spark cluster
82. Caching, partitioning, avoiding shuffle
83. Use map-side aggregation
84. Use salting or repartition
85. Broadcast join or indexing
86. Using logs and Spark UI
87. dropna() or fillna()
88. Spark Streaming or Structured Streaming
89. Add more executors/nodes
90. Tune executor memory settings
91. Use broadcast joins
92. Cache frequently used DataFrames
93. Broadcast smaller dataset
94. Avoid unnecessary transformations
95. Retry mechanisms + checkpointing
96. Spark UI monitoring
97. Use select and filter early
98. Extract → Transform → Load pipeline
99. Fast, scalable, distributed processing
100. Batch = static data, Streaming = real-time data