Databricks Certified Associate-Developer for Apache Spark 3.5 Online Practice Questions

Explanation:
A broadcast join is a type of join where the smaller DataFrame is replicated (broadcast) to all worker nodes in the cluster. This avoids shuffling the large DataFrame across the network.
Benefits:
Eliminates shuffle for the smaller dataset.
Greatly improves performance when one side of the join is small enough to fit in memory.
Correct usage example:
df_result = df2.join(broadcast(df1), "id")
This is a map-side join, where each executor joins its local partition of the large dataset with the broadcasted copy of the small one.
Why the other options are incorrect:
A: Broadcasting does not change partition sizes.
B: Joins always match on key equality; this is not specific to broadcast joins.
D: Broadcasting does not filter; it distributes data for faster joins.
Reference: Databricks Exam Guide (June 2025): Section “Developing Apache Spark DataFrame/DataSet API Applications” ― broadcast joins and partitioning strategies.
PySpark SQL Functions ― broadcast() method documentation.

Explanation:
The message "GC overhead limit exceeded" typically indicates that the JVM is spending too much time in garbage collection with little memory recovery. This suggests that the driver or executor is under-provisioned in memory.
The most effective remedy is to increase the driver memory using:
--driver-memory 4g
This is confirmed in Spark's official troubleshooting documentation:
“If you see a lot of GC overhead limit exceeded errors in the driver logs, it’s a sign that the driver is running out of memory.”
― Spark Tuning Guide Why others are incorrect:
A may help but does not directly address the driver memory shortage. B is not a valid action; GC cannot be disabled.
D increases memory usage, worsening the problem.

Explanation:
In Spark’s DataFrame API, the dropna() (or equivalently, DataFrameNaFunctions.drop()) method removes rows containing null values.
Behavior:
how="any" → drops rows where any column has a null value.
how="all" → drops rows where all columns are null.
Since the data scientist wants to drop records with any null field, the correct parameter is how="any".
Correct syntax:
filtered_users = raw_users.dropna(how="any")
This will remove all records that have at least one null value in any column.
Why the other options are incorrect:
A: Uses na.drop("any") but missing parentheses context (works only as raw_users.na.drop("any"), which is equivalent to option C).
B/D: how="all" only removes rows where all values are null ― too strict for this use case.
Reference: PySpark DataFrame API ― DataFrameNaFunctions.drop() and DataFrame.dropna().
Databricks Exam Guide (June 2025): Section “Developing Apache Spark DataFrame/DataSet API Applications” ― covers handling missing data and DataFrame cleaning operations.

Explanation:
The correct way to overwrite an existing file using the DataFrameWriter is:
df.write.mode("overwrite").json("path/to/file")
Option D is also technically valid, but Option A is the most concise and idiomatic PySpark syntax.
Reference: PySpark DataFrameWriter API

Explanation:
Spark Connect enables decoupling of the client and Spark driver processes, allowing remote access.
Spark supports configuring the remote Spark Connect server in multiple ways:
From Databricks and Spark documentation:
Option B (--remote "sc://localhost") is a valid command-line argument for the pyspark shell to connect using Spark Connect.
Option C (setting SPARK_REMOTE environment variable) is also a supported method to configure the remote endpoint.
Option A is incorrect because Spark Connect uses the sc:// protocol, not https://.
Option D requires modifying the code, which the question explicitly avoids.
Option E configures the port on the server side but doesn’t start a client connection.
Final Answers: B and C