Use Spark in Dev environments
Connect a dev environment to a running Spark cluster and work with DataFrames, SQL, and pandas UDFs.
You can use Spark directly from a dev environment in Developer Workspace. You connect your dev environment to a Spark cluster that's already running and get a Spark session on it. This guide shows how to connect and work with data.
Before you start
You need the following:
- The Spark Operator or Spark Admin role and editor access to a profile. See Before you start.
- A running cluster that you created under Compute > Clusters. See Create and manage clusters.
- A dev environment created from an image that supports
Spark:
jupyterlab:python3.12, the default, or, where your administrator offers it, Databricks Runtimejupyterlab:17.3-lts. See Choose a notebook image.
These images include the bsqai.spark helper, so you don't install anything.
Connect to a cluster
Pass the cluster's name to get_spark_session(). On the cluster's page in
Studio, Copy notebook code gives you this code with the right name.
from bsqai.spark import get_spark_session
session = get_spark_session(cluster="sales-analysis")The call returns when your session is ready. You can also set the
BSQAI_SPARK_CLUSTER environment variable instead of passing cluster. You
can connect only to clusters in a profile of the active tenant that you
belong to.
Work with data
Use the session to create DataFrames and run SQL. Reuse the same session for related work instead of starting a new one each time.
orders = session.createDataFrame(
[("north", 120.0), ("south", 80.0), ("north", 45.5)],
["region", "amount"],
)
orders.groupBy("region").sum("amount").show()
orders.createOrReplaceTempView("orders")
session.sql("SELECT region, COUNT(*) AS n FROM orders GROUP BY region").show()Read and write tables
Your session can read and write the tables in the data catalog without any
setup. Refer to a table by its full name, space.dataset.table, the same
name that the data catalog shows:
payments = session.table("finance.invoices.payments_2026")
payments.groupBy("region").sum("amount").show()
summary = payments.groupBy("region").count()
summary.writeTo("finance.invoices.payments_by_region").createOrReplace()The session uses your own table permissions. See What your Spark work can access.
Results come back to your dev environment
The Spark driver runs in your dev environment and uses its CPU and memory,
not the cluster's. Methods that bring results back to the driver, such as
collect() and toPandas(), load all the data into your dev environment's
memory. Filter or aggregate the data on the cluster first, or use limit()
before you bring it back. To work with larger results in Python, give the
dev environment more memory.
Use pandas UDFs
With the jupyterlab:python3.12 image, sessions support pandas UDFs and
mapInPandas:
import pandas as pd
from pyspark.sql.functions import pandas_udf
readings = session.createDataFrame(
[(1, 50.0), (2, 86.0), (3, 104.0)],
["sensor", "fahrenheit"],
)
@pandas_udf("double")
def to_celsius(fahrenheit: pd.Series) -> pd.Series:
return (fahrenheit - 32) * 5 / 9
readings.select("sensor", to_celsius("fahrenheit").alias("celsius")).show()Code inside a UDF runs on the workers, not in your dev environment. Packages
that you install in the dev environment with pip aren't available there. The workers
have Python 3.12 with NumPy, pandas, and PyArrow.
When you're done
When you finish your work, you don't need to close your session or do anything with the cluster. The session ends when you shut down the dev environment, and the platform stops the cluster after its idle timeout. See Start a stopped cluster. Delete the cluster in Studio when you no longer need it.
Close a session
Close a session only when you need to end it while your dev environment keeps running:
session.stop()You need this in two cases:
- To connect to a different cluster. The dev environment keeps using its
current session, even if you call
get_spark_session()with another cluster's name. Close the session first, then connect again. - To free the cluster for another session. An open session holds all of the cluster's CPU cores, even when it's idle. Another session that connects to the same cluster, for example from your second dev environment, waits until you close yours.
Closing a session doesn't affect the cluster.
Session options
You pass options to get_spark_session() as keyword arguments. Alternatively,
pass one SparkSessionConfig object, but not both. An unknown option raises a
TypeError.
| Option | Default | Description |
|---|---|---|
cluster | None | The cluster to connect to. Falls back to BSQAI_SPARK_CLUSTER. |
app_name | "notebook" | The name shown for your Spark application. |
ready_timeout_seconds | 300 | How long to wait for the session to start. |
If the cluster has failed, the call fails right away instead of waiting.