Use Spark in Dev environments

Connect a dev environment to a running Spark cluster and work with DataFrames, SQL, and pandas UDFs.

Agentic Friendly

You can use Spark directly from a dev environment in Developer Workspace. You connect your dev environment to a Spark cluster that's already running and get a Spark session on it. This guide shows how to connect and work with data.

Before you start

You need the following:

  • The Spark Operator or Spark Admin role and editor access to a profile. See Before you start.
  • A running cluster that you created under Compute > Clusters. See Create and manage clusters.
  • A dev environment created from an image that supports Spark: jupyterlab:python3.12, the default, or, where your administrator offers it, Databricks Runtime jupyterlab:17.3-lts. See Choose a notebook image.

These images include the bsqai.spark helper, so you don't install anything.

Connect to a cluster

Pass the cluster's name to get_spark_session(). On the cluster's page in Studio, Copy notebook code gives you this code with the right name.

from bsqai.spark import get_spark_session

session = get_spark_session(cluster="sales-analysis")

The call returns when your session is ready. You can also set the BSQAI_SPARK_CLUSTER environment variable instead of passing cluster. You can connect only to clusters in a profile of the active tenant that you belong to.

Work with data

Use the session to create DataFrames and run SQL. Reuse the same session for related work instead of starting a new one each time.

orders = session.createDataFrame(
    [("north", 120.0), ("south", 80.0), ("north", 45.5)],
    ["region", "amount"],
)

orders.groupBy("region").sum("amount").show()

orders.createOrReplaceTempView("orders")
session.sql("SELECT region, COUNT(*) AS n FROM orders GROUP BY region").show()

Read and write tables

Your session can read and write the tables in the data catalog without any setup. Refer to a table by its full name, space.dataset.table, the same name that the data catalog shows:

payments = session.table("finance.invoices.payments_2026")
payments.groupBy("region").sum("amount").show()

summary = payments.groupBy("region").count()
summary.writeTo("finance.invoices.payments_by_region").createOrReplace()

The session uses your own table permissions. See What your Spark work can access.

Results come back to your dev environment

The Spark driver runs in your dev environment and uses its CPU and memory, not the cluster's. Methods that bring results back to the driver, such as collect() and toPandas(), load all the data into your dev environment's memory. Filter or aggregate the data on the cluster first, or use limit() before you bring it back. To work with larger results in Python, give the dev environment more memory.

Use pandas UDFs

With the jupyterlab:python3.12 image, sessions support pandas UDFs and mapInPandas:

import pandas as pd
from pyspark.sql.functions import pandas_udf

readings = session.createDataFrame(
    [(1, 50.0), (2, 86.0), (3, 104.0)],
    ["sensor", "fahrenheit"],
)

@pandas_udf("double")
def to_celsius(fahrenheit: pd.Series) -> pd.Series:
    return (fahrenheit - 32) * 5 / 9

readings.select("sensor", to_celsius("fahrenheit").alias("celsius")).show()

Code inside a UDF runs on the workers, not in your dev environment. Packages that you install in the dev environment with pip aren't available there. The workers have Python 3.12 with NumPy, pandas, and PyArrow.

When you're done

When you finish your work, you don't need to close your session or do anything with the cluster. The session ends when you shut down the dev environment, and the platform stops the cluster after its idle timeout. See Start a stopped cluster. Delete the cluster in Studio when you no longer need it.

Close a session

Close a session only when you need to end it while your dev environment keeps running:

session.stop()

You need this in two cases:

  • To connect to a different cluster. The dev environment keeps using its current session, even if you call get_spark_session() with another cluster's name. Close the session first, then connect again.
  • To free the cluster for another session. An open session holds all of the cluster's CPU cores, even when it's idle. Another session that connects to the same cluster, for example from your second dev environment, waits until you close yours.

Closing a session doesn't affect the cluster.

Session options

You pass options to get_spark_session() as keyword arguments. Alternatively, pass one SparkSessionConfig object, but not both. An unknown option raises a TypeError.

OptionDefaultDescription
clusterNoneThe cluster to connect to. Falls back to BSQAI_SPARK_CLUSTER.
app_name"notebook"The name shown for your Spark application.
ready_timeout_seconds300How long to wait for the session to start.

If the cluster has failed, the call fails right away instead of waiting.

Next steps

On this page