# Use Spark in Dev environments (/docs/data/spark/dev-environments)



You can use Spark directly from a dev environment in Developer Workspace. You
connect your dev environment to a Spark cluster that's already running and get
a Spark session on it. This guide shows how to connect and work with data.

## Before you start [#before-you-start]

You need the following:

* The Spark Operator or Spark Admin role and editor access to a profile. See
  [Before you start](/docs/data/spark#before-you-start).
* A running cluster that you created under **Compute** >
  **Clusters**. See [Create and manage clusters](/docs/data/spark/clusters).
* A dev environment created from an image that supports
  Spark: `jupyterlab:python3.12`, the default, or, where your administrator
  offers it, Databricks Runtime `jupyterlab:17.3-lts`. See
  [Choose a notebook image](/docs/data/developer-workspace/dev-environment/choose-a-notebook-image).

These images include the `bsqai.spark` helper, so you don't install anything.

## Connect to a cluster [#connect-to-a-cluster]

Pass the cluster's name to `get_spark_session()`. On the cluster's page in
Studio, **Copy notebook code** gives you this code with the right name.

```python
from bsqai.spark import get_spark_session

session = get_spark_session(cluster="sales-analysis")
```

The call returns when your session is ready. You can also set the
`BSQAI_SPARK_CLUSTER` environment variable instead of passing `cluster`. You
can connect only to clusters in a profile of the active tenant that you
belong to.

## Work with data [#work-with-data]

Use the session to create DataFrames and run SQL. Reuse the same session for
related work instead of starting a new one each time.

```python
orders = session.createDataFrame(
    [("north", 120.0), ("south", 80.0), ("north", 45.5)],
    ["region", "amount"],
)

orders.groupBy("region").sum("amount").show()

orders.createOrReplaceTempView("orders")
session.sql("SELECT region, COUNT(*) AS n FROM orders GROUP BY region").show()
```

### Read and write tables [#read-and-write-tables]

Your session can read and write the tables in the data catalog without any
setup. Refer to a table by its full name, `space.dataset.table`, the same
name that the [data catalog](/docs/data/data-catalog) shows:

```python
payments = session.table("finance.invoices.payments_2026")
payments.groupBy("region").sum("amount").show()

summary = payments.groupBy("region").count()
summary.writeTo("finance.invoices.payments_by_region").createOrReplace()
```

The session uses your own table permissions. See
[What your Spark work can access](/docs/data/spark#what-your-spark-work-can-access).

<Callout type="warning" title="Results come back to your dev environment">
  The Spark driver runs in your dev environment and uses its CPU and memory,
  not the cluster's. Methods that bring results back to the driver, such as
  `collect()` and `toPandas()`, load all the data into your dev environment's
  memory. Filter or aggregate the data on the cluster first, or use `limit()`
  before you bring it back. To work with larger results in Python, give the
  dev environment more memory.
</Callout>

### Use pandas UDFs [#use-pandas-udfs]

With the `jupyterlab:python3.12` image, sessions support pandas UDFs and
`mapInPandas`:

```python
import pandas as pd
from pyspark.sql.functions import pandas_udf

readings = session.createDataFrame(
    [(1, 50.0), (2, 86.0), (3, 104.0)],
    ["sensor", "fahrenheit"],
)

@pandas_udf("double")
def to_celsius(fahrenheit: pd.Series) -> pd.Series:
    return (fahrenheit - 32) * 5 / 9

readings.select("sensor", to_celsius("fahrenheit").alias("celsius")).show()
```

Code inside a UDF runs on the workers, not in your dev environment. Packages
that you install in the dev environment with `pip` aren't available there. The workers
have Python 3.12 with NumPy, pandas, and PyArrow.

## When you're done [#when-youre-done]

When you finish your work, you don't need to close your session or do
anything with the cluster. The session ends when you shut down the dev
environment, and the platform stops the cluster after its idle timeout. See
[Start a stopped cluster](/docs/data/spark/clusters#start-a-stopped-cluster).
Delete the cluster in Studio when you no longer need it.

### Close a session [#close-a-session]

Close a session only when you need to end it while your dev environment keeps
running:

```python
session.stop()
```

You need this in two cases:

* **To connect to a different cluster.** The dev environment keeps using its
  current session, even if you call `get_spark_session()` with another
  cluster's name. Close the session first, then connect again.
* **To free the cluster for another session.** An open session holds all of
  the cluster's CPU cores, even when it's idle. Another session that connects
  to the same cluster, for example from your second dev environment, waits
  until you close yours.

Closing a session doesn't affect the cluster.

## Session options [#session-options]

You pass options to `get_spark_session()` as keyword arguments. Alternatively,
pass one `SparkSessionConfig` object, but not both. An unknown option raises a
`TypeError`.

| Option                  | Default      | Description                                                     |
| ----------------------- | ------------ | --------------------------------------------------------------- |
| `cluster`               | None         | The cluster to connect to. Falls back to `BSQAI_SPARK_CLUSTER`. |
| `app_name`              | `"notebook"` | The name shown for your Spark application.                      |
| `ready_timeout_seconds` | `300`        | How long to wait for the session to start.                      |

If the cluster has failed, the call fails right away instead of waiting.

## Next steps [#next-steps]

* [Limits and troubleshooting](/docs/data/spark/limits-and-troubleshooting)
