# Create and manage clusters (/docs/data/spark/clusters)







A Spark cluster is a set of nodes, one master and one or more workers, that
stays running so you can connect to it from your dev environment. You create
and manage clusters under
**Compute** > **Clusters** in Studio. This guide shows how to create a
cluster, connect to it, change its size, start it again, and delete it.

## Before you start [#before-you-start]

You need the Spark Operator or Spark Admin role and editor access to a
profile. See [Before you start](/docs/data/spark#before-you-start).

Studio uses your own profile automatically, or the first profile of the
active tenant that you can use. There's no profile picker.

## Open the clusters page [#open-the-clusters-page]

The clusters page lists every Spark cluster in your profile.

1. Open **Studio** from the workspace switcher.
2. In the sidebar, under **Compute**, select **Clusters**.

Each cluster appears as a card with its name, status, number of workers,
whether its main node (**Master**) is ready, and who created it. While a
cluster is starting or changing size, the page refreshes itself every few
seconds. Select **Refresh** to reload it at any time.

| Status                | Meaning                                                                                 |
| --------------------- | --------------------------------------------------------------------------------------- |
| **Provisioning**      | The cluster is starting.                                                                |
| **Running**           | The cluster is ready to use.                                                            |
| **Scheduling failed** | The platform couldn't find room for the workers. Try fewer or smaller workers.          |
| **Failed**            | The cluster stopped because of an error.                                                |
| **Released**          | The cluster's nodes have been released.                                                 |
| **Stopped**           | The cluster stayed idle for its idle timeout and was stopped. Start it to use it again. |

## Create a cluster [#create-a-cluster]

Create a cluster when you want Spark capacity that stays available while you
work in a dev environment.

1. On the **Clusters** page, select **Create cluster**.
2. In **Cluster name**, enter a name. Use lowercase letters, numbers, and
   hyphens, start with a letter, and use at most 45 characters. For example,
   `sales-analysis`.
3. Set the size of the cluster:
   * **Worker count:** how many workers to start, from 1 to 50. The default
     is 2.
   * **Worker CPU:** CPU cores per worker, from 1 to 64. The default is 1.
   * **Worker memory:** memory per worker, such as `512m` or `4g`. The
     default is `2g`.
4. In **Idle timeout**, set how long the cluster may stay idle before the
   platform stops it, for example `30m`, `8h`, or `7d`. The default is
   `1h` and the maximum is `365d`.
5. Select **Create**.

<img alt="Create cluster dialog with Cluster name, Worker count, Worker CPU, Worker memory, and Idle timeout" src="__img0" />

The new cluster appears with the status **Provisioning** and changes to
**Running** when it's ready.

A cluster is idle when no work is running on it and no session is connected
to it. When it has been idle for the whole **Idle timeout**, the platform
stops it and releases its nodes. The cluster stays in the list with its
settings, and you can start it again. Any activity starts the idle period
again, so a cluster that's in use keeps running.

A larger cluster finishes work faster but uses more of your
[personal limits](/docs/data/spark/limits-and-troubleshooting#per-user-limits).

## Connect to a cluster from a dev environment [#connect-to-a-cluster-from-a-dev-environment]

The cluster page gives you the code that connects a dev environment to the
cluster.

1. On the **Clusters** page, select the cluster.
2. In the **Attach from a notebook** card, select **Copy notebook code**.
3. Select **Open Workspace** to open Developer Workspace in a new tab, and
   open a dev environment.
4. Paste the code into the dev environment and run it.

The code looks like this:

```python
from bsqai.spark import get_spark_session

spark = get_spark_session(cluster="sales-analysis")
```

When the code finishes, `spark` is ready to use. Continue with
[Use Spark in Dev environments](/docs/data/spark/dev-environments#work-with-data).

<img alt="A running Compute cluster with Master ready and the Attach from a notebook card" src="__img1" />

The cluster runs as your personal Spark identity, so your sessions read and
write tables with your own permissions. See
[What your Spark work can access](/docs/data/spark#what-your-spark-work-can-access).

The Spark cluster's own web interface isn't available through the Portal.

## Change the size of a cluster [#change-the-size-of-a-cluster]

You can change a cluster's workers and idle timeout without recreating it.

1. Open the cluster and select **Edit cluster**.
2. Change **Worker count**, **Worker CPU**, **Worker memory**, or
   **Idle timeout**.
3. Select **Save**.
4. If you changed the workers, CPU, or memory, confirm with
   **Continue editing**.

Changing the workers, CPU, or memory restarts the cluster's workers, which
can end jobs that are running on it. Changing only **Idle timeout** doesn't
restart anything; the new value applies to the current idle period.

While the change is in progress, the cluster shows **Pending resize** or
**Resizing**. When it finishes, a **Resize complete** or **Resize failed**
message appears.

## Start a stopped cluster [#start-a-stopped-cluster]

The platform stops a cluster that stayed idle for its **Idle timeout**. A
stopped cluster shows **Stopped**, releases its nodes, and keeps its settings.

1. On the **Clusters** page, select the stopped cluster.
2. Select **Start**.

The cluster shows **Provisioning** and then **Running** when it's ready.
Connect to it again from your dev environment.

## Delete a cluster [#delete-a-cluster]

Delete a cluster when you no longer need it, so it stops counting toward your
limits.

1. Open the cluster and select **Delete**.
2. Confirm with **Delete**.

Deleting a cluster is permanent and ends every session connected to it.

## Who can change a cluster [#who-can-change-a-cluster]

Only the person who created a cluster can edit or delete it. People with the
Spark Admin role can edit and delete every cluster in the tenant.

If you're missing a permission, the button stays visible but greyed out.
Point to it to see which permission you need.

## Next steps [#next-steps]

* [Use Spark in Dev environments](/docs/data/spark/dev-environments)
* [Run a PySpark job](/docs/data/spark/jobs)
* [Limits and troubleshooting](/docs/data/spark/limits-and-troubleshooting)
