Create and manage clusters

Create a Spark cluster in Studio, connect to it from a dev environment, change its size, and delete it.

Agentic Friendly

A Spark cluster is a set of nodes, one master and one or more workers, that stays running so you can connect to it from your dev environment. You create and manage clusters under Compute > Clusters in Studio. This guide shows how to create a cluster, connect to it, change its size, start it again, and delete it.

Before you start

You need the Spark Operator or Spark Admin role and editor access to a profile. See Before you start.

Studio uses your own profile automatically, or the first profile of the active tenant that you can use. There's no profile picker.

Open the clusters page

The clusters page lists every Spark cluster in your profile.

  1. Open Studio from the workspace switcher.
  2. In the sidebar, under Compute, select Clusters.

Each cluster appears as a card with its name, status, number of workers, whether its main node (Master) is ready, and who created it. While a cluster is starting or changing size, the page refreshes itself every few seconds. Select Refresh to reload it at any time.

StatusMeaning
ProvisioningThe cluster is starting.
RunningThe cluster is ready to use.
Scheduling failedThe platform couldn't find room for the workers. Try fewer or smaller workers.
FailedThe cluster stopped because of an error.
ReleasedThe cluster's nodes have been released.
StoppedThe cluster stayed idle for its idle timeout and was stopped. Start it to use it again.

Create a cluster

Create a cluster when you want Spark capacity that stays available while you work in a dev environment.

  1. On the Clusters page, select Create cluster.
  2. In Cluster name, enter a name. Use lowercase letters, numbers, and hyphens, start with a letter, and use at most 45 characters. For example, sales-analysis.
  3. Set the size of the cluster:
    • Worker count: how many workers to start, from 1 to 50. The default is 2.
    • Worker CPU: CPU cores per worker, from 1 to 64. The default is 1.
    • Worker memory: memory per worker, such as 512m or 4g. The default is 2g.
  4. In Idle timeout, set how long the cluster may stay idle before the platform stops it, for example 30m, 8h, or 7d. The default is 1h and the maximum is 365d.
  5. Select Create.

Create cluster dialog with Cluster name, Worker count, Worker CPU, Worker memory, and Idle timeout

The new cluster appears with the status Provisioning and changes to Running when it's ready.

A cluster is idle when no work is running on it and no session is connected to it. When it has been idle for the whole Idle timeout, the platform stops it and releases its nodes. The cluster stays in the list with its settings, and you can start it again. Any activity starts the idle period again, so a cluster that's in use keeps running.

A larger cluster finishes work faster but uses more of your personal limits.

Connect to a cluster from a dev environment

The cluster page gives you the code that connects a dev environment to the cluster.

  1. On the Clusters page, select the cluster.
  2. In the Attach from a notebook card, select Copy notebook code.
  3. Select Open Workspace to open Developer Workspace in a new tab, and open a dev environment.
  4. Paste the code into the dev environment and run it.

The code looks like this:

from bsqai.spark import get_spark_session

spark = get_spark_session(cluster="sales-analysis")

When the code finishes, spark is ready to use. Continue with Use Spark in Dev environments.

A running Compute cluster with Master ready and the Attach from a notebook card

The cluster runs as your personal Spark identity, so your sessions read and write tables with your own permissions. See What your Spark work can access.

The Spark cluster's own web interface isn't available through the Portal.

Change the size of a cluster

You can change a cluster's workers and idle timeout without recreating it.

  1. Open the cluster and select Edit cluster.
  2. Change Worker count, Worker CPU, Worker memory, or Idle timeout.
  3. Select Save.
  4. If you changed the workers, CPU, or memory, confirm with Continue editing.

Changing the workers, CPU, or memory restarts the cluster's workers, which can end jobs that are running on it. Changing only Idle timeout doesn't restart anything; the new value applies to the current idle period.

While the change is in progress, the cluster shows Pending resize or Resizing. When it finishes, a Resize complete or Resize failed message appears.

Start a stopped cluster

The platform stops a cluster that stayed idle for its Idle timeout. A stopped cluster shows Stopped, releases its nodes, and keeps its settings.

  1. On the Clusters page, select the stopped cluster.
  2. Select Start.

The cluster shows Provisioning and then Running when it's ready. Connect to it again from your dev environment.

Delete a cluster

Delete a cluster when you no longer need it, so it stops counting toward your limits.

  1. Open the cluster and select Delete.
  2. Confirm with Delete.

Deleting a cluster is permanent and ends every session connected to it.

Who can change a cluster

Only the person who created a cluster can edit or delete it. People with the Spark Admin role can edit and delete every cluster in the tenant.

If you're missing a permission, the button stays visible but greyed out. Point to it to see which permission you need.

Next steps

On this page