# Spark (/docs/data/spark)



Apache Spark spreads data processing across many machines, so you can process
data that is too big or too slow for one computer. BullSequana AI runs Spark
for you. You don't install or operate anything: you ask for the computing power
you need, run your Python code, and the platform cleans up afterwards.

This section explains what you can do with Spark, what you need before you
start, and which guide to follow for your task.

## Ways to use Spark [#ways-to-use-spark]

You can use Spark in three ways. Pick the one that matches how you work.

| You want to                                                             | Use                                     | Guide                                                              |
| ----------------------------------------------------------------------- | --------------------------------------- | ------------------------------------------------------------------ |
| Keep a Spark cluster running that you connect to from a dev environment | **Compute** > **Clusters** in Studio    | [Create and manage clusters](/docs/data/spark/clusters)            |
| Run a Python file once, without writing any setup code                  | **Compute** > **Jobs** in Studio        | [Run a PySpark job](/docs/data/spark/jobs)                         |
| Explore data step by step                                               | Dev environments in Developer Workspace | [Use Spark in Dev environments](/docs/data/spark/dev-environments) |

<Cards>
  <Card title="Create and manage clusters" href="/docs/data/spark/clusters" description="Start a cluster in Studio and connect to it from a dev environment." />

  <Card title="Run a PySpark job" href="/docs/data/spark/jobs" description="Upload a Python file and run it once with its own resources." />

  <Card title="Use Spark in Dev environments" href="/docs/data/spark/dev-environments" description="Start a Spark session from a dev environment." />

  <Card title="Limits and troubleshooting" href="/docs/data/spark/limits-and-troubleshooting" description="Per-user limits and fixes for common errors." />
</Cards>

## Key terms [#key-terms]

Spark and the Studio pages use a few terms that appear throughout these
guides.

* **Cluster:** a set of machines, called nodes, that work together as one
  Spark system. Every cluster has one master node, which manages the cluster,
  and one or more worker nodes, which run the computation. A cluster keeps
  running until you delete it or the platform stops it after its idle
  timeout, so you can connect to it many times.
* **Master:** the node that manages a cluster. It keeps track of the workers
  and gives each Spark application a share of them. A cluster is ready to use
  when its master is ready.
* **Worker:** a node of a cluster that runs your code. You choose how many
  workers a cluster has, and how many CPU cores and how much memory each one
  gets.
* **Driver:** the process that runs your main program. It splits the work into
  small tasks, sends them to the executors, and collects the results. When you
  connect from a dev environment, the driver runs in your dev environment. A
  Compute job gets its own driver.
* **Executor:** a process that runs tasks for one Spark application and keeps
  its data in memory or on disk while the application runs. Executors run on
  the workers.
* **Spark session:** the entry point to Spark in your code. You use it to
  create DataFrames, run SQL, and read data. It lives in the driver, and
  stopping it ends your Spark application.
* **Job:** in Compute, one run of one PySpark file. The platform starts a
  driver and executors just for that file and removes them when it finishes.
  Spark's own logs also use "job" for each computation that an action such as
  `count()` or `save()` starts inside your program.
* **Idle timeout:** how long a cluster may stay idle before the platform stops
  it, 1 hour by default. A stopped cluster keeps its settings, and you can
  start it again. A cluster is idle when no work is running on it
  and no session is connected to it. Any activity starts the idle period
  again.
* **Profile:** an isolated workspace in Developer Workspace that holds your dev
  environments, pipelines, and Spark clusters and jobs. You can have a
  personal profile and be a member of profiles that others share with you.
  Studio uses your personal profile, or the first profile you can use.

## Before you start [#before-you-start]

You need a Spark role and a profile you can work in. Ask your tenant
administrator if something is missing.

* **A Spark role.** Spark Operator lets you create and manage your own clusters
  and jobs. Spark Admin also lets you see, edit, and delete the clusters and
  jobs of everyone in the tenant. New tenants give these roles to nobody,
  including tenant administrators, so an administrator must assign one. See
  [Spark permissions](/docs/reference/roles-permissions/spark).
* **Editor or administrator access to a profile** in your active tenant, such
  as your own profile. With viewer access you can see clusters, but you can't
  create them. See
  [Multi-tenancy and tiered access in Data](/docs/data/multi-tenancy-and-tiered-rbac#spark-workload-identity).
* **Data services set up for your tenant** by the platform operator. Without
  them, Compute shows that it isn't available.

## What your Spark work can access [#what-your-spark-work-can-access]

Spark work runs under your personal Spark identity, which decides which
tables your code can read and write.

| Where the work runs                                        | Identity                     | Table access               |
| ---------------------------------------------------------- | ---------------------------- | -------------------------- |
| A job you start under **Compute** > **Jobs**               | Your personal Spark identity | Your own table permissions |
| A session connected to your cluster from a dev environment | Your personal Spark identity | Your own table permissions |

## Spark version [#spark-version]

BullSequana AI runs Apache Spark 4.0.4 with Python 3.12. Every cluster, job,
and session uses this version; you can't choose another one.

## Use the API [#use-the-api]

Studio and your own code manage Spark through the BSQAI API, under `/spark`
on the API host. Every BullSequana AI deployment serves its own API
reference, with all Spark routes and their parameters, at
`https://api.<platform-domain>/docs`. See
[Spark permissions](/docs/reference/roles-permissions/spark) for the
permission each operation needs.

## Next steps [#next-steps]

* [Create your first cluster](/docs/data/spark/clusters)
* [Run a PySpark job](/docs/data/spark/jobs)
