Spark
Process large amounts of data with Apache Spark in BullSequana AI: Compute clusters and jobs in Studio, and Dev environments.
Apache Spark spreads data processing across many machines, so you can process data that is too big or too slow for one computer. BullSequana AI runs Spark for you. You don't install or operate anything: you ask for the computing power you need, run your Python code, and the platform cleans up afterwards.
This section explains what you can do with Spark, what you need before you start, and which guide to follow for your task.
Ways to use Spark
You can use Spark in three ways. Pick the one that matches how you work.
| You want to | Use | Guide |
|---|---|---|
| Keep a Spark cluster running that you connect to from a dev environment | Compute > Clusters in Studio | Create and manage clusters |
| Run a Python file once, without writing any setup code | Compute > Jobs in Studio | Run a PySpark job |
| Explore data step by step | Dev environments in Developer Workspace | Use Spark in Dev environments |
Create and manage clusters
Start a cluster in Studio and connect to it from a dev environment.
Run a PySpark job
Upload a Python file and run it once with its own resources.
Use Spark in Dev environments
Start a Spark session from a dev environment.
Limits and troubleshooting
Per-user limits and fixes for common errors.
Key terms
Spark and the Studio pages use a few terms that appear throughout these guides.
- Cluster: a set of machines, called nodes, that work together as one Spark system. Every cluster has one master node, which manages the cluster, and one or more worker nodes, which run the computation. A cluster keeps running until you delete it or the platform stops it after its idle timeout, so you can connect to it many times.
- Master: the node that manages a cluster. It keeps track of the workers and gives each Spark application a share of them. A cluster is ready to use when its master is ready.
- Worker: a node of a cluster that runs your code. You choose how many workers a cluster has, and how many CPU cores and how much memory each one gets.
- Driver: the process that runs your main program. It splits the work into small tasks, sends them to the executors, and collects the results. When you connect from a dev environment, the driver runs in your dev environment. A Compute job gets its own driver.
- Executor: a process that runs tasks for one Spark application and keeps its data in memory or on disk while the application runs. Executors run on the workers.
- Spark session: the entry point to Spark in your code. You use it to create DataFrames, run SQL, and read data. It lives in the driver, and stopping it ends your Spark application.
- Job: in Compute, one run of one PySpark file. The platform starts a
driver and executors just for that file and removes them when it finishes.
Spark's own logs also use "job" for each computation that an action such as
count()orsave()starts inside your program. - Idle timeout: how long a cluster may stay idle before the platform stops it, 1 hour by default. A stopped cluster keeps its settings, and you can start it again. A cluster is idle when no work is running on it and no session is connected to it. Any activity starts the idle period again.
- Profile: an isolated workspace in Developer Workspace that holds your dev environments, pipelines, and Spark clusters and jobs. You can have a personal profile and be a member of profiles that others share with you. Studio uses your personal profile, or the first profile you can use.
Before you start
You need a Spark role and a profile you can work in. Ask your tenant administrator if something is missing.
- A Spark role. Spark Operator lets you create and manage your own clusters and jobs. Spark Admin also lets you see, edit, and delete the clusters and jobs of everyone in the tenant. New tenants give these roles to nobody, including tenant administrators, so an administrator must assign one. See Spark permissions.
- Editor or administrator access to a profile in your active tenant, such as your own profile. With viewer access you can see clusters, but you can't create them. See Multi-tenancy and tiered access in Data.
- Data services set up for your tenant by the platform operator. Without them, Compute shows that it isn't available.
What your Spark work can access
Spark work runs under your personal Spark identity, which decides which tables your code can read and write.
| Where the work runs | Identity | Table access |
|---|---|---|
| A job you start under Compute > Jobs | Your personal Spark identity | Your own table permissions |
| A session connected to your cluster from a dev environment | Your personal Spark identity | Your own table permissions |
Spark version
BullSequana AI runs Apache Spark 4.0.4 with Python 3.12. Every cluster, job, and session uses this version; you can't choose another one.
Use the API
Studio and your own code manage Spark through the BSQAI API, under /spark
on the API host. Every BullSequana AI deployment serves its own API
reference, with all Spark routes and their parameters, at
https://api.<platform-domain>/docs. See
Spark permissions for the
permission each operation needs.