Spark

Process large amounts of data with Apache Spark in BullSequana AI: Compute clusters and jobs in Studio, and Dev environments.

Agentic Friendly

Apache Spark spreads data processing across many machines, so you can process data that is too big or too slow for one computer. BullSequana AI runs Spark for you. You don't install or operate anything: you ask for the computing power you need, run your Python code, and the platform cleans up afterwards.

This section explains what you can do with Spark, what you need before you start, and which guide to follow for your task.

Ways to use Spark

You can use Spark in three ways. Pick the one that matches how you work.

You want toUseGuide
Keep a Spark cluster running that you connect to from a dev environmentCompute > Clusters in StudioCreate and manage clusters
Run a Python file once, without writing any setup codeCompute > Jobs in StudioRun a PySpark job
Explore data step by stepDev environments in Developer WorkspaceUse Spark in Dev environments

Key terms

Spark and the Studio pages use a few terms that appear throughout these guides.

  • Cluster: a set of machines, called nodes, that work together as one Spark system. Every cluster has one master node, which manages the cluster, and one or more worker nodes, which run the computation. A cluster keeps running until you delete it or the platform stops it after its idle timeout, so you can connect to it many times.
  • Master: the node that manages a cluster. It keeps track of the workers and gives each Spark application a share of them. A cluster is ready to use when its master is ready.
  • Worker: a node of a cluster that runs your code. You choose how many workers a cluster has, and how many CPU cores and how much memory each one gets.
  • Driver: the process that runs your main program. It splits the work into small tasks, sends them to the executors, and collects the results. When you connect from a dev environment, the driver runs in your dev environment. A Compute job gets its own driver.
  • Executor: a process that runs tasks for one Spark application and keeps its data in memory or on disk while the application runs. Executors run on the workers.
  • Spark session: the entry point to Spark in your code. You use it to create DataFrames, run SQL, and read data. It lives in the driver, and stopping it ends your Spark application.
  • Job: in Compute, one run of one PySpark file. The platform starts a driver and executors just for that file and removes them when it finishes. Spark's own logs also use "job" for each computation that an action such as count() or save() starts inside your program.
  • Idle timeout: how long a cluster may stay idle before the platform stops it, 1 hour by default. A stopped cluster keeps its settings, and you can start it again. A cluster is idle when no work is running on it and no session is connected to it. Any activity starts the idle period again.
  • Profile: an isolated workspace in Developer Workspace that holds your dev environments, pipelines, and Spark clusters and jobs. You can have a personal profile and be a member of profiles that others share with you. Studio uses your personal profile, or the first profile you can use.

Before you start

You need a Spark role and a profile you can work in. Ask your tenant administrator if something is missing.

  • A Spark role. Spark Operator lets you create and manage your own clusters and jobs. Spark Admin also lets you see, edit, and delete the clusters and jobs of everyone in the tenant. New tenants give these roles to nobody, including tenant administrators, so an administrator must assign one. See Spark permissions.
  • Editor or administrator access to a profile in your active tenant, such as your own profile. With viewer access you can see clusters, but you can't create them. See Multi-tenancy and tiered access in Data.
  • Data services set up for your tenant by the platform operator. Without them, Compute shows that it isn't available.

What your Spark work can access

Spark work runs under your personal Spark identity, which decides which tables your code can read and write.

Where the work runsIdentityTable access
A job you start under Compute > JobsYour personal Spark identityYour own table permissions
A session connected to your cluster from a dev environmentYour personal Spark identityYour own table permissions

Spark version

BullSequana AI runs Apache Spark 4.0.4 with Python 3.12. Every cluster, job, and session uses this version; you can't choose another one.

Use the API

Studio and your own code manage Spark through the BSQAI API, under /spark on the API host. Every BullSequana AI deployment serves its own API reference, with all Spark routes and their parameters, at https://api.<platform-domain>/docs. See Spark permissions for the permission each operation needs.

Next steps

On this page