Limits and troubleshooting
Per-user Spark limits and how to fix common problems with Compute clusters, jobs, and Spark sessions.
The platform limits how much Spark capacity each person can use at the same time, so one person's work doesn't block everyone else's. This page lists those limits and explains how to fix the problems people run into most often.
Per-user limits
These limits apply to each person in each profile. Your administrator can change them.
| Limit | Default |
|---|---|
| Spark clusters | 2 clusters |
| CPU cores across the workers of those clusters | 20 cores |
| Jobs | 5 jobs |
| CPU cores across the drivers and workers of those jobs | 40 cores |
| Workers per cluster or job | 1 to 50 workers |
| CPU cores per worker | 1 to 64 cores |
| CPU cores of a job's driver | 1 to 16 cores |
| Size of a PySpark file for a job | 256 KB |
For example, two clusters with 4 workers of 2 cores each use 16 of your 20 worker cores, so a third cluster isn't possible until you delete one.
Missing access in Studio
When you're missing a permission, Studio tells you which one. A page you
can't open shows the missing permissions by name, such as List Compute
clusters (spark.cluster.list). Ask your tenant administrator to grant
them, then select Check access again. A button you can't use is greyed
out; point to it to see the permission it needs.
| Message | What to do |
|---|---|
| No Compute namespaces available | You have no profile in the active tenant that Compute can use. Check that the right tenant is selected, and ask your workspace administrator for editor access to a profile. |
| Compute namespace access denied | You're missing a Spark role. Ask your tenant administrator for Spark Operator or Spark Admin. |
| Namespace discovery is unavailable | Spark isn't available right now. Try again later. If it persists, contact your platform operator. |
| Compute says it isn't set up for your organization | If you got editor access to a profile recently, wait a few minutes and try again. Otherwise, ask your platform operator to set up Data services for your tenant. |
Clusters and jobs
These problems relate to creating clusters and running jobs in Studio.
| Symptom | What to do |
|---|---|
| Job submission failed with This Compute operation would exceed the configured quota. | You've reached a per-user limit. Delete clusters you no longer need, wait for jobs to finish, or ask for fewer workers or cores. |
| A cluster shows Scheduling failed. | There isn't enough room for the workers you asked for. Delete the cluster and create a smaller one, or try again later. |
| A cluster shows Stopped. | It stayed idle for its whole Idle timeout. Open it and select Start. To keep it running longer, set a longer timeout in Edit cluster. |
| Choose a non-empty Python (.py) file. | Only non-empty .py files can be uploaded. |
| The PySpark source exceeds the allowed size. | The file is larger than 256 KB. Make it smaller, for example by moving data out of the code. |
| File-library permission is required to choose a Platform file. | Ask your administrator for permission to list and download files, or upload the file from your computer instead. |
| A job's output is empty or ends early. | Make sure your file prints its results, for example with show() or print(). Output longer than 1 MB is cut off. |
Sessions in Dev environments
These problems relate to starting Spark from a dev environment.
| Symptom | What to do |
|---|---|
| A connection warning appears while the session starts. | A brief warning can appear while a session starts. If the session starts and your code runs, ignore it. Persistent connection errors, a startup timeout, or All masters are unresponsive mean the session failed. |
pyspark is not installed in this environment, or No module named 'bsqai_spark'. | The dev environment's image doesn't support Spark. Use jupyterlab:python3.12, or Databricks Runtime jupyterlab:17.3-lts where your administrator offers it. |
No Spark cluster selected. | You didn't name a cluster. Pass cluster="<name>" or set BSQAI_SPARK_CLUSTER. |
A PermissionError says that you need the Spark Operator or Spark Admin role. | Ask your tenant administrator for one of these roles. |
Per-user workloads quota reached or Per-user job cores quota reached. | Wait for jobs to finish, or ask for fewer driver or worker cores. |
| A UDF fails on the workers with a missing module. | The package is installed only in your dev environment. Use packages that the workers include: NumPy, pandas, and PyArrow. |
| A table isn't found, or access to it is denied. | Check the table's full name, space.dataset.table, in the data catalog. If you don't have access, ask the table's owner. |
| The session isn't ready within the startup timeout. | Check in Studio that the cluster shows Running and its Master is ready. To wait longer, increase ready_timeout_seconds. |
If none of these helps, contact your platform administrator with the cluster or job name and the time the problem happened.