# Run a PySpark job (/docs/data/spark/jobs)





A PySpark job runs one Python file once. The platform starts a driver and
executors for the job, runs your file, and removes them when it finishes. You
don't need a cluster or a dev environment. This guide shows how to run a job from
**Compute** > **Jobs** in Studio, read its output, and run it again.

## Before you start [#before-you-start]

You need the following:

* The Spark Operator or Spark Admin role and editor access to a profile. See
  [Before you start](/docs/data/spark#before-you-start).
* A PySpark file (`.py`) of at most 256 KB, on your computer or saved in your
  Platform files, for example by Chat.

A minimal PySpark file looks like this:

```python
from pyspark.sql import SparkSession

spark = SparkSession.builder.getOrCreate()
orders = spark.createDataFrame(
    [("north", 120.0), ("south", 80.0), ("north", 45.5)],
    ["region", "amount"],
)
orders.groupBy("region").sum("amount").show()
spark.stop()
```

## Run a job [#run-a-job]

Fill in the **Run a PySpark job** form to start a job.

1. Open **Studio** and, under **Compute**, select **Jobs**.

2. In **Job name**, enter a name, for example `orders-by-region`.

3. Under **PySpark file**, select **Add file**, then choose one of these
   options:
   * **From this computer:** select **Choose file** and pick a `.py` file.
   * **From Platform files:** choose a ready `.py` file from the list. This
     option needs permission to list and download files in your file
     library.

4. Optional: select **Show code** to check that you picked the right file.

5. Optional: under **Job resources**, change the computing power for the job.

   | Field             | What it controls                                                                 | Default |
   | ----------------- | -------------------------------------------------------------------------------- | ------- |
   | **Driver CPU**    | CPU cores of the driver, which runs your file and coordinates the work (1 to 16) | 1       |
   | **Driver memory** | Memory of the driver                                                             | `1g`    |
   | **Worker count**  | Number of executors that process the data (1 to 50)                              | 2       |
   | **Worker CPU**    | CPU cores per executor (1 to 64)                                                 | 2       |
   | **Worker memory** | Memory per executor                                                              | `2g`    |

6. Select **Run job**.

<img alt="Compute Jobs page with the Run a PySpark job form, its Job resources, and Job history" src="__img0" />

The job's output panel opens below the form and updates every few seconds
until the job ends.

To prepare a job without starting it, select **Save for later** instead of
**Run job**. The job appears in **Job history**, and you start it later with
**Run again**.

The file you upload is stored privately and can't be changed afterwards. The
job runs with your personal Spark identity, so it can reach only the data
that you can reach. See
[What your Spark work can access](/docs/data/spark#what-your-spark-work-can-access).

## Follow a job and read its output [#follow-a-job-and-read-its-output]

The output panel and **Job history** show how your job is doing.

| Status      | Meaning                                                                     |
| ----------- | --------------------------------------------------------------------------- |
| `PENDING`   | The platform is starting the job's driver and executors.                    |
| `RUNNING`   | Your file is running.                                                       |
| `SUCCEEDED` | Your file finished without errors.                                          |
| `FAILED`    | Your file stopped with an error, or the driver or executors couldn't start. |

The output shows everything your program printed, such as the result of
`show()` or `print()`. Spark's own technical messages are left out. If the job
fails, the error from your program appears in the output. Very long output is
cut off after 1 MB.

## Use job history [#use-job-history]

**Job history** lists your jobs with their status and creation
time. People with the Spark Admin role see the jobs of everyone in the
profile. Select **Refresh** to reload the list.

On each job, you can:

* Select **Show logs** to read the job's output.
* Select **Show code** to see the file the job ran.
* Select **Run again** to start the same file with the same resources as a
  new job. The new job gets a new name based on the original one.

Jobs stay in **Job history** after they finish, together with their code and
output.

<Callout type="info" title="Jobs can't be stopped from Studio">
  You can't stop or delete a job from Studio. To limit how long a job uses
  resources, keep the work in your file bounded.
</Callout>

## Next steps [#next-steps]

* [Limits and troubleshooting](/docs/data/spark/limits-and-troubleshooting)
* [Use Spark in Dev environments](/docs/data/spark/dev-environments)
