Run AWS Glue jobs locally

Glue versions you can run locally

Pick the Glue version per job or per notebook, as in the AWS console. Every run happens in Docker on a real Glue runtime:

New jobs default to 5.1, which is what AWS uses for jobs created without a Glue version. The published images are pulled the first time you use a version, and they are large (5 to 7 GB each), so the first run takes a while. AWS publishes no Glue 6.0 image, so the first 6.0 job or notebook builds one from the official Glue 6.0 pieces. It downloads about 1 GB and takes roughly 3 minutes on a fast connection, and it only happens once.

Glue 6.0 behaves like Glue 6.0 on AWS: ANSI SQL mode is on, Python is 3.13, --extra-jars must be built for Scala 2.13, and getResolvedOptions rejects abbreviated argument names.

Start Goku and turn on Glue

Install the goku CLI (see Install) and start Goku. The CLI mounts the Docker socket and the /tmp/goku-glue work folder that Glue runs need. Glue and Athena are not part of the core services, so subscribe to them once:

$ goku start
goku services subscribe glue athena
eval "$(goku env)"

You can also subscribe on the console's Services page, where the Data & analytics preset turns on Glue, Athena and related services together.

Run a Glue job from the AWS CLI

Jobs in Goku are real Glue resources, so the AWS CLI, boto3, Terraform and Step Functions all see the same jobs and runs. Start with a small CSV file in local S3:

$ aws s3 mb s3://my-bucket
printf 'customer,amount\nalice,20\nbob,5\nalice,7\n' > orders.csv
aws s3 cp orders.csv s3://my-bucket/raw/orders/orders.csv

Then a Glue script that totals the amounts per customer and writes Parquet back to S3:

import sys
from pyspark.context import SparkContext
from awsglue.context import GlueContext
from awsglue.job import Job
from awsglue.utils import getResolvedOptions

args = getResolvedOptions(sys.argv, ["JOB_NAME"])
glue_context = GlueContext(SparkContext.getOrCreate())
spark = glue_context.spark_session
job = Job(glue_context)
job.init(args["JOB_NAME"], args)

orders = spark.read.option("header", True).option("inferSchema", True).csv("s3://my-bucket/raw/orders/")
totals = orders.groupBy("customer").sum("amount").withColumnRenamed("sum(amount)", "total")
totals.show()
totals.write.mode("overwrite").parquet("s3://my-bucket/curated/totals/")

job.commit()

Upload the script, create the job and run it. The run log goes to CloudWatch Logs, as on AWS:

$ aws s3 cp etl.py s3://my-bucket/scripts/etl.py
aws glue create-job --name etl --role GlueRole --glue-version 5.1 \
  --command Name=glueetl,ScriptLocation=s3://my-bucket/scripts/etl.py \
  --worker-type G.1X --number-of-workers 2
RUN=$(aws glue start-job-run --job-name etl --query JobRunId --output text)
aws glue get-job-run --job-name etl --run-id $RUN --query JobRun.JobRunState
aws logs get-log-events --log-group-name /aws-glue/jobs/output --log-stream-name $RUN
aws s3 ls s3://my-bucket/curated/totals/

Run get-job-run again until the state is SUCCEEDED. The run goes through STARTING (where the image pull or the 6.0 build shows), then RUNNING. Errors land in /aws-glue/jobs/error. To try Glue 6.0, create the job with --glue-version 6.0.

Or build it in the console, with notebooks

Open Glue in the console at http://localhost:8080. Under ETL jobs, Create job gives you a script editor, the Glue version picker, job parameters and run history with live logs. Under Notebooks, a new notebook starts an interactive session with a live, stateful kernel where spark and sc are already defined, which is the fastest way to explore data before writing a job.

The Data Catalog, crawlers and Athena

Goku emulates the Glue Data Catalog: databases, tables and partitions. Create them with the AWS CLI, or point a crawler at a prefix in local S3. Crawlers classify CSV, JSON, Parquet, Avro and ORC files and turn key=value folders into partitions.

$ aws glue create-database --database-input '{"Name": "analytics"}'
aws glue create-table \
  --database-name analytics \
  --table-input '{
    "Name": "totals",
    "StorageDescriptor": {
      "Location": "s3://my-bucket/curated/totals/",
      "InputFormat": "org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat",
      "SerdeInfo": {
        "SerializationLibrary": "org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe"
      },
      "Columns": [
        {"Name": "customer", "Type": "string"},
        {"Name": "total",    "Type": "bigint"}
      ]
    }
  }'

With the job parameter --enable-glue-datacatalog (the console sets it for new Spark jobs, and notebooks use it by default), spark.sql("SELECT ... FROM analytics.totals"), CREATE TABLE, INSERT INTO and glue_context.create_dynamic_frame.from_catalog read and write the local catalog. Athena uses the same catalog, so you can query a job's output with SQL in the console's query editor.

Libraries, data lake formats and orchestration

• Scripts and libraries from S3. ScriptLocation, --extra-py-files (modules and .whl files), --extra-jars and --extra-files are fetched at run time, and --additional-python-modules is installed with pip.
• Hudi, Delta Lake and Iceberg. Set --datalake-formats to hudi, delta or iceberg and Goku adds the framework jars and Spark settings for your Glue version.
• Python shell jobs run Python 3.9 with AWS's analytics library set.
• Orchestration. Triggers (on-demand, scheduled, conditional and event), workflows, bookmarks, retries and concurrency limits work, and Step Functions can run a job with arn:aws:states:::glue:startJobRun.sync.

Differences from AWS Glue

Goku is a local sandbox, so a few things behave differently:

• Ray jobs (glueray) are accepted, but their runs fail because Ray isn't available locally.
• Job bookmarks advance after each successful run, but the Glue library doesn't skip already-processed input locally.
• On Glue 6.0, Spark Connect and Spark Declarative Pipelines are not wired up, and the Snowflake JDBC driver is not bundled.
• JDBC, DynamoDB and MongoDB crawler targets are not crawled; S3 targets are.

More Goku guides