Getting Started with Dagster Cloud using a Hybrid Deployment#

This guide is applicable to Dagster Cloud.

Welcome to Dagster Cloud!

In this guide, we'll give you everything you need to get up and running with Dagster Cloud using a Hybrid Deployment. With a Hybrid Deployment, your Dagster code is executed in your infrastructure. Refer to the Getting started with Serverless Deployment guide if you want to use Serverless.

We'll walk you through setting up your Dagster Cloud account, setting up an agent, loading your code, launching your first run, and configuring continuous integration.

Prerequisites#

To complete the steps in this guide, you'll need:

  • Access to some Dagster code. You can either use your own code or a project scaffold. This guide uses a scaffold.

  • To have dagster and dagster-cloud installed. To install, run:

    pip install dagster dagster-cloud

  • The ability to create and configure an agent in your infrastructure. Using a Hybrid Deployment requires setting up an agent for use with Dagster Cloud. The agent runs in your infrastructure, executes your code, and streams metadata about code execution over HTTPS back to Dagster Cloud.

    Dagster Cloud supports a handful of agents, each with its own prerequisites for setup. Refer to the Agent documentation for more info.

  • For all but the local agent, you will need access to a container registry to store Docker images containing your Dagster code.

Step 1: Create a Dagster Cloud account and organization#

If you don't already have a Dagster Cloud account, sign up for one before continuing.

You'll also be asked to create a Dagster Cloud organization. When your organization is created, a single deployment, named prod, will also be created. Deployments are standalone environments, allowing you to operate independent instances of Dagster with separately managed permissions.

Step 2: Select Hybrid as your deployment type#

  1. When prompted, click Select under Hybrid deployment.

  2. The modal that displays will contain a pre-generated agent token and info about the agents currently supported by Dagster Cloud.

  3. Follow the steps for setting up and deploying your agent:

Continue to the next step when finished.

Step 3: Deploy your code#

For most Hybrid deployments, with the exception of those using a local agent, you will need to create a Docker image containing your code. Then you will add a code location to Dagster Cloud that references the image. Typically, this build and deploy is done through CI/CD, but it is also possible to deploy a code location manually.

For continunous integration using GitHub actions:

  1. Begin by copying the GitHub action deploy workflow in this repository.
  2. Uncomment the step that is relevant to your Docker container registry.
  3. Set the necessary GitHub action secrets, including your Dagster Cloud API Token and Organization ID.

Step 4: Launch a run#

In this step, you'll run your first job in Dagster Cloud.

  1. Click the icon in the top-left corner of the page to expand the navigation.
  2. In the nav, open a repository and click a job.
  3. On the job's page, click the Launchpad tab.
  4. Click Launch Run in the bottom right corner to launch the run.

Step 5: Set up Continuous Integration#

Branch Deployments allow you to quickly and collaboratively develop your Dagster jobs. When a branch is created or updated in your Dagster repository, Branch Deployments will create a testing environment in the current Dagster Cloud deployment. For example, prod.

Check out the Branch Deployment docs for more info. If you skip this step now, you can always set it up later.

Next steps#

Congrats - your Dagster Cloud deployment is all set up! From here, you can: