Dagger: The Modern CI/CD Engine Running in Containers

1. Why Traditional CI/CD Is Broken

If you've worked with CI/CD systems like GitHub Actions, GitLab CI, Jenkins, or CircleCI, you've experienced the pain. YAML-based pipelines that start simple but grow into unmaintainable nightmares. Configuration that works in one environment but fails in another. Debugging that requires pushing commits and waiting for slow remote runners. Secrets scattered across multiple platforms. And the vendor lock-in that makes migration feel impossible.

The fundamental problem is that traditional CI/CD systems treat pipelines as configuration rather than code. This leads to several critical issues:

The YAML Problem

YAML was never designed for programming. It lacks:

Type safety: Everything is a string until runtime
Composability: No functions, modules, or imports
Testability: Can't unit test YAML logic
IDE support: Limited autocomplete, no refactoring
Debugging: No breakpoints, no step-through

When your pipeline logic gets complex—and it always does—you're fighting the format instead of solving problems.

The Environment Problem

"Works on my machine" becomes "works on GitHub's machine." Each CI platform has different:

Base images and pre-installed software
Network configurations and egress rules
File system layouts and permissions
Environment variables and secrets handling

This makes local reproduction nearly impossible. You commit, push, wait, fail, guess, repeat.

The Vendor Lock-in Problem

Your pipeline logic is trapped in platform-specific YAML. Migrating from GitHub Actions to GitLab CI isn't a search-and-replace—it's a complete rewrite. The hundreds of hours invested in your CI configuration become sunk cost that keeps you tied to platforms you may no longer want.

The Hidden Cost

Organizations often spend 10-20% of engineering time dealing with CI/CD issues—debugging failures, optimizing slow pipelines, managing secrets, and working around platform limitations. This is time not spent building features.

2. What Is Dagger?

Dagger is a programmable CI/CD engine that runs your pipelines in containers. Instead of YAML, you write pipelines in real programming languages—Go, Python, or TypeScript. Instead of platform-specific runners, you get portable, reproducible execution that works the same everywhere: locally, in CI, on your laptop, or in production.

Dagger was created by Solomon Hykes, the founder of Docker, along with the original Docker team. They took the lessons learned from containerizing applications and applied them to CI/CD. The result is a system where pipelines are code, execution is containerized, and portability is native.

Key Principles

Pipelines as Code: Write your CI/CD logic in Go, Python, or TypeScript. Use functions, types, imports, tests, and all the tooling of a real programming language.

Container-Native Execution: Every step runs in a container. This guarantees reproducibility—if it works once, it works everywhere.

Local-First Development: Run and debug pipelines on your laptop with the same environment they'll have in CI. No more "commit and pray."

Universal Portability: The same Dagger pipeline runs on GitHub Actions, GitLab CI, Jenkins, CircleCI, or your local machine without modification.

How It Works

You write a Dagger module—a package of functions that define your pipeline. Each function can:

Create and configure containers
Execute commands in those containers
Mount directories and files
Expose network services
Chain operations into complex workflows

Dagger compiles your code, builds a dependency graph, and executes it using BuildKit—the same engine that powers Docker builds. This gives you sophisticated caching, parallel execution, and efficient resource usage.

3. Dagger Architecture Deep Dive

The Three Layers

Dagger's architecture consists of three layers:

1. The Dagger Engine is a containerized BuildKit daemon that executes your pipelines. It handles container creation, caching, networking, and file operations. The engine runs as a container itself, which means it can run anywhere Docker runs.

2. The Dagger API is a GraphQL interface exposed by the engine. Your code doesn't call Docker directly—it builds a GraphQL query that describes what you want to do, and the engine executes it.

3. The Dagger SDK provides idiomatic APIs in Go, Python, and TypeScript. The SDK translates your code into GraphQL queries and handles the communication with the engine.

┌─────────────────────────────────────────────────────────────┐
│                      Your Code (Go/Python/TS)                │
│  func (m *MyModule) Build(ctx context.Context) error {       │
│      return dag.Container().                                 │
│          From("golang:1.21").                               │
│          WithMountedDirectory("/src", m.Source).             │
│          WithWorkdir("/src").                                │
│          WithExec([]string{"go", "build", "-o", "app"}).     │
│          File("/src/app").                                   │
│          Export(ctx, "./app")                                │
│  }                                                           │
└─────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────┐
│                    Dagger SDK (Generated)                   │
│  Translates method calls to GraphQL queries                 │
└─────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────┐
│                    Dagger Engine (GraphQL)                  │
│  Receives: { container: { from: "golang:1.21", ... } }      │
│  Returns: { file: { export: { path: "./app" } } }           │
└─────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────┐
│                      BuildKit Executor                      │
│  - Creates container with golang:1.21 image                 │
│  - Mounts source directory                                  │
│  - Executes "go build"                                      │
│  - Exports resulting binary                                 │
│  - Caches layers for future runs                            │
└─────────────────────────────────────────────────────────────┘

The DAG (Directed Acyclic Graph)

Every Dagger pipeline is a DAG. When you write:

// This creates a DAG node for the base image
base := dag.Container().From("node:20")

// This creates a dependent node for dependencies
deps := base.WithExec([]string{"npm", "ci"})

// These two nodes depend on deps but not on each other
// They execute in parallel
test := deps.WithExec([]string{"npm", "test"})
build := deps.WithExec([]string{"npm", "run", "build"})

// This depends on both test and build
return dag.Container().From("nginx:alpine").
    WithDirectory("/usr/share/nginx/html", build.Directory("dist")).
    WithFile("/etc/nginx/nginx.conf", test.File("nginx.conf"))

Dagger analyzes these dependencies and executes independent branches in parallel. BuildKit's sophisticated caching means unchanged steps are skipped entirely.

Caching Strategy

Dagger's caching is automatic and intelligent:

Layer caching: Container layers are cached by content hash
Directory caching: Mounted directories are snapshotted and cached
Remote caching: Share cache across machines with registry caching
Cache volumes: Persistent volumes for things like npm_modules

Cache Invalidation

Dagger invalidates cache when inputs change. For WithExec, this includes the command, environment variables, and mounted files. For WithMountedDirectory, it's the directory's content hash.

4. Getting Started with Dagger

Installation

Install the Dagger CLI:

# macOS/Linux
curl -L https://dl.dagger.io/dagger/install | sh
sudo mv bin/dagger /usr/local/bin/

# Or with Homebrew
brew install dagger/tap/dagger

# Verify installation
dagger version

Your First Dagger Module

Initialize a new Dagger module:

# Create a new project
mkdir my-dagger-pipeline
cd my-dagger-pipeline

# Initialize with your preferred language
# Options: go, python, typescript
dagger init --sdk=go --source=./dagger

This creates a dagger/ directory with your module code. Let's look at the Go version:

// dagger/main.go
package main

import (
    "context"
    "fmt"
)

type MyModule struct{}

// Returns a container that echoes whatever you provide
func (m *MyModule) ContainerEcho(
    ctx context.Context,
    stringArg string,
) (string, error) {
    return dag.Container().
        From("alpine:latest").
        WithExec([]string{"echo", stringArg}).
        Stdout(ctx)
}

// Returns lines of a file from a directory
func (m *MyModule) GrepDir(
    ctx context.Context,
    directoryArg *Directory,
    pattern string,
) (string, error) {
    return dag.Container().
        From("alpine:latest").
        WithMountedDirectory("/src", directoryArg).
        WithWorkdir("/src").
        WithExec([]string{"grep", "-r", pattern, "."}).
        Stdout(ctx)
}

Run your function:

# Call the container-echo function
dagger call container-echo --string-arg="Hello, Dagger!"

# Call grep-dir on the current directory
dagger call grep-dir --directory-arg=. --pattern="func"

Understanding the Structure

A Dagger module has a specific structure:

my-dagger-pipeline/
├── dagger.json          # Module configuration
├── dagger/              # Source code
│   ├── main.go          # Entry point (Go)
│   ├── main.py          # Entry point (Python)
│   └── main.ts          # Entry point (TypeScript)
└── .gitignore

The dagger.json file defines your module:

{
  "name": "my-dagger-pipeline",
  "sdk": "go",
  "source": "dagger",
  "engineVersion": "v0.11.0"
}

5. Writing Dagger Pipelines

A Real-World Example: Build and Test a Go Application

Let's build a complete CI/CD pipeline for a Go application:

// dagger/main.go
package main

import (
    "context"
    "fmt"
)

type Ci struct{}

// Source returns the source code directory
func (m *Ci) Source() *Directory {
    return dag.Host().Directory(".")
}

// Lint runs golangci-lint on the codebase
func (m *Ci) Lint(ctx context.Context) (string, error) {
    return dag.Container().
        From("golangci/golangci-lint:v1.55").
        WithMountedDirectory("/src", m.Source()).
        WithWorkdir("/src").
        WithExec([]string{"golangci-lint", "run", "--timeout", "5m"}).
        Stdout(ctx)
}

// Test runs all tests with coverage
func (m *Ci) Test(ctx context.Context) (string, error) {
    return dag.Container().
        From("golang:1.21").
        WithMountedDirectory("/src", m.Source()).
        WithWorkdir("/src").
        WithExec([]string{"go", "test", "-v", "-race", "-coverprofile=coverage.out", "./..."}).
        Stdout(ctx)
}

// Build creates a compiled binary
func (m *Ci) Build(ctx context.Context) (*File, error) {
    return dag.Container().
        From("golang:1.21-alpine").
        WithMountedDirectory("/src", m.Source()).
        WithWorkdir("/src").
        WithExec([]string{"go", "build", "-ldflags", "-s -w", "-o", "app", "."}).
        File("/src/app"), nil
}

// BuildImage creates a minimal container image
func (m *Ci) BuildImage(ctx context.Context) (*Container, error) {
    binary, err := m.Build(ctx)
    if err != nil {
        return nil, err
    }
    
    return dag.Container().
        From("gcr.io/distroless/static:nonroot").
        WithFile("/app", binary).
        WithEntrypoint(["/app"]).
        WithUser("nonroot:nonroot"), nil
}

// Publish builds and pushes the image to a registry
func (m *Ci) Publish(
    ctx context.Context,
    registry string,
    username string,
    password *Secret,
    tag string,
) (string, error) {
    image, err := m.BuildImage(ctx)
    if err != nil {
        return "", err
    }
    
    return image.
        WithRegistryAuth(registry, username, password).
        Publish(ctx, fmt.Sprintf("%s/myapp:%s", registry, tag))
}

// All runs the complete CI pipeline
func (m *Ci) All(ctx context.Context) error {
    // Run lint and test in parallel
    lint := m.Lint(ctx)
    test := m.Test(ctx)
    
    // Wait for both
    if _, err := lint; err != nil {
        return fmt.Errorf("lint failed: %w", err)
    }
    if _, err := test; err != nil {
        return fmt.Errorf("test failed: %w", err)
    }
    
    fmt.Println("✓ Lint passed")
    fmt.Println("✓ Tests passed")
    
    // Build image
    _, err := m.BuildImage(ctx)
    if err != nil {
        return fmt.Errorf("build failed: %w", err)
    }
    
    fmt.Println("✓ Image built")
    return nil
}

Run the complete pipeline:

# Run everything
dagger call all

# Run individual steps
dagger call lint
dagger call test
dagger call build export --path=./app

# Build and publish (requires registry credentials)
dagger call publish \
    --registry=ghcr.io \
    --username=$GITHUB_USER \
    --password=env:GITHUB_TOKEN \
    --tag=latest

Python Version

The same pipeline in Python:

# dagger/main.py
import dagger
from dagger import dag, function, object_type

@object_type
class Ci:
    @function
    def source(self) -> dagger.Directory:
        return dag.host().directory(".")
    
    @function
    async def lint(self) -> str:
        return await (
            dag.container()
            .from_("golangci/golangci-lint:v1.55")
            .with_mounted_directory("/src", self.source())
            .with_workdir("/src")
            .with_exec(["golangci-lint", "run", "--timeout", "5m"])
            .stdout()
        )
    
    @function
    async def test(self) -> str:
        return await (
            dag.container()
            .from_("golang:1.21")
            .with_mounted_directory("/src", self.source())
            .with_workdir("/src")
            .with_exec(["go", "test", "-v", "-race", "./..."])
            .stdout()
        )
    
    @function
    async def build(self) -> dagger.File:
        return await (
            dag.container()
            .from_("golang:1.21-alpine")
            .with_mounted_directory("/src", self.source())
            .with_workdir("/src")
            .with_exec(["go", "build", "-o", "app", "."])
            .file("/src/app")
        )
    
    @function
    async def all(self) -> None:
        # Run in parallel
        await asyncio.gather(
            self.lint(),
            self.test(),
        )
        await self.build()
        print("✓ Pipeline complete")

Working with Secrets

Dagger handles secrets securely:

// Load from environment variable
apiKey := dag.SetSecret("API_KEY", os.Getenv("API_KEY"))

// Load from file
sshKey := dag.Host().File("/home/user/.ssh/id_rsa").Secret()

// Use in container
result := dag.Container().
    From("alpine").
    WithSecretVariable("API_KEY", apiKey).
    WithExec([]string{"sh", "-c", "curl -H \"Authorization: $API_KEY\" ..."}).
    Stdout(ctx)

// Mount as file
result := dag.Container().
    From("alpine").
    WithMountedSecret("/root/.ssh/id_rsa", sshKey).
    WithExec([]string{"ssh", "..."})

Secret Security

Dagger secrets are never written to disk or logged. They're stored in memory and only exposed to the containers that need them. Even the Dagger engine can't read secret values after they're set.

6. Migrating from GitHub Actions/GitLab CI

GitHub Actions to Dagger

Here's how common GitHub Actions patterns translate to Dagger:

GitHub Actions	Dagger Equivalent
`uses: actions/checkout@v4`	`dag.Host().Directory(".")`
`uses: actions/setup-go@v5`	`dag.Container().From("golang:1.21")`
`run: go test ./...`	`.WithExec([]string{"go", "test", "./..."})`
`uses: actions/cache@v4`	Automatic via BuildKit
`uses: docker/login-action@v3`	`.WithRegistryAuth(...)`

Example GitHub Actions workflow using Dagger:

# .github/workflows/ci.yml
name: CI

on: [push, pull_request]

jobs:
  ci:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      
      - name: Run Dagger pipeline
        uses: dagger/dagger-for-github@v5
        with:
          verb: call
          args: all
          version: "0.11"
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

Your entire CI logic lives in the Dagger module. The GitHub Actions workflow is just a thin wrapper that calls it.

GitLab CI to Dagger

# .gitlab-ci.yml
stages:
  - build
  - test
  - deploy

variables:
  DAGGER_VERSION: "0.11"

build:
  stage: build
  image: golang:1.21
  script:
    - curl -L https://dl.dagger.io/dagger/install | sh
    - ./bin/dagger call build export --path=./app
  artifacts:
    paths:
      - app

test:
  stage: test
  image: golang:1.21
  script:
    - curl -L https://dl.dagger.io/dagger/install | sh
    - ./bin/dagger call test

deploy:
  stage: deploy
  image: alpine:latest
  script:
    - curl -L https://dl.dagger.io/dagger/install | sh
    - ./bin/dagger call publish --tag=$CI_COMMIT_REF_NAME
  only:
    - main

7. Advanced Features

Cross-Platform Builds

Build for multiple architectures:

func (m *Ci) BuildMultiArch(ctx context.Context) error {
    platforms := []string{"linux/amd64", "linux/arm64"}
    
    for _, platform := range platforms {
        binary := dag.Container(dagger.ContainerOpts{
            Platform: platform,
        }).
        From("golang:1.21-alpine").
        WithMountedDirectory("/src", m.Source()).
        WithWorkdir("/src").
        WithEnvVariable("GOOS", strings.Split(platform, "/")[0]).
        WithEnvVariable("GOARCH", strings.Split(platform, "/")[1]).
        WithExec([]string{"go", "build", "-o", "app", "."}).
        File("/src/app")
        
        // Export each binary
        _, err := binary.Export(ctx, fmt.Sprintf("./app-%s", platform))
        if err != nil {
            return err
        }
    }
    return nil
}

Remote Caching

Share cache across CI runs and machines:

# Use S3 as cache backend
dagger call all \
    --cache-from=type=s3,region=us-east-1,bucket=my-cache \
    --cache-to=type=s3,region=us-east-1,bucket=my-cache,mode=max

# Or use a registry
dagger call all \
    --cache-from=type=registry,ref=ghcr.io/myorg/cache \
    --cache-to=type=registry,ref=ghcr.io/myorg/cache,mode=max

Services and Networking

Run integration tests with dependencies:

func (m *Ci) IntegrationTest(ctx context.Context) error {
    // Start PostgreSQL
    postgres := dag.Container().
        From("postgres:16").
        WithEnvVariable("POSTGRES_PASSWORD", "secret").
        WithEnvVariable("POSTGRES_DB", "testdb").
        WithExposedPort(5432).
        AsService()
    
    // Run tests against it
    return dag.Container().
        From("golang:1.21").
        WithMountedDirectory("/src", m.Source()).
        WithWorkdir("/src").
        WithServiceBinding("postgres", postgres).
        WithEnvVariable("DATABASE_URL", "postgres://postgres:secret@postgres:5432/testdb?sslmode=disable").
        WithExec([]string{"go", "test", "-v", "./...", "-run", "Integration"}).
        Sync(ctx)
}

Custom Modules and Reusability

Create reusable modules:

// dagger/main.go in a module called "go-ci"
package main

type GoCi struct {
    Source *Directory
}

// New creates a new GoCi instance
func New(source *Directory) *GoCi {
    return &GoCi{Source: source}
}

// Lint runs golangci-lint
func (g *GoCi) Lint(ctx context.Context) (string, error) {
    return dag.Container().
        From("golangci/golangci-lint:v1.55").
        WithMountedDirectory("/src", g.Source).
        WithWorkdir("/src").
        WithExec([]string{"golangci-lint", "run"}).
        Stdout(ctx)
}

// Test runs go test
func (g *GoCi) Test(ctx context.Context) (string, error) {
    return dag.Container().
        From("golang:1.21").
        WithMountedDirectory("/src", g.Source).
        WithWorkdir("/src").
        WithExec([]string{"go", "test", "-v", "./..."}).
        Stdout(ctx)
}

Use it from another module:

// In your project's dagger/main.go
func (m *MyModule) Ci(ctx context.Context) error {
    goCi := dag.GoCi(m.Source())
    
    _, err := goCi.Lint(ctx)
    if err != nil {
        return err
    }
    
    _, err = goCi.Test(ctx)
    return err
}

8. Dagger vs Traditional CI/CD

Aspect	Dagger	GitHub Actions	GitLab CI	Jenkins
Configuration	Go/Python/TS code	YAML	YAML	Groovy/DSL
Local execution	Native	Act (limited)	GitLab Runner	Complex setup
Reproducibility	Container-native	Runner-dependent	Runner-dependent	Plugin-dependent
Debugging	Local, breakpoints	Remote logs only	Remote logs only	Limited
Testing	Unit tests	None	None	Limited
Portability	Any CI platform	GitHub only	GitLab only	Self-hosted
Caching	BuildKit (advanced)	Actions cache	Cache API	Plugin-based
Ecosystem	Growing (Daggerverse)	Massive (Marketplace)	Large	Massive (Plugins)

9. Production Patterns

Pattern 1: Monorepo Support

Build only changed components:

func (m *Ci) ChangedServices(ctx context.Context) ([]string, error) {
    // Get changed files from git
    diff := dag.Git().
        WithSource(m.Source()).
        Diff("HEAD~1", "HEAD")
    
    // Determine which services changed
    var services []string
    files, _ := diff.Glob(ctx, "**/Dockerfile")
    for _, f := range files {
        services = append(services, filepath.Dir(f))
    }
    return services, nil
}

func (m *Ci) BuildChanged(ctx context.Context) error {
    services, _ := m.ChangedServices(ctx)
    for _, svc := range services {
        dag.Container().
            Build(m.Source().Directory(svc)).
            Publish(ctx, fmt.Sprintf("registry/%s:latest", svc))
    }
    return nil
}

Pattern 2: Progressive Delivery

func (m *Ci) Deploy(ctx context.Context, environment string) error {
    image, _ := m.BuildImage(ctx)
    
    switch environment {
    case "staging":
        // Deploy to staging immediately
        return m.deployToK8s(ctx, image, "staging")
    case "production":
        // Canary deployment: 10% traffic
        if err := m.deployCanary(ctx, image, 10); err != nil {
            return err
        }
        // Wait for metrics
        time.Sleep(10 * time.Minute)
        // Promote to 100%
        return m.deployCanary(ctx, image, 100)
    }
    return nil
}

Pattern 3: Security Scanning

func (m *Ci) SecurityScan(ctx context.Context) error {
    image, _ := m.BuildImage(ctx)
    
    // Trivy vulnerability scan
    trivy := dag.Container().
        From("aquasec/trivy:latest").
        WithMountedFile("/image.tar", image.AsTarball()).
        WithExec([]string{"trivy", "image", "--input", "/image.tar", "--exit-code", "1", "--severity", "HIGH,CRITICAL"})
    
    // Snyk dependency scan
    snyk := dag.Container().
        From("snyk/snyk:go").
        WithMountedDirectory("/src", m.Source()).
        WithWorkdir("/src").
        WithSecretVariable("SNYK_TOKEN", dag.SetSecret("SNYK_TOKEN", os.Getenv("SNYK_TOKEN"))).
        WithExec([]string{"snyk", "test", "--severity-threshold=high"})
    
    // Run both in parallel
    _, err1 := trivy.Sync(ctx)
    _, err2 := snyk.Sync(ctx)
    
    if err1 != nil || err2 != nil {
        return fmt.Errorf("security scan failed")
    }
    return nil
}

Pattern 4: Multi-Environment Promotion

func (m *Ci) Promote(ctx context.Context, version string) error {
    // Tag with environment-specific tags
    stages := []struct{
        env string
        tag string
    }{
        {"dev", fmt.Sprintf("dev-%s", version)},
        {"staging", fmt.Sprintf("staging-%s", version)},
        {"prod", fmt.Sprintf("prod-%s", version)},
    }
    
    for _, stage := range stages {
        // Require approval for production
        if stage.env == "prod" {
            fmt.Printf("Ready to promote to production. Run: dagger call promote --version=%s --approve\n", version)
            continue
        }
        
        image := dag.Container().
            From(fmt.Sprintf("registry/app:%s", version))
        
        _, err := image.Publish(ctx, fmt.Sprintf("registry/app:%s", stage.tag))
        if err != nil {
            return err
        }
    }
    return nil
}

The Dagger Advantage

Dagger transforms CI/CD from a configuration burden into a software engineering discipline. You get type safety, testability, IDE support, and local development—all while maintaining portability across CI platforms. The initial learning curve pays dividends in maintainability and developer experience.

Getting Started Checklist

Install Dagger CLI: curl -L https://dl.dagger.io/dagger/install | sh
Initialize your first module: dagger init --sdk=go
Port one simple workflow from your existing CI
Run it locally: dagger call
Integrate with your CI platform using the thin wrapper pattern
Gradually migrate more workflows

Dagger is the future of CI/CD—programmable, portable, and container-native. Start small, iterate, and experience the difference that treating pipelines as code makes.