It was 11pm on a Friday. Deploy failed. Missing environment variable. After twenty minutes of debugging, I found it: I’d added a secret to CI but forgot to add it to the server. Two-minute fix, twenty-minute hunt.

I’d been burned enough times to finally fix this properly.

The Core Insight

One source of truth for production secrets. 1Password holds the secrets. Kamal fetches them. GitHub Actions triggers the deploy. No scattered env files on servers, no secrets in CI config.

The payoff: zero drift between environments, trivial secret rotation, and no more touching servers manually. Whether I’m running kamal deploy from my terminal or GitHub Actions is deploying on push to main, the same .kamal/secrets file pulls from the same vault.

How Secrets Flow

flowchart LR A["1Password
Production Vault"] --> B[".kamal/secrets"] B --> C["Kamal"] C --> D["Production
Container"] E["GitHub Actions"] -.->|"service account token"| B F["Terminal"] -.->|"1Password app"| B

One file, one vault. Both deploy methods pull from the same source.

The .kamal/secrets file fetches secrets and exports them as environment variables. Kamal reads those variables and injects them into your production container. The deploy.yml just declares which secrets to look for.

Setup

Before you start: you need a 1Password account, a service account token (for CI), and Kamal 2.5+ installed locally.

1Password Structure

Create a vault with your secrets:

MyApp/
  Production/
    - KAMAL_REGISTRY_PASSWORD
    - RAILS_MASTER_KEY
    - DATABASE_PASSWORD
    - CHATKIT_PK
    - MAILGUN_INGRESS_SIGNING_KEY
  Server/
    - SSH_KEY (private key for server access)

I keep Production and Server separate because app secrets rotate more often than SSH keys, and it’s easier to audit access when they’re in different items.

Create a Service Account with read access to your vault. Service accounts are designed for non-interactive environments like CI—your personal 1Password login won’t work in GitHub Actions. Save the token for later.

Kamal Configuration

config/deploy.yml:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33

service: myapp
image: your-username/myapp

servers:
  web:
    - your.server.ip.address

proxy:
  ssl: true
  host: your-app.example.com

registry:
  server: ghcr.io
  username: [email protected]
  password:
    - KAMAL_REGISTRY_PASSWORD

env:
  secret:
    - RAILS_MASTER_KEY
    - DATABASE_PASSWORD
    - CHATKIT_PK
    - MAILGUN_INGRESS_SIGNING_KEY
  clear:
    SOLID_QUEUE_IN_PUMA: true
    DB_HOST: infra-postgres

volumes:
  - "myapp_storage:/rails/storage"
  - "/var/run/docker.sock:/var/run/docker.sock"

builder:
  arch: amd64

The env.secret array tells Kamal which variables to pull from your secrets file.

The Secrets File

This is the heart of the setup.

.kamal/secrets:

1
2
3
4
5
6

SECRETS=$(kamal secrets fetch --adapter 1password --account your-account.1password.com --from MyApp/Production KAMAL_REGISTRY_PASSWORD RAILS_MASTER_KEY DATABASE_PASSWORD MAILGUN_INGRESS_SIGNING_KEY CHATKIT_PK)
KAMAL_REGISTRY_PASSWORD=$(kamal secrets extract KAMAL_REGISTRY_PASSWORD ${SECRETS})
RAILS_MASTER_KEY=$(kamal secrets extract RAILS_MASTER_KEY ${SECRETS})
DATABASE_PASSWORD=$(kamal secrets extract DATABASE_PASSWORD ${SECRETS})
MAILGUN_INGRESS_SIGNING_KEY=$(kamal secrets extract MAILGUN_INGRESS_SIGNING_KEY ${SECRETS})
CHATKIT_PK=$(kamal secrets extract CHATKIT_PK ${SECRETS})

One file. Works for manual deploys, works in CI.

GitHub Actions Workflow

The workflow installs the 1Password CLI, loads the SSH key, then runs kamal deploy. The service account token authenticates everything.

.github/workflows/hetzner-cd.yml:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56

name: CD

on:
  push:
    branches:
      - main
  workflow_dispatch:

concurrency:
  group: production-deploy
  cancel-in-progress: true

jobs:
  Deploy:
    runs-on: [self-hosted, linux, x64, privileged]

    env:
      OP_SERVICE_ACCOUNT_TOKEN: ${{ secrets.OP_SERVICE_ACCOUNT_TOKEN }}
      VERSION: ${{ github.sha }}
      DOCKER_BUILDKIT: 1
      RAILS_ENV: production

    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Install 1Password CLI
        uses: 1password/install-cli-action@v2

      - name: Load SSH key from 1Password
        uses: 1password/load-secrets-action@v3
        with:
          export-env: true
        env:
          OP_SERVICE_ACCOUNT_TOKEN: ${{ secrets.OP_SERVICE_ACCOUNT_TOKEN }}
          SSH_PRIVATE_KEY: "op://MyApp/Server/SSH_KEY"

      - name: Set up Ruby
        uses: ruby/setup-ruby@v1
        with:
          ruby-version: 3.4.4
          bundler-cache: true

      - name: Install Kamal
        run: gem install kamal -v 2.5.3

      - name: Set up SSH agent
        uses: webfactory/[email protected]
        with:
          ssh-private-key: ${{ env.SSH_PRIVATE_KEY }}

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3

      - name: Deploy
        run: kamal deploy

In your repository settings, add one secret: OP_SERVICE_ACCOUNT_TOKEN. Everything else lives in 1Password.

Note: This workflow uses a self-hosted runner. If you’re on GitHub-hosted runners, change runs-on to ubuntu-latest and ensure the runner has network access to your server.

Adding New Secrets

This is the part people get wrong. Adding a secret requires changes in three places. Miss one and the deploy fails silently.

1Password: Create the secret in your vault (e.g., MyApp/Production/NEW_SECRET).

.kamal/secrets: Add to both the fetch and extract:

1
2
3
4
5

# Add to fetch command
SECRETS=$(kamal secrets fetch ... EXISTING_SECRETS NEW_SECRET)

# Add extraction line
NEW_SECRET=$(kamal secrets extract NEW_SECRET ${SECRETS})

config/deploy.yml: Add to the secret array:

1
2
3
4

env:
  secret:
    - RAILS_MASTER_KEY
    - NEW_SECRET  # Add here

The common mistake: adding to the fetch but forgetting the extract line. Or adding to extract but not to deploy.yml. The deploy runs, the secret is empty, and you spend fifteen minutes wondering why your app can’t connect to the database.

Teaching This to Your Coding Agent

I use AI coding agents for most implementation work. They’re great at editing code but can’t touch 1Password. This template draws a clear line between human-only steps and agent-only steps:

Add a new production secret called `YOUR_SECRET_NAME`.

Prerequisites (I will handle manually):
- [ ] Create secret in 1Password at MyApp/Production/YOUR_SECRET_NAME

Code changes needed:

1. Update `.kamal/secrets`:
   - Add YOUR_SECRET_NAME to the `kamal secrets fetch` command on line 1
   - Add extraction line: YOUR_SECRET_NAME=$(kamal secrets extract YOUR_SECRET_NAME ${SECRETS})

2. Update `config/deploy.yml`:
   - Add YOUR_SECRET_NAME to the `env.secret` array

Reference existing secrets in these files for the exact format.

The explicit split matters. The agent can’t access 1Password, and I don’t want it trying. Clear boundaries, clear ownership. After it commits, I verify with kamal shell and printenv | grep SECRET_NAME.

Troubleshooting

Secret not found during deploy. The deploy succeeds but the app crashes with a missing env var. Check: secret exists in 1Password with exact spelling, name matches in fetch command, extraction line exists, and deploy.yml includes it.

SSH connection failures in CI. Permission denied (publickey) in GitHub Actions. Check: field in 1Password is named SSH_KEY (not SSH_PRIVATE_KEY), it’s the private key not public, format includes -----BEGIN OPENSSH PRIVATE KEY-----, and server’s authorized_keys has the corresponding public key.

“op not found” in CI. The 1Password CLI install step must run before any secret access. Ensure 1password/install-cli-action@v2 comes first.

Kamal can’t authenticate with 1Password. For manual deploys: 1Password desktop app needs to be running and unlocked. For CI: verify OP_SERVICE_ACCOUNT_TOKEN is set and the service account has vault access.

What Changed

The first time the pipeline worked end-to-end, I pushed a commit and walked away. Came back to a successful deploy. No SSH’ing into servers, no copying env files, no “wait, which secret changed?”

Secrets live in one place. Deployments are fully automated. Rotating a secret means updating 1Password and redeploying.

The only friction left is remembering to update all three files when adding a secret. But it beats the 11pm debugging sessions. Fewer surprises, fewer late-night deploys, a pipeline I can trust.

For preview environments on pull requests, the pattern extends naturally. I wrote about that in Building Review Apps with Kamal and GitHub Actions.