Deploying a Ruby on Rails application has never been more exciting since the release of Kamal. It promises a seamless, zero-downtime deployment experience using Docker, directly to your own servers. However, integrating Kamal into a fully automated GitLab CI/CD pipeline can introduce a few unexpected "boss fights."
Recently, I set up an automated deployment pipeline for a Rails project using a self-hosted GitLab instance. What started as a straightforward task turned into a deep dive into Docker networking, YAML parsing, and database initialization quirks.
Here is my survival guide and the ultimate .gitlab-ci.yml configuration to achieve that glorious green "Passed" pipeline.
Battle #1: The Self-Hosted Registry Connection Reset
The first hurdle was getting the GitLab CI runner to push the built Docker image to our self-hosted Container Registry. The pipeline failed with a cryptic Connection reset by peer or invalid reference format error.
The Fix: If your self-hosted registry runs on a specific port (e.g., 5050) and doesn't use strict HTTPS internally, Docker's security mechanism will block it. You need to explicitly tell the Docker-in-Docker (DinD) service to trust your registry:
services:
- name: docker:24.0.5-dind
command: ["--insecure-registry=gitlab.example.com:5050"]
Note: Make sure your config/deploy.yml only specifies the registry path without duplicating the domain in the image attribute!
Battle #2: Escaping Hell in .kamal/secrets
Kamal securely injects environment variables using a .kamal/secrets file. In a CI environment, you typically generate this file on the fly using CI/CD variables. But using standard bash echo to write passwords containing special characters (like $, ", !) often leads to broken strings or YAML parsing errors (e.g., Psych::SyntaxError).
The Fix: Bypass bash escaping issues entirely by using a Ruby script within your CI pipeline to safely generate the secrets file:
- |
ruby <<'RUBY'
def env_line(key, value)
v = value.to_s.gsub("\r", "").chomp
return "#{key}=\n" if v.empty?
if v.match?(/\A[A-Za-z0-9_.+\/@:-]*\z/)
"#{key}=#{v}\n"
else
esc = v.gsub("\\", "\\\\").gsub('"', '\\"').gsub("\n", "\\n")
%{#{key}="#{esc}"\n}
end
end
pairs = [
["KAMAL_REGISTRY_USERNAME", ENV["CI_REGISTRY_USER"]],
["KAMAL_REGISTRY_PASSWORD", ENV["CI_REGISTRY_PASSWORD"]],
["RAILS_MASTER_KEY", ENV["RAILS_MASTER_KEY"]],
["POSTGRES_PASSWORD", ENV["POSTGRES_PASSWORD"]]
]
File.write(".kamal/secrets", pairs.map { |k, v| env_line(k, v) }.join)
RUBY
Battle #3: The Non-Interactive Trap
If your server's kamal-proxy is outdated, Kamal will kindly ask if you want to reboot it to upgrade. In a local terminal, you just press y. In a CI pipeline, the runner hangs indefinitely waiting for a user input that will never come: This will cause a brief outage on each host. Are you sure? [y, N]
The Fix: Always add the -y flag to bypass interactive prompts in CI scripts.
kamal proxy reboot -y
Battle #4: The Ghost in the PostgreSQL Volume (The Final Boss)
Even after the image was pushed and pulled successfully, the container kept failing the health check (target failed to become healthy within configured timeout). The logs revealed a database authentication failure: password authentication failed for user "postgres".
The Root Cause: The official postgres Docker image has a specific quirk: it only sets the password on the very first initialization when the data directory is empty. During my failed deployment attempts, Kamal had already started the database container with a faulty (or empty) password. Once the data volume was created, PostgreSQL permanently ignored any new passwords I passed through the CI variables.
The Fix: I had to SSH into the production server and physically eradicate the tainted database directory to force a fresh initialization:
docker rm -f my_app-db
rm -rf ~/my_app-db
Then, ensure the CI script boots the accessories before the main deployment:
- kamal accessory boot db
- kamal accessory boot redis
The Ultimate .gitlab-ci.yml for Kamal
Putting it all together, here is the robust, battle-tested GitLab CI/CD pipeline for deploying a Rails 8 app with Kamal:
stages:
- deploy
deploy_production:
stage: deploy
image: ruby:3.3.0
services:
- name: docker:24.0.5-dind
command: ["--insecure-registry=gitlab.example.com:5050"]
variables:
DOCKER_HOST: tcp://docker:2375
DOCKER_TLS_CERTDIR: ""
before_script:
- apt-get update -qq && apt-get install -y docker.io openssh-client curl
# Install Docker buildx plugin
- mkdir -p ~/.docker/cli-plugins/
- curl -sSLo ~/.docker/cli-plugins/docker-buildx https://github.com/docker/buildx/releases/download/v0.13.1/buildx-v0.13.1.linux-amd64
- chmod +x ~/.docker/cli-plugins/docker-buildx
# Configure Buildx for insecure registry
- mkdir -p /etc/buildkit
- echo '[registry."gitlab.example.com:5050"]' > /etc/buildkit/buildkitd.toml
- echo ' http = true' >> /etc/buildkit/buildkitd.toml
- echo ' insecure = true' >> /etc/buildkit/buildkitd.toml
- docker buildx create --name kamal-local-docker-container --driver=docker-container --config /etc/buildkit/buildkitd.toml
# Setup SSH Agent
- eval $(ssh-agent -s)
- echo "$SSH_PRIVATE_KEY" | tr -d '\r' | ssh-add -
- mkdir -p ~/.ssh
- chmod 700 ~/.ssh
- ssh-keyscan <YOUR_SERVER_IP> >> ~/.ssh/known_hosts
- gem install kamal
script:
# Securely write .kamal/secrets using Ruby
- mkdir -p .kamal
- |
ruby <<'RUBY'
def env_line(key, value)
v = value.to_s.gsub("\r", "").chomp
return "#{key}=\n" if v.empty?
if v.match?(/\A[A-Za-z0-9_.+\/@:-]*\z/)
"#{key}=#{v}\n"
else
esc = v.gsub("\\", "\\\\").gsub('"', '\\"').gsub("\n", "\\n")
%{#{key}="#{esc}"\n}
end
end
pairs = [
["KAMAL_REGISTRY_USERNAME", ENV["CI_REGISTRY_USER"]],
["KAMAL_REGISTRY_PASSWORD", ENV["CI_REGISTRY_PASSWORD"]],
["RAILS_MASTER_KEY", ENV["RAILS_MASTER_KEY"]],
["POSTGRES_PASSWORD", ENV["POSTGRES_PASSWORD"]]
]
File.write(".kamal/secrets", pairs.map { |k, v| env_line(k, v) }.join)
RUBY
# Boot database and redis first (Only required for the first deployment)
- kamal accessory boot db
- kamal accessory boot redis
# Upgrade proxy if necessary (non-interactive)
- kamal proxy reboot -y
# Deploy the main Rails app
- kamal deploy
# Boot or reboot Sidekiq using the newly pushed image
- kamal accessory boot sidekiq || kamal accessory reboot sidekiq
environment:
name: production
only:
- main
Conclusion
Deploying with Kamal is incredibly powerful once you understand how it orchestrates Docker under the hood. Solving these CI/CD issues taught me invaluable lessons about Docker network routing, volume persistence, and handling secrets dynamically.
Seeing that green pipeline status makes all the debugging entirely worth it. Happy deploying!