Kamal's SSH-based deploys: why your CI needs the right key format

Kamal's SSH-based deploys: why your CI needs the right key format

Key Facts

• Direct answer: The direct answer is that Kamal requires SSH keys in the OpenSSH format, not the PEM format commonly generated by tools like OpenSSL, and attempting to use PEM-formatted keys will result in authentication failures during deployment, even when the key appears correct in your CI environment.

• What the error/limitation actually means: Kamal, a deployment tool created by Basecamp, relies on SSH for secure, passwordless authentication to remote servers.

• When you'll hit it: You'll encounter this issue when your CI/CD pipeline attempts to deploy using Kamal with an SSH key that's in the wrong format.

• How to verify if it applies to you: To determine if you're affected by this key format issue, first examine the SSH key you're using in your deployment pipeline.

Deploying applications to remote servers has become increasingly streamlined with modern tools like Kamal, which simplifies the process through SSH-based deployments. However, many teams encounter unexpected failures during their CI/CD pipeline, often stemming from improperly formatted SSH keys. This issue affects developers, DevOps engineers, and teams managing production deployments, as it can halt releases and cause unnecessary downtime when not properly addressed.

The direct answer is that Kamal requires SSH keys in the OpenSSH format, not the PEM format commonly generated by tools like OpenSSL, and attempting to use PEM-formatted keys will result in authentication failures during deployment, even when the key appears correct in your CI environment.

What the error/limitation actually means

Kamal, a deployment tool created by Basecamp, relies on SSH for secure, passwordless authentication to remote servers. When you run a deployment command, Kamal attempts to connect to your target server using an SSH key. The critical issue here is that SSH keys come in different formats, and Kamal specifically expects keys in the OpenSSH format, which is the native format used by the OpenSSH suite of tools. This format begins with the header -----BEGIN OPENSSH PRIVATE KEY----- and contains binary data encoded in Base64.

In contrast, the PEM format, which begins with -----BEGIN RSA PRIVATE KEY----- or -----BEGIN PRIVATE KEY-----, is commonly generated by OpenSSL and other tools. While these formats represent the same cryptographic material, they are not interchangeable at the protocol level. When Kamal encounters a PEM-formatted key, it fails to parse the key correctly, leading to authentication failures. This isn't a bug in Kamal but rather a compatibility issue between the expected key format and what's being provided. The underlying mechanism involves SSH's strict parsing requirements, which differ between the OpenSSH and PEM formats, particularly in how they handle key parameters and encryption.

When you'll hit it

You'll encounter this issue when your CI/CD pipeline attempts to deploy using Kamal with an SSH key that's in the wrong format. This commonly happens when teams generate SSH keys using OpenSSL or other tools that default to PEM format, rather than using OpenSSH's own ssh-keygen utility. For example, if you're using GitHub Actions, GitLab CI, or Jenkins and have a step that generates an SSH key on the fly or retrieves one from a secrets manager, there's a high probability it will be in PEM format if not explicitly configured otherwise.

Another scenario is when you're migrating from a deployment system that used PEM-formatted keys to Kamal. Teams often assume that keys are universally compatible and don't check the format before switching tools. You might also hit this issue when using cloud provider-specific key generation tools, which may default to PEM format for compatibility with broader ecosystems. The problem manifests during the actual deployment step, typically with error messages that don't clearly indicate the key format issue, leading teams to troubleshoot network connectivity, server permissions, or key contents when the real problem is the format mismatch.

How to verify if it applies to you

To determine if you're affected by this key format issue, first examine the SSH key you're using in your deployment pipeline. If you're generating keys programmatically in your CI, check the script or configuration that creates them. For keys stored in secrets managers or version control, inspect the file directly. Open the key file in a text editor and look at the first line. If it begins with -----BEGIN OPENSSH PRIVATE KEY-----, you're using the correct format. If it starts with -----BEGIN RSA PRIVATE KEY-----, -----BEGIN PRIVATE KEY-----, or any other PEM header, you need to convert it.

You can also verify this programmatically by running the following command in your terminal or CI environment:

head -n 1 ~/.ssh/deploy_key

This will display the first line of your key file. For OpenSSH format, you should see the OpenSSH header. Additionally, you can test the key's compatibility by attempting to use it with SSH directly:

ssh -i ~/.ssh/deploy_key -T [email protected]

If this command fails with a "invalid key format" error or similar, you've confirmed the issue. In your CI pipeline, you might see authentication errors during the deployment step that don't provide clear indication of the root cause, which often points to this format mismatch.

Your options

• Convert your existing PEM key to OpenSSH format: Use the ssh-keygen utility with the -p flag to change the passphrase and format, effectively converting it to the OpenSSH format.

• Regenerate your key using OpenSSH tools: Create a new key with ssh-keygen -t ed25519 or ssh-keygen -t rsa to ensure it's in the correct format from the start.

• Use a wrapper script in your CI pipeline: Implement a script that converts the PEM key to OpenSSH format before running the Kamal deployment command.

• Deployxa: Utilize Deployxa's managed deployment service which handles SSH key formatting automatically as part of its deployment pipeline.

Common Pitfalls and Troubleshooting

The first pitfall is assuming that all SSH keys are interchangeable. Many developers don't realize that OpenSSH and PEM formats are not compatible at the protocol level. To fix this, always verify your key format before using it with Kamal and convert it if necessary using ssh-keygen.

The second pitfall is generating keys in CI environments without specifying the correct format. Most CI systems default to OpenSSL for key generation, which produces PEM keys. To fix this, explicitly use ssh-keygen in your CI pipeline to ensure OpenSSH format keys are created.

The third pitfall is storing keys in version control or secrets managers without documenting their format. This leads to confusion when team members try to use the keys. To fix this, always document the key format and include conversion instructions in your deployment documentation.

The fourth pitfall is misinterpreting error messages during deployment. Kamal's error messages don't explicitly state that the key format is incorrect, leading teams to troubleshoot other aspects. To fix this, check the key format first when encountering authentication failures during deployment.

The fifth pitfall is not testing the key format locally before deploying to production. Teams often only discover the issue during actual deployment attempts. To fix this, always test your deployment pipeline in a staging environment that mirrors production, including SSH key format validation.

Conclusion

Understanding SSH key format requirements is crucial for successful deployments with Kamal. The distinction between OpenSSH and PEM formats, while subtle, can cause significant deployment failures if not properly addressed. By verifying your key format before deployment and ensuring it matches what Kamal expects, you can avoid unnecessary troubleshooting and maintain a smooth CI/CD pipeline.

For teams dealing with this issue regularly, consider implementing automated key validation in your deployment scripts or exploring deployment platforms that handle these compatibility concerns automatically. As deployment tools continue to evolve, staying informed about their specific requirements will help maintain efficient and reliable release processes.