Bootstrapping and Secrets Management: solving the chef-vault chicken/egg problem with chef-metal

Intro

Automating server creation with Chef can be very satisfying when it works well. It saves time and avoids the need for tedious and repetitive tasks. Distribution of secrets to servers created this way has always been a hassle for me (or insecure, if laziness gets the best of me). Chef-vault does some fancy things with encrypted data bags and the public/private key system used by chef. When an item is updated in the vault, it encrypts the data for clients using their public keys from chef, allowing those clients to decrypt it with their private keys. The problem with this is, the client needs to exist before the vault is updated, but you also want the client to access things in the vault while it is running its initial run list. In this post I'll cover how I'm currently solving this problem with chef-metal and chef-vault. For those unfamiliar, chef-metal allows you to create machines, clusters, full stacks, or whatever, using the same type of code you are already familiar with in configuring machines.

Too long; not going to read

A single chef-metal recipe does the following

1. creates an instance with a role and empty run list
2. chef-vault is updated using said role
3. instance gets a new run list

Disclaimer

My knowledge of ruby is pretty limited, and this whole method seems a little convoluted. I'm hoping someone will come along and show me a better way.

Tutorial

First, to get chef-metal to interact with your virtualization provider of choice, you need to set a driver. This can be done in knife.rb as follows:

...
driver 'fog:AWS:example'
...

I'm using AWS, so I also have a file ~/.aws/config with profiles for different accounts. In this case, I've specified the 'example' profile

...
[profile example]
aws_access_key_id=ABC
aws_secret_access_key=123
region=us-west-2
availability_zone=us-west-2b

chef-vault also needs to know whether it should act in 'client' or 'solo' mode. Put something like this in your knife.rb as well (replacing values as needed)

...
knife[:vault_mode] = "client"
knife[:vault_admins] = [ "reese" ]
...

Now we need a recipe that bootstraps a machine, updates the vault, and then adds the application to the machine's run list once it can access items in the vault.

#secrets/recipes/metal.rb

%w{chef-metal chef-metal-fog fog}.each do |pkg|
    chef_gem "#{pkg}" do
        action :install
    end
end
require 'chef_metal_fog'
#require "cheffish"

#specify where the chef server is
with_chef_server 'https://chef.example.com',
    :client_name => 'reese',
    :signing_key_filename => '~/.chef/reese.pem'


#create a key pair for the default admin user (ec2-user in this case)
fog_key_pair "admin"

#put options in a variable for easier overriding for different environments
options = {
    :sudo => true,
    :ssh_username => 'ec2-user',
    :sudo => true,
    :bootstrap_options => {
        :key_name => 'admin',
        :availability_zone => 'us-west-2b',
        :security_group_ids => %w(sg-12345678), 
        :subnet_id => 'subnet-abcdef12', 
        :flavor_id => 't2.micro',
        :image_id => 'ami-d13845e1', #amazon linux ami, us-west t2.micro
        :virtualization_type => 'hvm' #required for t2.micro
    }
}

with_machine_options options

machine "lego-app-dev" do
    chef_environment "development"
    role 'keyholder-bootstrap'
    role 'lego-app-bootstrap'
    action [:converge]
end

def deep_clone(o)
    Marshal.load(Marshal.dump(o))
end

production_options = deep_clone(options) #someone good at ruby, tell me there's a better way
production_options[:bootstrap_options][:flavor_id] = "m3.medium"
production_options[:bootstrap_options][:image_id] = "ami-d13845e1"

machine "lego-app1" do
    machine_options production_options
    chef_environment "production"
    role 'lego-app-bootstrap'
    action [:converge]
end

include_recipe "secrets::update-vault"

machine "lego-app-dev" do
    role 'keyholder-bootstrap' #is this needed? or does it remain from before?
    role 'keyholder'
    role 'lego-app-bootstrap'
    role 'lego-app'
    converge true
end

machine "lego-app1" do
    role 'lego-app-bootstrap'
    role 'lego-app'
    converge true
end

Check out the documentation to see more driver/machine configuration options

I want to deliver sensitive data only to the clients which need it, so I have an empty 'bootstrap' role to go along with each role that accesses a set of secrets. These non-empty roles (e.g. 'keyholder') trivially run a single recipe with a name similar to the role. In this case, the role 'keyholder' lets emmet and wildstyle log in, while the role 'lego-app' runs the actual application.

The update-vault recipe takes care of making sure each client can access the items it should be able to. This code is probably cringeworthy (but it works, as far as I can tell).

#secrets/recipes/update-vault.rb
chef_gem "chef-vault" do
    action :install
end

secrets_path = "~/.chef/secrets"
vault_name = 'credentials'

#create or update the vault
def update_vault(vault, item, filePath, type, role)
    ruby_block "check_vault_#{vault}_#{item}" do
        block do
            #probably a terrible way to decide whether to create or update the vault
            Chef::Resource::RubyBlock.send(:include, Chef::Mixin::ShellOut)
            #see if item exists in vault
            command = 'knife vault show '+vault+' '+item
            command_out = shell_out(command)
            #check if command resulted in an error
            if command_out.stderr.to_s != ''
                puts "\ncreating vault and/or vault item\n"
                action = "create"
            else
                puts "\nvault exists\n"
                action = "update"
            end
            vault_command = "knife vault #{action} #{vault} #{item} --mode client --#{type} #{filePath} -A 'reese' -S 'role:#{role}' "
            vault_command_out = shell_out(vault_command)
            puts vault_command_out.stdout
        end
    end
end

#for each secret file, add an entry to the vault
secrets = {
    "emmet" => "emmet.pub",
    "wildstyle" => "wildstyle.pub"
}

secrets.each do |secret_name,filename|
    filePath = File.join(secrets_path, filename)
    type = "file"
    role = "keyholder-bootstrap"
    update_vault(vault_name, secret_name, filePath, type, role)
end

vault_name = 'config'

#for each json secrets set, add an entry to the vault
jsonSecrets = {
    "service_config" => "services.json"
}
jsonSecrets.each do |secret_name,filename|
    filePath = File.join(secrets_path, filename)
    type = "json"
    role = "lego-app-bootstrap"
    update_vault(vault_name, secret_name, filePath, type, role)
end

And here's an example services.json:

{
    "password":"spaceship",
    "username":"benny"
}

So since we run update-vault with chef-zero, it runs the knife commands using the local knife client.

The actual 'lego-app' and 'keyholders' recipes run as usual on the machines.

#secrets/recipes/keyholders.rb
#allow use of chef_vault_item resource
include_recipe "chef-vault"

users = %w(emmet wildstyle)
vault = "credentials"

users.each do |username|

    user username do
        action :create
    end

    dir = "/home/#{username}/.ssh"
    directory dir do
        owner username
        group username
        mode 0755
        action :create
    end

    private_key = chef_vault_item(vault, username)
    file File.join(dir,"authorized_keys") do
        content private_key['file-content']
        owner username
        group username
        mode "0600"
        action :create
    end
end

#secrets/recipes/lego-app.rb
include_recipe "chef-vault"

vault = "config"
secret_name = "service_config"
config = chef_vault_item(vault, secret_name)

#these attributes could be used in a template or something
node.set['service']['password'] = config['password']
node.set['service']['username'] = config['username']

#alternatively, do stuff directly with the secrets
username = config['username']
execute "configure_amazing_service" do
    command "echo username=#{username} > /usr/local/etc/service.conf"
end

To run the recipe, use chef-zero and run the following command

chef-client -z -o secrets::metal

The 'metal' recipe should create 2 servers with empty run lists, one micro for development and one medium for production. From there, it will update the vaults, allowing the two new machines to access items if they are in the right role. Finally, it will apply the roles with the run lists to create the users and install/configure the application.
Let me know if I missed anything important or can do things a better way.

Inspiration/Credits

The method used here was inspired by the orchestration scenario presented on this page and by this tweet conversation which I didn't fully understand.