Intro
Automating server creation with Chef can be very satisfying when it works well. It saves time and avoids the need for tedious and repetitive tasks. Distribution of secrets to servers created this way has always been a hassle for me (or insecure, if laziness gets the best of me). Chef-vault does some fancy things with encrypted data bags and the public/private key system used by chef. When an item is updated in the vault, it encrypts the data for clients using their public keys from chef, allowing those clients to decrypt it with their private keys. The problem with this is, the client needs to exist before the vault is updated, but you also want the client to access things in the vault while it is running its initial run list. In this post I'll cover how I'm currently solving this problem with chef-metal and chef-vault. For those unfamiliar, chef-metal allows you to create machines, clusters, full stacks, or whatever, using the same type of code you are already familiar with in configuring machines.Too long; not going to read
A single chef-metal recipe does the following- creates an instance with a role and empty run list
- chef-vault is updated using said role
- instance gets a new run list
Disclaimer
My knowledge of ruby is pretty limited, and this whole method seems a little convoluted. I'm hoping someone will come along and show me a better way.Tutorial
First, to get chef-metal to interact with your virtualization provider of choice, you need to set a driver. This can be done in knife.rb as follows:... driver 'fog:AWS:example' ...
I'm using AWS, so I also have a file ~/.aws/config with profiles for different accounts. In this case, I've specified the 'example' profile
... [profile example] aws_access_key_id=ABC aws_secret_access_key=123 region=us-west-2 availability_zone=us-west-2b
chef-vault also needs to know whether it should act in 'client' or 'solo' mode. Put something like this in your knife.rb as well (replacing values as needed)
... knife[:vault_mode] = "client" knife[:vault_admins] = [ "reese" ] ...
Now we need a recipe that bootstraps a machine, updates the vault, and then adds the application to the machine's run list once it can access items in the vault.
#secrets/recipes/metal.rb
%w{chef-metal chef-metal-fog fog}.each do |pkg|
chef_gem "#{pkg}" do
action :install
end
end
require 'chef_metal_fog'
#require "cheffish"
#specify where the chef server is
with_chef_server 'https://chef.example.com',
:client_name => 'reese',
:signing_key_filename => '~/.chef/reese.pem'
#create a key pair for the default admin user (ec2-user in this case)
fog_key_pair "admin"
#put options in a variable for easier overriding for different environments
options = {
:sudo => true,
:ssh_username => 'ec2-user',
:sudo => true,
:bootstrap_options => {
:key_name => 'admin',
:availability_zone => 'us-west-2b',
:security_group_ids => %w(sg-12345678),
:subnet_id => 'subnet-abcdef12',
:flavor_id => 't2.micro',
:image_id => 'ami-d13845e1', #amazon linux ami, us-west t2.micro
:virtualization_type => 'hvm' #required for t2.micro
}
}
with_machine_options options
machine "lego-app-dev" do
chef_environment "development"
role 'keyholder-bootstrap'
role 'lego-app-bootstrap'
action [:converge]
end
def deep_clone(o)
Marshal.load(Marshal.dump(o))
end
production_options = deep_clone(options) #someone good at ruby, tell me there's a better way
production_options[:bootstrap_options][:flavor_id] = "m3.medium"
production_options[:bootstrap_options][:image_id] = "ami-d13845e1"
machine "lego-app1" do
machine_options production_options
chef_environment "production"
role 'lego-app-bootstrap'
action [:converge]
end
include_recipe "secrets::update-vault"
machine "lego-app-dev" do
role 'keyholder-bootstrap' #is this needed? or does it remain from before?
role 'keyholder'
role 'lego-app-bootstrap'
role 'lego-app'
converge true
end
machine "lego-app1" do
role 'lego-app-bootstrap'
role 'lego-app'
converge true
end
Check out the documentation to see more driver/machine configuration options
I want to deliver sensitive data only to the clients which need it, so I have an empty 'bootstrap' role to go along with each role that accesses a set of secrets. These non-empty roles (e.g. 'keyholder') trivially run a single recipe with a name similar to the role. In this case, the role 'keyholder' lets emmet and wildstyle log in, while the role 'lego-app' runs the actual application.
The update-vault recipe takes care of making sure each client can access the items it should be able to. This code is probably cringeworthy (but it works, as far as I can tell).
#secrets/recipes/update-vault.rb
chef_gem "chef-vault" do
action :install
end
secrets_path = "~/.chef/secrets"
vault_name = 'credentials'
#create or update the vault
def update_vault(vault, item, filePath, type, role)
ruby_block "check_vault_#{vault}_#{item}" do
block do
#probably a terrible way to decide whether to create or update the vault
Chef::Resource::RubyBlock.send(:include, Chef::Mixin::ShellOut)
#see if item exists in vault
command = 'knife vault show '+vault+' '+item
command_out = shell_out(command)
#check if command resulted in an error
if command_out.stderr.to_s != ''
puts "\ncreating vault and/or vault item\n"
action = "create"
else
puts "\nvault exists\n"
action = "update"
end
vault_command = "knife vault #{action} #{vault} #{item} --mode client --#{type} #{filePath} -A 'reese' -S 'role:#{role}' "
vault_command_out = shell_out(vault_command)
puts vault_command_out.stdout
end
end
end
#for each secret file, add an entry to the vault
secrets = {
"emmet" => "emmet.pub",
"wildstyle" => "wildstyle.pub"
}
secrets.each do |secret_name,filename|
filePath = File.join(secrets_path, filename)
type = "file"
role = "keyholder-bootstrap"
update_vault(vault_name, secret_name, filePath, type, role)
end
vault_name = 'config'
#for each json secrets set, add an entry to the vault
jsonSecrets = {
"service_config" => "services.json"
}
jsonSecrets.each do |secret_name,filename|
filePath = File.join(secrets_path, filename)
type = "json"
role = "lego-app-bootstrap"
update_vault(vault_name, secret_name, filePath, type, role)
end
And here's an example services.json:
{
"password":"spaceship",
"username":"benny"
}
So since we run update-vault with chef-zero, it runs the knife commands using the local knife client.
The actual 'lego-app' and 'keyholders' recipes run as usual on the machines.
#secrets/recipes/keyholders.rb
#allow use of chef_vault_item resource
include_recipe "chef-vault"
users = %w(emmet wildstyle)
vault = "credentials"
users.each do |username|
user username do
action :create
end
dir = "/home/#{username}/.ssh"
directory dir do
owner username
group username
mode 0755
action :create
end
private_key = chef_vault_item(vault, username)
file File.join(dir,"authorized_keys") do
content private_key['file-content']
owner username
group username
mode "0600"
action :create
end
end
#secrets/recipes/lego-app.rb
include_recipe "chef-vault"
vault = "config"
secret_name = "service_config"
config = chef_vault_item(vault, secret_name)
#these attributes could be used in a template or something
node.set['service']['password'] = config['password']
node.set['service']['username'] = config['username']
#alternatively, do stuff directly with the secrets
username = config['username']
execute "configure_amazing_service" do
command "echo username=#{username} > /usr/local/etc/service.conf"
end
To run the recipe, use chef-zero and run the following command
chef-client -z -o secrets::metalThe 'metal' recipe should create 2 servers with empty run lists, one micro for development and one medium for production. From there, it will update the vaults, allowing the two new machines to access items if they are in the right role. Finally, it will apply the roles with the run lists to create the users and install/configure the application.
Let me know if I missed anything important or can do things a better way.
