Thursday, September 11, 2014

Bootstrapping and Secrets Management: solving the chef-vault chicken/egg problem with chef-metal

Intro

Automating server creation with Chef can be very satisfying when it works well. It saves time and avoids the need for tedious and repetitive tasks. Distribution of secrets to servers created this way has always been a hassle for me (or insecure, if laziness gets the best of me). Chef-vault does some fancy things with encrypted data bags and the public/private key system used by chef. When an item is updated in the vault, it encrypts the data for clients using their public keys from chef, allowing those clients to decrypt it with their private keys. The problem with this is, the client needs to exist before the vault is updated, but you also want the client to access things in the vault while it is running its initial run list. In this post I'll cover how I'm currently solving this problem with chef-metal and chef-vault. For those unfamiliar, chef-metal allows you to create machines, clusters, full stacks, or whatever, using the same type of code you are already familiar with in configuring machines.


Too long; not going to read

A single chef-metal recipe does the following
  1. creates an instance with a role and empty run list
  2. chef-vault is updated using said role
  3. instance gets a new run list


Disclaimer

My knowledge of ruby is pretty limited, and this whole method seems a little convoluted. I'm hoping someone will come along and show me a better way.


Tutorial

First, to get chef-metal to interact with your virtualization provider of choice, you need to set a driver. This can be done in knife.rb as follows:

...
driver 'fog:AWS:example'
...

I'm using AWS, so I also have a file ~/.aws/config with profiles for different accounts. In this case, I've specified the 'example' profile

...
[profile example]
aws_access_key_id=ABC
aws_secret_access_key=123
region=us-west-2
availability_zone=us-west-2b

chef-vault also needs to know whether it should act in 'client' or 'solo' mode. Put something like this in your knife.rb as well (replacing values as needed)

...
knife[:vault_mode] = "client"
knife[:vault_admins] = [ "reese" ]
...

Now we need a recipe that bootstraps a machine, updates the vault, and then adds the application to the machine's run list once it can access items in the vault.

#secrets/recipes/metal.rb

%w{chef-metal chef-metal-fog fog}.each do |pkg|
    chef_gem "#{pkg}" do
        action :install
    end
end
require 'chef_metal_fog'
#require "cheffish"

#specify where the chef server is
with_chef_server 'https://chef.example.com',
    :client_name => 'reese',
    :signing_key_filename => '~/.chef/reese.pem'


#create a key pair for the default admin user (ec2-user in this case)
fog_key_pair "admin"

#put options in a variable for easier overriding for different environments
options = {
    :sudo => true,
    :ssh_username => 'ec2-user',
    :sudo => true,
    :bootstrap_options => {
        :key_name => 'admin',
        :availability_zone => 'us-west-2b',
        :security_group_ids => %w(sg-12345678), 
        :subnet_id => 'subnet-abcdef12', 
        :flavor_id => 't2.micro',
        :image_id => 'ami-d13845e1', #amazon linux ami, us-west t2.micro
        :virtualization_type => 'hvm' #required for t2.micro
    }
}

with_machine_options options

machine "lego-app-dev" do
    chef_environment "development"
    role 'keyholder-bootstrap'
    role 'lego-app-bootstrap'
    action [:converge]
end

def deep_clone(o)
    Marshal.load(Marshal.dump(o))
end

production_options = deep_clone(options) #someone good at ruby, tell me there's a better way
production_options[:bootstrap_options][:flavor_id] = "m3.medium"
production_options[:bootstrap_options][:image_id] = "ami-d13845e1"

machine "lego-app1" do
    machine_options production_options
    chef_environment "production"
    role 'lego-app-bootstrap'
    action [:converge]
end

include_recipe "secrets::update-vault"

machine "lego-app-dev" do
    role 'keyholder-bootstrap' #is this needed? or does it remain from before?
    role 'keyholder'
    role 'lego-app-bootstrap'
    role 'lego-app'
    converge true
end

machine "lego-app1" do
    role 'lego-app-bootstrap'
    role 'lego-app'
    converge true
end

Check out the documentation to see more driver/machine configuration options

I want to deliver sensitive data only to the clients which need it, so I have an empty 'bootstrap' role to go along with each role that accesses a set of secrets. These non-empty roles (e.g. 'keyholder') trivially run a single recipe with a name similar to the role. In this case, the role 'keyholder' lets emmet and wildstyle log in, while the role 'lego-app' runs the actual application.

The update-vault recipe takes care of making sure each client can access the items it should be able to. This code is probably cringeworthy (but it works, as far as I can tell).

#secrets/recipes/update-vault.rb
chef_gem "chef-vault" do
    action :install
end

secrets_path = "~/.chef/secrets"
vault_name = 'credentials'

#create or update the vault
def update_vault(vault, item, filePath, type, role)
    ruby_block "check_vault_#{vault}_#{item}" do
        block do
            #probably a terrible way to decide whether to create or update the vault
            Chef::Resource::RubyBlock.send(:include, Chef::Mixin::ShellOut)
            #see if item exists in vault
            command = 'knife vault show '+vault+' '+item
            command_out = shell_out(command)
            #check if command resulted in an error
            if command_out.stderr.to_s != ''
                puts "\ncreating vault and/or vault item\n"
                action = "create"
            else
                puts "\nvault exists\n"
                action = "update"
            end
            vault_command = "knife vault #{action} #{vault} #{item} --mode client --#{type} #{filePath} -A 'reese' -S 'role:#{role}' "
            vault_command_out = shell_out(vault_command)
            puts vault_command_out.stdout
        end
    end
end

#for each secret file, add an entry to the vault
secrets = {
    "emmet" => "emmet.pub",
    "wildstyle" => "wildstyle.pub"
}

secrets.each do |secret_name,filename|
    filePath = File.join(secrets_path, filename)
    type = "file"
    role = "keyholder-bootstrap"
    update_vault(vault_name, secret_name, filePath, type, role)
end

vault_name = 'config'

#for each json secrets set, add an entry to the vault
jsonSecrets = {
    "service_config" => "services.json"
}
jsonSecrets.each do |secret_name,filename|
    filePath = File.join(secrets_path, filename)
    type = "json"
    role = "lego-app-bootstrap"
    update_vault(vault_name, secret_name, filePath, type, role)
end

And here's an example services.json:
{
    "password":"spaceship",
    "username":"benny"
}

So since we run update-vault with chef-zero, it runs the knife commands using the local knife client.

The actual 'lego-app' and 'keyholders' recipes run as usual on the machines.

#secrets/recipes/keyholders.rb
#allow use of chef_vault_item resource
include_recipe "chef-vault"

users = %w(emmet wildstyle)
vault = "credentials"

users.each do |username|

    user username do
        action :create
    end

    dir = "/home/#{username}/.ssh"
    directory dir do
        owner username
        group username
        mode 0755
        action :create
    end

    private_key = chef_vault_item(vault, username)
    file File.join(dir,"authorized_keys") do
        content private_key['file-content']
        owner username
        group username
        mode "0600"
        action :create
    end
end


#secrets/recipes/lego-app.rb
include_recipe "chef-vault"

vault = "config"
secret_name = "service_config"
config = chef_vault_item(vault, secret_name)

#these attributes could be used in a template or something
node.set['service']['password'] = config['password']
node.set['service']['username'] = config['username']

#alternatively, do stuff directly with the secrets
username = config['username']
execute "configure_amazing_service" do
    command "echo username=#{username} > /usr/local/etc/service.conf"
end

To run the recipe, use chef-zero and run the following command
chef-client -z -o secrets::metal
The 'metal' recipe should create 2 servers with empty run lists, one micro for development and one medium for production. From there, it will update the vaults, allowing the two new machines to access items if they are in the right role. Finally, it will apply the roles with the run lists to create the users and install/configure the application.
Let me know if I missed anything important or can do things a better way.


Inspiration/Credits

The method used here was inspired by the orchestration scenario presented on this page and by this tweet conversation which I didn't fully understand.

6 comments:

  1. 세마스포츠마케팅은 14일 솔레어 리조트 앤 카지노와 박성현의 후원 계약 조인식을 개최할 예정이다. 국내 팬들도 박성현도 잘 알지 못했던 솔레어 리조트 앤 카지노는 거액을 투자한 홍보 효과를 톡톡히 누리고 있다. 이를 지켜보는 국내 팬들과 업계의 시선은 마냥 편하지만은 않다. 카지노가 합법인 미국에서도 LPGA 투어나 미국프로골프 투어 선수들이 카지노 기업의 로고를 달고 필드를 누비는 사례는 거의 찾아 볼 수 블랙잭 없다.

    ReplyDelete
  2. I read your content. It's informative and interesting information about management. Thanks for sharing your content with us. Now it's time to avail Diy Shutters for more information.

    ReplyDelete
  3. When it comes to automating server creation with Chef, the benefits are undeniable. It eliminates repetitive tasks and saves valuable time. However, one aspect that has always posed a challenge for me is the secure distribution of secrets to these servers. That's where Chef-vault comes in, employing encrypted data bags and a reliable public/private key system. It's a powerful solution for updating and encrypting data, ensuring clients can access the vault while running their initial run list. In my experience, combining Chef-metal and Chef-vault has effectively tackled this assignment writing oman challenge.

    ReplyDelete
  4. Although automating server creation with Chef and managing secrets may not directly relate to "law dissertation topics," it's crucial to explore relevant legal themes such as data privacy, cybersecurity regulations, or intellectual property rights in the context of emerging technologies. These areas provide intriguing avenues for law dissertation topics, contributing to the understanding and implementing of legal frameworks in a technology-driven world.

    ReplyDelete
  5. Bootstrapping and secrets management, providing readers with practical guidance for building secure and scalable software systems. The author's depth of knowledge on the subject is evident, as they navigate through the nuances of handling sensitive information in a variety of contexts. Whether you're a seasoned developer or just starting out, this post equips you with the tools and strategies needed to succeed in today's rapidly evolving tech landscape.
    By: write my dissertation

    ReplyDelete
  6. Discover innovative solutions for managing server creation and secret distribution efficiently with Chef. This insightful post explores using Chef-vault alongside Chef-metal to streamline processes, ensuring secure access to encrypted data for servers in real-time. Ideal for those seeking robust solutions in server automation and data security. For more insights on college assignment Writing Service, delve into this detailed exploration.

    ReplyDelete