Exporting and importing resources

Puppet approaches the problem of sharing configuration information among multiple agent nodes by way of exported resources. The concept is simple. The manifest of node A can contain one or more resources that are purely virtual and not for realization in the manifest of this node A. Other nodes, such as B and C, can import some or all of these resources. Then, the resources become part of the catalogs of these remote nodes.

The syntax to import and export resources is very similar to that of virtual resources. An exported resource is declared by prepending the resource type name with two @ characters:

@@file { 'my-app-psk': 
  ensure  => file,
  path    => '/etc/my-app/psk', 
  content => 'nwNFgzsn9n3sDfnFANfoinaAEF', 
  tag     => 'cluster02', 
}

The importing manifests collect these resources using an expression, which is again similar to the collection of virtual resources but with double-angled brackets, < and >:

File<<| tag == 'cluster02' |>>

Tags are a very common way to take fine-grained control over the distribution of such exported resources.

Configuring the master to store exported resources

The only recommendable way to enable support for exported resources is PuppetDB. It is a domain-specific database implementation that stores different kinds of data that the Puppet master deals with during regular operation. This includes catalog requests from agents (including their valuable facts), reports from catalog applications, and exported resources.

Chapter 2, The Master and Its Agents, detailed a manual installation of the master. Let's add the PuppetDB with more style—through Puppet! On the Forge, you will find a convenient module that will make this easy:

puppet module install puppetlabs-puppetdb

On the master node, the setup now becomes a one-line invocation:

puppet apply -e 'include puppetdb, puppetdb::master::config'

As our test master uses a nonstandard SSL certificate that is named master.example.net (instead of its FQDN), it must be configured for puppetdb as well:

include puppetdb
class { 
  'puppetdb::master::config': 
    puppetdb_server => 'master.example.net' 
}

The ensuing catalog run is quite impressive. Puppet installs the PostgreSQL backend, the Jetty server, and the actual PuppetDB package, and it configures everything and starts the services up—all in one go. After applying this short manifest, you have added a complex piece of infrastructure to your Puppet setup. You can now use exported resources for a variety of helpful tasks.

Exporting SSH host keys

For homegrown interactions between clustered machines, SSH can be an invaluable tool. File transfer and remote execution of arbitrary commands is easily possible thanks to the ubiquitous sshd service. For security reasons, each host generates a unique key in order to identify itself. Of course, such public key authentication systems can only really work with a trust network, or the presharing of the public keys.

Puppet can do the latter quite nicely:

@@sshkey { $::fqdn: 
  host_aliases => $::hostname, 
  key          => $::sshecdsakey, 
  tag          => 'san-nyc' 
} 

Interested nodes collect keys with the known pattern:

Sshkey<<| tag == 'san-nyc' |>>

Now, SSH servers can be authenticated through the respective keys that Puppet safely stores in its database. As always, the Puppet master is the fulcrum of security.

Managing hosts files locally

Many sites can rely on a local DNS infrastructure. Resolving names to local IP addresses is easy with such setups. However, small networks or sites that consist of many independent clusters with little shared infrastructure will have to rely on names in /etc/hosts instead.

You can maintain a central hosts file per network cell, or you can make Puppet maintain each entry in each hosts file separately. The latter approach has some advantages:

A manually maintained registry is prone to become outdated every once in a while. It will also obliterate local additions in any hosts files on the agent machines.

The manifest implementation of the superior approach with exported resources is very similar to the sshkey example from the previous section:

@@host { $::fqdn: 
  ip           => $::ipaddress,
  host_aliases => [ $::hostname ],
  tag          => 'nyc-site',
}

This is the same principle, only now, each node exports its $ipaddress fact value alongside its name and not a public key. The import also works the same way:

Host<<| tag == 'nyc-site' |>>

Automating custom configuration items

Do you remember the Cacti module that you created during the previous chapter? It makes it very simple to configure all monitored devices in the manifest of the Cacti server. However, as this is possible, wouldn't it be even better if each node in your network was registered automatically with Cacti? It's simple: make the devices export their respective cacti_device resources for the server to collect:

@@cacti_device { $::fqdn: 
  ensure => present, 
  ip     => $::ipaddress, 
  tag    => 'nyc-site', 
} 

The Cacti server, apart from including the cacti class, just needs to collect the devices now:

Cacti_device<<| tag == 'nyc-site' |>>

If one Cacti server handles all your machines, you can just omit the tag comparison:

Cacti_device<<| |>>

Once the module supports other Cacti resources, you can handle them in the same way. Let's look at an example from another popular monitoring solution.

Simplifying the configuration of Nagios

Puppet comes with support to manage the complete configuration of Nagios (and compatible versions of Icinga). Each configuration section can be represented by a distinct Puppet resource with types such as nagios_host or nagios_service.

Each of your machines can export their individual nagios_host resources alongside their host and cacti_device resources. However, thanks to the diverse Nagios support, you can do even better.

Assuming that you have a module or class to wrap SSH handling (you are using a Forge module for the actual management, of course), you can handle monitoring from inside your own SSH server class. By adding the export to this class, you make sure that nodes that include the class (and only these nodes) will also get monitoring:

class site::ssh { 
  # ...actual SSH management... 
  @@nagios_service { "${::fqdn}-ssh": 
    use       => 'ssh_template', 
    host_name => $::fqdn, 
  } 
} 

You probably know the drill by now, but let's repeat the mantra once more:

Nagios_service<<| |>>

With this collection, the Nagios host configures itself with all services that the agent manifests create.

Maintaining your central firewall

Speaking of useful features that are not part of the core of Puppet, you can manage the rules of your iptables firewall, of course. You need the puppetlabs-firewall module to make the appropriate types available. Then, each machine can (among other useful things) export its own required port forwarding to the firewall machines:

@@firewall { "150 forward port 443 to ${::hostname}": 
  proto       => 'tcp', 
  dport       => '443', 
  destination => $public_ip_address, 
  jump        => 'DNAT', 
  todest      => $::ipaddress, 
  tag         => 'segment03', 
}

The title of a firewall rule resource conventionally begins with a three-digit index for ordering purposes. The firewall machines collect all these rules naturally:

Firewall<<| tag == 'segment03' |>>

As you can see, the possibilities for modeling distributed systems through exported Puppet resources are manifold. The simple pattern that we've iterated for several resource types suffices for a wide range of use cases. Combined with defined resource types, it allows you to flexibly enable your manifests to work together in order to form complex cluster setups with relatively little effort. The larger your clusters, the more work Puppet lifts from you through exports and collections.

Removing obsolete exports

When a node manifest stops exporting any resource, that resource's record is removed from PuppetDB once the node's catalog is compiled. This usually happens when the agent checks in with the master.

However, if you take an agent out of commission permanently, this will never happen. That's why you will need to remove those exports from the DB manually. Otherwise, other nodes will keep importing the old resources.

To clean such records from PuppetDB, use the puppet node deactivate command on the master server:

puppet node deactivate vax793.example.net