Building secure CI/CD pipeline with Powershell DSC. Part 2: CI for IaaC

In the previous post, I described how to build CI-as-a-code with DSC Pull server with a focus on the security by using partial configurations and updating the on-demand.

Here comes the next challenge. Now, we want the DSC States that define Infrastructure configuration to be easily modifiable and deployable. As if we wanted to patch against Wannacry simply by adding a corresponding Windows update patch to the DSC Security state. “git push” – and in a short while all nodes in the CI pipeline would be secured. Below, I show how it can be done with script examples that can help you kick-start your CI for IaaC.

Before we start…

First, I’ll remind you some terminology.

DSC Configuration (State) is a Powershell-compiled file that needs to be placed on the Pull server. It defines the system configuration and is expected to get updates frequently.

Local Configuration Manager (LCM) is a Powershell-compiled file that has to be deployed to a target node. This file tells the node about the Pull server and which Configurations to get from it. This file is used once to register the node. Updates to LCM are rare and happen only in the case when the Pull server needs to be changed or new states added to the node configuration. However, we previously split the states into three groups – CI configuration state, Security state, and General system configuration. Given this structure, you can update only states without creating new types of them. Therefore, no LCM changes are required.

Also, keep in mind the fact that LCMs are totally different for targets with Powershell v4 (WMF4) and v5. And you need different machines to build them.

Looking from the delivery perspective, the difference between LCM and States is that LCMs need to be applied with administrative permissions and require running some node-side Powershell. In one of my previous posts, you can find more info on the most useful cmdlets for LCM.

On the contrary, States are easy to update and get to work – you only need to compile and drop them to the Configurations folder of the Pull server. No heavy-weight operations required. So, we design our CI for IaaC for frequent deployment of States in mind.

Building CI for IaaC

For the CI the first thing is always the source control. One of my former colleagues loved to ask the question at interviews: “For which type of a software project would you use source control?” And one and the only correct answer was: “For absolutely any”.

So, we don’t want to screw up that interview, and also our CI pipeline, therefore, we got the DSC States and LCMs under source control. Next, we want States and LCMs to be built on the code check-in time. The states will be published to the Pull server immediately,  while LCMs can be stored on the secure file share without direct applying them to the node.

ci_for_dsc

Building the artifacts is not a big deal – I’ll leave the commands out of this blog post. But what is still missing is how the nodes get LCMs. My way of doing it is to have a script that iterates over nodes and applies corresponding LCMs from the file share to them. I call it “Enroll to DSC”. Which is pretty fair since it happens when we need either to enroll a new node to a server or get some new states into it.

Here is the example of such script that uses Windows remoting in place from my Github. You can find details in README.md

Summary

By creating CI for IaaC we bring the best of DevOps practices to the way we handle the build infrastructure. In fact, having an abstract CI in place already simplifies our job, and after we are done – the CI itself becomes more reliable and controllable structure. You can deliver any updates to CI with CI it within seconds – isn’t it what CI supposed to be for? Quite a nice recursion example, I think 🙂

Install Powershell 5.0 on Windows 8.1 with a non en-US locale

Windows Management Framework 5 (aka Powershell 5) fails to install if your Windows was installed with a different locale than en-US. In my case, it was en-GB, so it is not a big deal, right?

Well, not exactly. After downloading WMF 5.0 update package,  it fails to apply the update   – saying “Update is not applicable to your computer”. Do not expect to get something more verbose, neither find any useful information in system logs.

After desperate surfing through MS support tickets and trying different fixes, it turned out that the last suspect was the locale configuration. Ironically, that MS support engineers from Seattle couldn’t verify the problem since they all have en-US Windows installed.

So far, the only working way to install WMF 5 (which means, PS 5.0) if it fails to apply the update is to change the locale setting – which is a non-trivial task. It requires running the system DISM utility in the Offline mode (when current Windows installation is not loaded). Also, it requires obtaining en-US language pack .cab archive. And finally, you may even brick your boot configuration if don’t run it properly. Sounds exciting, let’s start then!

  1. Set the default language and interface language to en-US (Control Panel – Language – Advanced Options)
  2. Prepare a bootable Win 8.1 USB installation drive – I used the same image as for the initial installation. Just write it to USB (Win32DiskImager is a great tool for this).
  3. Download en-US language pack. It can’t be found as a separate package from the official resources. What I did was to use MSDN subscription downloads page and grab an installation media of Windows 8.1 Language Pack – it is DVD with a bunch of language packs on it. After, mount the ISO and navigate to “langpacks/en-US”. save the .cab file from it to the convenient location on your drive. I.e. C:\lp.cab
  4. Boot into troubleshooting mode with command prompt – from running Windows session, press Restart while pressing the “Shift” key. The system will log out and troubleshooting options menu will be loaded from the USB. troubleshoot1 Navigate to Troubleshoot -> Advanced options -> Command Prompttroubleshoot
  5. In the command prompt, run the DISM utility: dism /Image:C:\ /Add-Package /PackagePath:C:\lp.cab.
  6. Do not change the locale here. It is possible and sometimes described as one of the steps to apply, i.e using dism ... /Set-Syslocale , but better don’t. – it made my machine to fail to boot until I reverted this fix back.
  7. Boot normally – the language pack was installed but not yet applied. Open “Control Panel\All Control Panel Items\Language”, select “English – United States” and click “Options” on the right side. Under “Windows display language” there will be a link to set the current locale to it. In a different situation, I’ve seen it being done from the “Control Panel -> Language -> Advanced Settings -> Change Locale” menu.
  8. After signing in and out – you may check that the locale has been changed from the elevated command prompt: `dism /Online /Get-Intl`.troubleshoot3
  9. Now, the WMF 5 update can be applied – it might firstly install some sort of fix and ask for a reboot. Afterward, run the installer again – and you will get your precious PS 5.0

 ps

This is it! I hope, MS folks will fix this issue soon – so the update can be applied to a system with any locale.

DSC Pull Server for managing… Linux nodes

Let’s face it, Powershell DSC and Linux is definitely not the most favorite duo when speaking about CM. In this article, I will explain how to set up Linux node with existing DSC Pull Server and why this is a win-win for Win-dominated tech stack.

Powershell DSC is well-documented for Windows, you may find some information about using it in the Push mode (when the configuration is applied TO the node FROM the server via CIM session) but Linux Pull Server is rarely getting more than just a paragraph of attention. So, I’m here to fix it.

The main prerequisite of this exercise is a configured DSC Pull Server – you may want to check out this excellent MSDN tutorial. We run it on Windows Seever 2012 R2, with WMF (Windows Management Framework 5.0). The hostname for the means of this exercise is set to DSCPULLSERVER. Also, you got a Linux node – in rhis case, CentOS 7, with a hostname: DSCPULLAGENT.

The first discovery about DSC On Linux is that the client doesn’t run in Powershell for Linux. It is actually a set of Python and shell scripts that provide an interface to OMI (Open Management Infrastructure) CIM Server and control an execution of pulled configurations. Therefore, it works in a slightly different way comparing to “native” DSC.

To start with, we would need to obtain both OMI and DSC rpm’s in accordance with an $ openssl version installed – it might be either 0.98 (then  use rpms with ssl_98 in the name) or >=1.00 (use ssl_100 rpms).

rpm -Uvh https://github.com/Microsoft/PowerShell-DSC-for-Linux/releases/download/v1.1.1-294/dsc-1.1.1-294.ssl_100.x64.rpm
rpm -Uvh omi-1.1.0.ssl_100.x64.rpm

After installing the packages, DSC files can be found under /opt/microsoft/dsc/.

The next step is to prepare a DSC Local Configuration manager file. It tells DSC client where to pull the configuration from and which configuration. To generate it (and all kinds of other .mof files), I use a separate Windows machine.

Here is an example PS DSC script that creates a local configuration mof.

[DSCLocalConfigurationManager()]
configuration LinuxPullAgent
{
Node localhost
Settings {
RefreshMode = 'Pull'
ConfigurationMode = 'ApplyAndAutocorrect'
}
ConfigurationRepositoryWeb main {
ServerURL = 'https://DSCPULLSERVER:8080/PSDSCPullServer.svc'
RegistrationKey = "xxx-xxx-xxxxxx-xxxxxx"
ConfigurationNames = @('DSCLINUXSTATE')
}
}
}
LinuxPullAgent

Here, DSCLINUXSTATE is the name of the configuration placed to a Configurations folder of the DSC Pull Server. RegistrationKey is the key you obtain from the Pull Server. Also, you can specify here how ofter to pull the configuration, and a bunch of other parameters – read more here.

After running this snippet on a machine with PS 4.0 or higher, it produces a local configuration manager file localhost.mof that you need transfer to the Linux node. Basically, it should be the same file for all your Linux nodes using the DSCLINUXSTATE configuration.

Assume, the transferred file is located under ~/localhost.mof. Now, you need to install this Pull Server configuration on the node. We navigate to /Scripts where you can found various Python scripts for different tasks – for example, do not run Register.py, it is intended for use with Azure DSC server. Also, GetDscConfiguration.py and StartDscConfiguration.py – are the scripts for running DSC in the Push mode.

So here we are interested only in SetDscLocalConfigurationManager.py. This script performs a configuration of the pull agent basing on the .mof files generated for this agent. The second command in the next listing displays all the parameters of an active configuration of the LCM and helps to troubleshoot issues related to it.

SetDscLocalConfigurationManager.py -configurationmof ~/localhost.mof
GetDscLocalConfigurationManager.py

After installing the configuration, the pulls are done periodically with a 30-minute minimal interval. To trigger the pull of the configuration manually we need to execute

/opt/microsoft/dsc/bin/ConsistencyInvoker

This command pulls the configuration state from the pull server that we just installed and check if the current state corresponds to it.

And this is it! If everything went well, your Linux is now managed by the DSC. Using nx, nxNetworking and nxComputerManagement modules from Powershell Gallery, you can cover basic scenarios of configuration for Linux machines.

You can troubleshoot the issues related to DSC states or server connectivity by checking the error messages in the log file /var/opt/omi/log/dsc.log. OMI server instance log is located under /var/opt/omi/log/omiserver.log.

Pros and cons of DSC on Linux:

  • Pros:
    • Powershell and DSC ecosystem – very convenient when the company’s tech stack mainly includes Windows machines. Managing Linux with the same set of technologies sounds like a dream for decreasing complexity and adding transparency to the CM solution.
    •  Simplicity configuration of the Pull model – server only provides the state, the node does all the work.
    • Extensibility of DSC framework – allows to create custom scripts and utilize existing modules, but for Linux!
    • Microsoft’s open-source direction – it looks very determined, and after Powershell release for Linux, I believe, the integration between these two worlds will only improve, adding more functionality and features to both. Therefore, it might be that tomorrow DSC for Linux will be the next big thing.
    • Free of charge comparing to 120$+/node for Puppet or Chef.
  • Cons:
    • nx family of modules is still limited comparing to what can do native CM tools (Puppet, Chef, Ansible, Salt).
    • Poor logging and transparency of a node state – it is hard to trace the results of applying states, especially from the DSC Pull server. Logs need to be examined on the agent. Also, requires setting up a reporting server for DSC which is also not being the most user-friendly interface.
    • Requires an orchestration tool for machine deployment (in the case of using VMs). DSC can provision a new node by itself.

In general, it worked for us quite well – we were aware of general DSC limitations and didn’t find anything unexpected in its Linux specific implementation – given our tech stack it turned out to be a perfect solution.

In the next post, I will cover the examples of using nx modules for configuring CentOS node.

4 ways of developing DevOps competences

I’ve been in DevOps for about 5 years so far and recently summed up the main ways of learning in this business.

The main challenges of DevOps learning are related to extremely wide technology and skills coverage that are presumed to be useful in this occupation.

While the toolset of DevOps highly depends on the technology stack, there are common areas of knowledge that (basing on my fails and trials) turned out to be crucial for system vision and approach.

First of all, DevOps is not only about development, as we can see from the name but also there is an Ops part. Therefore, any reading related to Operations Management is beneficial. You need to know how the company works from the inside out in order to be efficient in DevOps when improving delivery processes. One of my favorites are:

They will tell you how to build operations of your company as a factory – and you may argue that you work in lean/agile/cool startup and have nothing to do with blue collars from industrial areas – and would be mistaken. It is important to start looking at work being done by your company, as at something that you can measure and improve – not ephemerally but by means of time,  throughput and quality.

The second thing I learned about DevOps – is to always keep asking yourself – am I doing a right thing? Am I delivering value and improving the processes?  Do it all the time even out of work – for example, when attending tech conferences and reading literature. Try to make sure you are on track with the latest trends, and here I mean being on track conceptually. Do not try to chase each and every new tools or release – it rarely makes a really big difference comparing to using the old ones. Concentrate on following the concepts – whenever someone introduces a new way to doing DevOps – it is time to start digging in. When you introduce new concepts and take the best ideas to work – you change the rules of the game and you may get much better results than simply replacing the tools.

The third point – is to constantly and continuously improve your OWN operations. Think about how you handle the work that lands on your desk, think about yourself as a factory and the output that you produce. What do you need to improve? You may start with Time management, Memory skills, and Communication.

And the last, fourth way of learning is to always communicate your ideas – in blogs, comments, discussions with colleagues, and see how people react to them. This will help you understand the main pain points and weak spots of your concepts and improve them, with an assistance of others and from their perspective.

Ansible: loops and registering results of commands

Ansible has a great feature of registering results of a performed step. But recently I stumbled upon a need to do it against actions from a loop and then using the registered result in the next steps which are also looped. Most often, it can be used to perform some action only in case if a previous action changed a file.

Firstly, we register `action_state` variable. Its sub-dictionary `results` contains attribute `changed` which can be accessed as `action_state.results.changed`. Now, we need to get it into use in the next looped step. It can be done with ease using `with_together` operator:

- name: copy a file
  copy:
    src: '{{ source }}'
    dest: '{{ target }}'
  with_items: yourlist
  when: item.state == 'present'
  register: action_state

- name: create .ssh dir if file changed in the previous step
  sudo_user: '{{ item.0.user }}'
  file:
    state: directory
    path: '/home/{{ item.0.user }}/.ssh'
    mode: 0700
  with_together:
  - yourlist
  - action_state.results
  when: item.1.changed and item.0.state == 'present'

Here, for the `yourlist` we copied some files to a target machine (i.e. public keys to some temp location) in the loop and then for each of the files performed an action only if a file was changed.

Removing unused Docker containers

Those who work with Docker know how fast is free space disappearing, especially, in case you are experimenting with some new features and don’t pay much attention to cleaning up. As the result, a lot of empty untagged containers can eat tens of gigabytes of space.

Here is the one-liner I found to be very useful for removing old stuff:

docker ps -a | grep 'weeks ago' | awk '{print $1}' | xargs --no-run-if-empty docker rm

Actually, I’m not the one who came up with this answer, but I spent a few hours trying different commands and googling for the answer.

Here is the full Docker cheat sheet that I bumped into and found it being quite good itself:

https://github.com/wsargent/docker-cheat-sheet

Using renderables in Buildbot custom steps

Currently, I’m being widely involved in customization of Buildbot instances. Many questions arise, but no answers are usually available on the Web.

Recently, I designed a custom build step which by the definition needed to work with values from properties passed to it on the init time. The challenge was to make it understanding everything that can theoretically be passed – build properties (IProperty), plain values etc.

In this case, we can use ‘renderables’ list to specify mapping from variables that can contain properties to variables that are used to store rendered values.

We can use passed values without caring to handle it as property or some other renderable. Buildbot does it automatically

class MyStep(BuildStep):
    ...
    renderables = ['renderable1' , 'renderable2']

    def __init__(self, passed_var1, passed_var2):
        self.renderable1 = passed_var1
        self.renderable2 = passed_var2

Invoke the step in the build factory as:

MyStep(util.Prop(property1), util.Prop(property2))

or

MyStep('actual value 1', 'actual value 2')

Now we can use passed values simply as without caring to handle it as property or some other renderable. Buildbot does it automatically.