Building secure CI/CD pipeline with Powershell DSC. Part 1: overview

From my experience, way too often CI/CD pipelines suffer from the lack of security and general configuration consistency. There still might be an IaaC solution in place but it usually focuses on delivering a minimal functionality that is required for building a product and/or recreating the infrastructure if needed as fast as possible. Only a few of CI pipelines were built with security in mind.

I liked Powershell DSC for being native to the Windows stack and intensively developing feature modules to avoid gloomy scripting and hacking into the system’s guts. This makes it a good choice for delivering IaaC with Windows-specific security in mind.

DSC crash course

First, a short introduction of Powershell DSC in the Pull mode. In this configuration, DSC States or Configurations are deployed to and taken into use by the Pull server. States define what our node system configuration needs to look like, which features to have, which users to be admins, which apps installed etc – pretty much anything.

Configuration FirewallConfig
{


Import-DscResource -ModuleName PSDesiredStateConfiguration -ModuleVersion 1.1
Import-DscResource -ModuleName xNetworking -ModuleVersion 3.1.0.0

 xFirewall TCPInbound
 {
     Action = 'Allow'
     Direction = 'Inbound'
     Enabled = $true
     Name = 'TCP Inbound'
     LocalPort = '443'
     Protocol = 'TCP' 
 }

}

Each State needs to be built into the configuration resource of specific “.mof” format. Then, the state.mof need to be placed into “Configurations” folder on the Pull server together with its checksum file. Once the files are there, they can be used by nodes.

The second piece of config is the Local Configuration Manager file. This is the basic configuration for a Node that instructs it where to find the Pull server, how to get authorized with it and which states to use. More information is available in the official documentation.

To start using the DSC, you need to:

  • setup a pull server (once)
  • build a state .mof and checksum file and place it on the pull server (many times)
  • build a LocalConfigurationManager .mof and place it on the node (once or more)
  • instruct node to use LCM file (once or more)

After this, a node contacts the Pull server and receives one or more configuration according to its LCM. Then, a node starts a consistency against the states and correcting any difference it finds.

Splitting states from the Security perspective

It is the states that are going to be changed once we want to modify the configuration of enrolled nodes. I think it is a good practice to split a node state into pieces – to better control security and system settings of machines in the CI/CD cluster. For example, we can have the same security set of rules and patches that we want to apply to all our build machines but keep their tools and environment configurations corresponding to their actual build roles.

Let’s say, we split Configuration of Build node 1 into the following pieces:

  • Build type 1 state – all tools and environment settings required for performing a build (unique per build role)
  • Security and patches – updates state, particular patches we want to apply, firewall settings (same for all machines in the CI/CD)
  • General setting – system setting that (same for all machines in the CI/CD)

dsc_states

In this case, we make sure that security is consistent across CI/CD cluster no matter what role a build machine has. We can easily add more patches or rules, rebuild a configuration mof file and place it on the Pull server. The next time the node checks for the state, it will fetch the latest configuration and perform the update.

In the next post, I will explain how to build a simple CI pipeline for the CI – or how to deliver LCMs to nodes and configuration mof’s to Pull server with using a centralized CI server.

 

Install Powershell 5.0 on Windows 8.1 with a non en-US locale

Windows Management Framework 5 (aka Powershell 5) fails to install if your Windows was installed with a different locale than en-US. In my case, it was en-GB, so it is not a big deal, right?

Well, not exactly. After downloading WMF 5.0 update package,  it fails to apply the update   – saying “Update is not applicable to your computer”. Do not expect to get something more verbose, neither find any useful information in system logs.

After desperate surfing through MS support tickets and trying different fixes, it turned out that the last suspect was the locale configuration. Ironically, that MS support engineers from Seattle couldn’t verify the problem since they all have en-US Windows installed.

So far, the only working way to install WMF 5 (which means, PS 5.0) if it fails to apply the update is to change the locale setting – which is a non-trivial task. It requires running the system DISM utility in the Offline mode (when current Windows installation is not loaded). Also, it requires obtaining en-US language pack .cab archive. And finally, you may even brick your boot configuration if don’t run it properly. Sounds exciting, let’s start then!

  1. Set the default language and interface language to en-US (Control Panel – Language – Advanced Options)
  2. Prepare a bootable Win 8.1 USB installation drive – I used the same image as for the initial installation. Just write it to USB (Win32DiskImager is a great tool for this).
  3. Download en-US language pack. It can’t be found as a separate package from the official resources. What I did was to use MSDN subscription downloads page and grab an installation media of Windows 8.1 Language Pack – it is DVD with a bunch of language packs on it. After, mount the ISO and navigate to “langpacks/en-US”. save the .cab file from it to the convenient location on your drive. I.e. C:\lp.cab
  4. Boot into troubleshooting mode with command prompt – from running Windows session, press Restart while pressing the “Shift” key. The system will log out and troubleshooting options menu will be loaded from the USB. troubleshoot1 Navigate to Troubleshoot -> Advanced options -> Command Prompttroubleshoot
  5. In the command prompt, run the DISM utility: dism /Image:C:\ /Add-Package /PackagePath:C:\lp.cab.
  6. Do not change the locale here. It is possible and sometimes described as one of the steps to apply, i.e using dism ... /Set-Syslocale , but better don’t. – it made my machine to fail to boot until I reverted this fix back.
  7. Boot normally – the language pack was installed but not yet applied. Open “Control Panel\All Control Panel Items\Language”, select “English – United States” and click “Options” on the right side. Under “Windows display language” there will be a link to set the current locale to it. In a different situation, I’ve seen it being done from the “Control Panel -> Language -> Advanced Settings -> Change Locale” menu.
  8. After signing in and out – you may check that the locale has been changed from the elevated command prompt: `dism /Online /Get-Intl`.troubleshoot3
  9. Now, the WMF 5 update can be applied – it might firstly install some sort of fix and ask for a reboot. Afterward, run the installer again – and you will get your precious PS 5.0

 ps

This is it! I hope, MS folks will fix this issue soon – so the update can be applied to a system with any locale.

Uninstall MS Exchange Server 2016

Today, I read through the support thread of one guy who ended up paying to a customer support from MS to get rid of a nasty Exchange server.

And, I must admit, it is not something that can be removed with ease.

! The following manual will help you to annihilate the Exchange server and its data. If you still wish to save some of it, don’t use this instruction.

Let’s say, you need to completely wipe it out from the machine – and uninstaller always fails to remove so-called Default Mailboxes.

There are a few types of them which you need to take care of manually:

Get-Mailbox -Archive | Disable-Mailbox
Get-Mailbox -Monitoring | Disable-Mailbox
Get-Mailbox -AuditLog | Disable-Mailbox
Get-Mailbox -PublicFolder | Disable-Mailbox

Now,  you need to get rid of “-Arbitration” mailbox but it is not as easy as previous cases. Firstly, go and find out the name of your Mailbox Database:

Get-MailboxDatabase

It will show you something like “Mailbox Database 12212842873428”. Use the entire name, no0t just a number! Now, you can remove all Arbitration mailboxes:

Get-MailboxDatabase -Database "NAME FROM THE PREVIOUS COMMAND" -Arbitration | Disable-Mailbox -Arbitration

It will fail at the last mailbox which is using some default address book (are you still following the actual meaning of these error messages? Not sure if it is physically possible…)

So, to remove the default Offline Address Book we need to get its name. At least, this is what the support says. Forget the support, use the wild card!

Remove-OfflineAddressBook -Identity "*"

And finally:

Get-Mailbox -Database "NAME" -Arbitration | Disable-Mailbox -Arbitration -DisableLastArbitrationMailbox

Phew, that was it. Now, nothing can stop you from wiping out the Exchange server from your server.