Standard Operating Environment – Part I: Concepts and Structures

This article was originally posted on OpenSourceArchitects.org.

What is an SOE?

Modern IT departments face huge changes in the way they deploy and maintain servers. When I first entered the industry, UNIX, and hence Linux servers were in the parlance of cloud workloads, ‘pets’: few in number, lovingly tended machines that were individually configured and maintained by hand. They often ran many workloads at the same time (mail server, file server, database, shell accounts) on expensive hardware. A typical ratio of system administrators to servers might be 1:10 yet hardware costs, rather than labour, accounted for the greatest share of the IT department’s budget.

A typical current day bank may have 10,000 Linux servers deployed, with Sysadmin:server ratios as high as 1:1000. Modern deployments are largely treated as ‘cattle’: numerous, single workload devices that  should be deployed quickly, replaced quickly and require no manual configuration. Commodity servers and virtualisation have completely changed the economics of IT departments from being major capital cost centres to majority operating expenditure cost centres: labour costs, software licences, maintenance contracts now form the bulk of the expenditure of most IT organisations.

And yet the majority of IT departments are still stuck in the mindset of the 1990-2000s where servers are deployed manually or semi-manually from a “gold image”, are configured more or less manually, and are patched manually (if at all!) on an ad hoc basis.

A Standard Operating Environment aims to permit any IT organisation to adopt automation for the deployment and maintenance of their servers, to effect rapid patching and configuration change, to manage configuration drift, and to be able to respond rapidly to demands from their user base. The SOE consists of the following components:

  • A set of Concepts to define and describe the artefacts associated with the deployment and maintenance of operating infrastructure.
  • A set of Workflows to release and maintain standard builds, to maintain deployed services, and to automatically test software and configuration changes.
  • A set of Tools to maintain artefacts such as standard builds and configurations, to deploy these to servers, to track versions and history, and to perform automated testing.

SOE Concepts

SOE provides consistent definitions of various terms that can be used to describe infrastructure deployment and maintenance.

Build

Build is a set of installable software and configurations. A server is built using a specific build. The build may change over time as it is maintained, similarly servers built with a specific build may change in time as they are maintained or their configuration is altered.

In SOE, we use a single build to deploy different kinds of servers. Thus a webserver and a database server will be deployed using exactly the same build, only configuration will change. A builds consist of:

  • A base operating system e.g. Red Hat Enterprise Linux 7.2
  • Additional software such as application software, internally developed software, agents etc.
  • A set of configurations required to deploy servers into different Roles

Within Red Hat Satellite, builds are modelled using Content Views.

Build Version

Builds are versioned and any change to a build – for example the addition of a patch, results in a new version of the build being released. Every deployed server is registered to a specific build version, permitting drift analysis and management. To apply patches to a server, it is exposed to a later build version which contain the required patches.

Within Red Hat Satellite build versions are modelled using Content View Versions.

Role

Role is a specific software configuration that is directed at fulfilling a business role. An example role is “EMEA HR Production SAP Server”. This would be a server used to provide SAP server to the European HR division of an organisation. A server has only a single role. Within SOE, role is considered immutable: if a deployed server is required in a different role it is re-deployed from scratch. However, the definition of the role can change over time, and servers deployed into that role will change with the chaning definition.

A server role is defined by 3 broad areas:

  • The server is deployed with initial characteristics suitable for the role. These include physical characteristics such as RAM and number of CPUs, and operating system characteristics such as partition tables, specific OS versions, networking configuration etc.
  • The server is installed with all the software required to perform its role.
  • The server operating system and application software is correctly configured to perform its role.

Within Red Hat Satellite, roles are implemented using Hostgroups.

Profile

Roles tend to have a great deal of commonality – for example, server hardening, registration to monitoring system, and even the software to be installed will likely be common across roles. In order to encourage re-use, we introduce the concept of Profiles. (NB This concept of roles and profiles was proposed by Craig Dunn in this influential conference presentation.) A role contains one or more profiles. While a server can only have one role, that role will include one or more profiles. The following is a list of typical profiles:

  • Base server
  • Websphere server
  • SAP Cluster server
  • DMZ hardening
  • Netbackup client
  • Splunk client

Our fictional role “EMEA HR Production SAP Server” might include the following profiles: Base server, SAP Cluster server, Netbackup client, Splunk client.

In a Puppet environment, each profile would be implemented as a module. In an Ansible environment, each profile would be implemented, confusingly as a role. For the purposes of clarity, I will use the term Ansible-Role to distinguish the Ansible concept of a role from the SOE concept described above.

Build Lifecycle

A build goes through the following lifecycle stages:

Inception -> Development -> Release -> Maintenance -> Retirement

Organisations usually have multiple builds, each at different lifecycle stages at any one time. The stages are described in further detail below.

Inception Stage

The inception stage is the point at which a new build is initiated. There are several possible triggers for inception of a new build, however the majority of organisations use one of the two following schemes:

  • In a time-based scheme a new build is created at regular time intervals. For example an organisation may choose to release a new build every 6 months. NB Do not confuse this with patch frequency which will be described in the Maintenance Stage section.
  • In a Vendor release-based scheme the inception of a new build is triggered by an upstream vendor releasing a new version of software. For most Red Hat customers, this is simply when Red Hat releases a new major or minor version of Red Hat Enterprise Linux.

The name of the build should be settled on at the point of inception. The name of the build can be anything, however most organisations will want to settle on a specific naming convention. For vendor release-based build, the naming convention may be as simple as ACME-RHEL<maj>,<min>e.g. ACME-RHEL7.2. This indicates that this is the ACME company’s build of RHEL7.2.

For a time-based scheme, a common convention is ACME-RHEL<maj>.<min>-<date>e.g. ACME-RHEL7.2-20161027. This indicates that this is the ACME company’s build of RHEL7.2 incepted on the 27th October 2016. The datestamp is required if the build release period is shorter than the vendor’s software release period. I tend to use the datestamp of the inception stage, rather than the release date, as I usually do not know the release date in advance, and it may of course slip.

Development Stage

The development stage is the period during which a new build is designed, developed and tested. This consists of the following steps:

  • New Requirements are gathered. These will typically be design and functional requirements such as the need for the operating system build to support a specific end-user application, or function with specific hardware. The new requirement may simply be that the build be based on a newly released version of the base OS.
  • Requirements are condensed into a Task Backlog describing the work that needs to be done to create the new Build.
  • New base Build is created. Usually this will involve the cloning and updating of the existing build, however a build may be created from scratch. Within Red Hat Satellite, this would involve the creation of a new Content View, possible a copy of an existing content view. Kickstarts, configuration scripts, Ansible playbooks and tests will be held in a revision control system, usually Git. The branching mechanism of the revision control system is used to create a development stage branch.
  • The new build is Tested. Automated testing is key to rapid development of an SOE. A typical testing configuration is to use a group of virtual machines, one deployed into each of the Roles that the SOE supports. On every change to the SOE (for example a change in the Content Views, or configuration repositories), the test VMs are redeployed and acceptance and unit tests run against them. Re-deployment of test machine and test execution is handled using a Continuous Integration engine, usually Jenkins or Bamboo.
  • Development takes place iteratively until the Task Backlog is run down and all Tests are passing. Many organisation will use an Agile approach such as Scrumban to manage this phase.
  • The new Build is released.

Maintainance Stage

During the maintainance stage, the new build is continuously update. Updates come in a variety of flavours:

  • Vendor Patches are triaged and added to the build as required
  • Defects arising from end user issues may result in configuration and software changes
  • Enhancements resulting from end user requests may result in configuration and software changes to a current build rather than being deferred to the next build

Maintenance workflows need to answer several questions and are highly dependent on the organisation’s attitude towards risk and stability. These workflows will be expanded upon in detail in the next article in this series, but some of the questions that will be answered are listed below:

  • What changes need to be applied to the current build and what can be deferred to the next build?
  • How do I assess vendor patches for inclusion in the current build?
  • What is my schedule for releasing a new version of the current build?
  • How do I treat emergency patches that might need to be released outside of a formal schedule? For example a patch that is required to fix a critical security flaw, or a bug that is affecting a revenue-generating line of business?
  • After including a patch into the current build and incrementing the build version, what is my policy on actually applying the patch to deployed servers?

Retirement Stage

Once the build is retired, it will no longer be updated. No new servers should be built using a retired build and servers currently on the retired build should be migrated to currently maintained build, or redeployed.

Managing Multiple Releases

One of the challenges of complex environments is that multiple builds need to be managed simultaneously. For example, in a large organisation following the vendor-released base build stragegy, the build list might look like:

The workflows and processes required to manage complex environments such as this will be described in much greater detail in the next article in the series.


The workflows and processes required to manage complex environments such as this will be described in much greater detail in the next article in the series.

Connect with Red Hat Services

Learn more about Red Hat Consulting
Learn more about Red Hat Training
Learn more about Red Hat Certification
Subscribe to the Training Newsletter
Follow Red Hat Services on Twitter
Follow Red Hat Open Innovation Labs on Twitter
Like Red Hat Services on Facebook
Watch Red Hat Training videos on YouTube
Follow Red Hat Certified Professionals on LinkedIn
Creative Commons License

2016 Red Hat EMEA Training Partner Conference Recap & Award Winners

The Red Hat EMEA Training Team hosted the 2016 Training Partner Conference in Rome on September 29th, 2016 through October 1st, 2016. Over the course of the three days, over 115 EMEA training partners, resellers and instructors gathered for keynote and break-out sessions held by global and regional leaders, fellow partners and emerging technology specialists.

Continue reading “2016 Red Hat EMEA Training Partner Conference Recap & Award Winners”

Consulting Customer Snapshot: Red Hat Empowers an Energy Company

Welcome to our monthly series of blinded stories to highlight how customers are using Red Hat solutions to address business challenges and promote innovation.

This month highlights a migration from TIBCO to Red Hat JBoss Fuse with a Midwestern energy company, whose productivity was being drained by an outdated, unsupported version of TIBCO middleware.

Continue reading “Consulting Customer Snapshot: Red Hat Empowers an Energy Company”

BPM, EA and Compliance

A few years ago, I was invited to join a debate forum belonging to a group of companies that despite not being involved with the financial segment, discussed the workings of each other’s internal controls areas, SOX compliance and benchmarking practices.

First of all, why were BPM and EA specialists invited to this forum, and second, why am I publishing matters of internal controls in a BPM area that contributes to the subjects of Enterprise Architecture?

Continue reading “BPM, EA and Compliance”

BPM and its contribution to Enterprise Architecture

When employing the homonym “BPM,” it is important to clarify the difference between Technical BPM and Business BPM. Technical BPM covers not only the IT tools for business process automation, but also contributes as another available resource for a company to perfect its process management.

Continue reading “BPM and its contribution to Enterprise Architecture”

BPM in a Microservice World: Part 3

This article was originally published on Diabolical Labs.

Many BPM practitioners are used to utilizing a software suite that has some sort of Process Manager component that has control of the transaction as it progresses through activities. The process is generally authored and visualized graphically in BPMN or BPEL. When applying BPM in the microservice world we don’t have that visibility or control.

Continue reading “BPM in a Microservice World: Part 3”

BPM in a Microservice World: Part 2

This article was originally published on Diabolical Labs.

Back in the early days of “workflow” we had control of the transaction, usually a document, from the start of the process to the end. As IT evolved into the SOA/ESB era, we had a little bit less control but for the most part the process engine orchestrated everything.

There were frequent hand-offs to message queues but normally the message would come back to the process engine so it would continue to orchestrate the process.

The microservice world is different. Instead of having a process engine or an ESB controlling a small number of large services, we have many small services that can potentially send and receive messages or respond to events from any of the other services.

It’s more like a web. One initiating message or event to a particular service could affect the exchange of many hundreds of messages between the microservices before the initial request is considered complete. That can make BPM practitioners a bit uneasy due to the loss of control.

We may not have control any longer but we still can have visibility into the process. We can still apply our usual patterns for SLA and exception management, and human and compensating workflows. This can be accomplished through what I call a “tracking” process.

Continue reading “BPM in a Microservice World: Part 2”

BPM in a Microservice World: Part 1

This article was originally published on Diabolical Labs.

Business Process Management (BPM)-enabling software has been around for decades, having started as document centric workflow. It’s progressed into the Service Oriented Architecture (SOA) age to become an orchestrator of services and human tasks.

Now, we see that the Service Architecture is evolving to include a relatively new concept called Microservice Architecture (MSA). That architecture along with enabling technologies like Cloud Services and Application Containers is allowing us to apply process management practices to solutions in a much more lightweight and agile way.

In the upcoming blog post series, I’ll be exploring the application of BPM principles to solutions that can implemented with MSA. In this first part, I’ll review traditional BPM practices and their pitfalls, followed by a guide to begin the convergence of BPM and MSA. re with MSA.

You can also learn more in the webinar I’ll be hosting on 9/27 at 11am ET.

Screen Shot 2016-05-26 at 2.01.22 PM

Continue reading “BPM in a Microservice World: Part 1”