News

CI/CD in OpenShift with Gitlab and Terraform

We’re always searching new ways of implementing CI/CD at Eurotux, and in this post I’ll describe one of those by leveraging 3 components that we are using in our customers:

• Gitlab
• Terraform
• Openshift

Application

The application we wanted to deploy is Fess Enterprise Search Server , (“Fess is Elasticsearch-based search server”) and we use it to scan our internal Wiki server and allow our teams to have a “google”-like search engine for that wiki. Fess supports all sorts of targets (file servers, web sites, databases) and it supports several authentication methods such as BASIC/DIGEST/NTLM/FORM (keep this in mind for the next minutes).

We use OpenShift which is a container based orchestration, so the first thing to do is to create a container for the application. Fortunately Fess already provides a base container image which I’ll use as the base container for the project and I will only improve on that. The first thing is to create a Dockerfile:

FROM docker.io/codelibs/fess:latest
RUN perl -i -p -e "s/crawler.document.cache.enabled=true/crawler.document.cache.enabled=false/" /etc/fess/fess_config.properties
ADD logo-head.png /usr/share/fess/app/images/logo.png
ADD osdd.xml /usr/share/fess/app/WEB-INF/orig/open-search/osdd.xml
ADD logo-head.png /usr/share/fess/app/images/logo-head.png
RUN apt-get update && apt-get -y install libjson-perl
COPY entrypoint.sh /usr/share/fess/run.sh

ADD insert.sh /insert.sh

# We are using oc 3.9 because the later ones require libcrypto.so.10 (see https://github.com/openshift/origin/issues/21061)
RUN wget https://mirror.openshift.com/pub/openshift-v3/clients/3.9.63/linux/oc.tar.gz && tar zxf oc.tar.gz -C /usr/bin && rm oc.tar.gz

ADD fess_config.properties /etc/fess/fess_config.properties

I did do some customization (like changing the logo to our company one), changing the entrypoint and adding the oc (openshift client) command. As one can easily understand, our internal wiki is password protected. It is a form-based username/password (you see why it is great that Fess supports form-based authentication) and I only need to provide the Fess server the username and password to access the wiki.

The entrypoint is changed so that when the container starts, will get the username and password from OpenShift secrets (that’s why the container installs the oc command), update the Fess server configuration and start indexing the wiki. As this is a stateless service, I don’t need to worry about saving state and using Persistent Volumes. If the container dies or gets redeployed, the search engine will re-index our wiki. This keeps this project simpler and cleaner. Here is a snippet of the insert.sh script:

if [ -z "$WIKIUSER" ]; then
    export WIKIUSER="`oc get secret wikiuser --template='{{.data.username}}' | base64 -d`"
fi
if [ -z "$WIKIPASS" ]; then
    export WIKIPASS="`oc get secret wikiuser --template='{{.data.password}}' | base64 -d`"
fi

curl -XPOST "http://localhost:9200/.fess_config.web_authentication/web_authentication" -H 'Content-Type: application/json' -d "
{
           \"webConfigId\" : \"$CONFIGID\",
           \"updatedTime\" : 1509224726193,
           \"hostname\" : \"wiki.eurotux.com\",
           \"password\" : \"$WIKIPASS\",
           \"updatedBy\" : \"admin\",
           \"createdBy\" : \"admin\",
           \"createdTime\" : 1509224726193,
           \"protocolScheme\" : \"FORM\",
           \"username\" : \"$WIKIUSER\",
           \"parameters\" : \"encoding=UTF-8\\nlogin_method=POST\\nlogin_url=https://wiki.eurotux.com/Special:UserLogin\\nlogin_parameters=username=\${username}&password=\${password}&auth_id=1&deki_buttons%5Baction%5D%5Blogin%5D=login\"
}"

Terraform

We use Terraform to bootstrap the infrastructure required for the deployment of this application, which is responsible for the following:

• OpenShift Project (Namespace)
• Secrets (wiki username and password)
• Granting permissions to the container default service account to access the secret (so that the container can fetch that info)
• Granting the gitlab runner service account to edit this namespace objects (so that the deployment pipeline can deploy to this namespace)
• Adding the anyuid scc to the deployer service account. The Fess container runs several services (actually this is an anti-pattern in the container world), and requires to run as root inside the container (later on it changes the uid to another)

Unfortunately, the terraform kubernetes provider is somewhat lacking in features comparing to others (like aws or azure provider). Because of that, I use a mix of internal resources like the kubernetes_namespace and null_resource as a wrapper to the occommand:

# Create namespace
resource "kubernetes_namespace" "search" {
  metadata {
    annotations {
      name = "search-engine"
    }

    labels {
      owner = "npf"
    }

    name = "${var.namespace}"
  }

  lifecycle {
    # because we are using openshift, we have to ignore the annotations as openshift does add some annotations
    ignore_changes = ["metadata.0.annotations"]
  }
}
# This container requires root, so we need to allow anyuserid
resource "null_resource" "add-scc-anyuid" {
  provisioner "local-exec" {
    command = "oc -n ${kubernetes_namespace.search.id} adm policy add-scc-to-user anyuid -z deployer"
  }

  provisioner "local-exec" {
    command = "oc -n ${kubernetes_namespace.search.id} adm policy remove-scc-from-user anyuid -z deployer"
    when    = "destroy"
  }
}

As you can see, I use local-exec to spawn the oc command when there isn’t support for those features in the kubernetes terraform provider. As a result of a terraform apply:

Gitlab

At Eurotux we are using an internal gitlab server to house all our projects. As so, we make extensive use of its’ CI/CD capabilities. To implement the CI/CD I’ve created a .gitlab-ci.yml file to describe the pipeline:

image: $CI_REGISTRY/docker/base-builder

stages:
  - review
  - staging
  - production
  - cleanup

variables:
  OPENSHIFT_SERVER: https://oshift.install.etux:8443
  OPENSHIFT_DOMAIN: oshift.install.etux

.deploy: &deploy
  tags:
    - kubernetes
  before_script:
    - ci-bootstrap
  script:
    - "oc -n $CI_PROJECT_NAME get services $APP 2> /dev/null || oc -n $CI_PROJECT_NAME new-app fess --name=$APP --strategy=docker"
    - "oc -n $CI_PROJECT_NAME start-build $APP --from-dir=fess --follow || sleep 3s && oc -n $CI_PROJECT_NAME start-build $APP --from-dir=fess --follow"
    - "oc -n $CI_PROJECT_NAME get routes $APP 2> /dev/null || oc -n $CI_PROJECT_NAME create route edge --hostname=$APP_HOST --insecure-policy=Redirect --service=$APP"
......
......
staging:
  <<: *deploy
  stage: staging
  tags:
    - kubernetes
  variables:
    APP: staging
    APP_HOST: $CI_PROJECT_NAME-staging.$OPENSHIFT_DOMAIN
  environment:
    name: staging
    url: http://$CI_PROJECT_NAME-staging.$OPENSHIFT_DOMAIN
  only:
    - master

production:
  <<: *deploy
  stage: production
  tags:
    - kubernetes
  variables:
    APP: production
    APP_HOST: $CI_PROJECT_NAME.$OPENSHIFT_DOMAIN
  when: manual
  environment:
    name: production
    url: http://$CI_PROJECT_NAME.$OPENSHIFT_DOMAIN
  only:
    - master

The pipeline will create a review application when working on a git branch other than master so that I can review and fix things. When a merge (or a commit for that matter) occurs in master, it will deploy automatically to staging and then I can press play to deploy to production. Here is an example of the pipeline:

Here is a snippet of the pipeline running:

After that, i can browse to https://search.oshift.install.etux/ and I’m presented with the search engine webpage:

OpenShift

As you’ve figured out by now, all of this is running in our testing OpenShift cluster. We are using the 3.11 version of OpenShift, which features monitoring using Prometheus and Grafana (later on, I will detail some other interesting features, such as integration with Keycloak). OpenShift automatically provides some Grafana dashboards so that you can see what are the usage patterns:

One of the interesting things that these dashboards present is the lifecycle of the application (starting new containers and stopping the older ones).

Written by Nuno Fernandes

April 8, 2022