Jenkins pushing to branch overwrites commits made in interim A while ago I had written about how commits were being merged (successfully) into our develop branch in Git, but would later go missing, with no evidence of the missing code other than all the Bitbucket logs proving the merge happened.

We just figured out yesterday that what's happening is Jenkins clones our repo, then holds it for up to 40+ minutes while jobs run, then pushes back with git push origin +develop. This is essentially overwriting any commits we've made in that 40+ minutes as Jenkins pushes back the branch exactly as it saw it on the clone, and minus our additions.

We thought the solution would be to change to

git pull --rebase origin
git push origin develop

but that's returning

+ git pull --rebase origin
error: Cannot pull with rebase: You have unstaged changes.

To clarify, Jenkins' "branch" has files on it that we do need merged into the repo. The ideal situation is to find the right way to do a fresh pull of develop into Jenkins' copy before pushing back.

What's the correct git procedure to avoid this issue?

Note: My original post was deleted for being too vague, and rightfully so since at the time I had zero idea why things weren't working. In this case what happens is Jenkins checks out the repo, runs a script that makes parameter files needed for a second job, then checks it back in so the files (and associated logs) exist in the repo for use/auditing. Not having Jenkins do the process isn't an option.

It really depends on your situation, but I'll go with the generic/naive approach and say do exactly what your developers do in the same situation: I pulled an hour ago, I made some changes, now I want to push my changes, but other changes have been pushed in the meantime. I commit, pull/rebase, then push. Assuming that whatever Jenkins is committing isn't tied to the commit it built from, this is the same process it should follow to get the same result.


How to check out GitHub repo after specifying skipDefaultCheckout in Jenkins declarative pipeline? I have a Jenkins pipeline in which I build in one stage and test in another. I'd like them to be different machines since they have different capabilities. I have something like the following so far:

pipeline {
  agent none
  stages {
    stage('Build') {
      agent { node { label 'builder' } }
      steps {
        sh 'build-the-app'
        stash(name: 'app', includes: 'outputs')
      }
    }
    stage('Test’) {
      agent { node { label 'tester' } }
      steps {
        unstash 'app'
        sh 'test-the-app'
      }
    }
  }
}

The problem I have is that since it runs on two different nodes, by default it checks out the repository on both nodes. I'd like for it to not do that because it's not needed. So I added the following option:

options { skipDefaultCheckout() }

But now no repositories are checked out, causing the build to not work. How do I, after specifying skipDefaultCheckout, manually tell the pipeline to checkout the repository in stages which I specify?

You can use the checkout scm step whenever you need the source:

pipeline {
  agent none
  options { skipDefaultCheckout() }
  stages {
    stage('Build') {
      agent { node { label 'builder' } }
      steps {
        checkout scm
        echo 'build-the-app'
        stash(name: 'app', includes: 'outputs')
      }
    }
    stage('Test') {
      agent { node { label 'tester' } }
      steps {
        unstash 'app'
        echo 'test-the-app'
      }
    }
  }
}


Use K8s Deployment Hostname In YAML Config I am trying to convert some K8s examples from ReplicationControllers to Deployments. These examples are for a Selenium grid located: https://github.com/kubernetes/examples/tree/master/staging/selenium

When I run the examples as-is, my selenium-hub and selenium-nodes connect successfully. However, when I tried to convert selenium-hub-rc.yaml and selenium-node-chrome-rc.yaml into Deployments, the selenium-nodes will not connect to the selenium-hub.

In selenium-node-chrome-rc.yaml there are these variables:

...
env:
- name: HUB_PORT_4444_TCP_ADDR
value: "selenium-hub"
- name: HUB_PORT_4444_TCP_PORT
value: "4444"
...

HUB_PORT_4444_TCP_ADDR is the ip/hostname of the selenium-hub. The selenium-hub-rc.yaml shows the phrase selenium-hub in many locations. Question 1: which of these is actually defining the host name? (.metadata.name, .spec.selector.app, ...?)

apiVersion: v1
kind: ReplicationController
metadata:
  name: selenium-hub <-------------------
  labels:
    app: selenium-hub <-------------------
spec:
  replicas: 1
  selector:
    app: selenium-hub <-------------------
  template:
    metadata:
      labels:
        app: selenium-hub <-------------------
    spec:
      containers:
- name: selenium-hub <-------------------
....

Now, I changed this selenium-hub-rc.yaml to a deployment:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: selenium-hub-deployment <---------- this is the hostname?
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: selenium-grid
    spec:
      containers:
      - name: selenium-hub <--------- this is the hostname?
        image: selenium/hub
        ports:
        - containerPort: 4444
...

But now the selenium-node won't connect and I believe it is due to HUB_PORT_4444_TCP_ADDR. I have tried:

- name: HUB_PORT_4444_TCP_ADDR
  value: "selenium-hub"

and

- name: HUB_PORT_4444_TCP_ADDR
  value: "selenium-hub-deployment"

Neither worked. Question 2: What is the hostname for a deployment? It seemed straight forward in the ReplicationController yaml e.g. selenium-hub, but I can't get it to work as a deployment.

Note, I have not changed the service in any way. Actually I DID change it and that partially caused my initial issue.

My idea of where the hostname was coming from was wrong. According to the docs:

  Every Service defined in the cluster (including the DNS server itself) is assigned a DNS name.

and

  Currently when a pod is created, its hostname is the Pod’s metadata.name value.

So the hostname is actually the name of the service. The last sentence in my question says I did not change the service in anyway. That was an unintentional lie. I did change the service name, or else selenium-hub as the hostname would have worked.

Changing HUB_PORT_4444_TCP_ADDR to the .metadata.name of the service got me closer. The connection was timing out, though.

I believed some of the configuration was wrong in my service yaml. So instead of using my yaml file, I created the service via:
kubectl expose deployment <deployment name here>

This created a service and I checked the selenium-hub and the node was finally connected. So there was an issue in my service yaml. To get the correct configuration I exported the currently working service:
kubectl get svc <service name here> -o yaml --export

I copied the exported configuration to my service yaml file and it also worked. Here was the difference between the service yamls

old yaml (copied from GitHub example above)
---
apiVersion: v1
kind: Service
metadata:
  name: selenium-hub-service
  labels:
    app: selenium-grid
spec:
  ports:
  - port: 4444
    targetPort: 4444
    protocol: TCP ------------- [added in new config]
    name: port0 ------------- [deleted in new config]
  selector:
    app: selenium-hub
  type: NodePort ------------- [deleted in new config]
  sessionAffinity: None

I am not sure why adding .spec.ports[0].protocol and deleting .spec.type and .spec.ports[0].name fixed the issue. If anyone could clarify in the comments I would appreciate it.


Terraform: How to use a variable inside of another variable? I'm trying to design it so that all configurable parameters exist within variables.tf...the issue I'm running into currently is for provisioning EC2 instances. I'd like to be able to define the security group to use, but the input variable only accepts strings, so I wasn't able to reference the security group id.

I moved on to trying this, which is looking up the resource id dynamically by inserting the name (look at security_groups_name and subnet_name:

variable "ansible_ami" {
    type =  "map"
    default = {
            name = "ansible"
            ami = "ami-4836a428"
            size = "t2.micro"
            ebs_size = "8"
            ebs_type = "gp2" #general purpose ssd
            enable_public_ip = true
            security_groups_name = "JumpBox"
            subnet_name = "pubnet1"
    }
}

resource "aws_instance" "ansible" {
    ami = "${var.ansible_ami["ami"]}"
    instance_type = "${var.ansible_ami["size"]}"
    security_groups = "${aws_security_group.${var.ansible_ami["security_groups_name"]}.id}"
    subnet_id = "${aws_subnet.${var.ansible_ami["subnet_name"]}.id}"
    associate_public_ip_address = "${var.ansible_ami["enable_public_ip"]}"
    #user_data = ""

    ebs_block_device {
        device_name = "/dev/sdb"
        volume_size = "${var.ansible_ami["ebs_size"]}"
        volume_type = "${var.ansible_ami["ebs_type"]}"
    }

    tags {
        Name = "${var.ansible_ami["name"]}"
    }
}

However, that is not working either since i'm getting syntax errors from the lines where I'm trying to next variables. I'm looking for ideas or direction on how to accomplish this.

Largely inspired by the documentation the idea is to create a terraform datasource from a filter of the available resources and then use this as entry point in your ami resource:

variable "security_group_id" {}
variable "subnet_id" {}

data "aws_security_group" "selected" {
  filter {
    name = "${var.ansible_ami["security_groups_name"]}}"
  }
}

data "aws_subnet" "selected" {
  filter {
    name = "${var.ansible_ami["subnet_name"]}"
  }
}

resource "aws_instance" "ansible" {
    ami = "${var.ansible_ami["ami"]}"
    instance_type = "${var.ansible_ami["size"]}"
    security_groups = "${data.aws_security_group.selected.id}"
    subnet_id = "${data.aws_subnet.selected.id}"
    associate_public_ip_address = "${var.ansible_ami["enable_public_ip"]}"
    #user_data = ""

I'm not using terraform myself, so this could be plain wrong.


Docker image two-steps build I have a third party PHP application to install in a container. The application is shipped without its dependencies. There is a provided script to automate dependency installation, but it relies on a couple of tools not available in my base image (both general utilities like unzip as well as some PHP-specific tools like composer). Those tools are only required for the dependency installation. There are not required afterward.

A simple solution would be to install the required tools and run the installation script in a RUN section of my Dockerfile. But I don't want to clutter the image with software only useful during the image build step.

  RUN apt-get update && apt-get install --yqq unzip
  RUN wget https://getcomposer.org/download/1.8.3/composer.phar
  RUN php installer.php
  # I end up polluting my container with unused tools and possible
  # behind the scene changes in some files (`php.ini`, ...)

Another solution would be to perform the installation on the host and to COPY the application alongside its dependencies into the container. But I want a reproducible build.

  COPY app/ ./
  # Not reproducible: I do not control the version of
  # the installed app neither the version of the tools
  # and dependencies

So, I envision a two-steps process:

using a first container to perform the installation;
then copying the application alongside its dependencies into another "clean" container.

But I don't see exactly how to achieve that. Do you have any suggestions?

So you can use two build stages in your Dockerfile so you don't end up having unnecessary things inside container.  Anything from first stage is gone until you move it to second stage. Something like:

FROM whatever as build-stage
WORKDIR /app
RUN apt-get update && apt-get install -y unzip
RUN wget https://getcomposer.org/download/1.8.3/composer.phar
RUN php installer.php

FROM whatever as production-stage
COPY --from=build-stage /app ./


Build Docker Images with Ansible + Vagrant So I am working on a research project and I am trying to automatically generate specific environments on different virtualization techniques.

So, first I made a Vagrantfile that pulls an Ubuntu image and then runs an Ansible  playbook. I have a playbook that defines everything for my environment.
Which means it copies files, installs and configures everything.
For VirtualBox this is no problem, - it spins up my environment in VirtualBox.

Now I want to do this exact same thing, but on Docker. Normally one would write a docker file to define the environment. What I want to do is re-use the Ansible script to define the environment (despite being a bit counter-intuitive). While there seems to be some support for Docker in Vagrant - what is the best way to achieve what I want. I want to create a Docker Container - exactly how specified in the Ansible playbook. (Basically re-using the playbook I also used to setup VirtualBox). The output should then be a docker image that I can just start with docker run.

Whats the best way to do this? What do I have to look out for?

My current idea is the following:

1) Setup a VM with Vagrant.
2) Install Ansible and Docker in that VM.
3) Create a new Docker Image with Ansible. It just defines the Container OS and maybe opens a port so Ansible can SSH into the container?
4) Run the playbook in the container to configure everything.
5) Package the entire thing as a new image
6) Transfer packaged docker image to the host machine

If it's within scope, I would use Packer to build the artifacts.

Write a Packer template using the builders that you want:

Vagrant builder for the Vagrant image
Docker builder for the Docker image.

Use the same Ansible playbook as you already have to provision each of the builders.

Note that you need to make sure that the Ansible playbook will work in both types of virtualised environments.

This has the benefit of producing arbitrary types of artifacts, based off of a single input. You can use a single workflow for both the Vagrant and the Docker images, and they will be built off of the same state of the codebase (i.e. git hash).


What is the difference between EKS and EC2 installed kubernets? I'm newbie in AWS usage.
I don't understand the difference between EKS and EC2 installed kubernets?

  Amazon EKS makes it easy for you to run Kubernetes on AWS without needing to install and operate your own Kubernetes clusters.

Does this means that we could install Kubernets to EC2 and use EC2 instead of EKS?

You absolutely could install Kubernetes to EC2 and setup the configuration yourself.  However, EKS will handle the bulk of the setup, configuration, maintenance, and monitoring for you.  Check out the feature page to determine if it is worth it to you.


Evolution of DevOps? I head that the present DevOps jobs were previous know as build and release engineer. I want to go back to past 10 years. What would be the different job position roles a personal might have played from past 10 yes?

I don't view DevOps as a job position. Yes, I also have job offers out there for titles like "DevOps Engineer" [sic], but that just means I am looking for people who mainly can take a team with little DevOps knowhow and guide them along (be it through just doing the bulk of the work, or evangelism or whatever). But at the end of the day, everybody on the team needs to have a working understanding at least of what's going on.

DevOps is more of a feeling, a culture, and a set of common tools and techniques. A guideline if you wish. With memes like having the same environment across dev, qs, prod, immutable containers being shipped across all those, "everything as code", applying Agile and Lean to ops, having devs and ops in the same team and such.


How to update frontend servers behind a load balancer I have multiple frontend servers churning out the HTML pages , behind a load balancer.

Just wondering what is the best practice to update the frontend app while users are surfing the site.

The objectives:

ensure each user see either the new or old version, but not flip between both.
ensure load is still evenly distributed amongst servers that are not being updated.
no disruption to service.

My thoughts:

Divide the servers into groups.
Take out a group from load balancing and wait for it to finish off current requests.
Update the app and OS, etc.
put the group back to load balancing.
Pick another group and repeat steps 2-5.

While in process, some servers will have the new version, so to ensure users do not flip between the old and new ones, do I enable sticky session?

What you have above seems like a reasonable update procedure, however you can also use persistence or a sticky session so that when a user goes to a web server, they will be routed to the same web server for the life of their browser session (or for the next hour, or whatever you decide to configure). This also allows for some A/B testing if you need to get user feedback on a new feature or just allow for the possibility of a bug (just pull server group B from the pool if an issue is discovered during A/B testing with that group).

There are some concerns you should be aware of with sticky sessions: Eg, source address persistence can be problematic if you have a thousand users behind a firewall - then you have a huge group that always goes to a single web node. Maybe you really don't want a cookie to do persistence, etc.

But, there really is no agreed "best practice" on this - the best practice is going to the the one that meets your business need in this case. You should just collect all of your options and choose the workflow that best meets your needs.


Passing secrets to a Docker container I have a base docker image which is used to run image analysis software. For each container created from the image, there are a set of configuration settings, some of which are secrets (encryption keys, customer information, etc.), that are used by the software to analyze and distribute the processed images. How can I safely pass these secrets to a container?

You have 3 methods to get secrets to an app inside a docker container. The first 2 involve docker configuration. The last one is to have your apps directly fetch secrets from a secret store.

1 - Environment variables

According to "The 12 Factor App" guide, secrets are merely config, and they should always be set in the environment. You could set your secrets as environment variables during the docker run, and your app accesses them from there.

2 - Mounted volumes

You could have your secrets all within a particular configuration/secrets file, then mount that to your instance as a mounted volume.

3 - Fetch from secret store

As @030 mentioned, you can use Hashicorp Vault (or "Amazon Secrets Manager", or any service like that).
Your app, or a sidecar app can fetch the secrets it needs directly, without having to deal with any configuration on the Docker container. This method would allow you to use Dynamically created secrets (a very appealing feature of such systems) and without having to worry about the secrets being view-able from the file system or from inspecting the env variables of the docker container.

Personal Opinion

I believe env variables is the way to go. It's easier to manage, and you can still pull from a secret store like Hashicorp Vault, if you have you CI build system pull the secrets during the build and set them when you deploy. You get the best of both worlds, and the added benefit of your developers not needing to write application code to fetch secrets. Devs should be focused on their code functionality, and not dealing with admin tasks like fetching passwords.

Your application's code should be focused on it's own app functionality itself, and not dealing with backend tasks like fetching passwords. Just like the 12 Factor App states.

Edit: changed last sentence to remove implication of Developer vs SysAdmin silo-ing. The tasks themselves should be separate from a code perspective, but DevOps is about the same persons keeping both in mind and not be limited.

Personal Opinion (Update)

Per @Dirk's excellent comment (Passing secrets to a Docker container), there is a very strong argument to prioritize a secret store over ENV vars, due to not wanting to leak them.


Is making kubernetes masters schedulable a really bad idea? It's a cash strapped start-up. What are the worst consequences of running only three nodes, three schedulable masters?

Are there some ways can one mitigate against the risks?

https://kubernetes.io/docs/tasks/administer-cluster/highly-available-master/

  Best practices for replicating masters for HA clusters

  Try to place master replicas in different zones. During a zone failure, all masters placed inside the zone will fail. To survive zone
  failure, also place nodes in multiple zones (see multiple-zones for
  details).

  Do not use a cluster with two master replicas. Consensus on a two-replica cluster requires both replicas running when changing
  persistent state. As a result, both replicas are needed and a failure
  of any replica turns cluster into majority failure state. A
  two-replica cluster is thus inferior, in terms of HA, to a single
  replica cluster.

  When you add a master replica, cluster state (etcd) is copied to a new instance. If the cluster is large, it may take a long time to
  duplicate its state. This operation may be sped up by migrating etcd
  data directory, as described here (we are considering adding support
  for etcd data dir migration in future).

I would like to emphasize the following quote

  A
  two-replica cluster is thus inferior, in terms of HA, to a single
  replica cluster.

If less then 3 nodes will be used then it is basically more safe to run 1 master, but if one goes down there will be an issue as well. So it is better to run at least three masters.


How do I get more control over the NodePorts my Service assigns to my Deployment? I have a Deployment with 5 replicas. All have ssh and telnet. They should not be load balanced. I would like each to select from a predictable list of 5.

Here is my Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
spec:
  selector:
    matchLabels:
      app: myapp
  replicas: 5
  template:
    metadata:
      labels:
        app: myapp
    spec:
      containers:
      - name: myapp
        image: myapp:1.0
        ports:
        - name: ssh
          protocol: TCP
          containerPort: 22
        - name: telnet
          protocol: TCP
          containerPort: 23

This is my Service with invalid nodePort values for illustrative purposes.

apiVersion: v1
kind: Service
metadata:
  name: myapp
  labels:
    app: myapp
spec:
  type: NodePort
  ports:
  - name: ssh
    port: 22
    nodePort: [30022, 30122, 30222, 30322, 30422, 30522]
  - name: telnet
    port: 23
    nodePort: [30023, 30123, 30223, 30323, 30423, 30523]

I am hoping to be able to accomplish 2 things:

Each pod replica instance will only get an ssh port from [30022, 30122, 30222, 30322, 30422, 30522] and a telnet port from [30023, 30123, 30223, 30323, 30423, 30523]
A pod replica instance that gets an ssh port of 30022 also gets the telnet port 30023. A pod replica instance that gets an ssh port of 30122 gets a telnet port of 30123 and so on.

Thank you!

As far as I understand kubernetes services, that trick is not possible.

What could help you to have preserved hostnames is a StatefulSet.
See more here

  The Pods’ ordinals, hostnames, SRV records, and A record names have
  not changed, but the IP addresses associated with the Pods may have
  changed. In the cluster used for this tutorial, they have. This is why
  it is important not to configure other applications to connect to Pods
  in a StatefulSet by IP address.


Ansible No such file or directory error message for any executable in Amazon Linux AMI I'm trying to run this command in a Amazon Linux AMI instance
  - name: Run docker
    shell: systemctl enable docker && systemctl start docker

And receiving this error message
"stderr": "/bin/sh: systemctl: No such file or directory"

I already tried with command, ansible systemd module and /usr/bin/systemctl.

After struggling for a time I solved the problem by changing the image in the ci from Ubuntu to ansible-runner.


docker container does not open port for php:latest I'm working with Docker containers for a while know, but I do not notice, what I am doing wrong here:

In my current folder on the host system, I have a simple index.php file which contains:

<?php echo 'hi'; ?>

Why is this not working?

docker run --name php -v $(pwd):/app -w /app -p 8080:8080 -d --rm php:cli php -S localhost:8080 -t . index.php

This results in an empty reply:

$ curl localhost:8080
curl: (52) Empty reply from server

If I exec in the container and apply the curl there it succeeds in replying hi.

So, what am I missing?

Try to use php -S 0.0.0.0:8080:

docker run --name php -v $(pwd):/app -w /app -p 8080:8080 -d --rm php:cli php -S 0.0.0.0:8080 -t . index.php

Explanation:

The localhost(or 127.0.0.1) address is attached to the container namespace, so anything outside of this namespace cannot have access to it. Furthermore, the 0.0.0.0 (in this context) is a specific address that means all IPv4 addresses on the local machine. If you check inside the container, you could see:

$ ip a
lo:
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
eth0:
    link/ether 02:42:ac:11:00:02 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 172.17.0.2/16 brd 172.17.255.255 scope global eth0
       valid_lft forever preferred_lft forever

The eth0 interface is the interface used when you forward the port. As you don't specify the IP address of the container, you can use the 0.0.0.0 to listen on the eth0 interface.

I don't know if I make my self clear, but you can (should) read this if you want to know more:

https://pythonspeed.com/articles/docker-connection-refused/
https://superuser.com/questions/949428/whats-the-difference-between-127-0-0-1-and-0-0-0-0


What are tools that lend themselves for backup and recovery in a CI/CD environment? I have used tar, dump, Amanda, Legato (EMC) Networker and other backup software for making backups of systems.  However, I don't know the best tools for making backups in a CI/CD (Continuous Integration/Continuous Deployment) environment and doing "rolling recoveries" of systems "as you go" which is the case in a DevOps oriented system.

Many of these backup utilities are not necessarily best suited for a CI/CD environment because of the continuous changes taking place on both the development and production environment.

I'm looking at programs like Borg
and Git based backups like Bup
These programs would allow for many incremental backups and finer granularity of backups and work well with CD tools like Ansible.  Even Ansible has a couple of way to make backups.  However, there is no clean way to recover the most recent backup in an automated way.

Relying on some form of caching or mirroring of your content on the cluster and hope it's not corrupted is not a way to go :(

The things that would be backed up would be the config files of the servers, the database and uploaded content as these will almost always be unique and necessary.
The containers, VMs, even the webapps would NOT be backed up as these should all be uniform and constantly updated under the CI/CD principles.
Currently I use a combination of scp and tar into an archive directory sorted by date and machine.  I like to find something better if possible.

I would like to know what are the best backup and recovery tools used in the CI/CD environment?  and an example of how you would configure it?

I don't expect a "magic bullet" just some possible solutions.

The things that would be backed up would be the config files of the servers, the database and uploaded content as these will almost always be unique and necessary. The containers, VMs, even the webapps would NOT be backed up as these should all be uniform and constantly updated under the CI/CD principles. Currently I use a combination of scp and tar into an archive directory sorted by date and machine. I like to find something better if possible.

Config files, containers and VMs should be backed up and versioned by a tool like git or SVN if you are doing configuration management properly.. Backing up config files and versioning them is the whole point of configuration management tools like, salt, puppet, ansible, and chef.

As you duly note however, dynamic content such as database data and similarly dynamic data is a more challenging issue. These however, are outside of the scope of configuration management, CI, and CD principles. These principles purposefully remain silent on dealing with these issues: because they have no new answers for these problems. Instead, you should use the many battle-tested techniques and strategies for dealing with this kind of data instead of expecting CI/CD/CM tools to solve problems that they were never intended to address.


Route53 Routing Policies - Best Policy for US-based Ecommerce? When I look over the documentation for Route 53 Routing Policies they list 7 different policies. How should I pick the right one if I am building out new infrastructure in AWS for a US-based e-commerce website? Here is the list of offerings:

Simple
Failover
Geolocation
Geoproximity
Latency-based
Multivalue Answer
Weighted

These different routing policies are pretty confusing. I would of course always want failover for my DNS. Why would I ever choose Simple? Is there some major difference between Geolocation and Geoproximity? What do they mean by "Traffic Flow Only" next to Geoproximity?

Because of the confusion around all of this I am for now choosing latency-based routing. That seems simple and intuitive. I have ELBs to handle failover for me, and with my latency routing I have associated health-checks, so it seems like my bases are covered.

Notes:

We have a handful of ELBs but they are mostly in us-west-2 region, some in us-east-1.
These ELBs are serving the same site out of multiple regions.

To answer your question, ill first try to give simple explanations for each policy to help with the confusion (https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/routing-policy.html).

Simple: a basic routing policy that randomly sends a user to one of
your addresses.
Failover: You have a primary (active) route and a secondary (passive)
route. When the primary fails, it routes to the passive. If passive
fails, you have an issue.
Multivalue: Multiple routes that each have health checks. It will randomly select an address (Simple routing policy) from the healthy routes. A more
complex failover policy.
Geolocation: Route users dependent on where they are located. If a
user is in England and gets better latency in Germany, they will
still be routed to England.
GeoProximity: A more fine grained Geolocation. You would be able to    route people located on the border of France and Germany (based
on longitude) to a    specific route. "Traffic Flow Only" is to
specify that you must use the Route 53 traffic flow to configure this
policy. It is the most complicated of the route 53 policies
Weighted: This policy allows percentage based balancing. Routing 30%
of people to route-1, 20% to route-2 and 50% to route-3.
Latency: Route to the address that gives the user the best latency.

Now for your architecture, your current implementation would handle instance failures with the ELBs and the latency based routing (with health checks) would handle the routing failures. If you don't require routing your users to the lowest latency address, use the multivalue policy instead.


Unable to create container after installing mysql and phpmyadmin | Ubuntu I have installed docker successfully and try to install phpmyadmin and mysql.

I have did following steps.
Using below link I have pulled php version

https://hub.docker.com/r/codepulse/php-7.2.0-apache-mysql/

docker pull codepulse/php-7.2.0-apache-mysql
docker pull phpmyadmin/phpmyadmin

After installing it when I execute docker ps -a I could not found any container there.

Could any one help to install phpmyadmin and mysql.

Pulling the images only make them available for you to use locally. Therefore, to use them within a container you should run the command docker run with the appropriate configuration for each services


How to run "docker/compose" container on RPi3? I have pull the image https://hub.docker.com/r/docker/compose/
docker pull docker/compose

But how can I run it to build my images on Raspberry Pi/rpi3?

According to this reply it should run by issuing:

docker run -ti --rm -v `pwd`/docker-compose.yaml:/docker-compose.yml -v
/var/run/docker.sock:/var/run/docker.sock docker/compose:1.16.1 up


Tool preferences for rapidly spinning up and spinning down k8s clusters My team hosts several completely different projects on a production k8s cluster with a given configuration. We use AWS (and are starting to use Google Cloud). We have a QA cluster as well, that "sort of" mimics the production configuration, and has a few test projects deployed on it as well. I say "sort of" because right now we just have a collection of shell scripts and custom commands that an engineer manually ran to create the clusters. This had a bad consequence when we made a change to the configuration on the QA and production cluster, and it prevented deployment to that cluster until we reverted the change.

I am looking to implement Infrastructure as Code (IAC) and a corresponding CI/CD pipeline to prevent this. The goal is that a PR will trigger a build that will spin up a cluster by following the IAC steps, deploy some projects on it, run some tests, then spin down the cluster and provide a test report.

I am investigating Kops, Ansible, Chef, and Puppet as candidates for this task. Right now the shell scripts I mentioned before are mostly kops commands. I have been spending the last few days reading about the other three technologies as well, and I am not quite sure how they might fit into what I need.

Are there any recommendations for my particular use case? Implement IAC and a CI/CD pipeline that can spin up a new k8s cluster with the proposed changes and then spin it down once tests finish?

You could try this:

https://github.com/heptio/aws-quickstart

Their CTO uses it to spin up K8S clusters for his weekly youtube live stream called TGIK, in order to showcase whatever he wants to talk about.

I myself use GKE so I do not need it.


Why does TFS only support 1-to-many Predecessor /Successor relationships? The TFS task tracker only supports 1-to-many Predecessor/Successor relationships (docs).

I totally do not understand why is so.

For me it's obviously a many-to-many relationship, since a task may be blocked by more than one predecessor tasks.

This is the first and only time I've seen such a restriction in such a tracker (for instance, Meistertask supports many-to-many).

Since I can't think of any technical problem to do that (it should be trivial), I'm pretty sure there is a real reason for that. I'd just like to know that reason: I guess it's public, but I couldn't find it.

Note: this is not a rant. This is a very specific question, that may very well have a known answer (although it's possible it doesn't). If you think this is a rant either you misread it, or I miswrote it, or a combination of both.

Because this link has direction from one side to another. Only one work item can on the start of this direction. However, work item:

Can has many successor
Can has many predecessor.

This type of link can not create the cycle, like:

Wi1 successor to WI2
Wi2 successor to WI1


Salt - Need state to check pillar and perform a if/else check This is salt 101 I think, but I'm really a salt N00b and need assistance.
I need to disable specific services on all of our servers by default, but I need to create a pillar that allows me to override this, and allow the specific servers mentioned in pillar to have those services running.  For the sake of discussion, let's assume the service is httpd.
Currently I have a VERY simple state named "disable_httpd.sls" that says this...
{% from "security/map.jinja" import security with context %}
stop and disable httpd-server:
  service.dead:
    - name: httpd
    - enable: False

Now we do have a few web servers where we need to allow it, and obviously my state, which is called as part of our security items, will break those servers.
I need to figure out how to write a pillar that will exempt specific servers.
Thoughts?

Update: Here's what I have tried so far, but it's not working.
Note that this is being tried in a lab, and the system I'm playing with is named "saltminion1.local".

Changed the state (disable_httpd.sls) to say this...

{% from "security/map.jinja" import security with context %}

{% set allow_httpd = salt['pillar.get']('security:options:allow_httpd', "") %}
{% if "True" in allow_httpd and allow_httpd is string %}
Make sure httpd service is running:
  service.enabled:
    - name: httpd

{% else %}

stop and disable httpd:
  service.dead:
    - name: httpd
    - enable: False

{% endif %}

Then in pillar I created a file named "saltminion1.sls" in the "security" directory.
security:
  options:
    allow_httpd: 'True'

I modified pillar/top.sls to add this section...
prod:
  'saltminion1.local':
    - match: nodegroup
    - security.saltminion1

The thought is that it should pick up the "True" in the saltminion1.sls pillar file, and allow httpd.  What it's actually doing is the "else" section of the state, and killing the service.

It looks like you are on the right approach here, the issue seems to be one of debugging. I have found this jinja2 live templating engine to be helpful with debugging in the past.
In python (on which jinja is based) "True" is a keyword which is probably getting interpreted as a boolean data type. This would be in contrast to the string, integer, or floating point (decimal) data types which are a C/C++ concept that python inherits since it's interpreter is written in C. Python tries to be agnostic to data type and variables are not strongly typed. This appears to be a situation in which the lack of strong typing may be getting you into some hot water.
In your boolean logic in the "if" block says to only render the section if allow_httpd is a string - but it isn't! It's a boolean! This is likely the error. Instead, you may wish to try simply:
{% if allow_httpd %}
Make sure httpd service is running:
  service.enabled:
    - name: httpd

{% else %}

stop and disable httpd:
  service.dead:
    - name: httpd
    - enable: False

{% endif %}

I'd also recommend you remove your quotes in the pillar data if you use this approach.


Is there a Radiators for Jenkins and Bamboo? I need to bring our companies rather old build radiators upto standard so it reflects what our current toolstack looks like.
However, we're currently inside a weird limbo between using both Jenkins and Bamboo.

Is there an open source, off the shelf build radiator that can interact with both Bamboo and Jenkins? I've already had a crack at making my own system and decided that my JS skills are not up to the task.

Not essential but bonus points for pulling metrics from other sources like SonarQube and Jira...

It really depends on what information you are trying to "Radiate":

For an off-the-shelf solution that integrates with everything you mentioned you could check out Prometheus.  It has plugins for Jira, Bamboo, Jenkins, and SonarQube.

You could also opt for something quite a bit simpler and use the built in dashboards for the tools you are trying to display metrics for.  Just put them on different tabs and use an extension such as Revolver.

If you want to get a little more customized you can check out Elastic Stack's Kibana.  The book Agile Metrics in Action has several examples of how to capture and display Atlassian data inside of Kibana.


How to perform continuous Integration using Jenkins? I've the basic idea about the Continuous Integration using Jenkins like- basic project setup, setting up source code management bit-bucket and build the project. I have done this so far and following the same approach, whenever any commit done in bit-bucket, Jenkins trigger the build and i get the build status But in my organization there is no expertise who clearly tell us about how CI tool actually used in real time. So we are not aware of the scope and expansions of the Automation project in terms of CI tool.

Can some one guide us about this and guide us to manage the release using Jenkins ? also somewhere the terms Jenkins pipeline confused me what it is and why it used for?

I'm working on Agile, So after completion on each sprint we have to commit out code in master branch only. We are planning to create new branch for each sprint. The vague in my is, we can create the new branch, sprint wise and also create the Jenkins job as per branch, So after committing the new branch Jenkins will trigger the build for that branch. It is the right approach ?

Continuous integration means roughly what you said. Whenever something is committed, it is automatically built and tested, with a red/green light showing up at the end.

Continuous deployment takes this a step further and whenever a green light turns up, automatically deploys that.

Note that "tested" above can mean several steps: smoke tests (does it crash and burn right away), automated unit/feature/integration tests, and then, if you are not superbly confident, also maybe some final manual tests. Also, obviously, for CD, your test setup needs to be sufficiently conclusive so you can deploy with good confidence. You can also do the fully automatic deploy to internal testing/staging areas first, and then only do a production deploy at a later date.

To your questions:

  But in my organization there is no expertise who clearly tell us about how CI tool actually used in real time.

You need to figure that out for yourself as it depends very much on the persons, processes and software you are building/deploying. At first, just build the tools, work with them, and see what happens. If everything works splendidly, continue along the path you laid out yourself; if you hit problems, solve them as they come up.

  So we are not aware of the scope and expansions of the Automation project in terms of CI tool.

The more automation, the merrier. Automation not only means shaving off a few minutes here and there (but this factor also pays of hugely surprisingly fast), but also that everything that is automated, is codified in some script or tool; which means you are much less dependent on individual persons (who may become ill, overloaded, or leave the company) and scale much better if your team grows.

  manage the release using Jenkins ?

I view Jenkins one step "lower" than managing releases. In "pure" CI/CD, the final stage, there would be no releases anymore as every commit would directly and in real-time end up on production anyways. At the other end of the spectrum, you have something like 3 releases per year, each one of them late by 2 months ;). Every project will be somewhere along that spectrum, as dictated by circumstances (not the least of which are your stakeholders and the available money - running a very fast CI/CD is not free).

  the terms Jenkins pipeline confused me what it is and why it used for?

"Pipeline" is just a fancy word for saying that you commit - build - test - deploy. It's just figuratively speaking. The imagery is that you drop your commit into the pipe at the one end, and on the other end you get a running production server.

  So after committing the new branch Jenkins will trigger the build for that branch. It is the right approach ?

Sure, it's worthwhile to have every commit on every branch tested, by the default, if you have the resources for that. It lets you know exactly which build broke everything. Coupled with very small commits, this is a quite powerful tool.

Another useful trigger is to not build every commit, but have Jenkins look at the repository every 15 minutes or so, and build if there is one more more new commits. This may be more realistic when your test suites take longer to run, and if you do not have unlimited amounts of test environments that you can spin up and down regularly.

(By the way, if you are completely new to all of this, including "agile" and branching, then maybe it would be good for you to pick up an existing branching strategy, for example Gitflow, so you don't complicate things too much by (re-)inventing too much stuff at the same time.)


How to connect to postgres container from a node container using docker-compose? I'm trying to connect from a node app container to a postgresql container using docker-compose but I'm not able to do it.
This is my docker-compose.yml file:

version: '3.7'
services:
  db:
    image: postgres:12.1
    ports:
      - "5432:5432"
    env_file:
      - "./.env"
    networks:
      - tcp-modbus
    volumes:
      - ./modbustcp.sql:/docker-entrypoint-initdb.d/init.sql
  redis:
    image: redis:5.0.6
    ports:
      - "6379:6379"
    networks:
      - tcp-modbus
  rabbit:
    image: "rabbitmq"
    environment:
      RABBITMQ_ERLANG_COOKIE: "SWQOKODSQALRPCLNMEQG"
      RABBITMQ_DEFAULT_USER: "*"
      RABBITMQ_DEFAULT_PASS: "*"
      RABBITMQ_DEFAULT_VHOST: "/"
    ports:
    - "15672:15672"
    - "5672:5672"
    volumes:
      - "./enabled_plugins:/etc/rabbitmq/enabled_plugins"
    networks:
      - tcp-modbus
  modbus-tcp:
    image: customImage
    ports:
      - "8080:3000"
    links:
      - "db"
    volumes:
      - "./modbus.env:/usr/src/app/.env"
    networks:
      - tcp-modbus
networks:
  tcp-modbus:
    driver: bridge

I'm using typeorm to set the connection from the node app and was working fine before dockerizing the app, maybe has something to do with the host name of the container, cause I'm using localhost to connect to the postgresql container. Can I use somehow the container network as a hostname to connect to postgres?

docker-compose contains an internal DNS. In the example the service names are also the DNS names. In this case, the DNS of the database is db. If one configure this as a hostname in the node app then one should be able to access the database.

Back in the day, the wordpress example of docker-compose helped me understanding this. When I changed WORDPRESS_DB_HOST: db:3306 to WORDPRESS_DB_HOST: db2:3306, the wordpress was not able to access the mysql db anymore as depicted in the log. When it was changed back and forth to db and to db2 it was not able to access it and with db it was. Then I realized that the service names are actually added to an internal DNS that can be resolved by other services that are defined in the docker-compose.yml.

Recently I realized that running docker-compose up and specifying multiple files using -f also ensures that all services get added to the internal DNS. Initially I thought that the resolving was restricted to the same docker-compose.yml, but probably the docker network is responsible. If multiple apps reside in the same network, in this case the default, then they can resolve each other.


Deployment custom App in Microsoft Teams I am developing a chatbot Azure service, which I want to integrate within Microsoft Teams.
So far everything is working but the "re-install" of the package in the Microsoft Teams.
I created a publish "folder-profile". Then I zip the result with the manifest.json and the icon files inside. I go to the Manage Team section and in the Apps tab I select upload a custom app. Then I choose the .zip file and the service seems to be there (actually it is there).
It works, but when I create a new version and I repeat the described steps, it seems like Microsoft Teams is still using my old code.
I test the chatbot in the Chat by using @"APP-ID" and I see how my changes work, but installed as an App for the "Team" keeps the old version.
I tried to uninstall it, check if the bot is gone(it is gone) and then upload again, but some kind of cache is there and the bot behaves like in the previous version.
Any idea how to deploy new versions of my app in Microsoft Teams?

One could use NuGet. This tool is embedded in various tools, including Chocolatey.


MSSQL and AD in a Docker Container I am attempting to use Active Directory authentication with SQL Server on Linux running inside the official MSFT Docker container for MSSQL (https://hub.docker.com/_/microsoft-mssql-server)

I am in the midst of working through this tutorial, but I heard from a coworker that it wouldn't work in a container. He did not say why. He did say which particular step he was doubtful about ('Join SQL Server host to AD domain')

https://docs.microsoft.com/en-us/sql/linux/sql-server-linux-active-directory-authentication?view=sql-server-2017

Is there any reason why I can't complete the 'Join SQL Server host to AD domain' step if I'm using containers?

You absolutely can do this.  However, you will need to make sure that the following is happening:

Setup the Docker host on the MSSQL computer in such a way that it can reach the AD server.  Using a bridge network may help (especially if you are using Docker for Windows).  If using Docker Toolkit you may need to look into a solution like port forwarding.
Commit your configuration changes to a new Docker image.  This will allow you to move the MSSQL container to a new computer or even run it with Kubernetes/Orchestration tooling.
Map your MSSQL container to a Volume for persistence.  Otherwise the data in the container will be eliminated between container restarts.


Can't get P3 spot instance on AWS I'm new to aws-ec2.

I'd like to make spot request for a p3.2xlarge instance in my local region (EU-Ireland) to benchmark V100 Tesla GPUs against my home GPU for deep learning.

I have already requested and have been granted a limit increase for 1 instance of this type in my region.

I fill in the usual spot request application form changing only the instance type to p3.2xlarge (8 vCPU, 61 GiB EBS only) and the security group to SSH for logging in via my IP.

I can see the instance p3.2xlarge instance type listed among the instance types in the fleet request settings.

p3.2xlarge  8 vCPUs 61GiB   $0.9915/hr  70 %

p3.2xlarge  8 vCPUs 61GiB   $0.9915/hr  70 %

r3.4xlarge  16 vCPUs    122GiB  $0.3019/hr  80 %

r5d.2xlarge 8 vCPUs 64GiB   $0.1512/hr  76 %

r5a.4xlarge 16 vCPUs    128GiB  $0.3023/hr  70 %

r3.2xlarge  8 vCPUs 61GiB   $0.159/hr   79 %

r4.2xlarge  8 vCPUs 61GiB   $0.1903/hr  68 %

i3en.2xlarge    8 vCPUs 64GiB   $0.3/hr 70 %

m4.4xlarge  16 vCPUs    64GiB   $0.3056/hr  66 %

I submit and when the request is fulfilled, I can see the instance type that was assigned is an r5d.2xlarge not a p3.2xlarge. This instance doesn't even have an Nvidia graphics card and is therefore useless for my comparison.

How do I make a spot request for a p3.2xlarge that will be filled by the same instance type as I have requested?

Spot Fleet aims to meet your capacity requirements across a range of instances. This reduces the risk of any unavailability of a specific instance type or family affecting your workload.

As you are interested in only one specific machine type, you will need to remove the other types from your spot fleet request; or just create a separate spot instance request for the p3.2xlarge (a plain spot instance request, not a spot fleet request)


How to have multiple git repos in an Azure DevOps project? My organisation uses Azure DevOps and we want to use git as our repo.

But, Azure projects seem to only allow one git repo per project.

This is bizarre to me! Projects and repos are usually separate. One project may involve development on several tools, and a tool may be used by several projects.

How can I get projects to engage with multiple git repos?

Following works for me:

Click on your Project and then [Repos] in the left hand menu.
Along the top, you'll see the bread-crumbs: Organization / Project / Repos / Files / {NameOfCurrentRepo}
Click the down arrow next to that and you'll see the option to add a new Repository

If that's not available, then it must be a permissions setting that I'm not aware of.


User Data in launch configuration of amazon web services I was trying to create an EBS volume by writing a shell script in the user data of my launch configuration. I don't think there is anything wrong with my bash script. I kept looking up creating launch configuration and writing user data in AWS websites, but could not find anything more than what I did below. For my launch configuration, I used Amazon Linux 2 AMI which has pre-built AWS CLI. So I just wrote the following code in my user data:
    #!/bin/bash
    aws ec2 create-volume \
        --volume-type gp2 \
        --size 100 \
        --encrypted \
        --availability-zone us-east-1c

    aws ec2 create-volume \
        --volume-type gp2 \
        --size 100 \
        --encrypted \
        --availability-zone us-east-1d

As you can see I am trying to create EBS volumes in two availability zones, but I just could not find out why it does not create any volume for me?

Make sure the EC2 instance has permissions to create the volumes. See https://aws.nz/best-practice/ec2-instance-roles/
Check for the errors in /var/log/cloud-init-output.log and (much more verbose) /var/log/cloud-init.log
Be aware that you can't attach volumes from other AZs to your EC2, so not quite sure why you would want to create them from the launch script.
(added based on comments) Specify region: aws --region us-east-1 ec2 create-volume ...

Hope that helps :)


Personalize Jenkins job view per user I am creating a Jenkins job.

This will build some artifacts and will deploy to one of our environments, i.e. qa, staging, production etc. (I am using a choice parameter for this purpose)

However we need to restrict access to environments to specific user, i.e. the QA team should only be able to deploy to the qa env and so on; hence, the choice parameter should display to each user, only the environments he/she can deploy to.

Is this feasible in Jenkins?

Is there any other tool that could someone use to create jobs that present a custom UI (hiding/revealing options/choices) to users based on permissions?

Have a look at the Role Strategy Plugin. It allows you to assign roles per project or node, so you can configure detailed permission as required.

There are other CI tools with extended permission-management as well, but I see no need for them: Jenkins is an open source, stable CI system which works well on all important platforms and became a de facto standard. It has many plugins to archive nearly anything. Even rare use-cases can be handled by writing custom plugins/batch scripts.


What are some methods to measure the ROI for DevOps? DevOps is complex, and involves many non-deterministic aspects like culture and process.
What are some ways to measure DevOps initiatives for success?
How do you prove to a business that the investment they have made is returning (or saving) real dollars?

Great question! Most of us know investing in DevOps practices is a highly worthwhile pursuit for myriad reasons; we don't often justify DevOps on the impact to the bottom line alone, though.

Note: this is something of an opinionated question, and my answer is likewise, opinionated.

Tensibai wisely suggested we find the right metrics, and he used time-to-market as an example. This is a great big-picture approach.

As an alternative approach, my experience with bean-counters is that they don't need to--or necessarily want to--know the big-picture, they just want evidence of fiscal responsibility. They want to see a trend in the right direction.

Here are just a few fiscal wins:

calculate the server costs saved by leveraging auto-scaling in the cloud
for income-generating sites, extrapolate the cost-per-minute of downtime, then show improvements in MTTI and MTTR
again, for income-generating sites, estimate the cost-per-minute saved by leveraging highly-available architecture based on past incidents
if you've improved on your build and deploy pipeline, show that you've reduced regressions and errors in production caused by already-tracked faults
if you've made improvements to developer test environments, or even the tools and configuration on developer laptops, look at commit histories to see if new engineers are making their first contributions sooner after joining
perform a full cost comparison between cloud and on-prem, much like Gitlab did recently, to justify your infrastructure spending (a.k.a. savings!)

Showing that you're money-conscious and you have a few clear wins is often sufficient. I've certainly missed some obvious examples; feel free to add comments below.


What options are available to encrypt the content of git repositories? git-crypt, git secret were found. What is your personal preference and why?

A couple of months ago, I have applied git-crypt. It was quite hard to set it up and the secrets are stored as binaries. I wonder whether git-secret is less cumbersome to set up. What other options are available?

I personally am a fan of Mozilla SOps, we've introduced it to two teams and it's gone down well both times, it has proven easy to implement both with local development and our CI/CD pipelines.

It has support for PGP/GPG to encrypt secrets but also supports Amazon KMS, GCP KMS and Azure KeyVault thus works well in an enterprise cloud-native infrastructure.


Gitlab - Is there any reason not to default enable 'Allow Users to Request Access' for Projects? Is there any reason not to default enable 'Allow Users to Request Access' for Projects?

I can conceive that there may be a reason this is not enabled by default. But I am considering enabling it if possible. If not that may be a feature request.

Anybody know why it is default disabled?

It's just a safeguard for preventing unwanted user accounts from accessing projects. It's disabled by default so that you can control from the start which accounts can access a project and not have unwanted spam of requests for access.

One example I can think of, is if you only wanted users to be able to use their company email/account to access projects, and not their personal account. It could even happen by accident that someone unknowingly requests access with the wrong account.


ERROR: Unable to read database state: No such file or directory I was trying to install Photon by pulling the docker image. But when I run docker build -t photon . I am getting the below error.

 ---> Running in 3076fb42ef8b
ERROR: Unable to read database state: No such file or directory
ERROR: Failed to open apk database: No such file or directory
The command '/bin/sh -c apk add git && git clone https://github.com/s0md3v/Photon.git Photon' returned a non-zero code: 99

I observed the same error before when I tried to install some other application. So, I am guessing this is not something specific to Photon. What is causing this error? How to fix this?

Please try splitting this line:

RUN apk add git && git clone https://github.com/s0md3v/Photon.git Photon

into two:

RUN apk add git

RUN git clone https://github.com/s0md3v/Photon.git Photon

This will tell you exactly which piece of the command fails. If it is the apk add git part maybe try this: RUN apk update && apk add git.

Also you may fixing to a specific tag in the FROM python:3-alpine as 3-alpine follows the latest alpine release which may have some unknown/unfixed issues.

See https://hub.docker.com/_/python/ for available tags.


In a Dockerfile, is there a way to avoid copying files to make them accessible to the RUN command? I need to build a Docker image containing a pre-populate a database.
For now, I am using the following commands in our Dockerfile:

COPY db-dump.gz /tmp
RUN zcat /tmp/db-dump.gz | mysql

But is there a way to achieve the same result without copying first the (potentially large) dump into the container?

At shell-level, I could pipe the data into the container using something along the lines of (untested):

zcat db-dump.gz | docker exec -i $CID mysql

But I don't know if there is any similar option available in a Dockerfile.

You can do this with BuildKit and the experimental frontend. As of 18.09, BuildKit can be enabled with either:

export DOCKER_BUILDKIT=1

for a single shell, or to change the default for the host, you can add to /etc/docker/daemon.json:

{
  "features": {"buildkit": true}
}

You'll need to reload the docker engine after changing the above.

With BuildKit enabled, your Dockerfile would look like:

# syntax=docker/dockerfile:experimental

RUN --mount=type=bind,source=db-dump.gz,target=/tmp/db-dump.gz \
    zcat /tmp/db-dump.gz | mysql

You then build with a docker build . command as before, but with BuildKit enabled.

To see more about BuildKit's experimental features, see https://github.com/moby/buildkit/blob/master/frontend/dockerfile/docs/experimental.md


How to handle different credentials in continuous integration? I am using teamcity to automatically build and test a nodejs application. The problem I am facing is that the application needs to have some environment variables set during the build phase, as well as some secret variables that need to be set during the actual deployment (this is a manual step). These environment variables should be different depending on where the application should be deployed and I have no idea on how to solve this during the actual build.

I guess one solution would be to set up different builds for different environments such as test, stage and production.
Looking into using vault to store my secrets in something like <PROJECTNAME>/<ENVIRONMENT>/secret.

Any ideas on how to solve this problem?

I'm not familiar with teamcity, but you can :

add a config folder in your repo, and inside this repo
create three files (staging.yml, preprod.yml, prod.yml).
on each file you will write down variables specific to each
environment, and apply them within pipeline.

For secrets, it's cautious to handle that manually or better use Vault.

To not throw on error your pipeline/pods, set secrets as optional that could help if you want to set them later.


Build Design - Docker vs Fedora's Mock This post is migrated from a reddit post.  Parts in bold are to adopt it to SE's format.

I'm talking with companies to help streamline/combine their build process.  They must do it together and their current processes are scary.  We've convinced them to go with AWS/jenkins/bitbucket/artifactory.

Now one company wants to use fedora's mock for their build environment deps and build artifact management.  Another wants to use docker for their build environment but does not have a build artifact solution yet.

I suggested a combination of docker (for build env deps) and mock (for product related artifacts and deps) to both.  They scoffed at it.  BTW one of the products is a custom rhel .iso.

What are the pros/cons for mock vs docker in the way the two companies are debating using them?
What about pros/cons for my proposal.. or if it is even a smart thing to do?
Any other suggested tools/build flow that may work better?

Ultimately I am interested in what is the best practice/most adopted method for doing this (build env deps mgmt/product artifact dep mgmt) in the industry that will have continued support and advancements?

Docker is going to be far more well-known than Mock. I consistently see Docker used during build in companies that I work for. I think I have seen Mock before, but only because I am a Fedora user.

However, Docker is not a build tool in of itself, so you can still use Mock inside a container. This is a great way to avoid the "snowflake" build server. By using Docker containers, your builds are consistent no matter what build tool you are using. You can standardise on Docker as a platform for builds, and leave developers to choose the best build tool (whether that's Mock or NPM).

The only other suggestion I'd have is to run the builds on a CI server, and Docker makes this really easy.

Here's one example of a company using Docker for builds (as well as tests and deploys).


What git command could be used to check when a remote branch was created? Git is used at work. The number of branches is growing and growing. The aim is to remove branches that are older than X weeks.

Attempt 1

Running git branch -h will probably indicate what to run to check when a branch has been created.

user@localhost $ git branch -h
usage: git branch [<options>] [-r | -a] [--merged | --no-merged]
   or: git branch [<options>] [-l] [-f] <branch-name> [<start-point>]
   or: git branch [<options>] [-r] (-d | -D) <branch-name>...
   or: git branch [<options>] (-m | -M) [<old-branch>] <new-branch>
   or: git branch [<options>] [-r | -a] [--points-at]

Generic options
    -v, --verbose         show hash and subject, give twice for upstream branch
    -q, --quiet           suppress informational messages
    -t, --track           set up tracking mode (see git-pull(1))
    --set-upstream        change upstream info
    -u, --set-upstream-to <upstream>
                          change the upstream info
    --unset-upstream      Unset the upstream info
    --color[=<when>]      use colored output
    -r, --remotes         act on remote-tracking branches
    --contains <commit>   print only branches that contain the commit
    --abbrev[=<n>]        use <n> digits to display SHA-1s

Specific git-branch actions:
    -a, --all             list both remote-tracking and local branches
    -d, --delete          delete fully merged branch
    -D                    delete branch (even if not merged)
    -m, --move            move/rename a branch and its reflog
    -M                    move/rename a branch, even if target exists
    --list                list branch names
    -l, --create-reflog   create the branch's reflog
    --edit-description    edit the description for the branch
    -f, --force           force creation, move/rename, deletion
    --merged <commit>     print only branches that are merged
    --no-merged <commit>  print only branches that are not merged
    --column[=<style>]    list branches in columns
    --sort <key>          field name to sort on
    --points-at <object>  print only branches of the object

Attempt 2

This post https://stackoverflow.com/questions/3184555/cleaning-up-old-remote-git-branches was found, but this is not an option as it removes branches without checking the age.

Note: this answer shows how to find branches that have been *updated* a long time ago, not branches that have been *created* (i.e., spliced off of some parent) a long time ago. I believe this is what the OP actually wanted (as opposed to the title he chose for the question), as this would lead to good candidates for deletion.

You have to recall that a branch in git is nothing "physical". It is, by definition, and literally, only a simple text file in .git/refs which only has its file name (which is the branch name) and the hash of the tip/head commit. It has no date information or anything else.

So git branch will not help you here.

The object type that has dates associated with it are commits, in git. So you have to list commits. This is done with git log.

So a working command would be:

git log --remotes --before 2018-01-01 --no-walk --decorate

You can read up the details on https://www.git-scm.com/docs/git-log , but in short, it means

--remotes: start with any and all remote branches (substitute --branches for local branches)
--before ...: only list commits older than the date given
--no-walk: only list the very first commit for each branch, don't walk back the history
--decorate: display all tag and branch names associated, just in case

This will give you all branch heads that are older than your given date.


Set container registry auth key via command line/env variable I'm trying to set up Jenkins CI pipeline which will be used to build docker images. I have my auth key stored in Jenkins credentials and I want it to be loaded on start in placed in some environment variable (already done).

Is it possible that Docker will read that environment variable and use the key for authenticating to my container registry?
Or maybe I can pass it somehow via command line?

I've tried searching for that in the docs, but only thing I found is that I can specify that values in config.json and specify the location of that. This is not the best option IMO.

Any ideas? Thanks in advance.

You mentioned that the auth key is stored in Jenkins credentials and are available as environment variables, if so, then your half task is already done; what you can probably do from here is, use those environment variables to perform docker login (assuming USERNAME and PASSWORD are the Jenkins credentials variable), use the below command to login

docker login your.docker-registry.com -u $USERNAME -p $PASSWORD

This assumes, that your registry has SSL enabled, if so, don't forget to place the registry certificates under the below name on your Jenkin host

/etc/docker/certs.d/your.docker-registry/ca.crt

Post this, you would be able to push image to the registry


Is it possible to trigger an AWS Lambda function directly from CloudWatch alarm? Currently, for triggering a lambda function in response to a CloudWatch alarm, I need to route the alarm via SNS, like this:

AWS CloudWatch Alarm --> Send to a topic in SNS --> SNS topic triggers the lambda function

So, is there a way I can do it directly, without SNS in the middle?  [Reason: Less touch points and a bit of cost optimization].

CloudWatch events trigger lambda but there is no way I can trigger alarms via it. The only options available in CloudWatch events for SQS are:

As far as I know, it is not possible. The the offcial AWS Lambda documentation says it support CloudWatch Events, but no mention of CloudWatch Alarms (and CW Events != CW Alarms)

What you are currently doing, sending the alarms on a SNS and using listening on a topic seems the way to go. In fact, CloudWatch Alarms only outputs to SNS so far.


Download Google Cloud Bucket as a file (zip) I use Firebase.
If you want to backup a Firebase bucket, they recommend doing this: https://firebase.google.com/docs/firestore/manage-data/export-import

What is the problem? You are backing up a bucket inside other bucket.
I want to backup my bucket in an external device. I just want to download all my bucket content (or a folder inside that bucket) as a zip file, and save it in my awesome pendrive.

Is there a way to do it? I could not find a way to:
1. Download a bucket as a file in my computer
2. Or, download a folder that is inside a bucket.

How I'm currently doing the backup? Running this in the Google Cloud console

gcloud beta firestore export gs://my-product-name-backup/12-31-2018 --collection-ids=users,problems,seeds,feeback

So, my bucket is called my-product-name-backup. I'm OK with downloading the whole bucket, or just a folder (for example: my-product-name-backup/03-03-2019)

You seem to be confused a bit about how Firestore stores data and what Google is suggesting you do.

  What is the problem? You are backing up a bucket inside other bucket.

This is not correct.  Firestore stores its data inside some sort of mostly opaque database format.  If you are copying this out to a GCS bucket, that really is an export into a different storage medium and a different format.

That being said, it's still reasonable that you might want to keep your data backups outside of Google (although GCS has a good SLA).  To do that, after you've exported your Firestore data to a GCS bucket, use the gsutil utility to copy that bucket down:

gsutil cp -r gs://your-bucket-name .


What is the difference between a node and a machine on OpenShift? OpenShift 4.2 documentation defines:

  Machines

  A fundamental unit that describes the host for a Node. A machine has a providerSpec, which describes the types of compute nodes that
  are offered for different cloud platforms. For example, a machine type
  for a worker node on Amazon Web Services (AWS) might define a specific
  machine type and required metadata.

https://docs.openshift.com/container-platform/4.1/machine_management/creating-machineset.html

While Kubernetes doc defines:

  Nodes

  A node is a worker machine in Kubernetes, previously known as a
  minion. A node may be a VM or physical machine, depending on the
  cluster. Each node contains the services necessary to run pods and is
  managed by the master components. The services on a node include the
  container runtime, kubelet and kube-proxy. See The Kubernetes Node
  section in the architecture design doc for more details.

https://kubernetes.io/docs/concepts/architecture/nodes/

I don't know whether Kubernetes has the concept of machines, and I can't find on OpenShift's documentation its own definition of node (perhaps it uses Kubernetes'?)

And I can't get what's the difference between them.  My current understanding is that a machine represents a raw physical or virtual machine; just the computational resources like cpu, memory and storage.  Nodes, then, would represent a machine that is configured for using with OpenShift, that is, a machine that contains the services necessary to run pods and is managed by the master components.

Another theory would be that machines are just an abstraction to the existing infrastructure, so that persistent labels can be assigned to them for organizing the infrastructure, and as a way to provide information on the underlying infrastructure properties for the nodes.

So...  What is really the difference between nodes and machines?  Is a node in OpenShift exactly the same thing as in Kubernetes?

Your interpretation is correct:

  My current understanding is that a machine represents a raw physical or virtual machine; just the computational resources like cpu, memory and storage. Nodes, then, would represent a machine that is configured for using with OpenShift, that is, a machine that contains the services necessary to run pods and is managed by the master components.

The differences arise on the level of abstraction.

A Node in OpenShift is exactly the same thing as a Kubernetes node. It refers to the abstraction of a worker/master "Node" in a "Cluster". The Node abstraction only exists within the boundaries of a running cluster.
A Machine is an abstraction introduced by the Cluster API which is an ongoing effort to allow deploying K8S clusters using a declarative approach. OpenShift is currently adopting the Cluster API hence they expect you to define the host infrastructure required to run your Nodes using the Machine spec.


How to automate the process of patching Jenkins Slave instances in AWS I've been working on an automated patching solution for some infrastructure in one of our AWS accounts. This AWS happens to host the Jenkins instance we use for CI/CD actions, among other things.

Our instance runs off AWS EC2 instances and are persistent - basically they live long enough that they would be out of compliance if we never patched them.

I have patched Jenkins slaves manually in the past, where the workflow looked something like this:

Log into Jenkins instance
Make sure there are no running jobs on the target node
Mark the target node as offline
Patch the target node
Log into Jenkins instance
Mark the node as online again

This process involves interacting with the Jenkins instance itself. Since I want this solution to be automated, I don't want to have to manually log in and facilitate this process before and after patching. Right now, I have the ability to execute arbitrary bash code from the target node locally and I've been working on a way of using the target node to mark itself offline in some way, and right now my best bet looks like using the environment variables that Jenkins sets while it is running a job to determine whether or not the node is busy.

That will solve the problem of the making sure the node isn't busy, but I have no idea how I will prevent it from accepting new jobs while automation patches the machine. I don't want to use API calls because I believe that would require me to put credentials on all of the Jenkins machines, which I am adverse to doing (Secrets management is a pain). Another possible method could be to prevent the Jenkins master from assigning the slave more tasks by shutting off the slave's SSH, but that sounds like a terrible idea.

Anyone have any ideas as to how I could mark Jenkins slaves offline and online, with my only real tool being local bash execution on the slave itself?

If you use the Self-Organizing Swarm Plug-in to connect your build agents to your master, you can spin agents up/down without having to make any changes on your master. Agents will automatically register themselves with the master. You will have to supply credentials to the agents to self-register. I would suggest using SSM Parameter Store to store them and then retrieve them in userdata.

Your agents can then be completely ephemeral, and updates performed in userdata or by updating the AMI.

See this Terraform module for a more detailed example of implementing this.


How to get GitLab CI to rebuild on JIRA issue status change? I would like to start builds in GitLab CI in response to issue transitions in JIRA.
I have not found a direct way to do that from searching online. I have found ways to get GitLab to tell JIRA what to do, but seemingly no solutions in the other direction.
We have a constraint which would make JIRA to GitLab messages very helpful: we must deploy to servers under the rule that a branch gets deployed

to ...test.example.com when the JIRA issue has status QA
to ...stage.example.com when the JIRA issue has status Stage
to ...beta.example.com when the JIRA issue has status Beta
to ....example.com when the JIRA issue has status Done

Note that we have added a Merge Request field to issues in our JIRA, so we know "the branch for an issue" (and "the issues for a branch").
To satisfy this constraint we run a python script to do deploys for QA, Stage, Beta every hour in a crontab for every project. That script has too many top-level loops, and too many features (being old).
Instead I would prefer to have JIRA notify that an issue is ready for re-build, and then ask our GitLab CI to re-build the relevant Merge Request.
In a JIRA workflow status transitions can "Trigger A Webhook", but it seems we will have to supply our own REST api to which JIRA can post messages, and which can then ping GitLab. That's a new internal app, which will need maintenance, and gain features, so we would like to find out if there are any other solutions.

I didn't get what's wrong with webhooks
You just create a webhook on Gitlab CI side and provide this URL to your Jira account.
https://gitlab.com/api/v4/projects/PROJECT_ID/ref/REF_NAME/trigger/pipeline?token=TOKEN

PROJECT_ID - unique project id on Gitlab
REF_NAME - your qa / stage / beta branches within repository
TOKEN - uniquely generated access token (gitlab provided)
Look here for more details how to trigger build on Gitlab via this call


Aggregating JUnit test results from multiple projects We have a Jenkins setup which utilizes multiple projects for running various suites of tests for our application.

There is one 'parent' project, which is triggered by Git commits. This project in turn builds our 'child' project. Each child projects runs a series of defined tests.

Each project is setup with JUnit to output onto the build page the results of the tests.

In order to get a comprehensive view of commits across our entire project, you need to go to the build page of EACH of our 'child' projects to see the results.

What I would like to do is aggregate ALL the JUnit reports, from builds triggered by out 'parent' project across all the 'child' projects to one place so we can view all the test results together.

Does anyone have experience with something like this?

The JUNIT plugin has a built-in publisher for aggregating test results called "Aggregate Downstream Test Results". You need to make sure that the 'parent' and 'child' jobs are linked together with a downstream/upstream relation. This is usually done by archiving an artifact in the parent build and copying it to the children builds using the 'Archive Artifact' publisher and the 'Copy Artifact' buildstep respectively.


How to implement Continuous Deployment in Angular, PHP API and Ansible projects? I am working on an Angular2 project with Yii2 (PHP/MySQL) as its API point. We will need to deploy the Angular app in different languages for different customers (using AOT compilation from angular-cli).

What is the best way to achieve this? I am looking at Docker or Ansible(-container) to make this work. But the examples are mainly for a 1:1 setup. But I would need a 1:n, something like:

deploy app-de new_costomer prod_server

Additional to that, when a new version of the app is available i want to update the entire network with the new app (that requires uploading of the src/ folder AOT compiled, and db migration scripts on the Yii2 side).

Ideally it would do something like:

deploy update network-all prod_server

Any ideas?

To clarify: I have a number of customers, each customer should get its own container (docker, through ansible-container). There is the AOT compiled Angular stuff (just JS) and the PHP backend with MySQL.

Each customer gets this setup on a server (so 3 customers = 3 docker containers with the Angular + PHP + MySQL).

We push updates to both Angular and PHP quite frequently (i.e. the entire DIST folder needs to be uploaded or at least a diff, migrations scripts for the PHP/MySQL backend needs to run, etc.).

And this from one command (because I obviously do not want to have SSH into each container to do all that semi-manually).

Since I have never done this kind of setup before, I would like to get some ideas on how this can achieved with things like Ansible (or the like).

Let's assume you have the following scenario: You have many customers, but most use the exact same app, except for some configuration changes. You would like each of your customer to get the latest of the app ASAP.

If this is your scenario then try make everything that needs to be differentiated between the apps a configuration change. You build one (or one set of) docker containers that can handle any of your customer based on the configuration it was given. Then on deployment time you simply supply the proper configurations for this container for each of your customer.

The benefit of this approach is that you don't really incur massive overheads when a new customer comes. You generate their configuration, create the new servers, add some entries into an inventory file, and the next time you deploy, it gets there as well. The more well your CD pipeline is set up the more you can automate any of this process.

So how would the flow generally look like:

You have your application codebase, that can cater for all of your customers
You build the docker container for your application (one for the frontend, one for the backend, possibly a few more supporting ones)
You have a deployment script that uses a configuration file to determine where to deploy and what.
You run this deployment script. It will go over all of your customers, and simply deploy the latest version of your app with the appropriate config.

If you have a new customer, the only place you have to change is the deployment configuration file. If you are using ansible most likely you have to change your inventory and your vars file. Let's say you have get a new customer, who would like the app to run under Spanish locale, and has access to the beta features. This customer's configuration would maybe look like this:

config.js:

SETTINGS = { locale: 'es', beta: true };

config.php:

define('CUSTOMER_LOCALE','es');
define('CUSTOMER_BETA','true');

You then make sure that during deployment your container gets these files mounted on startup.


Are manual changes to an aws IAM policy detected by Terraform plan? I know that Terraform dose some checks before the plan and can detect some changes but not others such as detecting changes to variables on a lambda but not detecting a new secret in a secret manager.
Are changes to the IAM policy something it detects?

Yes, Terraform will detect drift in IAM policies.
Here is my starting point, a policy that allows full S3 and SQS access to specific resources.
data "aws_iam_policy_document" "task" {
  statement {
    effect  = "Allow"
    actions = ["sqs:*"]
    resources = [
      aws_sqs_queue.my-queue.arn
    ]
  }

  statement {
    effect  = "Allow"
    actions = ["s3:*"]
    resources = [
      "${aws_s3_bucket.my_bucket.arn}",
      "${aws_s3_bucket.my_bucket.arn}/*",
    ]
  }
}

resource "aws_iam_policy" "task_role" {
  name   = "my-task-policy"
  policy = data.aws_iam_policy_document.task.json
}

I then manually added ec2:DescribeInstances to this policy via the AWS management console.
To test whether it was detected or not, I ran terraform apply -target aws_iam_policy.task_role
An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  # aws_iam_policy.task_role will be updated in-place
  ~ resource "aws_iam_policy" "task_role" {
        arn    = "arn:aws:iam::111111111111:policy/my-task-policy"
        id     = "arn:aws:iam::111111111111:policy/my-task-policy"
        name   = "my-task-policy"
        path   = "/"
      ~ policy = jsonencode(
          ~ {
              ~ Statement = [
                  ... REDACTED...
                  - {
                      - Action   = "ec2:DescribeInstances"
                      - Effect   = "Allow"
                      - Resource = "*"
                      - Sid      = "VisualEditor0"
                    },
                ]
                Version   = "2012-10-17"
            }
        )
    }

Plan: 0 to add, 1 to change, 0 to destroy.

So Terraform correctly detected the drift.


How to revert the Jenkins setup if plugin/Jenkins updates went wrong. I would like to upgrade my Jenkins version(From 2.60.2 to 2.121.1) and some of my plugins. Currently, there are some jobs configured in it. I know there are some configuration changes needed for jobs after upgrading the plugins. Is there any way to restore all the settings(data + conf + plugins) to the previous state if my updates went wrong.

As Tensibai mentioned in the comment - better way is backup you Jenkins instance before an update

There are different ways to do this:

Plugins

Jenkins has a lot of backup plugins, most popular is:

PeriodicBackup Plugin - be aware, has status This plugin is up for adoption right now, that means there is can be lack of support

thinBackup Plugin - the same about current status of plugin

SCM Sync configuration plugin - see attention on the plugin page, this is deprecated plugin, the suggestion about migrate their jobs to Pipeline Plugin and use the Configuration as Code Plugin also make sense

All of them easy to use, also you can discovery any Jenkins plugin here

Your own solution

Jenkins has good structure for backup, it is not hard to understand what exactly you need to backup and how.

You can write script\job with simple steps copy, archive, move to achieve backup of your Jenkins.

More information and discussion about how to backup jenkins can be found here: How do you back up Jenkins jobs & master configs?

As for me

I'm using Periodic Backup plugin + my own scripts for transfering backups into different places. But this plugin is not really stable and has opened issues, so i'm not recommend this for highload production use. That's fine for personal not-critical systems. Now I decided to migrate to my own solution with simple job to archive and copy all configs of my Jenkins instances.


Is Azure's B-2 app service plan better than a S-1 for production environments? When using Azure's WebApps to run C# MVC based applications on production environments, I'm always wondering about when to use the B-2 (Basic-2 with 2-cores and 3.5 GB) and when to use S-1 (Standard-1 with 1-core and 1.75GB).

It sure sounds like the B-2 would result in better performance, still all official recommendations for production use is to run it on S-1 or above. Maybe there's some CPU throttling on B-* plans? Or other 'bottle-neck' that I'm not aware?

Or should I be better off just using the B-2 for production (over a S-1)?

The B-series has this name from burstable adjective.

It means that you do not have 100% of the cores for 100% of the time (see Introducing B-Series, our new burstable VM size).
For example B-2 you have guaranteed only 40% out of a potential 200%.

The suggestion to pick a different serie for production workload, is to guarantee the CPU available.


Same Docker image, multiple configurations I got an application server image built with Docker. However, this image contains an OpenVPN client which needs a different certificate for each client, among other minimal configuration files.

I push this image to multiple servers across the globe. Every of them communicates with each other, thats the reason to use the VPN.

How should I put the configuration for such servers? I think this is out the scope of Docker. Must I use a configuration server like Ansible or so?

One could pass environment variables when running docker, e.g.:

  Additionally, the operator can set any environment variable in the
  container by using one or more -e flags, even overriding those
  mentioned above, or already defined by the developer with a Dockerfile
  ENV:

$ docker run -e "deep=purple" --rm ubuntu /bin/bash -c export


What package manager does the gitlab docker CI image use? I'm trying to install additional software but apt-get is not available.

So in my .gitlab-ci.yml file I have:

image: docker:stable

What is the image based off from?

https://hub.docker.com/_/docker/

Seems it's just a reference to the above... Which uses APK as the package manager


Looking for a Desired State and Configuration as Code central configuration management framework I am looking for a central configuration management framework that supports the following scenarios:

Desired State Configuration. My expectation is to express the desired configuration in code and have the framework ensure it constantly to prevent the configuration drift. And, of course, it should support the pull mode.
All the configuration is managed in the Source Control (git, for example). I.e. no ad hoc scripts.

I come from the software development environment where having code outside of the Version Control is unthinkable. I want to treat my DSC specs the same - everything is in the version control, so when the configuration is changed by anyone it is easy to know exactly when, why, how and who changed it.

I started to look at the Azure Automation DSC, not sure if it fits the bill though - see Is it possible to keep all the DSC configurations in Version Control and make Azure Automation fetch it from there automatically?

Our servers are Windows Servers. Mostly 2012R2 with 2016 being slowly introduced.

Ansible is one example of a tool that that allows you to define all your configuration as scripts stored in your SCM and can deal with configuration drift.  Ansible has modules to work with Windows in multiple ways including DSC through run books.  Full information is available at:  https://docs.ansible.com/ansible/latest/user_guide/windows.html.

For desired state configuration specifically - https://docs.ansible.com/ansible/latest/user_guide/windows_dsc.html#what-is-desired-state-configuration

There is also additional information in writing scripts to insure the system stays in the desired state.  It is important to insure the way your writing the script does not in itself make differences in the system. - https://shadow-soft.com/ansible-idempotency-configuration-drift/

I know of many environments where the full system is defined and deployed with Ansible and it is used to remove any drift.

There are other ways to also help with the situation including using dynamic provisioning to shorten the life of systems.  Only keeping them up for the length of a test or other short time frame.


What would be the best way to improve this Gitlab CI/CD (Gitlab.com) pipeline's efficiency and consistency? This project has a pipeline that runs integration tests against a spun up instance of Gitlab. Each Job in the integration test stage has its own instance testing different build distributions (es, cjs, original source). Unfortunately, the shared runners have limited resources and inconsistently cause these jobs to fail because the spun up GitLab instances have insufficient resources. This occurs more often when I have more than one instance spin up at a time since the jobs run in parallel.
Here is a test repo showcasing the problem and it's pipeline
Gitlab Support has suggested using a dedicated runner OR refactoring my pipeline. Since I'm trying to maintain a $0 cost for this open-source project, the only feasible suggestion (that I'm aware of) is the latter one.
I've tested doing just this, specifically splitting these integration tests into separate stages to stop creating multiple GitLab instances at the same time. This, however, has not improved the results. The only other idea I can think of is running all the integration tests in the same script, instead of separate stages or jobs, which would only require spinning up one instance of GitLab. In this solution, I would have to add some sort of id's to the data being created in the tests to avoid having clashes (i.e., creating a project must use a unique name)
What I want to figure out is if I'm overlooking something? Are there any better ways to improve the pipeline to avoid the resource bottlenecks causing the crashing?

Can you explain the reason for why the "step_script" installs the dependencies and not use a prebuild images that has all the dependencies installed?
Solution
Based on the information you provided, Your test infrastucture rebuild even the test container from scratch.
You tests take a long time to be done as those lines indicate also a fresh composing of the docker container:

(1/11) Installing ca-certificates (20191127-r2)
(2/11) Installing nghttp2-libs (1.40.0-r1)
(3/11) Installing libcurl (7.67.0-r3)
(4/11) Installing expat (2.2.9-r1)
(5/11) Installing pcre2 (10.34-r1)
(6/11) Installing git (2.24.3-r0)
(7/11) Installing c-ares (1.15.0-r0)
(8/11) Installing libgcc (9.3.0-r0)
(9/11) Installing libstdc++ (9.3.0-r0)
(10/11) Installing nodejs (12.20.1-r0)
(11/11) Installing yarn (1.19.2-r0)

If you build a docker image with those thing preinstalled, your Tests will run much faster.
This Process creates clean environment where the only change to it will be the code that has changed!!
The new Docker image then need to be only pulled ones and can be reused.
Build/test infra
docker run -v "$(pwd)"/target:/app:ro -name BUILDID jdalrymple/docker-compose-with-node yarn test
run that for each of your tests.


Can jenkins control baremetal with ipmi I have a setup with a bunch of blades that pxe boot into coreos using terraform/matchbox setup.  I currently control them all manually using ipmitool from my desktop.

Is there a way to let jenkins control this?

an example use case would be if a bunch of different builds are being run and a new one is kicked off jenkins will know to use one of the slaves not running a build so it will power reset the machine and wait for it to come up and then run my docker stuff and the build.

Generic answer yes, it is possible with a lot of work...

Is it something wise to do? In my opinion no, you should convert your bare metal nodes to hypervisors with low maintenance and use common tooling to start slaves Virtual Machines on those hypervisors.


Docker fails to mount volume inside Jenkins scripted pipeline I'm working on a CI Jenkins build with scripted pipeline. From Jenkins on one machine, I'm connecting with already configured Jenkins slave on another machine (M2). I'm able to successfully execute docker commands within the jenkins slave, however it seems that docker is unable to mount the volumes from the slave to the freshly started docker image.

node('testImage') {
   ... //do some job including creation of volumes within the "testImage" container
    stage('Run WildFly instances') {
        //run wildfly relevant
        sh "docker run -d --restart=unless-stopped \
           -v /some/path/to/volume/in/jenkins/slave/container:/some/path/to/volume/within/wildfly14/container \
           -p 8380:8080 \
           -p 9380:9990 \
           -e JAVA_OPTS='-Xms256m -Xmx512m -Djava.net.preferIPv4Stack=true' \
           --name wildfly14 \
          ${DockerRegistryBase}${WildflyImageName}:${WildflyImageVersion}"
}

/some/path/to/volume exists within the jenkins slave docker image, because it's created within the lifecycle of the pipeline, but the wildfly14 container that I'm trying to run is unable to access /some/path/to/volume, because when I run

docker exec -it wildfly14 ls -lR /some/path/to/volume/within/wildfly14/container

it's simply empty. Any suggestions on why is it even happening? I've noticed that the docker containers are set up outside of the jenkins slave container, on the actual machine (M2). Could it be the reason that docker tries to map a volume from outside of the container (/some/path/to/volume) on M2 instead of the location within the jenkins slave?

In configuration settings of Docker plugin for Jenkins, there were missing mountings for the volumes, because I was deploying docker images within docker slave container, so that the "master" docker container (with Jenkins slave agent) couldn't access the real files from the host machine.


Docker - which cache is used to build image I have got my docker image called samplecontainer,
during build I am not specifying --cache-from option.

Is it possible to know which image is used during build as cache?

Answer

The default location of the image cache is under /var/lib/docker.

More answer

The "cache" features in docker are more like a communication vehicle for the documentation, than a proper, separate data structure. There is no special storage which makes up this cache.

For every filesystem-modifying line (RUN/ADD/...) in your Dockerfile, docker run creates a new image, with the image of the previous line (or the FROM line at the beginning) as the base. Docker uses the hashes of the previous image together with the hash of the RUN line to calculate the hash of the new image. If that new hash already exists, then it is reused.

Those intermediate images are exactly the same kind of object in storage as your final image (i.e., a subdirectory of /var/lib/docker, by default), they just don't have a tag pointing to them, so they can be cleaned away somewhat more aggressively by docker image prune -a, for example. But technically they are just the same as tagged images.

All options which talk about caches in the docker documentation (i.e., build --rm=false, build --squashetc.) just modify the behaviour so that said image handling a bit different, but under the hood there's just images, nothing special related to caching.