How can I convert a production Linux server into a docker Image or container? I've a production Ubuntu that contains a huge number of software and dependencies, I want to 'clone' that VPS into a Docker image to be able to replicate the same environment and distribute them in different servers.

Is that possible without any hack?
If yes, what are the big lines to follow?

Is that possible without any hack?

Yes, it is possible. While Docker is often used for microservice-oriented images (i.e., one single service per image), nothing keeps you from converting a monolithic old application to Docker by building a large image, and running arbitrarily many processes inside.

While it is true that a Docker image is not a standalone VM, it still contains everything an equivalent VM would contain - excluding the kernel. I.e., all OS libraries (libc, everything else...) are in there anyways.

  If yes, what are the big lines to follow?

A Dockerfile is just a funky version of a plain old shell script which you could ordinarily use to provision your software with all its dependencies. That is, you chain a lot of commands, which are executed during the build process, and whatever ends up in the file system at the end gets packed up into an image.

So the general route is:

Pick a base image. Depending on your needs and security requirements, you can pick your favourite distribution from Dockerhub. At first, you can pick any Debian, Ubuntu or whatever distribution you prefer and are used to. See below for further considerations.
Those base images usually are pretty small. Hence, your next step is to use RUN apt install... (or whatever package manager your base image uses) to add any standard packages you like.
Finally, you have to install your own software somehow. If you follow the usual approach, you would prepare any binaries and other files you need in the host-side directory alongside your Dockerfile, and then ADD them in.
The last step is to (optionally) create a ENTRYPOINT, i.e., a command that is run when the image is started up, as well as EXPOSE routing information, run as non-root USER or set ENVironment variables.

These are the big lines. If you just do it naively, you will end up with a huge image (gigabytes), and create a new huge image everytime you rebuild, wich can pose serious problems. The next steps are thus:

Switch the base image to a smaller one. alpine is good.
For each RUNcall, make sure there are no temporary/unneccessary files left afterwards. For example, apt install leaves you with a lot of cache junk in /var (same for the other package managers). Due to the way Docker works with its onion-style layered file systems, you have to get rid of the junk within the same RUN command, you cannot clean up in a second RUN. If you google, you will find plenty of examples.
You probably pulled in too many packages with your package manager, fix that.
For installing your own software, the same applies. Avoid having temporary .tar archives lying around which you don't need anymore.

These steps should significantly reduce your image size already. Depending on what you are actually doing, 500'ish MB is a good target to avoid over-optimizing at this stage.

The final step is to optimize everything:

Make sure that you do not have to rebuild every single intermediate image each time, by ordering the lines in the Dockerfile appropriately. You can get away with large images if the "largeness" happens in the first few image layers, if those change seldom, and can be re-used by many final images.
For an intermediate image to be re-used, it must be bitwise identical to another one. So be sure to fix all variable information (like version numbers etc.) as best you can.
Obviously, if you get the hang of all of this, and get your complex, many-part application running inside a single Docker image, then it's time to split it apart across multiple images (rinse, repeat). Anything that communicates by TCP/IP should be an easy choice to do so.


Ansible nested playbooks with variables I have an ansible playbook add_dns_record.yml that register the A record and relative PTR of an host to our MS DNS server.

- hosts: my_ad_dns
  remote_user: "{{ lookup('env', 'USER') }}@MYDOMAIN.COM"
  gather_facts: false
  tasks:
  - name: Create A record
    win_dns_record:
      computer_name: "{{ inventory_hostname }}"
      name: "{{ dns_a_name }}"
      type: "A"
      value: "{{ dns_ip_address }}"
      zone: "MYDOMAIN.COM"
      state: present

  - name: Create PTR
    ...

I have another playbook debian_base.yml used to prepare debian servers:

- hosts: debian
  tasks:

    - name: install python
      raw: test -e /usr/bin/python || (apt -y update && apt install -y python-minimal)

    - name: Set hostname
      hostname:
        name: "{{ inventory_hostname }}"

    - name: Update repositories cache and install pre-requirements
      apt:
        name: "{{ packages}}"
        update_cache: yes
      vars:
        packages:
        - sudo
        - apt-transport-https
        - cloud-guest-utils
        - open-vm-tools
        - screen
        - dirmngr
        - openjdk-8-jdk-headless
        - dos2unix
        ....

I want that every debian server deployed using the playbook debian_base.yml get registered on the MS DNS.

I had a look to import_playbook but it doesn't accept variables.

What are the best practices to achieve that?

the best way I think is to create a var  yml file and include it in your ansible-books:

    - name: Include vars of stuff.yaml into the 'stuff' variable (2.2).
      include_vars:
        file: stuff.yaml
        name: stuff

see include_vars_module


How to skip play in ansible-playbook? I have 2 plays in my sites.yml file:

- name: First play
  hosts: all
  remote_user: root
  roles:
       - role1

- name: Second Play
  hosts: all
  remote_user: ansible
  become: yes
  become_method: sudo
  connection: ssh
  gather_facts: yes
  roles:
     - role2
     - role3

The reason behind this is that, the first play will create ansible user with sudo access. Here, I have used root user to run the play. Then I have to run another play as ansible user.

My question is how to run these play one after another? I first want to run first one then after that second one.

If I run the playbook then it will get stuck because it will first run first play and tries to run second play where it gets stuck.

I am running playbook with the following command:

ansible-playbook -i inventory.ini -l web sites.yml --ask-pass -c paramiko

This is for 1st part.

Although the comments have highlighted the fact that this playbook is fine, as long as it is designed to do what it needs to do:

  The reason behind this is that, the first play will create ansible user with sudo access. Here, I have used root user to run the play. Then I have to run another play as ansible user.

In the first play, you have only one role: role1. If in role1 you have a task that creates the ansible user, then you can use that user to connect the next time. This is actually something many people do, including myself, to put a known baseline on a service:

- name: Ensure Ansible user is present (RedHat)
  user:
   name: ansible
   comment: "ansible user created by bootstrap playbook"
   generate_ssh_key: yes
   groups: wheel
 tags:
 - bootstrap
 when: ansible_os_family=="RedHat"

- name: Ensure Ansible user is present (Debian)
  user:
   name: ansible
   comment: "ansible user created by bootstrap playbook"
   generate_ssh_key: yes
   groups: sudo
  tags:
  - bootstrap
  when: ansible_os_family=="Debian"

- name: update sudoers to ensure ansible user can sudo
  lineinfile:
    dest: /etc/sudoers
    state: present
    regexp: '^ansible'
    line: 'ansible ALL=(ALL) NOPASSWD: ALL'
  tags:
  - bootstrap

This will ensure that the user is added, and included in the right groups, with sudo rights -- it is just an example of how you could achieve this, there are other strategies of course.

  My question is how to run these play one after another? I first want to run first one then after that second one.

That's exactly how playbooks work.

  If I run the playbook then it will get stuck because it will first run first play and tries to run second play where it gets stuck.

If you do not have a task to configure the user required in the next play, then yes, it will fail (not get stuck).
The point is that since Ansible is idempotent, you don't have to worry about skipping the play because if the user is properly configured, the first play will simply have no effect.

However, to answer the actual question

  How do I skip a play in an Ansible playbook ?

This is done using the conditional execution using when


Combine Docker Swarm and Kubernetes My company is trying to play a little catch-up in the DevOps space. I have been doing a lot of research on containerization of applications and the orchestration systems that go with it. I came across an article (one that I wish I had saved) where they were talking about combining Swarm with Kubernetes to get better functionality. In this article they did not define what they gained by doing that.

I wondered what benefits would this provide? Is adding the extra layer of complexity really going to give you much return?

EDIT: I am looking for technical pro's/con's. KISS is a good motto but doesn't hold up in a debate with your CEO or Board.

I am almost positive that we will be selecting Docker for our containers and Swarm for an orchestration. I, however, would like to see Kubernetes in our space so the proposition that you could merge the technologies together for a more robust solution intrigues me.Thanks for any insight.

Update: Docker just released support for Kubernetes as scheduler, which changes the situation and makes Kubernetes just an alternative scheduler to Docker Swarm.

TL;DR: DON'T DO IT. Engineers always try to create these dog-pigs. Every unnecessary technology you bring will bring another whole set of faults. If you can pick one, then pick one and be happy you don't have to do both. If you like to play with Kubernetes, just get a private account on Google Cloud and play with it as much as you like. But don't make everyone at your company suffer through unnecessary complications.

Those are two parallel and mostly equivalent technologies. If your business had legitimate business reason to deploy in multiple Cloud providers for reliability for example and wanted to deploy into both AWS ECS (Elastic Container Service - based on Docker) and Google GKE (Container Engine - based on Kubernetes) and you were asking how do you build a pipeline, which would build your software and package in containers for deployment in both, that would be something else, but doing it just because you want to play with a new technology is very irresponsible.


Bitbucket Pull Request Builder Jenkins job is not getting invoked I have enabled web hooks pull request in Bitbucket like below -

And my Jenkins job configurations are as follows -

Bitbucket hooks are getting invoked as soon as I create pull request but Jenkins job is not getting invoked.
Can anyone help please?

It took me a while to understand this as well as I had experience with Bitbucket Server and not Bitbucket Cloud which is different.

You do not need to enable the webhook for the Bitbucket Pull Request Builder plugin in Jenkins.

Just ensure you set the RepositoryName and RepositoryOwner properly from the URL of your repo as such: https://bitbucket.org/RepositoryOwner/RepositoryName

Once you have that set, a new PR or commit to branch of existing PR will trigger your job.

P.S. There's a bug in the plugin where you can't use git for other repos such as workflow library or pipeline from scm. Git will try to use the same hash to checkout all repos which will fail the build.

P.P.S. The webhook in Bitbucket cloud uses a POST with Json data so that can't be used with the git plugin in Jenkins unless you don't care about the hash or branch.

EDIT:
One last note, I've since moved to use the multibranch pipeline with bitbucket branch plugin and that works great with the webhook as well as allows us to use a workflowlib library


Which S3 buckets is Lambda allowed to read code from? There doesn't seem to be any way to define a policy about which S3 buckets Lambda is allowed to run code from.

With most services in AWS, the usual practice is to define an IAM role that can be assumed by the AWS service so that it can perform work on user-created resources in the account. For instance, Kinesis Firehose isn't allowed to write into one of your S3 buckets until it's explicitly granted permission with an IAM role.

Even with Lambda we have the concept of an execution role, that is the IAM role that can be used by custom code in our Lambda. But that still takes place after the code has been loaded.

This represents a misunderstanding of when Lambda code is loaded.

AWS Lambda does not download a copy of the code from S3 every time a function is executed. Instead, Lambda has its own internal storage that is updated when invoking CreateFunction or UpdateFunctionCode either through the API or the Console.

Update with further clarification from jdog:

  Lambda is never given access to S3 buckets -- the user or role that runs CreateFunction is.


Is there a [ci skip] option in GitLab CI? I'm debugging the use of a report creator. I forgot a line and would like to edit the code without triggering the build pipeline.

I know that it's possible to write [ci skip] in the commit pipeline to pass the pipeline execution in Jenkins CI, but I'm using GitLab CI.

Does the same mechanism exist for GitLab CI? I couldn't find it in the docs.

Yes, there is:

If your commit message contains [ci skip] or [skip ci], using any capitalization, the commit will be created but the pipeline will be skipped.
Alternatively, one can pass the ci.skip Git push option if using Git 2.10 or newer:
git push -o ci.skip

From: GitLab CI YAML documentation - "Skip Pipeline"


Jenkins cron works only for the 1st line I'm trying to use Jenkins to run a small monitoring script every minute on two separate ECS clusters.
In the text box that opens when you enable "Build periodically with parameters" I wrote:

* * * * * %CLUSTER=Cluster1;

* * * * * %CLUSTER=Cluster2;

This works fine for Cluster1 but it does not run for Cluster2. It does not fail, it just doesn't run. If you switch the lines it works for Cluster2 (which is now at the top) but not for Cluster1. I made sure the node it runs on has enough runners, and that "Execute concurrent builds if necessary" is enabled. The closest thing I found while googling is this, and I checked but there is a blank line after these and adding another or removing it or adding a space between them does not help.
I have a bad workaround which is just squeezing both runs in an unparametrized build and that works but it is inflexible.
So to reiterate the question, why does Jenkins not run concurrent builds from cron, and what can I do to change that?
Thanks.
EDIT:
When you insert a cron schedule of * * * * *, Jenkins posts the following:

Do you really mean "every minute" when you say "* * * * *"? Perhaps you meant "H * * * *" to poll once per hour.

As I need it to be on a per minute basis this is (for the 1st one, but not for the 2nd) the desired result.
Based on Ian W's suggestion to do replace the schedule from "* * * * * " to "H/2 * * * *". I checked it but this still led to only the 1st one being run, and would have caused it to run every other minute and not every minute as intended.
H/1 doesn't run at all, and H/0.5 returns an error of

Invalid input: "H/0.5 * * * * %CLUSTER=Cluster1;": line 2:3: step must be positive, but found 0

I also tried H/1.1 but that returned:

Invalid input: "H/1.1 * * * * %CLUSTER=Cluster1;": line 2:4: unexpected char: '.'

The closest I got so far was staggering the cron jobs like so:

H/2 * * * * %CLUSTER=Cluster1;

* * * * * %CLUSTER=Cluster2;

Which caused both jobs to run every 2 minutes.
BTW, the Jenkins version is 2.204.1, the Parametrized Scheduler plugin is on version 0.8, if any other plugins are relevant I can post their versions as well.

I believe your issue is the use of * * * * *.
The parameterized-scheduler-plugin example has two different times specified.
#lets run against the integration environment at 15 past the hour
15 * * * * % env=int
#run QA too
30 * * * % env=qa

So one job runs @15 and one @30.
Yours specify the same time, so only the first is executed.
You could try the random time parameter "H" or fix some different interval on each.
Or, it does say "Leave a space. Put in a %. Then add the name=value pairs". So maybe it's cronexp space percent space name=value ?
OR you could just copy the job job and schedule two separate jobs w/fixed parameter.
But Jenkins does sound like the wrong tool for this.
The above will address why only the 1st line was executed.
NOTE :There is a known issue attemptimg to to every minute, see JENKINS-22129 H/2 will run every 2 minutes, but H/1 will run every hour.


Jenkins Rerun Old Build with Specific Commit I am pretty new to jenkins and trying to figure out the best way to accomplish rollbacks.  Currently this is my pipeline:

Obtain a webhook from beanstalk (git versioning)
Build the project
Obtain the build artifact and deploy to Azure

This is working great - the last thing I would like to solve is being able to rollback the project to a specific build.

When I do a "replay" however, it doesn't grab the actual commit the build ran on - it grabs the current commit in the repo.  So it doesn't build the project from the commit it originally ran on - but the latest commit in the repo.

pipeline {
agent any
stages {

    stage('Install Dependenciess') {
        steps {
            bat'npm install'
        }
    }

    stage('Build the Project') {
        steps {
            bat'npm run build'
        }
    }

    stage('Deploy to Azure Prod') {
        when {
            environment name: 'REPOSITORY_BRANCH', value: 'master'
            beforeAgent true
        }
        steps {

            timeout(time:5, unit: 'DAYS') {
                input message: 'Approve Deployment?'
            }

            azureWebAppPublish azureCredentialsId: env.AZURE_CRED_ID,
            resourceGroup: env.RES_GROUP,
            appName: env.WEB_APP,
            filePath: "**/*.*",
            sourceDirectory: "build"
        }
    }

    stage('Deploy to Azure Development') {
        when {
            environment name: 'REPOSITORY_BRANCH', value: 'development'
            beforeAgent true
        }
        steps {

            azureWebAppPublish azureCredentialsId: env.AZURE_CRED_ID,
            resourceGroup: env.RES_GROUP,
            appName: env.WEB_APP,
            filePath: "**/*.*",
            sourceDirectory: "build"
        }
    }

}

post {
    always {
        archiveArtifacts artifacts: "build/**/*.*", onlyIfSuccessful: true
        cleanWs()
    }

    success {
        mail bcc: '', body: "Example
Project: ${env.JOB_NAME}
Build Number: ${env.BUILD_NUMBER}
 URL de build: ${env.BUILD_URL}", cc: '', charset: 'UTF-8', from: '', mimeType: 'text/html', replyTo: '', subject: "ERROR CI: Project name -> ${env.JOB_NAME}", to: "";
    }

}

}

As a short term solution you can replay you last good build. Read the docs for more information

Update: There is a bug , so the proposed way will work only in multibranch pipeline. So as a short term solution you can convert your job to a multibranch pipeline and to use a replay when it’s needed

As a long term solution I would separate build and deployment job and store the build artifact in private npm registry. In that way you can always pick whatever version is needed to deploy


How to add rules to Kubernetes NGINX Ingress controller from different yml definitions? I'm trying to implement a Kubernetes cluster with a fanout pattern using an NGINX Ingress which routes traffic on multiple different hosts and paths to different microservices. The code for each microservice is hosted in their own Git repo, including the kubernetes definition yml files which deploys the pods (Deployment) and services for each service respectively.

Considering that there will be a single Ingress which routes traffic to all the different Services, where (by convention or industry standard) should the definition file be stored? I would like to keep the Ingress rules related to a single microservice in the git repo where the rest of the code is. I don't particularly want a separate git repo just for the Ingress.

Is it possible to define only a subset of an Ingress' rules in a file which would then get "appended" to the list of rules instead of overwriting them.

For example, could I have these two different definitions which creates a single Ingress with two rules (based on having the same name):

foo-ingress-rules.yml

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: nginx-ingress
spec:
  tls:
    - hosts:
      - foo.bar.com
      secretName: tls-secret
  rules:
    - host: foo.bar.com
      http:
        paths:
        - path: /
          backend:
            serviceName: foo-web-svc
            servicePort: 80
        - path: /api
          backend:
            serviceName: foo-rest-svc
            servicePort: 80

baz-ingress-rules.yml

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: nginx-ingress
spec:
  tls:
    - hosts:
      - baz.bar.com
      secretName: tls-secret
  rules:
    - host: baz.bar.com
      http:
        paths:
        - path: /
          backend:
            serviceName: baz-web-svc
            servicePort: 80
        - path: /api
          backend:
            serviceName: baz-rest-svc
            servicePort: 80

I was able to combine your two example files by first converting them to JSON.

foo.json

{
  "apiVersion": "extensions/v1beta1",
  "kind": "Ingress",
  "metadata": {
    "name": "nginx-ingress"
  },
  "spec": {
    "tls": [
      {
        "hosts": [
          "foo.bar.com"
        ],
        "secretName": "tls-secret"
      }
    ],
    "rules": [
      {
        "host": "foo.bar.com",
        "http": {
          "paths": [
            {
              "path": "/",
              "backend": {
                "serviceName": "foo-web-svc",
                "servicePort": 80
              }
            },
            {
              "path": "/api",
              "backend": {
                "serviceName": "foo-rest-svc",
                "servicePort": 80
              }
            }
          ]
        }
      }
    ]
  }
}

baz.json

{
  "apiVersion": "extensions/v1beta1",
  "kind": "Ingress",
  "metadata": {
    "name": "nginx-ingress"
  },
  "spec": {
    "tls": [
      {
        "hosts": [
          "baz.bar.com"
        ],
        "secretName": "tls-secret"
      }
    ],
    "rules": [
      {
        "host": "baz.bar.com",
        "http": {
          "paths": [
            {
              "path": "/",
              "backend": {
                "serviceName": "baz-web-svc",
                "servicePort": 80
              }
            },
            {
              "path": "/api",
              "backend": {
                "serviceName": "baz-rest-svc",
                "servicePort": 80
              }
            }
          ]
        }
      }
    ]
  }
}

You can then combine the rules for each by using jq.

levi@La-Tower:~$ jq 'reduce inputs as $i (.; .spec.rules += $i.spec.rules)' foo.json baz.json
{
  "apiVersion": "extensions/v1beta1",
  "kind": "Ingress",
  "metadata": {
    "name": "nginx-ingress"
  },
  "spec": {
    "tls": [
      {
        "hosts": [
          "foo.bar.com"
        ],
        "secretName": "tls-secret"
      }
    ],
    "rules": [
      {
        "host": "foo.bar.com",
        "http": {
          "paths": [
            {
              "path": "/",
              "backend": {
                "serviceName": "foo-web-svc",
                "servicePort": 80
              }
            },
            {
              "path": "/api",
              "backend": {
                "serviceName": "foo-rest-svc",
                "servicePort": 80
              }
            }
          ]
        }
      },
      {
        "host": "baz.bar.com",
        "http": {
          "paths": [
            {
              "path": "/",
              "backend": {
                "serviceName": "baz-web-svc",
                "servicePort": 80
              }
            },
            {
              "path": "/api",
              "backend": {
                "serviceName": "baz-rest-svc",
                "servicePort": 80
              }
            }
          ]
        }
      }
    ]
  }
}

While yaml is more common JSON is perfectly valid for the Kubernetes API. All credit for the jq logic goes to @Jeff Mercado and his answer.


Where would DevOps Engineers mostly look for their next opportunity and what would it be? Where do DevOp Engineers find their next opportunity? Users Groups, Associations, Job Boards?

What do DevOps look for in their next role?

Understand that the term DevOps refers to both a Strategies and Tactics.
For instance, the strategies of Continuous deployment and Consistent development/production environment.
The tactics may be using Kubernetics for deployment to the cluster with Docker for development/production environment.
With that said, a DevOps shop will likely use Kubernetics and Docker as their tactical tools and would look for people with those skillsets.

ALSO, they would look for people who are a CULTURAL fit who would work well in a collaborative environment.  If your background is let's say operational and your not open to learning programming skills(or learn what the programmers need), don't take ownership of a project or work well in a team...you will NOT be a fit for the company (you won't get hired).

As far as the best place to find a DevOps related job?

I would look at going to a DevOps Meetup, go to trainings/seminars where DevOps tactics and tools are taught (i.e. Docker, Vagrant, Ansible, Kubernetics, etc.), look for certain keywords in the posts such as SRE (Site Reliability Engineer), Systems Automation, etc... and do some research on companies that are moving to or have embraced DevOps Strategy.

If you understand these things, I believe it will give you a great advantage at landing a job in one of those companies.


How to attach two (or more) networks to a container so they are available when the container starts? I noticed if you try to attach a container to two networks, the following syntax does not cause an error, but effectively attach the container only on the network specified by the last --network:

sudo docker run \
        --network front-end \
        --network back-end \
        --name app1 ${IMAGE}

sudo docker network inspect front-end | jq '.[].Containers[].Name'
# <-- nothing here
sudo docker network inspect back-end | jq '.[].Containers[].Name'
"app1"

How to attach two (or more) networks to a container so they are available when the container starts?

Do that:

You cannot attach several networks using the --network option. You have to rely on the docker network connect command. So if you want the multiple network interfaces to be available at container startup, you have to do it in three steps:

create the container,
connect the extra network needed,
and finally run the container.

# create a container, attaching it to one user-defined bridge:
docker create --name app3 \
              --network back-end \
              ${IMAGE}

# connect the other bridge networks:
docker network connect front-end app3

# start the container
docker start app3

docker exec -it app3 ip addr show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
465: eth1@if466: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
    link/ether 02:42:ac:13:00:03 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 172.19.0.3/16 brd 172.19.255.255 scope global eth1
       valid_lft forever preferred_lft forever
467: eth0@if468: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
    link/ether 02:42:ac:14:00:03 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 172.20.0.3/16 brd 172.20.255.255 scope global eth0
       valid_lft forever preferred_lft forever

Dont' do that:

It is tempting to do that instead:

# You probably DON'T want this:
# create a container with the default network settings
docker create --name app3 \
              ${IMAGE}

# connect the user-defined bridge networks:
docker network connect back-end app3
docker network connect front-end app3

# start the container
docker start app3

However, there is a pitfall here: when creating the container without an explicit --network option, Docker attaches it to the default bridge. Then the two network connect commands will add two more networks. And the container end-up bein attached to the front_end and back_end networks as expected. But also to the default bridge, something you probably don't want when using user-defined networks.


How to avoid OOMKilled with azcopy on k8s When running azcopy copy with huge repos, or when running azcopy bench, the pods are killed with the error OOMKilled.  (azcopy v10)

The pod has to following resource limitations:

limits: { cpu: 1, memory: 2Gi }

azcopy (v10) tries to use all available memory to buffer the blobs on transfer. It seems that the calculated memory size is incorrect and exceeds the pod memory limit.

This can be controlled with the option AZCOPY_BUFFER_GB, as described in the azcopy env output below.

  INFO: Name: AZCOPY_BUFFER_GB

  Description: Max number of GB that AzCopy should use for buffering data between network and disk. May include decimal point, e.g. 0.5.
  The default is based on machine size.

So for a pod limit of 2GiB, you can set the buffer a bit lower:

export AZCOPY_BUFFER_GB=1.5
azcopy bench ...


GitLab CI/CD page listing many gitlab runners that no longer exist I am running GitLab Runner using a Helm chart as described on the GitLab Runner Helm Chart page.

Everything seems to be working, but after about two months of running I see on my GitLab group that controls the GitLab runner on the "CI / CD Settings|Runners" page the message

  "Available group Runners: 54".

Here is the first couple on that page:

Rather than listing all 54 here is the list run through uniq -c:

      3 gitlab-runner-gitlab-runner-9fcb644b9-4f5ss
      2 gitlab-runner-gitlab-runner-9fcb644b9-4gqqq
      1 gitlab-runner-gitlab-runner-9fcb644b9-889rc
      2 gitlab-runner-gitlab-runner-9fcb644b9-grb5j
      1 gitlab-runner-gitlab-runner-9fcb644b9-hhv8f
      1 gitlab-runner-gitlab-runner-9fcb644b9-kzcdv
     13 gitlab-runner-gitlab-runner-9fcb644b9-nvrms
      7 gitlab-runner-gitlab-runner-9fcb644b9-pl2k4
      2 gitlab-runner-gitlab-runner-9fcb644b9-sxlmp
      9 gitlab-runner-gitlab-runner-9fcb644b9-vdmsj
      3 gitlab-runner-gitlab-runner-9fcb644b9-vgdnv
      8 gitlab-runner-gitlab-runner-9fcb644b9-wst8q
      2 gitlab-runner-gitlab-runner-9fcb644b9-xnzhx

Looking at my GitLab Runner Kubernetes container there seems to be only one runner:

$ /usr/bin/gitlab-runner list
Runtime platform                                    arch=amd64 os=linux pid=4953 revision=1b659122 version=12.8.0
Listing configured runners                          ConfigFile=/home/gitlab-runner/.gitlab-runner/config.toml
gitlab-runner-gitlab-runner-9fcb644b9-grb5j         Executor=kubernetes Token=XXXXXXXXXXXX-XXXXX URL=https://myserver.example.com/

Another potentially important piece of information: the Kubernetes node that hosts the GitLab Runner container is preemptible meaning it gets killed and restarted about every 24 hours.

How do I convince GitLab that these old runners are not still around? And how do I avoid this problem in the future?

Just unregister offline runners:
As per GitLab Runner commands manual:

To delete the old and removed from GitLab runners, execute the following command.

    gitlab-runner verify --delete

You are not alone
P.S. The problem is common: there are many issues with questions like yours.
Moreover, there are several custom recipes to unregister "offline" runners:

bash script
API oneliner with curl
python script

API for "zombie" runners
As rlandster noted, sometimes you'll need an API
to unregister "zombie" runners.
curl -S --header "PRIVATE-TOKEN:<token>" "https://gitlab.example.com/api/v4/runners/all" | jq '.[] | select(.status == "offline") | .id' | xargs -I runner_id curl -S --request DELETE --header "PRIVATE-TOKEN:<token>" "https://gitlab.example.com/api/v4/runners/runner_id"`

This will list all runners, instead of the personal ones: https://docs.gitlab.com/ee/api/runners.html#list-all-runners
P.S. note about runners
Pls note that Gitlab itself doesn't manage runners. So, e.g. to restart or shutdown the runner, use appropriate commands on the runner's hosts.


Gitlab CI Runner command line doesn't care about --parameters I'm trying to write a windows batch file to automate the gitlab runner registration :

SET /p token=Token ?:
gitlab-runner register --locked false --run-untagged true --tag-list java --name foo --token %token% --url https://myurl --executor

I found the parameters in help documentation (gitlab-runner register --help) but it doesn't care about them. It still continues to ask question whose answers are in the parameters.

Thanks for helping

You're missing the --non-interactive tag to your registration command.

Full command should be:

gitlab-runner register --non-interactive --locked false --run-untagged true --tag-list java --name foo --registration-token %token% --url https://myurl --executor shell

Relevant documentation for non-interactive registration.


CI/CD with a GPU cluster With typical continuous integration environments, you configure an environment capable to execute compilation and test batches (agent, slave..) coordinated by a scheduler (master, server..)

But what if your "client" environment is a Graphics Processing Unit (GPU) cluster used to perform model trainings in different configurations? Is there any difference or would you just for example let the head cluster node incorporate a Jenkins slave? (or Bamboo agent etc)

Since you're talking about CI/CD I presume you have the possibility to automate the model trainings in those configurations. Let's call the scripts able to do that train_model_config_A, train_model_config_B, etc.

Then you could have a wrapper script which checks an environment variable used to select which client environment you desire and invokes the corresponding train_model_config_<blah> script. Ideally translating the outcome of the training (whatever that is) into one or more results of the pass/fail type. Then such wrapper script can be integrated in a CI/CD pipeline like a custom test step/stage (or even a build one, if it produces any artifacts you might want to archive). Just like any test executed on a testbed incorporating some non-generic piece of hardware. In other words the CPU cluster makes no real difference.

You might not need to install the slave directly on the GPU cluster if the train_model_config_<blah> scripts (and thus the wrapper script as well) can be executed on some other hosts and remotely control the GPUs - you can then have the slave on some other host, leaving the GPU cluster free to only do its stuff.


How to pass variables between build/deploy tasks in a TaskGroup in VSTS 16.131.28601.4 I have a multi-configuration web application I'm using parallel deployments, leveraging Task Groups for reusing my deployment code.

I've been using Powershell tasks and the ##vso[task.setvariable variable] but those only are readable by other tasks (particularly non-PowerShell tasks) if they correspond to top-level variables at the Build/Release level.  Variables that are only local to the task-group.

Basically, the behavior described in Issue #3116

Does anybody have a better workaround for this issue besides binding all your variables right to the root with identical names?

I'm not sure my approach is any "better". However, what I have done in the past was to use Table Storage or Cosmos DB to store the key-value pairs that needed to be persisted between each stage.

The general schema was:

Key: buildNum-variableName
Value: variableValue

I then created various wrappers for the SDK to access from PowerShell and MSBuild, this also enabled chaining multiple builds together to create a CI/CD pipeline.


Looping through hosts from an inventory group in a group_vars file Is it possible to do something like this:

# inventories/inventory1/group_vars/group1

filewatches:
    {% for host in groups['group2'] %}
    - type: "copy"
      directory_name: "{{ host }}"
      shell_script: "{{ script_dir }}/ascript.sh"
     {endfor %}
    - type: "anotherone"
      ...

# inventories/inventory1/hosts

[group1]
somemachine01

[group2]
machine1
machine2
machine3
...

Basically, I want to generate a bunch configuration that is similar except for just one variable (in this case the directory_name variable). I would rather do this than making 20+ different entries which all differ by JUST the directory_name variable.

I did not include the playbook or roles information for the sake of clarity/terseness. I hope I was clear enough and gave enough context, and I appreciate any help.

Thanks!

Yes. It is possible to create a YAML data structure with Jinja and read the variable in with the filter from_yaml.

The play

- hosts: group1
  vars:
    script_dir: /tmp
  tasks:
    - debug:
        var: filewatches

with the group_vars

$ cat group_vars/group1.yml
filewatches: "{{ filewatches_var|from_yaml }}"
filewatches_var: |
    {% for host in groups['worker'] %}
    - type: "copy"
      directory_name: "{{ host }}"
      shell_script: "{{ script_dir }}/ascript.sh"
    {% endfor %}

and with the inventory

$ cat hosts
...
group1:
  hosts:
    test_01

group2:
  hosts:
    test_02
    test_03

gives

ok: [test_01] => {
    "filewatches": [
        {
            "directory_name": "test_02",
            "shell_script": "/tmp/ascript.sh",
            "type": "copy"
        },
        {
            "directory_name": "test_03",
            "shell_script": "/tmp/ascript.sh",
            "type": "copy"
        }
    ]
}


Tunnel docker traffic though VPN or SSH, during build stage Well, the title says all. I have no success trying to route docker's network traffic through an active VPN connection (openconnect) or SSH tunnel (sshuttle). Docker creates a virtual network interface called docker which seems to be ignoring all the iptables rules that sshuttle adds. In VPN case however the entire docker process disconnects while OS's traffic is being routed through VPN.

Please note that I am not trying to push a contained behind VPN. The problem occurs specifically on build stage.

Docker actually manipulates iptables behind the scenes, so there is a good chance that sshuttle is interacting with docker in unpredictable ways.  Any rules that need to be added to IP Tables that affect the docker interface need to be added to the DOCKER-USER chain as per Docker and iptables.

It's quite an involved topic so I would also consider reading an exploration of Docker networking and the overview of the docker network.

There is also an old question on StackOverflow but based upon the comments the current highest voted answer no longer works.


Unable to mount docker folder into host using docker-compose Here I created docker container which use the mariadb image and created three volumes as below.
docker-compose.yml
version: '2.0'

services:
  mariadb:
  image: mariadb:latest
  restart: always
  container_name: mariadb
  environment:
    - MYSQL_ROOT_PASSWORD=root
    - MYSQL_DATABASE=testdb
  ports:
   - 3307:3306
  volumes:
   - ./database:/var/lib/mysql
   - ./_conf/mariadb.cnf:/etc/mysql/my.cnf:ro
   - ./logs:/var/log/mysql

First two volume works successfully but I can't able to find mariadb-logs files into logs folder. I seems logs folder showing blank on host as well as on container(/var/log/mysql). I think host folder override into docker file system.
If I remove this volume(./logs:/var/log/mysql) from docker-compose then logs files are showing on container.
My plan is to mount /var/log/mysql/ folder to local machine.
Thanks !

If you use CentOS 7 (as I have ). You may have SELinux turned on by default.check https://www.rootusers.com/how-to-enable-or-disable-selinux-in-centos-rhel-7/
or mount like this:

  volumes:
   - ./database:/var/lib/mysql:Z
   - ./_conf/mariadb.cnf:/etc/mysql/my.cnf:Z
   - ./logs:/var/log/mysql:Z


How to set ping response of AWS ec2 instance? I hosted one website in AWS. When I try to open in a browser it's working properly. But when I try to ping same website that time it's not sent a response. so how to enable ping response.

When we created a new instance in aws ec2, by default aws security group block all protocol and port.

AWS security groups block ICMP (including ping, traceroute, etc.) by default. You need to explicitly enable it.

So, enable ICMP protocol (ping response) follow below steps.

Go to EC2 Dashboard and click "Running Instances"

on "Security Groups", select the group of your instance which you need to add security.

click on the "Inbound" tab

Click "Edit" Button (It will open a popup window)

click "Add Rule"

Select the "Custom ICMP rule - IPv4" as Type

Select "Echo Request" as the Protocol (Port Range by default show as "N/A)

Enter the "0.0.0.0/0" as Source

Click "Save"


How to prune all but two most recent docker images? I'm building images on a small server and spinning them up with docker-compose.
When it's disk gets full I run docker prune -a so all the stopped containers, dangling image are cleaned out.
But I'd like to keep one or two recent images in case I need to roll back quickly.
The docker prune documentation says --filter until=<timestamp>. But the timestamps from the previous images could be days, weeks or months old.
Then suggested method for pruning by date is:
a) find the timestamp by listing the images with this format
docker images --format 'table {{.Repository}}\t{{.Tag}}\t{{.ID}}\t{{.CreatedAt}}\t{{.Size}}'
which produces
REPOSITORY          TAG                 IMAGE ID            CREATED AT                      SIZE
foo                 latest              2f287ac753da        2017-01-04 13:42:23 -0800 PST   3.98 MB
alpine              latest              88e169ea8f46        2016-12-27 10:17:25 -0800 PST   3.98 MB
busybox             latest              e02e811dd08f        2016-10-07 14:03:58 -0700 PDT   1.09 MB

How can I automatically select the second created at timestamp in the list it produces?
...to then pass it as a variable into
b)
docker image prune -a --force --filter "until=<2rd timestamp from list>"
Is there an way to use awk ? Is there another way to leave an extra image ?

The problem is that the date format isn't ideal.

Supported formats for date formatted time stamps include RFC3339Nano, RFC3339, 2006-01-02T15:04:05, 2006-01-02T15:04:05.999999999, 2006-01-02Z07:00, and 2006-01-02

Reference: https://docs.docker.com/engine/reference/commandline/system_prune/#filtering
So this is really just a half answer, since it will use the date and the local timestamp.

The local timezone on the daemon will be used if you do not provide either a Z or a +-00:00 timezone offset at the end of the timestamp.

Thankfully, this comes sorted.
~: docker images --format '{{.CreatedAt}}'
2020-06-09 12:44:57 -0700 PDT
2020-05-29 14:19:46 -0700 PDT
2020-05-05 13:40:47 -0700 PDT
2020-04-28 11:55:11 -0700 PDT
2020-03-09 10:35:34 -0700 PDT
2020-03-09 10:14:43 -0700 PDT
2020-03-09 10:14:43 -0700 PDT
2020-03-09 10:14:43 -0700 PDT
2020-02-26 07:10:58 -0800 PST

I don't know how to transform timestamps in bash off the top of my head, so let's just do this the quick and messy way... the whole date!
 ~: docker images --format '{{.CreatedAt}}' | sed -n '2p' | awk '{print $1;}'
2020-05-29

Now we can do command substitution.
~: docker image prune --force --filter "until=`docker images --format '{{.CreatedAt}}' | sed -n '2p' | awk '{print $1;}'`"
Total reclaimed space: 1.344GB

I didn't put a heck of a lot of thought in to this answer, so there may be some edges cases I didn't consider. Also, if someone with better bash-fu can transform that timestamp, that would be awesome.


Secure RDS (private endpoint) and allow connecting from local machine for specific predefined users I'm looking for some way to allow access to RDS (Private endpoint) instances from local machine for a specific set of people (for the internal team.)
ways I know: Create Bastion EC2 and through SSH tunneling connect from local.
as I'm already using k8s from where the instances are reachable. is there any way to do some kind of port forwarding and finally connect RDS instance from local machine?
what I was trying: as I'm already having k8s which has access to those RDS instances, I tried creating a service in that, and through the Loadbalancer, I tried to connect using ssh but failed. (Probably I misunderstood/missing something in the process.)
is there any other secure easy way to do it?

I am not sure it will be helped. I have similar design in my office.
I have some Kubernetes Cluster. all node and master running on local ip from a PVC. And my rds too, only have local ip run on same PVC. Kubernetes Cluster already connect to RDS using local ip.
Then, I build VPN server on bastion server. Have local ip and public ip, run on same PVC with RDS and Kubernetes cluster. So when need connect, I just start my vpn connection on my laptop.
Only vpn/bastion server have IP public. + Load balancer too.
I hope this will give you inspiration.


How to have Ansible move a microservice? I am working with a microservices application that is using Ansible to deploy to a fleet of VMs. We have different numbers of replicas of each microservice type running across the VM fleet. Deployments are working fine using a single deploy playbook, passing it extra variables to say which service to install, and giving naming it a group of hosts in the inventory to run against.

Occasionally we need to move microservices between hosts to rebalance load. This isn't clean with how we have set up Ansible. Moving one microservice instance means adding one new host entry, and removing one old host entry, from the inventory group. The new instance is installed but no logic will run against the removed host. We have to run another playbook to uninstall the service from the removed host.

I am looking for a way to reorganize ansible so that it will quickly move a microservice. I am thinking that we should use a single inventory group of all hosts and use a list variable to say which hosts a microservice runs on. A custom fact can be set when a service is installed. The install task can check the fact and that the current host is in the list. The uninstall task can also check the fact and that the current host isn't in the list. Is this the right way to organise the logic or is there a better way?

I have created a demo with a role that can install many springboot microservices on a fleet of VMs where you name a subset of the VMs where you want to place each service. The playbook can then run on all hosts, compute whether each service should be on each host, and will either run the install.yaml or uninstall.yml tasks.

To install two of the same microservices with different runtime args you play something like:

roles:
    - {
      role: "ansible-role-springboot",
      sb_app_name: "microservices-registration",
      sb_app_group_id: "org.springframework.samples.service.service",
      sb_app_artifact_id: "microservices-demo",
      sb_app_version: "2.0.0.RELEASE",
      sb_app_run_args: '"registration 8082"',
      sb_app_healthcheck_urls: [
        "http://localhost:8082/actuator/health"
      ],
      sb_hosts: [
        "my-test-vm"
      ]
    }
    - {
      role: "ansible-role-springboot",
      sb_app_name: "microservices-web",
      sb_app_group_id: "org.springframework.samples.service.service",
      sb_app_artifact_id: "microservices-demo",
      sb_app_version: "2.0.1.RELEASE",
      sb_app_run_args: '"web 8083"',
      sb_app_healthcheck_urls: [
        "http://localhost:8083/actuator/health"
      ],
      sb_hosts: [
        "my-test-vm"
      ]
    }

The role is based upon orachide/ansible-role-springboot which seems very comprehensive. The new bit is in bold which is simply a list of host that should run the microservice.

Normally we would expect to use different artefact ids and versions each time we apply the role. In the example above, it is running an official springboot microservices mega jar that takes different RUN_AS args to run as different services. That seems a bit odd and not something I would recommend yet the role is agnostic to such details.

What thing that the original role requires is that you use an executable jar file:

        <plugin>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-maven-plugin</artifactId>
            <configuration>
                <executable>true</executable>
            </configuration>
        </plugin>

It role will create a symlink to that executable and create an environment files to run it and all that good stuff.

The modifications that I made were quite simple as the original role has logic to install or uninstall based on sb_app_state:

- import_tasks: install.yml
  when: >
        sb_app_state == 'present'

- import_tasks: uninstall.yml
  when: >
        sb_app_state == 'absent'

So it was simply a matter of adding some logic to check whether the current hostname is in the sb_hosts list then update the value of sb_app_state as appropriate. That is done with a single line change.

The demo codebase uses test kitchen ansible with testinfra to unit test that the playbook works. Getting that working was well worth the effort to do automated acceptance tests of the playbook logic.


Terraform deletes AWS lambda layers instead of keeping the older versions I am using terraform version 12.26 for AWS deployment. Whenever I try updating a lambda layer, the lambda layer version is incremented, but the older layer is deleted. Even though my current version of layer is for example 8, the older versions from 1 to 7 are automatically deleted.
I am using steps as mentioned here.
resource "aws_lambda_layer_version" "lambda_layer" {
  filename   = "lambda_layer_payload.zip"
  layer_name = "lambda_layer_name"
  source_code_hash    = filebase64sha256("lambda_layer_payload.zip")
  compatible_runtimes = ["nodejs12.x"]
}

How do I retain my older versions of layers?

Terraform's job is to maintain one remote object per resource block, either by updating that same object over time or by replacing it with a new object that meets new requirements.
If you wish to gradually accumulate new objects over time then Terraform is not the correct tool for the job. Your need would be better served by an application deployment tool, such as the Serverless framework.


How to download a VM image from GCP? I do not see a download button. I would like to download a VM image that was created on GCP using Packer and I would like to run it locally in Virtualbox.

I don't know of a way to get the actual GCE VM images.

But it's possible to export the images to GCS (as a tar.gz archive):

  If you need to move your Compute Engine boot disk data outside of your
  Compute Engine project, you can export a boot disk image to Cloud
  Storage as a tar.gz file.

  You can export a custom image as backup or for sharing by exporting
  the image to Cloud Storage. This method is ideal for sharing
  individual images with other projects that do not have access to your
  images.

You can then download this tar.gz file and, I guess, create a VM image from it to run on your local machine. I imagine the process would contain steps similar  to those detailed in Migrating VMs to Compute Engine and Importing Boot Disk Images to Compute Engine, only seen from your VirtualBox perspective instead of the GCE one.

These guides are making it easy to transfer VM loads to GCE. I guess they don't have incentives to do the same for transfers from GCE, which probably explains why there is no guide for it ;)


How to install a private managed Kubernetes on-premise? Where do you start in case you need something like a managed Kubernetes solution to setup a multi-tenant orchestration solution on-premises?

Any entity with its own container infrastructure has an access to the centrally managed orchestration service
Entities do not see each other's containers

Note: if "managed Kubernetes" is kind of reserved term by AWS, maybe it's better to say "orchestration as a service"?

Take a look at OpenShift container platform (https://www.openshift.com/products/container-platform/). They seem to provide on-premise options. You might want to start with OpenShift Origin - the community edition (https://github.com/openshift/origin) to test for free if it fits your plan.


What is the future of DevOps in the age of serverless? My senior recently argued that DevOps related roles such as DevOps Engineer, SRE, etc, will be less relevant once serverless matures. He pointed me to Containers won the battle, but will lose the war to serverless.

Can you see less and less DevOps roles being available in the age of serverless?

Also serverless needs some infrastructure and for that you also need software components to develop and rollout.

Beyond yet another website or application server I'd also think of energing advanced ecosystems related to:

Internet of things -  embedded systems and edge servers
Artificial intelligence and data engineering in distributed systems. Could be some ETL (extract, transform, load) stuff or training and logistics for machine learning models.
Gaming but also working in virtual environments.

So given the cloud market and these ecosystems will grow,  even if servetless would dominate, there will be a DevOps market, maybe smaller than now.

Otherwise it is in general very hard to predict how (IT) world and job market might look like in say 20 years.


What are immutable servers? There are some questions about immutable servers, such as:

How to implement the immutable server pattern without loosing the ability to do post-mortems?
What role do Configuration Managment tools play in immutable infrastructure?

It seems obvious that it has to do with servers (that part I get). And just digesting the grammar of immutable, I would think it has something to do with "not possible to mute". If that guess is close, I'd have no clue what exactly cannot be muted (and I doubt it has to do with soundcards or something ...).

My questions:

What is actually a "immutable servers" (in the context of DevOps)?
Why are they used?

Immutability is a term often used in computer science circles, which generally boils down to "not possible to change after creation".  It is typically used in reference to parallelism, concurrency, and thread safety.

The discussion of that topic is fascinating, but can generally be found elsewhere on Stack Overflow.  I'm resisting the urge to dive into it here.  The key concept is 'not possible to change after creation'.

Imagine if, on Amazon, you deploy a web service by baking it into a Machine Image (AMI - a pre-built instance you can repeatedly re-provision).  It connects to a backend database via credentials it gets out of a registry on startup.  It dumps logs into a logging tool like Splunk.  For normal day-to-day operation, you have no reason to ssh into this box.  If you need to scale up that service, you just create more instances of that AMI and adjust a load balancer.  Spinning it down is simply the destruction of instances and load balancers.

For the day-to-day operation, this box has no reason to change.  We can just fire up more off of the AMI.

What happens when you need to provide an OS-level security patch?  This is when you have a decision to make... do you bake a new AMI with the patch installed and redeploy all running instances, or do you ssh into existing images and update the patch?  There are plenty of people who would just ssh in.  The adherents of 'immutable architecture' just yelled at me for even suggesting such a thing is possible.

Immutabilists (if there is such a word) advocate baking the new ami.  They advocate removing all reason to ssh into a machine ever.  They advocate that any specific machine configuration should happen on the startup of that machine by pulling config details out of a repository.  This is the ultimate expression of 'cattle, not pets'.

Immutable architecture is specifically about machine configurations that have no reason to change after the creation of the machine image.  If something needs to change, bake a new instance image, shut down the old, bring up the new.


Why is Docker best practice to minimise the number of layers in an image? I see here that

Docker best practices says .. minimise the number of layers in our images

The tips given are to chain multiple lines in the dockerfile with \ so as to combine many lines of code into single commands. A specific example being:
Acceptable:
RUN pip install jupyter

RUN pip install pandas

Better:
RUN pip install jupyter && \
pip install pandas

From this knowledge, I understand what should be done, however, I don't understand why. What are the advantages of minimising the number of layers in a Docker image? (or, conversely, what are the disadvantages of not doing so?)

you should minimize the Layers of an Image for following Reasons:

first, you can only have (afaik) 127 Layers in an Image.
second: yes, you should include severals Steps in a Single run because of the Copy-On-Write functionality:

if you run:
RUN dd if=/dev/zero of=output.dat  bs=1M  count=10
RUN rm -rf output.dat

your image is still 10MB (plus some bytes for the delete Marker)
instead, you should run:
RUN dd if=/dev/zero of=output.dat  bs=1M  count=10 &&\
    rm -rf output.dat

then, your image size is not increasing.
surely, this is not the Usual Use-Case, but, for example from a Real-World-Example:
# install terraform
ENV TF_VERSION="0.12.16"
RUN curl -LO https://releases.hashicorp.com/terraform/${TF_VERSION}/terraform_${TF_VERSION}_linux_amd64.zip \
  && unzip terraform_${TF_VERSION}_linux_amd64.zip \
  && chmod +x ./terraform \
  && mv ./terraform /usr/local/bin/terraform \
  && rm terraform_${TF_VERSION}_linux_amd64.zip

more informations here: https://docs.docker.com/storage/storagedriver/#the-copy-on-write-cow-strategy


What are reasons for not not using Bitbucket-server for storing artifacts? I'm working with a company setting up a brand new project, and we're talking about what tools to use for what. I was talking about Artifactory or Nexus for storing built artifacts (APKs in this case), and they asked why they can't just use Bitbucket like they will use for the software, to reduce the number of tools they need to install and maintain.

My initial response was "Source goes in source repo and artifacts go in artifact repo", but that's not actually an answer why or why not.  In fact Googling around didn't offer any reasoning either.

ADDING TO THIS: This product is for a regulated industry.  That means we need a complete versioned, auditable record of the artifacts built.  This is one of the reasons we're looking into this.  Also, as I said in some comments but will add here, we will install Bitbucket in a private GCP not accessible by the public, so there are no concerns about others accessing it.

Any input into this? Anything that would bite us later on if we store them in Bitbucket?

Reasons not to store large binaries in a git repository:

Everbody cloning your repository will download all those binaries, by default. Binaries, if built regularly, tend to consume massive amounts of storage, compared to source code - git cannot compress them, or calculate deltas, to reduce their size.
git goes to great lengths to make sure history is not lost, it will never delete anything that is part of the history of any ref (branches or tags). That also means that if you later decide to delete old, long-useless binaries due to running out of space, you will find that, while not impossible, very hard indeed.
One point of proper artifact stores is that they help phasing out outdated or compromised parts. git cannot really do that for you - you'd have to write your own tooling for that - and you will never get rid of the old versions in history.
The concept of a "commit" does not really apply to binaries. You need operations like "upload" and "download", as they are regularly involved in the build processes of your software (i.e., bundler in the ruby world, or maven in Java). Those builders know how to fetch their 3rd party libraries from artifact repositories easily, and how to upload new versions. They may be convinced to work with a  git repository for downloads, but since git has no versioning information, you again need to have your own tooling, specify the commit in which to find that binary manually, or have all binaries ever in use checked out locally. And uploading to a git repository, again, would use your own tooling (creating spurious commits as well).

In total, using git for that is just the wrong tool. If your colleagues are afraid of adding yet another complex tool like Nexus, then at least convince them to use a simple tool instead of git - like some arbitrary (s)ftp/scp repository in a VM somewhere, with https access via a simple webserver.

If you must use git, then at least make sure the build binaries are not committed together with the source code, but in their own part of history. Check out this older answer to see how to create an orphan branch. Those can at least be deleted later, and the binaries themselves removed by garbage collection; and not every client is forced to download all of it all the time.


How well can agile pair programming work in interdisciplinary DevOps contexts? In agile, pair programming can be performed like the following:

two specialists of same discipline change the place before the terminal and each has a chance to code and to conceptualize/reflect;
two specialists of different disciplines work together, but one is rather a customer and cannot code much in the target language; also the coder might not know much about the requirements which then discussed directly and coded and tested and deployed.

How is this relevant/appliable in DevOps context? (facts)

Agile fits very well with a DevOps culture and pair programming is no exception.

The following are some examples of "pair programming" principles applied in a DevOps context:

A pair of individuals working together on a Dockerfile for their service.
A pairing sits together to work on defining their infrastructure as code.
A traditional developer and an IT administrator building a CI/CD pipeline together.
A specialist and tester sit together to define what the testing environment should look like.
A specialist and a customer sitting together and working on a deployment to the customers infrastructure.


Which plugin is used to copy file from different environment in jenkins I want to copy file from different environment in jenkins. Can you please guide me to choose the plugin for copying

You are probably looking for something like the Config File Provider. Its plugin page describes it as a plugin which:

  Adds the ability to provide configuration files (i.e., settings.xml for maven, XML, groovy, custom files, etc.) loaded through the Jenkins UI which will be copied to the job's workspace.

If you need something more like a script (as opposed to the configuration file scenario above), you may be interested in the Managed Script plugin. This allows jobs to consume scripts which are centrally stored and defined by the Jenkins admins:

  Managed scripts are shell scripts which are managed centrally by an administrator and can be referenced as a build step within jobs.

If these do not suit your needs, try looking at the Jenkins plugins list


Docker tags for multi-stage builds for series of Docker images out of one Dockerfile Thanks to Docker multi-stage builds, we can compile some artefacts in one environment and put them for execution in other one.

As result, there are two images, where the final corresponding to the last stage and provided tag gets tagged as defined; there is no tag for intermediate (ones), but they remain there as well.

Question: is it possible to create a series of tagged images for all defined stages using multi stage build out of one Dockerfile?

You can build and tag any stage of a multi-stage build. Just use the --target option to docker build. E.g.

docker build --target jdk -t myapp-jdk:v1 .
docker build --target jre -t myapp:v1 .

With the layer caching, the the earlier steps will be reused from the cache and not rerun from scratch each time, so there's little downside to running the build to different targets multiple times.

If you absolutely need to run this from a single build command, you can parse the output of the build, get the image id of the intermediate step, and tag that image id:

docker tag <some_image_id> myapp-jdk:v1


How to verify hashes of Ansible Galaxy dependencies? From a security perspective, I don't like the idea of relying only on Git tags, as they could be moved later. Unfortunately, I'm not aware  of an official solution to enforce checks.

This is the best that I could come up with:

# requirements.yaml:
- src: git+https://github.com/geerlingguy/ansible-role-nodejs.git
  version: 405a113ccbde0c99614f6f815fb4285cb742ad25 # == release 5.0.0
  name: geerlingguy.nodejs.trusted_hash

And in the playbook, use it as:

- roles:
  - role: geerlingguy.nodejs.trusted_hash
    nodejs_version: "12.x"

It works, but feels a bit overly complicated. Is there an easier way?

The feature is currently (Nov 2019) not directly supported in Ansible Galaxy.

There have been related discussions in the past (#14604 and #1358), but the latest state can be followed in the new issue #2108.

In the meantime, the workaround described in the question can be used, at the cost of some verbosity.


Store and analyize logs data from diffenrent application We have different applications that generate logs intensively, We do a lot of analytics, visualizations, monitoring on these logs.

Currently, we are using azure app insights to do the visualization but it's not enough for the advanced visualization we need, so we decided to move the data to another database that it's suitable for analyzing the data and be able to connect with tableau.

Initially, we are exploring elasticsearch but there is no straight forward connector to tableau so we need to build our connector and also I found some more options like ClickHouse, PrestoDB, InfluxDB, Apache HBase and Tarantool

So my question is, what is the best practice a storage system for logs data which hot active, continuous analysis, ability to monitor and could integrate with tableau?

Can you not use Kibana with Elasticsearch or does that not give you the 'advanced' features you need?

Have you looked at this connector for ES and Tableau? https://www.dremio.com/tutorials/unlocking-tableau-on-elasticsearch/


Jenkins managing a pool of servers across restarts I have a pool of servers to test a provisioning process my team owns.  One server is needed per branch or PR.  The process requires reboots.  Once the process is run, we run some tests.  A Jenkins pipeline automates the provisioning process and test.

So my problem is how to manage the pool of servers so that

Any available pool server can be selected for a build
A server claimed for a build is unavailable for other builds
Upon successful build, the server is made available again
Failed builds are not made available so they retain the failed state for investigation (with manual return to the pool after)

I can't just use the executor on the slave node because we reboot the server and that would kill the job.

My current solution is:

Pool servers are setup as Jenkins slaves with one executor each and a common label
A pipeline stage is assigned to the common label, choosing one of the slaves with a free executor--this stage moves slave.jar to protect against Jenkins restarting the assigned slave (which it annoyingly will do)
The next pipeline stage uses node.getComputer().disconnect() to disable the slave--it seems this must run on master so must be in a different stage
A post { success { ... } } uses ssh to move slave.jar back into place on the server and node.getComputer().connect(true) to allow it to be picked again

This mostly works and is kinda cool, but it adds a ton of noise to the pipeline and worse, suffers from a race condition between steps 2 & 3.  When jobs queue up for the pool, a new one can be assigned to the same node just before a prior jobs disconnects the slave.  When this happens, things break badly.

Does anyone else use Jenkins to manage a pool of servers where you need to support reboots in the pool and how do you do it?

Any ideas for ways to manage the pool outside Jenkins, but somehow get the assigned node into Jenkins for the test run?  Plugins I've missed, etc.

I've examined the few results google returns for the slave disconnect problem and the most promising one was https://groups.google.com/forum/#!topic/jenkinsci-dev/ch2lQZvZdkw -- the suggestion there failed for me because OfflineCause.UserCause class could not be resolved in my pipeline.  I only know enough Java and Groovy to be dangerous!

I found the missing piece in the Lockable resource plugin.

lock(label: 'server-pool', quantity: 1) {
   ...
}

Now the servers to be used for testing don't need to be managed as slaves at all and that greatly simplifies things.


How to set a Git repository when creating a job in Jenkins from a Groovy script? I'm trying to setup a Jenkins server with Groovy script:

def repository = 'git@somerepo'
import hudson.tasks.Shell;

job = Jenkins.instance.createProject(FreeStyleProject, 'TestJob')
job.setDescription("Some description")
job.displayName = 'SomeTestJob(TESTING groovy)'

job.scm = new hudson.plugins.git.GitSCM(repository)
job.save()

Now I need to set a repo branch - How do I do this?

def repository = 'git@somerepo'
import hudson.tasks.Shell
job = Jenkins.instance.createProject(FreeStyleProject, 'TestJob')
job.setDescription("Some description")
job.displayName = 'SomeTestJob(TESTING groovy)'
job.scm = new hudson.plugins.git.GitSCM(repository)
job.scm.branches = [new BranchSpec('*/master')]
job.save()

Thanks a lot Michael Durrant for giving that link !


How to avoid wasting time/resources in a CI/CD workflow? Ideally, a team should be able to run all the pipelines on every commit and automatically deploy to production.

However, I don't understand how to most effeciently use our resources  so that hours are wasted running tests on code which is about to be overwritten by the next commit.

If a build or test takes 20 minutes to go through the pipeline, the system gets "clogged".

What mechanism is used to solve this problem? Do tests/jobs get canceled as new code gets pushed? If so, then how do you deploy multiple times a day?  Add to this the need for Pull requests or sign off.  What intermediate steps can be taken until it's possible to automate everything?

We break ours down into stages, and each stage pulls, rather than pushes.

Our build takes 20 minutes. So if I commit once a minute, I will still only get three builds an hour, but each build will contain 20 commits.

The tests then pull on the latest successful builds and run them, and the same for the deployment scripts.

The main point is that we do NOT build every commit individually. We do NOT test every build individually.

I probably got all this from Continuous Delivery by Jez Humble and David Farley.


What processes or tools enable Segregation of Duties when engineers both deploy and run code? In highly regulated environments, such as the Financial Services sector, Segregation of Duties is an essential mechanism for avoiding collision amongst individuals with development responsibilities and production privileges.
Traditionally this has meant that Developers develop code and then hand it over to Operations however in many DevOps Operating Models the segregation between Development and Operations is, at a minimum, blurred:

In Google's Site Reliability Engineering, or SRE, practice there is a separate SRE function within Google, however, Developers are brought in to backstop the SREs in times of high operational load.

In the "You Build It, You Run It" model there is no separate operations function.

After spending months drilling down to the root causes of a Segregation of Duties mechanism it seems it predominantly exists to satisfy Sarbanes Oxley Section 404: Management Assessment of Internal Controls:

(a) Rules Required.
The Commission shall prescribe rules requiring each annual report required by section 13(a) or 15(d) of the Securities Exchange Act of 1934 to contain an internal control report, which shall--
(1) state the responsibility of management for establishing and maintaining an adequate internal control structure and procedures for financial reporting; and
(2) contain an assessment, as of the end of the most recent fiscal year of the issuer, of the effectiveness of the internal control structure and procedures of the issuer for financial reporting.
(b) Internal Control Evaluation and Reporting. On the internal control assessment required by subsection (a), each registered public accounting firm that prepares or issues the audit report for the issuer shall attest to, and report on, the assessment made by the management of the issuer. An attestation made under this subsection shall be made in accordance with standards for attestation engagements issued or adopted by the Board. Any such attestation shall not be the subject of a separate engagement.

Based upon the comments it is important to call out a couple of assumptions I am making:

I am predominantly considering mass-market financial services, i.e. transaction volumes are high but relatively low value.  This would be as opposed to commercial financial services which has a different transaction value profile.
A financial institution's online offering will be made up of many components that have differing risk considerations:

Move Money - Moving money between accounts or transfers between accounts of different owners.  An operation that has to consider Anti-Money Laundering, Fraud Protection, and Embargo'ed countries to name a few.
Customer Acquisition - Less "Risky" as it has low transaction volumes compared to Move Money but still needs consideration.
Internet Banking - Covers a wide range of services with varying levels of risk, Move Money would be considered part of this.

Conceivably a different approach could be taken for each depending on the risk, however, in the interests of keeping it simple, I am working towards a solution that would apply to some of the riskiest operations.

TL;DR: It is the responsibility of the management to ensure that adequate internal controls are in place that comply with the Securities and Exchange Commission's regulations.
Sarbanes Oxley 404 is normally satisfied through the completion of a Top-Down Risk Assessment, part of which will assess the risk of collusion and puts forward mitigation strategies.
Within a company which employs DevOps practice and culture, where Developers routinely have access to both source control and production how can Segregation of Duties be achieved, or more generally how can the risk of collusion be mitigated.

Your question doesn't seem to make any assumption about the platform/OS it is about. Which is why it may make sense to add an answer about how this is typically done/addressed in a mainframe environment, where the "engineers" (as in your question title) are actually groups of people were dozens (possibly hundreds) of people are involved. My answer is based on using the SCM product where I'm most familiar with (not sure if it's needed to disclose the product name).

1. Architecture

Here are the highlights of how I'd answer your question:

All code (and related artifacts like executables, etc) is stored in files which, all together is what we call the library structure.
For each environment on each (possibly remote) target system, there is a server (a "started task" in mainframe speak), which takes care of ALL (repeat: ALL) updates to anything in the library structure. There are a few exceptions (like security people, or space management team), but apart from that, nobody (repeat: nobody) has authorization to apply updates to any file within that library structure. In other words: the server gets exclusive update authority to the entire library structure. Attention: OPS-people will go bonkers if you walk in to limit their access (at first they are going to resist ...), so make sure you're covered by upper management (CxO) to impose those access rules ...
The actual software changes my consist of a single component (a tiny code fix in the middle of the night ...), or it may also be hundreds or thousands of sources, executables, or whatever other artifacts (during a release weekend). To make them manageable, things that should be moved (more or less) together, at the same time, are bundled together in what's called a software change package.

With the above in place, any kind of update to be applied by the server to the library structure, will only be possible via a well defined workflow, which we call the lifecycle of a software change package (SDLC if you prefer). To actually execute the various steps in that workflow, this is what it takes to make it happen:

Only the server will execute the required (and preconfigured) steps.
The server will only do a specific step (= update something somewhere in the library structure), after the required approvals (from human beings) have been gathered to perform such step.
The approvals can only be given by users who have a role that allows them (= permission) to issue such approvals.

2. Roles and permissions

The server will ensure that the user trying to make something happen (like 'approve something') will only be able to do so, if the user's permissions are appropriate. That part is easy. But you don't want to use the SCM system to administer all those permissions for all the users involved, that's what belongs in your security system (not the SCM system!), so that you can adapt your workflow (in your SCM system) to go check those permissions whenever appropriate. The steps below provide some more details on that.

Step 1: Configure the permissions (in the security system)

Define security entities in your security system, with well defined names for those entities. A few samples (add as many similar ones to fit your own needs):

PrmUnit, used for getting permission to request a Promote to say Unit-testing.
PrmQA, used for getting permission to request a Promote to say Qa-testing (let's assume this is the highest level of testing).
PrdEnduser, used by end-users involved in some level of testing, to indicate that they are satisfied by the results produced by some kind of testing. And because of that, those end-users agree with the change moving forward in the library structure.
PrdRelmgnt, used by release managers to authorize an Activation in production (= the last/highest level in the library structure).

Define groups of users in your security system. A few samples (add as many similar ones to fit your own needs):

GrpDevs, which (say) corresponds to your developers (probably more then just 1).
GrpEnduser, which (say) corresponds to your end-users (at least 1, preferably with more similar users).
GrpRelMgnt, which (say) corresponds to your release managers (at least 1, preferably a few more users).

Grant permissions, also using your security system, to allow access to selected "security entities" for selected "groups of users". To continue the example above, here is what seems appropriate (adapt to fit your own needs):

Group GrpDevs gets access to (only!) security entity PrmUnit.
Group GrpEnduser gets access to (only!) security entity PrdEnduser.
Group GrpRelMgnt gets access to (both!) security entity  PrmQA and PrdRelmgnt.

Step 2: Configure the workflow (in the SCM system)

After the permissions are configured in your security system (as in Step 1), all that's left to do in your SCM system is to configure how the various steps in the lifecyle match with the related security entities in your security system. That is, only those users who have the appropriate access to the required security entity, are allowed to request the server to perform the corresponding step in the workflow.

Here are some examples of how you'd configure your SCM system to make some magic happen:

If a user has access to PrmUnit, then such user is allowed to request a Promote to Unit-testing. Obviously, the users in group GrpDevs are the users authorized for this (note: not, e.g, the users in group GrpRelMgnt).
If a user has access to PrmQA, then such user is allowed to request a Promote to QA-testing. Obviously, the users in group GrpRelMgnt are the users authorized for this (note: not, e.g, the users in group GrpDevs, or in group GrpEnduser).
If a user has access to PrdEnduser, then such user is allowed to authorize the change moving forward in the library structure (which is typically a prereq for users in group GrpRelMgnt to even be able to review a change). Obviously, the users in group GrpEnduser are the (only) users authorized for this.
If a user has access to PrdRelmgnt, then such user is allowed to authorize an Activation in production (= the last/highest level in the library structure).

3. Expect the unexpected, and be ready for it

The above is just a blueprint, which hopefully helps to understand how in the end it is the server that takes care of the segregation of duties ... provided you have the CxO cover you to impose some access rules that not everybody will like.

To complete the picture as explained above, the server creates an audit trail (logging) of anything that's happening in the system. So that at any point in time, it is always possible to answer questions like

  What happened when and why, and which authorized user actually approved it ... upfront?

But, probably the toughest part is to have adequate reporting tools available (and know how to use them). At least to (easily) satisfy requests from IT auditors (their questions can be very challenging). But also to point to relevant log records in your SCM system to answer all sorts of "What happened"-questions in crisis situations where (part of) production is down.

PS: I leave it to everybody's own judgement if my answer is yes or no DevOps-compliant.


What are the ways to get authorized access to mlflow through Azure Active Directory? What are the ways to get authorized access to mlflow through Azure Active Directory?

Requirements:

I need to grant access to the mlflow web interface installed on a Ubuntu 16 Virtual Machine in Azure cloud to only allow authorized users.
Granting access will be done through group membership in Azure Active Directory.

It looks like MLFlow doesn't natively support any authentication schemes and the recommendation is to use a reverse proxy such as Nginx to only forward requests to MLFlow if a valid authentication cookie is provided, redirecting any requests without a cookie or an invalid cookie back to the Identity Provider.

Azure AD Application Proxy

You can configure Azure AD Application Proxy to add authentication, although you will also need to protect MLFlow from unauthorised access by putting it in a virtual network or creating rules to only accept connections from Azure AD Application Proxy.

Used managed ML Flow in Azure

There is also the potential of using MLFlow that is part of Azure Databricks, which probably also includes authentication.

Nginx

There are complete instructions on setting this up on the Nginx for a generic application. In summary, however, you will need to:

configure Nginx as a reverse proxy
prevent access to MLFlow from anything other than Nginx
create an application in Azure Active Directory for Nginx
configure Nginx to look for the JWT token and validate it

You would also be able to use other products such as Google's Identity Aware Proxy and ForgeRock's Identity Gateway.


How to get instance's route table id using CLI? I'd like to find a route table id associated with the given EC2 instance.

How can this be achieved using AWS CLI?

I've tried to use aws ec2 describe-vpcs, but the route tables are not there.

I'm using aws-cli/1.11.13.

The route table is associated with VPC, which is associated with the instance.

Given $id shell variable has the instance ID, e.g.

id=i-0xyz # Replace i-0xyz with the real Instance ID.

here is the shell command to get VPCs associated with the instance and assign to $vpcs variable:

vpcs=$(aws ec2 describe-instances --instance-id $id --query 'Reservations[].Instances[].NetworkInterfaces[].VpcId' --output text)

Then to list the route tables associated with VPC, run:

aws ec2 describe-route-tables --filters "Name=vpc-id,Values=$vpcs"

To get just the route ids, add: --query "RouteTables[].RouteTableId[]" parameter.


What are the different types of work in a development team? I'm currently on my annual re-read of The Phoenix Project and we're going through the different types of work that an Ops Team face. These are:

Project Work
Changes
Internal Projects
Unplanned Work

This time around I'm not convinced that these types apply directly to development teams (as opposed to Ops work).

We obviously have Project Work (pretty much everything in our backlogs). Unplanned Work is a no-brainer for anyone who's ever supported an incident...

But do we have an equivalent of "Changes" or "Internal Projects"? Internal Projects kind of correlates to Tech Debt...

Have the authors of The Phoenix Project (or other highly regarded figure in the DevOps community) compared the different types of work a development team may face and commented on whether it is different?

Nice question!

Possibly, Changes are further strategic product development beyond current work which is more than required by technical debt? For example, introducing a new framework, or nice abstraction layer. (which could break a lot though)

Internal projects could be custom helper tools to automate something invisible in the backlog. Sometimes you have also to fill in time sheets.

By the way Unplanned work could be also a request to support sales or HR or some unexpected meeting or all hands social skills training event.

There is anyway a sequel to "Phoenix Project" called "Unicorn Project" by same author, focusing on the Dev part of the equation.

I haven't read it myself yet but hope you might find the answer there.


How to mount a single 100TB SSD on AWS and access the device over nfs from different machines within the same VPC? I have a bad experience with Storage Gateway (SGW) and would to consider the possibility to setup a 100TB SSD device (via EFS or similar) on AWS and allow an access to the storage device from different clients such as: Amazon Workspaces sessions, EC2 machines, AWS Batch and other station within the same VPC.
The main issue is that SGW fails quite frequently to mirror the s3 bucket properly even after 24 hours upon update of the bucket.

create storage in AWS Console
mount created storage into VM
configure nfs deamon to present the storage volume in the VPC
configure Clients to use/mount NFS volume coming from server

sometimes the easiest is the best, this even works on other systems, not just AWS.


Installing azcopy in Gitlab Runner (Alpine Linux) How can I install azcopy and use it in my gitlab runner CI/CD script?

First I have two scripts that download the tar file and extract it.

wget -O azcopy.tar.gz https://aka.ms/downloadazcopy-v10-linux
tar -xf azcopy.tar.gz

However, if I try running executable ./azcopy_linux_amd64_10.1.0/azcopy I get an error saying /bin/sh: eval: line 102: ./azcopy_linux_amd64_10.1.0/azcopy: not found
I know this directory/file exists because I've ran ls commands.

I've also tried copying the binary to /usr/bin and/or /usr/local/bin but I still get the error: /bin/sh: eval: line 102: azcopy: not found

What am I doing wrong? How can I extract and run the executable, or otherwise install azcopy in my gitlab CI/CD script

This is a glibc dependency issue on Alpine.

This azcopy binary is compatible with Linux distributions that use glibc, GNU's C standard library. Alpine Linux uses a different libc implementation, musl-libc, which is generally not compatible to glibc. When trying to run non-Alpine-built binaries on Alpine, they'll usually fail to link since the glibc shared object, libc.so.6, is missing. azcopy: not found is actually the result of the dynamic linker failing to resolve the requires libraries.

You could verify by running ldd ./azcopy:

    /lib64/ld-linux-x86-64.so.2 (0x7fa0bc9c3000)
    libpthread.so.0 => /lib64/ld-linux-x86-64.so.2 (0x7fa0bc9c3000)
    libc.so.6 => /lib64/ld-linux-x86-64.so.2 (0x7fa0bc9c3000)

Note the libc.so.6 dependency. On vanilla Alpine, there's only the musl libc binary, libc.musl-x86_64.so.1.

Fortunately, Alpine provides a light compatibility layer for glibc, through the libc6-compat package. It installs a libc.so.6 link which is redirected to the musl shared object, and adds some missing glibc interface functions. This seems to be enough to running azcopy.

Therefore, for running azcopy on Alpine, just install the libc6-compat package, using:

apk add libc6-compat


What is self hosted version control system? What is self hosted version control system and how is it different from github? I mean if I use self-hosted VCS then where does my code go? I mean: does it get uploaded to the server, or it is saved on my local machine? If it is uploaded to the server, then if I reset the computer, will I able to access my code again?

Let's take it word by word:

Self: You or your organisation owns this thing. By contrast, a cloud-based offering by GitHub.com, Azure, GitLab.com, etc. (see this list on Wikipedia for a more comprehensive list) is not owned by you, but rather the service provider in question. Do not confuse ownership of the code (dictated by it's license – typically owned by you), with ownership of the service for performing version control on the code.
Hosted: You or your organisation are responsible for operating and maintaining this thing. In some cases, this is exactly the same software (e.g. one can self-host GitLab or run GitHub Enterprise “on premise” in your own datacentre). This means keeping it secure, making any changes to it, serving user requests for change, etc. You are also responsible for some part of the infrastructure necessary to provide this service. This may vary, depending on who you have service provision agreements with, from everything down to the power and networking, or only the last layer of the service (here, the version control system itself), or anything in between.
"version control system" (or VCS): anything (software or otherwise, but most likely software) which provides the functionality necessary to control versions. Note there is a subtle difference between Version Control and Version Control Software, although the meaning is often conflated.

The term VCS can refer to any version control system, even when it is not applied to software, however in our context, it can be assumed that you are referring to a version control system applied to software.

In other words, a “self-hosted VCS” can be thought of as:

  A service and all of its components which is under your own ownership and control, acting as a version control system for software.

Often, this can take the form of:

base infrastructure (network/power/server),
operating system,
authentication and authorisation mechanism, and
the VCS server itself (Git, SVN, etc.).

The second part of your question is about how it is different from GitHub. The following contrast can be made:

Functionally there may be differences between what you can do with GitHub an what you can do with your self-hosted VCS. GitHub, as is the case for almost all VCS hosted by third parties (i.e., not hosted by you) goes a few steps further than simply doing version control. There are collaboration tools, identity management, issue tracking, etc. built into their service. While this is strictly speaking not part of VCS, it greatly improves the experience of using one, almost to the point where doing version control without them feels impossible.
GitHub owns and operates the service. You may make suggestions to change their service, API, etc., and perhaps even contributions to their open source products, but you do not own the service. It is provided to you based on the SLA for the offering that you choose. You cannot make your own changes to it. This is overall a very good thing, unless you are in the business of providing the VCS itself – but chances are that you just want to use the service, not compete with it.

Update to answer the edited question:

  does it get uploaded to the server, or it is saved on my local machine? If it is uploaded to the server, then if I reset the computer, will I able to access my code again?

You are conflating two issues here: self-hosting and distributed version control. Of course, if you self-host on your own computer, your run the risk of losing everything if the machine dies. However, of someone else hosts your code in only one place, then you still risk losing everything if their machine goes down.

To address this (and several other issues related to centralisation of source control), distributed version control was invented. A good discussion is in the git book.


Are there any best practices and/or tools for 'nohup' monitoring? I use nohup for starting Ansible playbooks inside the Master which is spawned via boto3 from AWS Lambda.

This playbook creates multiple workers, make them do some tasks, and terminates them.

So, I want to design/build a reliable monitoring system for the same. For now, I am writing the logs of nohup to a tasks.log file inside master, and pushing it to a store(for now, S3) once the Ansible plays are completed.

But, I wanted to know whether there is a much more elegant (and/or) better method for doing the same?  And also, whether there are good tools which would help me in the process?

I'm not sure it's a best practice but what I would do is using filebeat to read the log in near real time and push it to ElasticSearch.
AWS provides an ElasticSearch Service which can save you from this part.

The drawback of using filebeat directly to elasticsearch is that the time of events will be the time the prospector read them and if your logs have a timestamp they can be "ingested" in different order than produced (from multiple workers).

You can add Logstash inbetween, sending from filebeat to logstash and then to elasticsearch. Logstash is able to interpret your logs and split them into fields to search again, in the same manner you can parse your date in log so events will be stored with a correct timestamp.

You may use logstash directly to read the logs, but it's quite heavy on resources so I would keep it separated to no impact the workers.

If you buy a license from x-pack you can add alerts on messages directly from elasticsearch.

If you want to save you from the burden of maintaining logstash there's services like logmatic.io which provides you with the full stack, you just have to push your logs and a bunch of parsing are handled from stock.


How to run Ansible file in Jenkins using Ansible plugin Please look into this error trying to implement the deploy.yml file with ansible plugin which is installed in the jenkins.

In ansible console I can implement the app successfully to the target machine but while same deploy.yml file using in jenkins getting error .

Tried :

ssh connection from target and master machine able to connect
Add the rsa key/finger print key to the target machine as well as in master controller .
From terminal i can able to access target machine as well as can able to deploy any app to any VM but while switched to jenkins ansible plugin getting error.

Simple one.
You should either add ansible_ssh_extra_args="-o StrictHostKeyChecking=no" to inventory or disableHostKeyChecking: true into Jenkins pipeline script (inside ansiblePlaybook block).

Like this:

ansiblePlaybook(
  colorized: true,
  installation: 'ansible27'
  inventory: '[hostsfile]'
  playbook: '[playbook]'
  disableHostKeyChecking: true
)


Trigger gitlab downstream pipeline from a upstream pipeline in a multi-project using trigger, rules and custom configuration variables I have this scenario where I want to run the trigger in build stage only when $FLAG variable has been set by the .pre stage. However, the build stage never runs.
How should I conditionally trigger a downstream pipeline?
checkArtifactPresent:
  stage: .pre
  script:
    - >
      set +e;
      if curl -s -S -f $NEXUS_RAW_PICKER_REPOSITORY/${PRODUCT_FLAVOR}/${PRODUCT_FLAVOR}-${BUILD_TYPE}v${PICKER_TEMPLATE_TAG}.apk --output ${PRODUCT_FLAVOR}-${BUILD_TYPE}v${PICKER_TEMPLATE_TAG}.apk;
      then
        export FLAG= true;
      fi

buildArtifact:
  stage: build
  only:
   variables:
     - $FLAG
  trigger:
   project: dev/project_name
   strategy: depend

You could use dynamic child pipelines. (https://docs.gitlab.com/ee/ci/parent_child_pipelines.html#dynamic-child-pipelines).
You could create a job for triggering your job "buildArtifact" in a yml-file created with jsonnet in your "checkArtifactPresent".
With using "--ext-str" or "--ext-code" in jsonnet.
You can control the yml-file creation (https://jsonnet.org/learning/tutorial.html#parameterize-entire-config).
Read more https://jsonnet.org/


How to configure a pull request quorum making an organizational constraint? Consider the following situation.

Alice and Anna, Bob and Bart, and finally Eve and Evelyn work on same Git project in three different organizations.

How can they configure a quorum for pull requests, which requires at least 2 votes but from two different organizations?

Example:

Votes from Alice and Anna - no quorum achieved possible as both votes come from the same organization;
Votes from Alice and Bob - quorum has been achieved because team members from two different organizations have voted.

Although I like the idea, based on what I know while working with Gitlab, Bitbucket and Github I do not believe that it is currently possible as I cannot remember that such integrations were available.

Also Googling: github bitbucket pull request approvals integrated did not return any positive results.

I think that the users should always be part of the same organization in order to approve Pull Requests. However, when I think about github I can remember that I was able to invite people as a collaborator even when they were not part or part of any organization.

In conclusion, I think that it is possible, but only in a certain git-server providers like GitHub. My suggestion is to try it and otherwise report an issue in the associate Github project.