How to allow a non-root user to write to a mounted EFS in EKS I am having trouble configuring a statically provisioned EFS such that multiple pods, which run as a non-root user, can read and write the file system.
I am using the AWS EFS CSI Driver. My version info is as follows:
Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.18", GitCommit:"6f6ce59dc8fefde25a3ba0ef0047f4ec6662ef24", GitTreeState:"clean", BuildDate:"2021-04-15T03:31:30Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"18+", GitVersion:"v1.18.9-eks-d1db3c", GitCommit:"d1db3c46e55f95d6a7d3e5578689371318f95ff9", GitTreeState:"clean", BuildDate:"2020-10-20T22:53:22Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}

I followed the example from the github repo (https://github.com/kubernetes-sigs/aws-efs-csi-driver/tree/master/examples/kubernetes/multiple_pods) updating the volumeHandle appropriately. The busybox containers defined in the specs for the example are able to read and write the file system, but when I add the same PVC to a pod which does not run as the root user the pod is unable to write to the mounted EFS.
I have tried a couple other things to get this working as I expected it to:

just applying the annotation described here: https://kubernetes.io/docs/tasks/configure-pod-container/configure-persistent-volume-storage/#access-control to the persistent volume definition.
applying the aforementioned annotation and including securityContext.runAsGroup to the pod definition (with the appropriate value)
applying the annotation, the runAsGroup, as well as fsGroup for the Pod

None of these configurations allowed a non-root user to write to the mounted EFS.
What am I missing in terms of configuring a statically provisioned EFS so that multiple pods, all of which run as a non-root user, can read and write in the mounted EFS?
For reference here are the pod definitions:
apiVersion: v1
kind: Pod
metadata:
  name: app1
spec:
  containers:
  - name: app1
    image: busybox
    command: ["/bin/sh"]
    args: ["-c", "while true; do echo $(date -u) >> /data/out1.txt; sleep 5; done"]
    volumeMounts:
    - name: persistent-storage
      mountPath: /data
  volumes:
  - name: persistent-storage
    persistentVolumeClaim:
      claimName: efs-claim
---
apiVersion: v1
kind: Pod
metadata:
  name: app2
spec:
  containers:
  - name: app2
    image: busybox
    command: ["/bin/sh"]
    args: ["-c", "while true; do echo $(date -u) >> /data/out2.txt; sleep 5; done"]
    volumeMounts:
    - name: persistent-storage
      mountPath: /data
  volumes:
  - name: persistent-storage
    persistentVolumeClaim:
      claimName: efs-claim
---
apiVersion: v1
kind: Pod
metadata:
  name: app3
spec:
  containers:
  - name: app3
    image: busybox
    command: ["/bin/sh"]
    args: ["-c", "while true; do echo $(date -u) >> /data/out3.txt; sleep 5; done"]
    volumeMounts:
    - name: persistent-storage
      mountPath: /data
  securityContext:
    runAsUser: 1000
    runAsGroup: 1337
    fsGroup: 1337
  volumes:
  - name: persistent-storage
    persistentVolumeClaim:
      claimName: efs-claim

And the SC/PVC/PV:
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: efs-sc
provisioner: efs.csi.aws.com
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: efs-claim
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: efs-sc
  resources:
    requests:
      storage: 5Gi
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: efs-pv
  annotations:
    pv.beta.kubernetes.io/gid: {{ .Values.groupId | quote }}
spec:
  capacity:
    storage: 5Gi
  volumeMode: Filesystem
  accessModes:
    - ReadWriteMany
  persistentVolumeReclaimPolicy: Retain
  storageClassName: efs-sc
  csi:
    driver: efs.csi.aws.com
    volumeHandle: fs-asdf123

If anyone comes across this later I resolved my issue by using an initContainer for any Pod that needed to write to the file system.
For example:
apiVersion: v1
kind: Pod
metadata:
  name: app3
spec:
  containers:
  - name: app3
    image: busybox
    command: ["/bin/sh"]
    args: ["-c", "while true; do echo $(date -u) >> /data/out3.txt; sleep 5; done"]
    volumeMounts:
    - name: persistent-storage
      mountPath: /data
    securityContext:
      runAsGroup: 1337
  initContainers:
  - name: fs-owner-change
    image: busybox
    command:
    - chown
    - "root:1337"
    - "/efs-fs"
    volumeMounts
    - mountPath: /efs-fs
      name: cog-data
  securityContext:
    fsGroup: 1337
  volumes:
  - name: persistent-storage
    persistentVolumeClaim:
      claimName: efs-claim

The rest of the definitions match what was in my question.


Docker preserve images policy Can I configure Docker in a such way that it will store only certain amount of tags for the given image e.g. docker images

REPOSITORY                     TAG                                      IMAGE ID            CREATED             SIZE
ecr.us-east-1.amazonaws.com/myapp   02e56bac4ee3f27a4d5670a6ac0d578e3dd5b2e5   af154c422af8        2 days ago          1.25GB
ecr.us-east-1.amazonaws.com/myapp   d43d33c8e15a5578cf9f6831f94a18f94e9b932f   e9958837c2c5        3 days ago          1.25GB
ecr.us-east-1.amazonaws.com/myapp   3a97c3ac36f6c241fb061199c5d3347b9bbff94a   3e80e7a25667        3 days ago          1.25GB
ecr.us-east-1.amazonaws.com/myapp   12f9a2890c67d42eb050e7d06dd593ec932b1ee7   8c07b05cd756        4 days ago          1.25GB
ecr.us-east-1.amazonaws.com/myapp   7b55fb6a75c73d16cbc731050d29323ae6b90632   426530a5cc66        4 days ago          1.22GB
ecr.us-east-1.amazonaws.com/myapp   c222e88d6b24984f338091cc5600aabb62e0ac72   e65e650d2dd1        2 weeks ago

and I want to store locally only N most recent images. Can I configure some kind of policy or is there any common script that runs rotation with cron or something like that? Other best practices/suggestions are welcome.

I do not believe that there is a way to set this as a global policy.  However, there is a simple command you can run that will "prune" or remove your unused images.

You will want to use docker image prune --filter [options] to achieve this.  Check out the official documentation for information on how to best filter.  As a best practice you should be tagging your images which will make managing of them easier using tools like this.


Jenkins configuration to support multi-config multi-branch CI Here are my requirements and I go back-and-forth on how to achieve them:

Must build Java spring-boot apps
Artifacts must be released to artifactory repository
Docker image to be built directly relating to the artifact group/artifactId/version
Docker image to be deployed to AWS ACR
Container deployment via cloud formation
Maintain 2 branches in a git repo: dev and master. dev releases to our development environment and master is for prod (I have tried to explain that this approach made sense when code was app (php) but not when you have a release artifact)
All branches off dev must build and test independently but NOT release even snapshots (doesn't apply to master as no one should be working off master)
best part dev and prod builds must be done under entirely different AWS credentials and accounts into totally different VPCs

Given these requirements I believe need a multi-configuration pipeline - one for the {prod/master} and {development/dev} arrangement to successfully capture the credential and deployment differences and point them at different branches. What I can't seem to do is get the dev configuration to ALSO build the branches from dev - I have no pattern I can match - the branches don't look like dev-.* - they are mapped directly to Jira tickets.

Can I even do such a thing? I feel like this last part is where my multi-config is breaking down.

You can use a Jenkinsfile for this. You can add logic  with groovy for every condition you've listed above.

You'll want to use a multi-branch pipeline build for sure, and just write in conditions / switches to determine what you want to do based on the current branch. Jenkins provides a global environment variable for all builds: BRANCH_NAME

The job will run on any branch that contains the Jenkinsfile and you can write conditions in for steps if you  want to include / exclude certain branches. You can use the when directive for full-step conditions: https://jenkins.io/doc/book/pipeline/syntax/#when

For handling credentials: https://jenkins.io/doc/book/pipeline/jenkinsfile/#handling-credentials


Open source microservice visualization We run various microservices on rancher and kubernetes, and I am wondering if there are any open source projects that assist in visualizing network traffic between services to help detect dependencies. Similar to the project below

https://docs.netsil.com/

I believe some of the monitoring tools like Dynatrace and Scope show some neat stuff. This really depends on your platform (I'm assuming Kubernetes).

However, if you are running something else checkout Visceral. Here is a neat video of it in action. It's more complicated to get up and going.


What's the difference between DevOps and Automation? I see that whenever someone does DevOps, it's mostly about automating things like deployment etc.

But where does automation end and DevOps begin?

A big part of DevOps is making it possible to release very often. That comes with automated build, automated testing, etc. You can say that to achieve its goals, DevOps need to use automation to be efficient.

Here's how DevOps and automation are related. DevOps is not just automation, there's more to it. Conversely, automation is not exclusively used by "DevOps people". A lot of automation was taking place in IT before DevOps came around.

Please don't consider to above diagram to represent all that is DevOps, or all this is automation. It is to help the reader picture how the two concepts are related.


Set job start timeout in Gitlab CI My pipeline consists of very long jobs that build docker images by compiling a set of packages; all of them require ~ 3 hours on my runner.

I set up my CI so that only one job can run at a time, in order to reduce the disk I/O overhead. But in this way many jobs fail, supposedly because they hit a job start timeout which seems to be bout 1h, with this message:

  There has been a timeout failure or the job got stuck. Check your timeout limits or try again

I have not been able to find how to increase the job start timeout, since all the documentation I found is relative to the job duration timeout.

So I’d need help with this, thanks. I'm using GitLab CE 12.8.1 and Gitlab Runner 11.5.1.

There are three types of timeouts in Gitlab CI:

Project timeout
Runner timeout
Jobs timeout

Project timeout

According to GitLab docs:

  Timeout defines the maximum amount of time in minutes that a job is able run. This is configurable under your project’s Settings > CI/CD > General pipelines settings. The default value is 60 minutes. Decrease the time limit if you want to impose a hard limit on your jobs’ running time or increase it otherwise. In any case, if the job surpasses the threshold, it is marked as failed

Runner timeout

According to GitLab docs:

  For each Runner you can specify a maximum job timeout.

Runner timeout settings are defined in runner's edit page

Jobs timeout

GitLab CI/CD Pipeline Configuration Reference | GitLab

Job's timeout allows you to configure a timeout for a specific job. For example:

build:
  script: build.sh
  timeout: 3 hours 30 minutes

test:
  script: rspec
  timeout: 3h 30m

Precedence of different types of timeout

The job-level timeout can exceed the project-level timeout but can not exceed the Runner-specific timeout.

If runner timeout smaller than project defined timeout, will take the precedence.

Examples of precedence of timeout directive

See Configuring GitLab Runners | GitLab

name: **Example 1 - Runner timeout bigger than project timeout**
  project_timeout: 2h
  runner_timeout: 24h
  job_timeout: 4h
  resulting_timeout: 4h

name: **Example 2 - Runner timeout not configured**
  project_timeout: 2h
  job_timeout: 4h
  resulting_timeout: 4h

name: **Example 3 - Runner AND job timeout are not configured**
  project_timeout: 24h
  resulting_timeout: 24h

name: **Example 4 - Runner timeout smaller than project timeout**
  project_timeout: 2h
  runner_timeout: 30m
  resulting_timeout: 30m

name: **Example 5 - Runner timeout smaller than Project timeout, Job timeout is bigger than Runner timeout**
  project_timeout: 2h
  runner_timeout: 30m
  job_timeout: 1h
  resulting_timeout: 30m


Way to Connect to all ec2-instances one by one Is there any way through which I can connect to all ec2-instances one by one through Python script to extract some server information from there. Please note that all ec2-instances lies behind a bastion host. Using shell script I can do that but how about python. Is there any specific library?

You can benefit from EC2 inventory script provided by Ansible - click
It will allow you to to connect to all ec2-instances, one by one (in serial: 1 mode)


How to determine which packaged pip to install in order to run the Ansible pip module? I am struggling to find a reasonable method to detect which of python-pip or python3-pip I should install depending on the contents of the ansible_python fact. This is on Debian and Ubuntu.

Looking at ansible_python contents, I did not see anything that would let me correlate the Python version Ansible uses with the package name used to install that Python version. So even though I could glean the path to the Python executable from the fact I don't see how to determine the package used to install it.

The rationale is trying to avoid hardcoding stuff into my playbook. But failing everything else I'd also be willing to hardcode these assumptions into my playbook. The "best" I've come up with so far was to use the package_facts module to collect the information on installed packages, then use ansible_python.version.major to pick from either python-pip or python3-pip, which still contains the assumptions: 1.) that ansible_python.version.major==2 maps to python-pip and ansible_python.version.major==3 maps to python3-pip and 2.) that ansible_python.executable does name either of the packaged Python versions available in the python and python3 packages respectively.

Is there an objectively better way? Better isn't meant to elicit opinions here, anything getting rid of the implied assumptions qualifies as objectively better.

There are two options either you make sure you use the same Python version on each host or you make sure the right packages are installed on each platform.

Chose same Python interpreter

Provide the python binary to use. Since you have a Debian on each host, the location is always the same. If you know a Python version that is available on each host, go for this.

Either in your inventory hosts

[webserver]
srv1.example.com
srv2.example.com

[webserver:vars]
ansible_python_interpreter=/usr/bin/python3

Or in your Ansible configuration ansible.cfg

ansible_python_interpreter=/usr/bin/python3

See also https://docs.ansible.com/ansible/latest/reference_appendices/python_3_support.html#using-python-3-on-the-managed-machines-with-commands-and-playbooks

Package per Python version

If you do not know whether either Python 3 or 2 is available on each host, ensure to get the right package for each Python version.

- apt:
    name: python-pip
  when: ansible_python.version.major==2

- apt:
    name: python3-pip
  when: ansible_python.version.major==3

You may face the same issue when installing Python packages. So you can put your Pythone version specific tasks in a v2 and in a v3 file and conditionally include one for version 2 and the other for version 3.


Docker ADD always invalidates cache I've got problem with my Dockerfile.
After building an image it is pushed to repository as cache-image, so on next builds I can pull it and use with --cache-from.

Problem is that build stops using cache after first ADD command.

Step 20/30 : WORKDIR $PROJECT_PATH
---> Using cache
---> d79e2ba28d96
Step 21/30 : ADD requirements $PROJECT_PATH/requirements
---> 6261bb0bd3e6
Removing intermediate container 26707f4107c9

I ran both images (cache and built one) and checked content of requirements directory, they're both identical.
Whole build is done on Bamboo worker.

Check the image history of your two images with docker image history. Pay careful attention to the sha256 checksum of the files being added on the ADD line. Docker performs a checksum on the files, including some of the metadata on these files, so a single byte change or a permission update will result in a cache miss.

Once you've checked this, if it is not your issue, please update the question with the pull command, the build command and full output, and the history command on both images.

Side note: you likely want to use COPY instead of ADD unless you are pulling remote files or extracting a tar file into a directory.


Jenkins Pipeline and stash Pull Request Builder not working on PR create/update Below is the requirement needed to achieve using the Jenkins Pipeline and i am new bee into Jenkins Pipeline.

After completing development work and pushing his changes to
Bitbucket the user creates a pull request.
In order to approve a pull request we require at least one successful
Jenkins build. Thereby we would like to get only the build result of
the code checked in for the pull request.
When a pull request is created/updated Jenkins shall be triggered
automatically for real continuous integration.
The build result shall be reported back to Bitbucket.

Used Stash Pull Request Builder and stash Notifier for the above process which is working for Normal Freestyle Project.
We need to migrate the similar functionality using Jenkins pipeline, So have created the jenkins job as below.

The pipeline script to checks out the PR branch and trigger build is as below
node {
    stage('Checkout') {
        checkout(
        [
            $class: 'GitSCM',
            extensions: [
                [$class: 'CleanCheckout'],
            ],
            branches: [
                [name: '']
            ],
            userRemoteConfigs:
            [[
                credentialsId: 'id',
                url: 'repourl.git'
                refspec: ('+refs/pull-requests/*/from:refs/remotes/origin/pr/*/from'),
                branch: ('origin/pr/${pullRequestId}/from')
            ]]
        ])
    }

    stage('Build') {
        sh 'make'
    }
    stage('notify') {
    step([$class: 'StashNotifier'])
        try {
            // Do stuff
            currentBuild.result = 'SUCCESS'
        } catch(err) {
            currentBuild.result = 'FAILED'
        }
    step([$class: 'StashNotifier'])
   }

}

Though i have done the above configuration when i create/update the PR, the build is not automatically triggered in jenkins. I guess the notification from stash to jenkins did not happen because we specify "origin/${pullRequestId}/from" in free style project. But i do not have that option to specify in pipeline job.
Tried few alternatives as below.
Instead of stash Pull Request Builder i tried with just "Poll SCM" project and specified cron job to trigger as "H/2 * * * *". Upon commits the job is triggered at jenkins. It means that for every commit the jenkins job gets triggered. But Jenkins should trigger the job when PR is created/updated.
I am missing something here certainly which could be basic and new to jenkins pipeline.
Any hint on achieving the desired behavior?

To expand on the answer above - try using:

branch: "origin/pr/${pullRequestId}/from"

Because in groovy, string interpolation is not done for simple strings - i.e. strings within single quotes. See http://groovy-lang.org/syntax.html#_string_interpolation for details on that.


What is the best way to put on a repository for a docker image with multiple tags? I've seen a lot of repositories with Docker images and I think (correct me if my guessing is wrong) that the projects that handle a docker image with multiple tags can be grouped into three groups:

Dockerfiles with different names in the same directory.
Dockerfiles with the same name (i.e. "Dockerfile") in different directories.
Dockerfiles with the same name (i.e. "Dockerfile") in different branches.

I've looked at the Docker documentation page but there are no suggestions on how one should manage multiple tag images. I'm not looking for a subjective answer. Of course, any advice will be appreciated but what I want is to know the pros and cons of each of the three possibilities.

Of course, there is the possibility that I've understood nothing of how docker images should be deployed. In this unfortunate case, please be merciful and point me in the right direction.

Case study

I wanted to pose a general question but I realized that maybe it was too broad. For this reason, I explain here what is my case hoping that could help a bit more what I wanted to achieve and what I've done now (this question is about only a part of what I wanted to achieve).

What I want is to develop a docker hub automated build linked to a Git Hub repository. The images are about LaTeX on Ubuntu. The first build, tagged base, starts from Ubuntu 18.04 and comes with texlive-base. Another build, tagged full, starts from the build tagged base and comes with texlive-full. Another two builds, tagged develop-base and develop-full, will start from the respectively builds without 'develop' and will come with writing and visualization tools. All the builds will probably have some semantic versioning system.

What I've done right now is implement base and full through two different Docker files into two different directories in the same Git Hub repository. Whenever a tag base or full is created/moved, the corresponding build is triggered. This implementation has a problem: if I move the tag base, the image base is correctly built but all the images that depend on it are not built again and I have to manually trigger them. What I've thought is to have a branch for each tagged image (with the corresponding automated builds triggered at each new push on that branch) and a tag system to keep track of the version number (with the corresponding automated builds triggered at each new tag). The tags will be base, full, develop-base, etc. for each latest version and with semantic versioning for each version, including the latest, i.e. 1.0-base, 2.3-full, 1.3.1-develop-base, ecc. In the latter case, the automated build triggering will probably have to handle regular expression to automatically label the images with a version number.

I realize that my question isn't very related to what I wanted to achieve but I want to learn to walk before trying to run. Thus, I want to understand when and why one should choose one way or the other to put on a docker hub repository with several images and which are those ways. The project of LaTeX images itself, that maybe you could think that is very naive, is to deeply understand the way docker, docker hub, docker cloud, docker compose, ecc. work.

If you think of the Dockerfile as a kind of "bill of materials" for your containers, then there is the following consideration:

container environments are mostly same - you can use many Dockerfiles in same folder, or even just one Dockerfile with parametrized builds
container environments are or might divert in terms of further "local assembly parts" configuration files, scripts etc., then you are better off with using separate folders.

Branching has rather to do with developing versions, or also independent features which then should be anyway merged together. See Git flow on this topic.

As a reference project laoyut, check for example the GitHub project linked from Tomcat Docker Hub site. Here you find different folders, even with single Dockerfiles.

See also this TeX-Docker distro:

https://hub.docker.com/r/raabf/texstudio-versions/

Side note: strange enough there is no "official" Docker distro for TeX, while many other well established open source communities have this library entry.


Create multiple S3 buckets at once I am trying to create a static website using S3 buckets.  I want to create a bucket for www and non-www versions.  I can't work out how to create two buckets at once.  I read that you can chain two entries together using square brackets.  And I read that you can use variables in the json file.

So far I haven't managed to get variables or multiple entries to work.  Can anyone suggest what I am doing wrong?

I have used this code:

Main.tf

provider "aws" {
  shared_credentials_file = "C:/Users/Eoin.Oliver/.aws/terraform.tfvars"
  profile                 = "customprofile"
  access_key = "${var.access_key}"
  secret_key = "${var.secret_key}"
  region     = "${var.region}"
}

resource "aws_s3_bucket" "b" {
  count      = "${length(var.domain_name)}"
  bucket     = "${var.domain_name[count.index]}"
  acl        = "public-read"
  policy     = "${file("bucket-policy.json")}"

  website {
    index_document = "index.html"
    error_document = "error.html"

    routing_rules = <<EOF
EOF
  }
}

Variables.tf

variable "access_key" {}
variable "secret_key" {}
variable "region" {
    default = "eu-west-2"
}
variable "domain_name" {}

terraform.tvars

domain_name             = ["website.com", "www.website.com"]
region                  = "eu-west-2"
access_key              = "xxxxxxxxx"
secret_key              = "xxxxxxxxxxx"

bucket-policy.json

{
    "Version": "2012-10-17",
     "Statement": [
        {
             "Effect": "Allow",
            "Principal": {
                "AWS": ["arn:aws:iam::240479180938:user/${iams_user_name}"]
            },
            "Action": "s3:PutBucketPolicy",
            "Resource": "arn:aws:s3:::${domain_name}"
        }
    ]
}

You can make changes in Main.tf as specified below:

resource "aws_s3_bucket" "b" {
  count      = "${length(var.domain_name)}"
  bucket     = "${element(var.domain_name, count.index)}"
  acl        = "public-read"
  policy     = "${file("bucket-policy.json")}"

  website {
    index_document = "index.html"
    error_document = "error.html"

    routing_rules = <<EOF
EOF
  }
}


What are reasons to prefer an ingress over using `expose` in k8s? If one uses expose in k8s then it is possible to assign an IP to an app so it could be accessed on the www.

It is also possible to configure an ingress. This is basically a rproxy.

What are reasons to use an ingress over expose?

As @Tensibai indicated:

  With an ingress you can have multiples names on same port, no possible
  by exposing a port tied to only one service


Using a combination of Terraform and Ansible to create dynamic AWS EC2 infrastructure The company I work for wants to prevent provider lock-in adn therefore told us to use Terraform+Ansible instead of Cloudformation. We already decided to use terragrunt as an addon.
While the motivation for the company decision is clear they didn't provide guidance how to achieve this and we have to find out on our own how to do this in an "elegant" way.
In my scenario this means I use the two tools like this:
Terraform:

provision EC2 instances (Ubuntu 20.04 in this case)
define security groups with ingress rules
define host name aliases (assigning them is a weak spot, I will come back to this later)

Ansible:

install additional OS level components
install and configure the application

All of this will be run from a devops machine, after that I have a working application setup.
But what happens when for example monitoring decides that one of ten instances is no longer healthy ? That component will then be nuked and created from scratch. In this scenario, how can I make sure that:

the replacement instance is configured using the same Ansible playbooks that were run before ?
host name aliases will be updated

For the host name alias topic I have to give you a little bit more background on what we need. In one scenario we use an application that does not load balance. Because of this we create for example three instances which we assign aliases:
inst-one   -> lg1.company.foo
inst-two   -> lg2.company.foo
inst-three -> lg3.company.foo

Users access the instances using the aliases
Let's assume that inst-two becomes unhealthy, then this will need to happen:
inst-one   -> lg1.company.foo
inst-two   -> terminated
inst-new   -> lg2.company.foo
inst-three -> lg3.company.foo

There are a lot of gaps in your deployment and configuration strategy. In order to provide a more robust and elegant strategy, I suggest you:

Define autoscaling groups with Terraform instead of provisioning individual EC2 instances. Auto-scaling groups make dynamically scaling instances a much simpler process. It will also allow you to define scaling policies and health checks for your instances to determine when to add/remove instances from your autoscaling group.
Create and define CloudWatch Metrics and CloudWatch Alarms. CloudWatch allows you to create custom metrics to monitor and alert via CloudWatch Alarms. This can be used in conjunction with autoscaling group scaling policies to define a threshold to scale your instances based on metrics such as cpu utilization, application response time, etc.
Instead of using Ansible to configure each EC2 instance, use Ansible to configure an Amazon Machine Image (AMI) to be uploaded and used by your autoscaling group. If your instances are configured identically, configuration is a one and done process. If your instances are configured differently, consider uploading multiple AMIs, and separating into multiple autoscaling groups. You can also use the User Data definition of an EC2 instance to run shell commands on each individual instance at launch (a one time process).
Last, but not least, instead of running this process from "a devops machine", consider using a build/deploy CI pipeline to run and configure this entire process. Tools such as AWS CodePipeline, Jenkins, GitLab CI, etc. allow for you and other team members to run your configuration in a configured and repeatable manner. It will also allow you define stages and deployment strategies should you choose to implement testing for your application later on in your development process.


How to set a Circle CI environment variable for a project? I've set the environment variable with a "context" (I have no idea if thats the right place - it's really confusing!)

None the less, it's now doing a build - and I don't see the environment variables set (I'm using the SSH build to inspect the vars with getenv).

Why is it not being picked up? Am I meant to set an environment variable by some other means rather than a "context"?

https://circleci.com/docs/2.0/env-vars/#setting-an-environment-variable-in-a-project

Seems "contexts" can be used but must then also be set to "reference" the context in the build yaml config file.

Instead to set a environment variable for a project:
Jobs > Cog / gear icon of a project > Build settings > Environment variable


How to check if pipeline parameter is empty I have a pipeline job which includes some parameters:

In the pipeline file I have the below code:

stage ("create bundle"){
            steps{
                script{
                        amd_distribution_create_bundle credential_id: params.DISTRIBUTION_CREDENTIAL_ID, distribution_url: params.DISTRIBUTION_URL, gps_credential_id: params.GPG_PASSPHRASE, bundle_name: params.BUNDLE_NAME, bundle_version: BUNDLE_VERSION
                }
            }
        }

how can I ask before calling the groovy method 'amd_distribution_create_bundle', if some field is empty?

While creating the credentials parameter in jenkins job, you can specify required: true, then jenkins should validate the credentials paramter.

parameters {
    credentials(name: 'GPG_PASSPHRASE', defaultValue: '', credentialType: "Username with password", required: true )
}

You can also check the details mentioned in the link https://docwhat.org/jenkins-user-credentials

Adding Parameter

/* EXAMPLE */
parameters {
    credentials(
        credentialType: 'com.cloudbees.plugins.credentials.impl.UsernamePasswordCredentialsImpl',
        defaultValue: '',
        description: 'The credentials needed to deploy.',
        name: 'deployCredentialsId',
        required: true
    )
}

Using in the stage

/* EXAMPLE */
steps {
    withCredentials([usernamePassword(
        credentialsId: '${deployCredentialsId}',
        usernameVariable: 'DEPLOY_USERNAME',
        passwordVariable: 'DEPLOY_PASSWORD',
    )]) {
        sh './my-command.bash --username="${DEPLOY_USERNAME}" --password="${DEPLOY_PASSWORD}"'
    }
}

If you just want to check if the variable is empty or not, you can check isEmpty() in groovy script. Your code can be as follows:

stage ("create bundle"){
        steps{
            script{
                    if ( GPG_PASSPHRASE.isEmpty() ) {
                             GPG_PASSPHRASE = 'custom_string'
                     }
                    amd_distribution_create_bundle credential_id: params.DISTRIBUTION_CREDENTIAL_ID, distribution_url: params.DISTRIBUTION_URL, gps_credential_id: params.GPG_PASSPHRASE, bundle_name: params.BUNDLE_NAME, bundle_version: BUNDLE_VERSION
            }
        }
    }


Transfer files and execute commands from jenkins scripted pipeline to AWS instance I want to transfer files and execute a few commands on remote AWS instance from Jenkins. I have the following methods in mind to do the same.
Note that my Jenkins server is running in the local server, and I don't have SSH access to Jenkins

1) Using hardcoded pem file
I have following scripted pipeline to transfer files from Jenkins pipeline to AWS EC2 instance.

stage('transfer docker image to ec2') {
  sh 'scp -oIdentityFile=/var/lib/jenkins/.key.pem docker_image.tar ubuntu@ec2-3-83-239-32.compute-1.amazonaws.com:'

However it is not working, and waiting for a long time without showing logs.

2) Using Jenkins "credential"

I have added the pem file in Jenkins credential, and then invoked the credential by the following method. (Referred from - https://stackoverflow.com/a/47627460/9403545)

withCredentials([sshUserPrivateKey(credentialsId: "bindu-test", keyFileVariable: 'keyfile')]) {
 stage('transfer docker image to ec2') {
  sh 'scp -i ${keyfile} test ubuntu@ec2-x-83-xxx-32.compute-1.amazonaws.com:'

Unfortunately, the above method also didn't work and got the following error.

  /tmp/../Hello directory doesn't exist.

I have tried to use Publish over SSH, but it doesn't seem to be supported in scripted pipeline.

Is there any way I can do this?
P.S: Please keep in mind that I'm AWS newbie, it would be nice if the answer is in detail :)

Below is sample groovy code for sending file or execute command over SSH in jenkins:

node(master)
{
  stage(Deploy)
  {
     sshPublisher(publishers: [sshPublisherDesc(configName: 'SERVER_NAME', transfers: [sshTransfer(cleanRemote: false, excludes: '', execCommand: 'apt-get update', execTimeout: 120000, flatten: false, makeEmptyDirs: false, noDefaultExcludes: false, patternSeparator: '[, ]+', remoteDirectory: '', remoteDirectorySDF: false, removePrefix: '', sourceFiles: '*.war')], usePromotionTimestamp: false, useWorkspaceInPromotion: false, verbose: false)])
  }
}


A comparison of idempotence and immutability Many in the DevOps apply the cattle-not-pets mentality by implementing immutable infrastructure and redeploying when changes are needed (instead of modifying).

Configuration management has a similar principle of idempotence. What are the comparative benefits, similarities and drawbacks of immutability vs. impotence and which is more efficient? Can these be used synergisticly (eg, periodic deletion and redeployment of VMs or Docker containers using configuration management?)

The two terms are very different.

Let us start with immutability, which literally means "no mutations" or "no changes". In the DevOps sense, it means that once you created an artifact, be that a container image, or a VM image, or maybe a package from compiled code - you declare that you will never ever change it. Often if any changes are required, you declare that a new version of "thing" will be created instead.

The term idempotence means that when changes are applied multiple times, the state is mutated (changed) just once. First, it already assumes that there are going to be changes applied which means that you cannot have something both immutable and have idempotent actions done to it (no actions are done to it by contract).

In the use of configuration management tools, idempotence is used in some cases when applying the same change multiple times. Like adding the line that says localhost to the /etc/hosts file, you don't really need multiple such lines and if one already exists it is safe to not try to append again.

Also idempotent is a term used to describe actions that attempt to change things, while immutable is used to describe nouns (objects) that are set against changes done to them.

Why is an immutable object useful? Because when you copy it around, for example from a dev environment to qa to production. You already know quite a lot (but not everything) regarding how it is going to behave. In many cases, the parts that are working will be consistent, and the parts that are broken are going to be consistent as well.

Why are idempotent actions useful? Because when you want to change a state of some object, in many cases it is useful to only verify that the change has been applied and apply changes just in case it is required. For example, when a configuration item in a file is missing or has the wrong value, it is useful to add it just once or change it just once while applying the action multiple times. In many other cases, like log files, you don't want to have idempotent actions because you often do want to append another line each time some event happens.


What can I use to forward a conection to port 3006 on the bastion to an RDS database? Forwarding a port 3006 database connection from a bastion server to a RDS database?

Trying to find the answer online I came across the term Proxy jump but can't find anything on proxy jump for 3006 to send a 3006 mysql conection to the bastion on port 3006 to an RDS?

It is possible on the Linux(Unix) bastion host.

Step-1: First ssh to the bastion host. eg. 16.23.14.20 is the bastion host.

# ssh root@16.23.14.20

Step-2: Create port tunneling by using the below command. eg. The RDS IP is 172.16.13.10

# ssh -L  3006:172.16.13.10:3006 localhost

Verify using netstat command.

netstat -ntulp | grep 3006

Note: If netstat command not available. Using sudo apt install -y net-tools or sudo yum install -y net-tools


Preventing an IAM user from overriding launch template parameters on RunInstance I'm trying to set up an IAM policy that allows a user to run an instance (ec2:RunInstances), but only from a specific launch template - and without making any changes to that launch template's parameters.

The relevant section of my policy looks like this:

{
  "Effect": "Allow",
  "Action": "ec2:RunInstances",
  "Resource": "*",
  "Condition": {
    "ArnLike": {
      "ec2:LaunchTemplate": "arn:aws:ec2:ap-southeast-2:xxxxxxx:launch-template/lt-xxxxxxx"
    },
  }
}

This works great. Running the following using the user's credentials works:

aws ec2 run-instances --launch-template LaunchTemplateName=test

...and specifying a different launch template fails. So far so good.

However, the user can override values in the launch template, and pretty much do whatever they like:

aws ec2 run-instances --launch-template LaunchTemplateName=test --instance-type t2.micro

This is despite me setting the instance type in the launch template as t2.nano.

The documentation has a solution for this: require the ec2:IsLaunchTemplateResource condition to be "true", which apparently only is so if the user has not overridden the launch template.

Perfect - exactly what I was after! The Condition statement in my IAM policy now looks like this:

  "Condition": {
    "ArnLike": {
      "ec2:LaunchTemplate": "arn:aws:ec2:ap-southeast-2:xxxxxxx:launch-template/lt-xxxxxxx"
    },
    "Bool": {
      "ec2:IsLaunchTemplateResource": "true"
    }
  }

However, with this additional condition present, the RunInstances call fails even if the user makes no changes. After decoding the authorization failure message, I can see that AWS is reporting ec2:IsLaunchTemplateResource was not "true":

{
  "key": "ec2:IsLaunchTemplateResource",
  "values": {
    "items": [
      {
        "value": "false"
      }
    ]
  }
}

EDIT: After posting this, I continued to iterate, and noticed that the specific resource that was hitting up against this issue was the subnet - which wasn't defined in the launch template. Taking cues from additional examples in the documentation I linked to, I tried adding another statement to allow the subnet resource even without ec2:IsLaunchTemplateResource being true, and this worked. I also tried, alternatively, explicitly defining the subnet in the launch template: this worked too.

However, now I am not getting access denied even when changing the instance type! So it appears that it was the subnet all along that was triggering the original denial. While I've sort of taken a step forward, the symptom of the issue has now flipped from what I was initially experiencing... and now although the user can launch, they can also launch instances I don't want them to. So my initial question below still remains. :)

So, my question is... have I done something wrong? Does ec2:IsLaunchTemplateResource work in the way I have understood it (which I think is fairly clear in the documentation)?

Alternatively, is there another way I could allow an IAM user to launch an instance in line with a template, while preventing them from making changes (such as choosing a super-expensive instance type)?

Alternatively, is there another way I could allow an IAM user to launch an instance in line with a template, while preventing them from making changes (such as choosing a super-expensive instance type)?

I've encountered this problem last year and have circulated up to our AWS account's architect, who contacted their IAM division.

AWS's response is that this is intended behavior. Reason given was that "An instance-type spec on a launch template is not a resource, or conditional that can be used to granted/deny permissions to". Notice that inside the IAM policy, specifying a launch template is a condition to action "RunInstance", and the resource of said policy is an ec2 instance.

According to AWS IAM team, in order for you to restrict instance-type (a condition on the resource, "instance"), you must specify the restriction as part of the condition clause to be applied to the instance, like the following:

{
  "Effect": "Allow",
  "Action": "ec2:RunInstances",
  "Resource": "*",
  "Condition": {
    "ArnLike": {
      "ec2:LaunchTemplate": "arn:aws:ec2:ap-southeast-2:xxxxxxx:launch-template/lt-xxxxxxx"
    },
    "Bool": {
      "ec2:IsLaunchTemplateResource": "true"
    },
    "StringNotLikeIfExists": {
      "ec2:InstanceType": [
        "c5.*xlarge",
        "p3.*xlarge"
      ]
    }
  }
}

Obviously, it is near impossible to list all of the expensive instances we don't want our users to use, and kinda defeats the purpose of using a launch template.  But in a messed-up way, in the IAM world AWS created, this actually makes sense.

Consider the docs here: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ExamplePolicies_EC2.html#iam-example-runinstances-launch-templates

Note the language: The policy uses the ec2:IsLaunchTemplateResource condition key to prevent users from overriding any of the launch template *resources* in the RunInstances request.  So that clause only applies to resources (like NICs, IPs, etc), but not instance-type, which is not considered as a resource.

I asked that they revise their docs to explain this, and of course, nothing's changed.


Kubernetes on k3s can't resolve domains from custom dns server (fritz.box with dnsmasq) I have a dns server running at 192.168.0.19 for custom domains like .fritz.box. Having a single node cluster on k3s, Rancher was installed using a subdomain server2.fritz.box using this command:

    helm install rancher-latest/rancher \
      --name rancher \
      --namespace cattle-system \
      --set hostname=server2.fritz.box

Rancher itself shows that some services are not avaliable and the logs from cattle say server2.fritz.box is not avaliable. Since Kubernetes has its own dns system, I looked at the documentation and it seems that I need to set my .19 dns server so that Kubernetes knows how to resolve .fritz.box domains. Some questions also have similar problems like https://stackoverflow.com/questions/41448095/kube-dns-does-not-resolve-external-hosts-on-kubeadm-bare-metal-cluster

So I created the following yaml:

# https://github.com/kubernetes/kops/issues/4986
apiVersion: v1
kind: ConfigMap
metadata:
  name: kube-dns
  namespace: kube-system
data:
  stubDomains: |
    {"fritz.box": ["192.168.0.19"]}
  upstreamNameservers: |
    ["192.168.0.19"]

Loaded with kubectl apply -f dns.yml. Now created a busybox test pod:

~$ kubectl exec -it busybox -- ping server2.fritz.box
ping: bad address 'server2.fritz.box'

Why is this not working?
And what must be done to resolve a custom dns server in Kubernetes?

I believe this is an current bug in k3s that upstream DNS is hardcoded to 1.1.1.1.  this should be resolved shortly https://github.com/rancher/k3s/issues/53


Is \u the only way to output the user from .bashrc? I can't get it to work on my Mac (catalina version 10.15.2).
\u works on ubuntu when I test it but not on my Mac.
I want to customize my mac terminal. Colors work fine on my Mac and it's reading the .bashrc file when I test it by source .bashrc  .
For example with the following in .bashrc in the user home directory.

orange=$(tput setaf 166);
PS1="${orange}\u"
export PS1;

Outputs my username in orange on ubuntu but on my Mac it just outputs \u in orange so for some reason \u isn't working on my Mac? Is there an alternative I could try?

With the release of macOS Catalina (10.15) Apple has switched to Zsh as default shell (instead of Bash).
Instead of setting PS1 in Bash you can set PROMPT in Zsh:
PROMPT="${orange}%n"


How to run a Windows container inside Windows Subsystem for Linux (WSL)? I have managed to install docker on Windows, and I can run docker from my Ubuntu bash prompt of the WSL. But when I try to pull or run a Windows image from docker hub, I get the following error:

root@DESKTOP-J9B5HJG:~# docker info  -f '{{.OSType}}/{{.Architecture}}'
linux/x86_64
root@DESKTOP-J9B5HJG:~# docker pull mcr.microsoft.com/windows:1809-amd64
Error response from daemon: manifest for mcr.microsoft.com/windows:1809-amd64 not found
root@DESKTOP-J9B5HJG:~#

C:\Users\romain.prevost>docker info  -f '{{.OSType}}/{{.Architecture}}'
'linux/x86_64'

Now, if you are running Docker for Windows, right click the Docker icon in your tray, and select "Switch to windows containers"

C:\Users\romain.prevost>docker info  -f '{{.OSType}}/{{.Architecture}}'
'windows/x86_64'

Voilà! Essentially, Docker for Windows includes two different engines : Linux in a VM, and Windows, natively on your windows kernel. But soon, there will be options to run Linux containers natively on the Windows kernel. Exciting times!


Nginx /foo and /foo/ redirects I want both http://domain/foo and http://domain/foo/ requests to serve a given resource.
Here's the relevant part:

location / {
       rewrite /(.*) /ipns/QmdpoFuwY/$1 break;
       proxy_pass http://127.0.0.1:8080;
}

with this config only:

curl -X GET http://domain.io/foo/

returns a resource while:

curl -X GET http://domain.io/foo

gives a Path Resolve error.

One could replace

rewrite /(.*) /ipns/QmdpoFuwY/$1 break;

with

rewrite ^(.*[^/]) /ipns/QmdpoFuwY/$1 break;

and try again.

The issue was solved by inspecting the logs. If for example one navigated to /foo/ instead of /foo/ the log indicated:

2017/10/15 14:51:28 [error] 7#7: *1 "/etc/nginx/html/index.html/foo.html" is not found (20: Not a directory)

It turned out that the regex did not match /foo/. One could also enable the rewrite logging to facilitate debugging:

rewrite_log on;


How to explain the Twelve Factors to non techy people? The Twelve Factors is a sort of manifesto seen as a road-map for developers to follow when building modern web-based.

How would you explain each of the 12 factors in less than three sentences to random people?

As far as I can read between the lines, you want your upper management to support you in introducing the 12-factor approach to your company's software development.

In that respect, you do not want to explain each of the 12 factors to them. If they really were interested in that, then you could just send them the 12-factor website you have linked in your question - it already has 1-sentence summaries.

I'll answer the question you wrote in the title:

  How to explain the Twelve Factors to non techy people?

The Twelve Factors are not a soft "culture". It is a set of concrete, practical and pretty low-level techniques for solving problems relating to configuration management, deployment and high availability of applications. Instead of "culture" or "methodology", I would rather call it a pattern, or a kind of handbook for developers. It is obviously related to DevOps, cloud and similar buzzwords, but not necessarily so - you can also employ the techniques if you only have one developer doing his own thing, and even if there is no cloud or otherwise "modern" environment involved at all.

It is not even necessary to involve management in the question whether to employ some or all of the techniques mentioned in the 12-factors. Many of them are simply best practices which everybody who develops ("service"-like) software will end up over the years.  This is pretty evident in the I. Codebase, III. Config, XI. Logs. No manager is required to give a green light to employ these practices.

Other of the principles, namely II. Dependencies, IV. Backing services, VI. Processes, VII. Port binding, VIII. Concurrency, IX. Disposability are neatly combined into the keyword "microservices". Don't explain to your management those specifics, but tell them that you are going to make a move from some presumably monolithic application architecture you had before to a microservice based one. On a management level, this can be sufficiently done with a graphical representation (one big blob of complex software on the one hand - several individual small blobs of software on the other hand). You can explain why the later can be better for the job at hand with nontechnical terms; every manager can understand the concepts of dividing a big, convoluted mass of code into smaller pieces that can be handled (and scaled) much easier.

Your management will want to know what it costs and how long it takes. 12-factor tells you nothing at all about that. While there is a slight cost involved in some of the factors (e.g., establishing a central log stash), the cost is minor and will easily be caught up by savings later. Other costs may be the time needed for your developers to embrace some of the more tougher parts (i.e., II, VII, IX, depending on how much existing software you already have).

Very important: it is pretty simple to work in an environment with a big monolithic application, and only add new stuff with the full 12-factor approach, or convert bits and pieces from the original software as you go. This  might be more sensible if you have time or money constraints.


What is the best way to set up a CI/CD pipeline on ECS? There are so many options:

Docker-compose with ECS cli looks the easiest solution

Terraform

CloudFormation (looks complex!)

Ansible

I am only interested in setting up a basic ECS docker set-up with ELB and easily updating the Docker image version.
We all love technology here, but we're not all super geniuses when it comes to tech. So I'm looking to keep my set-up as simple as possible. We run Jenkins, 2 NodeJS applications, 2 Java applications in ECS and I know it involves IAM, Security Groups, EBS, ELB, ECS Service/Task, ECS Task Definition, but that already gets complex quickly in CloudFormation.
What are good technologies that will allow us to use Docker, keep things simple and don't require us to be very intelligent to understand our own programming code?

The question is a bit broad, but here are some suggestions to accomplish what I believe you are asking. If they are simple or not is for you to judge!
If the infrastructure referred to already exists and you would like to manage, update using infrastructure as code (IaC), you could try tools to convert the infrastructure to code. (These are new and at least one is in beta at the time or writing this)

Existing Infrastructure to Terraform:
https://github.com/GoogleCloudPlatform/terraformer
Existing
Infrastructure to CloudFormation:
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/cfn-using-cloudformer.html

If the infrastructure does not exist, I recommend using infrastructure as code (IaC). AWS was mentioned, as well as the complexity of CloudFormation, with these considerations, Terraform could be a good option.

Cloud Posse offers some good modules to simplify using Terraform to build infrastructure. This module may be a good place to start.

Finally, if you would like to abstract away the bulk of the infrastructure and just run containers in the cloud.

Google Cloud Platform (GCP) offers Cloud Run, providing a completely managed service to run containers in the cloud with out much of the underlying complexity. The GCP Provider for Terraform can help here.
AWS Fargate cloud be another option, with Cloud Posse again offering modules for Terraform.

Once the code and infrastructure option is selected, CI/CD is as easy as adding a buildspec.yaml to the repository and adding the Code Build infrastructure to the Terraform code (AWS) and similar for Cloud Build (GCP).
Hopefully these options help find you the simple solution you are looking for!


Assertions and constraints I'm constructing a template to build a configuration file, and the service that consumes this file places constraints on identifier lengths.

If an identifier is longer than, say, 6 characters, the service will get part-way through applying the configuration, fail, and leave the node in an inconsistent state.

How can I perform an assertion to trigger a deployment transaction failure, preventing the target nodes' service from being misconfigured?

My particular circumstance is Salt, but I would be curious to see how other systems solve the problem as well.

In Ansible: you can use assert or fail module.

- name: "Make sure web_sites is dictionary"
  fail: msg="web_sites should be dictionary"
  when: web_sites is not dict

- name: "cluster_name should be shorter than 6 chars"
  assert:
       that: cluster_name|len <= 6

In Puppet: there is fail function evaluated during parsing phase which cause parsing failure on server (see question on StackOverflow)

 if length($cluster_name) > 6 {
      fail("Cluster name is too long. Should be less than 6 chars.")
 }


Debug Docker networking and Windows 10 firewall Given:

Windows 10 / 64 bit
Docker edge 17.11-ce-win40

Experiment:

docker run -it --rm -p 8888:8080 tomcat:8.0

Expected behaviour:

If all network settings are okay, Tomcat must be reachable locally and from outside on the host port 8888.

Observed behaviour:

curl localhost:8080 inside container proves Tomcat to be up and running
ERR_EMPTY_RESPONSE locally/from outside
Flipping Windows Firewall leads to the question whether vpnkit should be allowed, allowing leads to no change
nmap from outside reveals port open
telnet is possible as well but no further interaction
docker inspect -f '{{ .NetworkSettings.Networks.nat.IPAddress }} <CONTAINER_ID> outputs <no value>
find ipconfig /all output below

Further debugging

ipconfig /all [excerpt]

   Hostname  . . . . . . . . . . . . : foo
   Primäres DNS-Suffix . . . . . . . : foo.com
   Knotentyp . . . . . . . . . . . . : Hybrid
   IP-Routing aktiviert  . . . . . . : No
   WINS-Proxy aktiviert  . . . . . . : No
   DNS-Suffixsuchliste . . . . . . . : foo.com

Ethernet-Adapter vEthernet (DockerNAT):

   Verbindungsspezifisches DNS-Suffix:
   Beschreibung. . . . . . . . . . . : Hyper-V Virtual Ethernet Adapter
   Physische Adresse . . . . . . . . : 00-15-5D-48-8F-A4
   DHCP aktiviert. . . . . . . . . . : No
   Autokonfiguration aktiviert . . . : Yes
   Verbindungslokale IPv6-Adresse  . : fe80::e93b:535c:4ea5:84cf%18
   IPv4-Adresse  . . . . . . . . . . : 10.0.75.1
   Subnetzmaske  . . . . . . . . . . : 255.255.255.0
   Standardgateway . . . . . . . . . :
   DNS-Server  . . . . . . . . . . . : fec0:0:0:ffff::1%1
                                       fec0:0:0:ffff::2%1
                                       fec0:0:0:ffff::3%1
   NetBIOS über TCP/IP . . . . . . . : Active

Ethernet-Adapter Ethernet:

   Verbindungsspezifisches DNS-Suffix: foo.com
   Beschreibung. . . . . . . . . . . : Realtek USB GbE Family Controller
   Physische Adresse . . . . . . . . : D4-81-D7-57-AC-C6
   DHCP aktiviert. . . . . . . . . . : Yes
   Autokonfiguration aktiviert . . . : Yes

Since Windows use a tiny VM called MobyLinux, the port mapping will happen with tiny VM and not with Windows Host. If you are not reachable from Windows Host to Docker MobyVM, check windows events for possible error. For more networking related information, please see -

https://docs.docker.com/engine/userguide/networking/


Find appropriate AWS policy I have created a user group in AWS ("developers"). Now I want to attach a policy to the group which will grant the users permission to list, create and update Amplify apps. I am struggling to determine the appropriate policy.

I see that the "AdministratorAccess" policy includes access to the AWS Amplify service, but I don't want the developers to have administrator level access. What are my options?

Edit:
When the specific user accesses the Amplify console the following error is displayed

List apps call failed User: arn:aws:iam::XXXXXXXXXX:user/myUser is not authorized to perform: amplify:ListApps on resource: arn:aws:amplify:eu-west-1:XXXXXXXXXX:apps/*

If you want to create your own IAM policy, you can use the visual editor present in IAM. Using visual editor, you can construct a new policy from scratch using the very intuitive GUI.

Goto IAM, then policies, then use visual editor.


IT staff sizing in the DevOps era - how and when would a DevOps team scale up? Given that DevOps allows IT staff actually to downscale and do more through all the automation, what are limits to that?

That is, what are rules of thumb - if there can be any - where you would know you need more people despite rapidly progressing rates of automation and introduction of RPA (Robotic Process Automation)?

Note: this is about the staff size of the whole IT departments altogether put in relationship to your business size, structure and function (size of a single team is well defined by the two-pizza team concept).

IMHO it's not a good idea to directly and immediately apply a team's gains from its DevOps transformation into scaling the team down. At best you'll lose just the team's motivation to do any such improvement in the future. But most likely you'll lose most/all the talent in that team.

Instead:

allow the team to appreciate the benefits of the difficult transformation they went through, so that they spread the word to your other teams that will have to go through (or are already struggling with) the transformation
focus on delivering better quality products/services, faster - with the same team - this is now possible due to the transformation.
in many cases your business will pick up speed because of the transformation - soon you'll more likely need to hire people - IMHO it's better to use the experience of the ones you already have rather than educating new hires
allow voluntary attrition to work for you and get the effective downscaling down the road.

But generally the DevOps transformations tend to not happen fast enough to allow for the above choices :)

Now onto scaling up/out.

Ideally the DevOps transformation should have greatly increased the predictability of your sw development and delivery process. You should be able to tell with a great degree of precision you existing team's capacity: how much sw/services of a certain quality can your current team get done in a certain amount of time.

When you need more stuff done, or done better/faster than the current team's capacity you need to scale the team up/out. The exact details come from whatever process/methodology you use for your day-to-day activities, probably some agile variant (see also Does my organization need adopt Agile Soft. Dev. before adopting DevOps?)

How - apparently easy: hire more people to handle the overflowing activities. If you're worried about how fast you can get them onboard and actually handling that overflow work at the expected rates: start hiring earlier, based on projections.

The reality can, however, a bit more difficult than that. Scaling a software product/service or the development process and tools supporting it is not necessarily straight-forward, even for a DevOps-empowered organisation.

One of my favourite examples is the weakest DevOps link when it comes to scalability: Continuous Integration. If the CI build fails there can be no Delivery/Deployment (kinda undermining the "continuous" term in the CD context).

Just because a certain team is happily using CI with decent results it doesn't mean the same will happen if the team size doubles. Nope, the results could be devastating - the delivery process can grind down to a halt (in extreme cases). I'm trying to explain this in Congestion in Traditional CI Systems (disclaimer: I'm the founder of the company owning the referenced page and offering a solution to this problem).

Another example could be, for example, the version control system (VCS) being used: as the software being stored into a repository evolves so does its change history, pushing the limits of the VCS system, sometimes beyond acceptable performance levels. Switching to a more scalable VCS system or splitting the codebase into multiple repositories could be an approach to address the problem.

But evolving the development processes and tooling supporting them alongside the evolution of the software products/services being delivered are fundamentally just what a mature DevOps team does. They just need to also be taken into consideration when in the team's scaling up/out strategy.

With that in mind, the how can be, indeed, reduced to just increasing the team size to cover the additional amount of work necessary for execution of the above-mentioned strategy.


Jenkins: Permission issue using Docker as build environment I installed Jenkins on an Ubuntu 16.04 machine. The Jenkins itself is not run in a container. What I want to do is simply call yarn install using a node image. So here is my Jenkinsfile:

pipeline {
    agent any
    stages {
        stage('install node modules...') {
            agent { docker 'node' }
            steps {
                sh 'cd /path/to/package.json; yarn install'
            }
        }
    }
}

Pretty straightforward, right?

jenkins user/group is 112:116, and the uid of the node container is 1000, hence yarn process (which is run as node user 1000) can't do its things, like mkdir /.config.

I tried to spin up the node container passing in argument -u 1000, it bumped into permission issues when trying to create durable directories.

It looks like one or the other kind of issue, how can I work around that?

Jenkins logs:

Below is where the build starts and fails.

[Pipeline] sh
[Pipeline_Test_Jenkins_test-4JTFYMX7KSJY6ZH44VINNGEB7WH2D2HWYZN5ABF6O32O2HBQJYXQ@2] Running shell script
+ docker inspect -f . node
.
[Pipeline] withDockerContainer
Jenkins does not seem to be running inside a container
$ docker run -t -d -u 112:116 -w /var/lib/jenkins/workspace/Pipeline_Test_Jenkins_test-4JTFYMX7KSJY6ZH44VINNGEB7WH2D2HWYZN5ABF6O32O2HBQJYXQ@2 -v /var/lib/jenkins/workspace/Pipeline_Test_Jenkins_test-4JTFYMX7KSJY6ZH44VINNGEB7WH2D2HWYZN5ABF6O32O2HBQJYXQ@2:/var/lib/jenkins/workspace/Pipeline_Test_Jenkins_test-4JTFYMX7KSJY6ZH44VINNGEB7WH2D2HWYZN5ABF6O32O2HBQJYXQ@2:rw,z -v /var/lib/jenkins/workspace/Pipeline_Test_Jenkins_test-4JTFYMX7KSJY6ZH44VINNGEB7WH2D2HWYZN5ABF6O32O2HBQJYXQ@2@tmp:/var/lib/jenkins/workspace/Pipeline_Test_Jenkins_test-4JTFYMX7KSJY6ZH44VINNGEB7WH2D2HWYZN5ABF6O32O2HBQJYXQ@2@tmp:rw,z -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** --entrypoint cat node
[Pipeline] {
[Pipeline] sh
[Pipeline_Test_Jenkins_test-4JTFYMX7KSJY6ZH44VINNGEB7WH2D2HWYZN5ABF6O32O2HBQJYXQ@2] Running shell script
+ cd /path/to/package.json
+ yarn install
yarn install v0.24.6
error An unexpected error occurred: "EACCES: permission denied, mkdir '/.config'".
info If you think this is a bug, please open a bug report with the information provided in "/var/lib/jenkins/workspace/Pipeline_Test_Jenkins_test-4JTFYMX7KSJY6ZH44VINNGEB7WH2D2HWYZN5ABF6O32O2HBQJYXQ@2/<path>/yarn-error.log".
info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.
[Pipeline] }
$ docker stop --time=1 c1147934ea689f71a449e486282db03338b12182368def31bdf8e8cf179ab46a
$ docker rm -f c1147934ea689f71a449e486282db03338b12182368def31bdf8e8cf179ab46a
[Pipeline] // withDockerContainer
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
ERROR: script returned exit code 1
Finished: FAILURE

I had the same issue with node. The thing is files in the container are owned by "root:root". Try adding docker args -u root:root:

docker {
    image 'node:8'
    args '-u root:root'
}


How to change the owner of VOLUME directory in Dockerfile? I've got the following Dockerfile:

FROM ubuntu:xenial
RUN useradd -d /home/ubuntu -ms /bin/bash -g root -G sudo -p ubuntu ubuntu
WORKDIR /home/ubuntu
USER ubuntu
VOLUME /opt/myvolume

Which I built it:

$ docker build -t vol-test .
Sending build context to Docker daemon  2.048kB
Step 1/5 : FROM ubuntu:xenial
 ---> 0b1edfbffd27
Step 2/5 : RUN useradd -d /home/ubuntu -ms /bin/bash -g root -G sudo -p ubuntu ubuntu
 ---> Using cache
 ---> d82e3ecc5fe8
Step 3/5 : WORKDIR /home/ubuntu
 ---> Using cache
 ---> ab1db29ee8bf
Step 4/5 : USER ubuntu
 ---> Using cache
 ---> 129393a35d9e
Step 5/5 : VOLUME /opt/myvolume
 ---> Running in 691a4cbd077e
Removing intermediate container 691a4cbd077e
 ---> 11bc9e9db9d3
Successfully built 11bc9e9db9d3
Successfully tagged vol-test:latest

However, when run, the /opt/myvolume directory is owned by root, not ubuntu:

$ docker run vol-test id
uid=1000(ubuntu) gid=0(root) groups=0(root),27(sudo)
$ docker run vol-test find /opt/myvolume -ls
    66659      4 drwxr-xr-x   2 root     root         4096 Jul 18 23:02 /opt/myvolume
$ docker run -u ubuntu vol-test find /opt/myvolume -ls
    66940      4 drwxr-xr-x   2 root     root         4096 Jul 18 23:12 /opt/myvolume

because it's created during the run.

Is it possible to define or change the default owner of VOLUME directory in Dockerfile?

I'm running it on macOS and Linux.

As stated in the documentation, VOLUME instruction inherits the directory content and permissions existing in the container, so you can workaround the problem with a dockerfile like this:

FROM ubuntu:xenial
RUN useradd -d /home/ubuntu -ms /bin/bash -g root -G sudo -p ubuntu ubuntu
RUN mkdir /opt/myvolume  && chown ubuntu /opt/myvolume
WORKDIR /home/ubuntu
VOLUME /opt/myvolume

The creation of the directory has to be done as root (to be able to write within /opt).


Jenkins Helm Chart - kubernetes-plugin pulling image from a private gcr I am failing to pull images from a private gcr repo through helm jenkins on a private k8s repo. I am getting:

rpc error: code = Unknown desc = Error response from daemon: unauthorized: You don't have the needed permissions to perform this operation, and you may have invalid credentials. To authenticate your request, follow the steps in: https://cloud.google.com/container-registry/docs/advanced-authentication.

Is there anyway I can mitigate this error without specifically specifying credentials on pipeline level?
Pulling image from host work and creating a deployment in K8s is pulling successfully, but when pulled within Jenkins it doesn't work.

Found the issue, had:
kubectl -n jenkins patch serviceaccount jenkins -p '{"imagePullSecrets": [{"name": "gcr-json-key"}]}'

... applied on default serviceaccount instead of jenkins.


AWS CLI --profile with duration > 1h With the AWS CLI you can assume a role using the --profile parameter:

aws --profile <profile_name> <service> <command>

e.g.:

aws --profile admin s3 ls s3://mybucket/

That profile can be an AWS role that os to be assumed. My problem with that is that the session of the role is only valid for one hour by default. Is there a way of explicitly setting the session duration without using the sts service directly?

You can set a maximum duration of up to 12 hours in the IAM Role config:

Hope that helps.


per-merge request GitLab CI pipeline customization We have a gitlab CI pipeline for building&testing our product. Our product can be, fairly clearly, divided into two distinct parts - let's say we call them A and B. The most time-consuming part of our pipeline are end-to-end (e2e) tests that take > 40 minutes to execute, and we want to cut this time down. They are currently executed in 1 job.

Because our product consists of A and B, for each merge request we want to run only a subset of our e2e tests (only tests for A or only tests for B or all) - we want to be able to specify this manually for each merge request.

We were looking for some idiomatic way to do this in GitLab CI, but didn't find one. What is the cleanest, least obtrusive way to achieve this? We ofc also want to be able to run all e2e tests - this would be the case of master branch.

Our ideas included encoding this information in commit messages or merge request title and extracting this from env variables - https://docs.gitlab.com/ee/ci/variables/predefined_variables.html but both seem like a hack.

Thank you :)

I'd go with what gitlab already offers, the only/except keyword.

Something like:

test:e2e:A:
  script:
    - whatever you do to test A part
  only:
   refs:
      - master
      - /^A-.*/

test:e2e:B:
  script:
    - whatever you do to test B part
  only:
   refs:
      - master
      - /^B-.*/

The only requirement is that work on A or B is done in branches prefixed with A- or B-.

the refs keyword accept regexes to match, more details on gitlab-ci docs here

If your gitlab is pretty recent (>12.3) the rules method would be prefered and the example above for A would become:

test:e2e:A:
  script:
    - whatever you do to test A part
  rules:
    - if: $CI_COMMIT_REF_NAME =~ /^A-.*$/
    when: always
    - if: $CI_COMMIT_REF_NAME =~ /^master$/
    when: always

Or with a single regex:

test:e2e:A:
  script:
    - whatever you do to test A part
  rules:
    - if: $CI_COMMIT_REF_NAME =~ /^(A-.*|master)$/
    when: always

I didn't test those entries above, let me know if one fail


Bitbucket Pipeline & Jenkins & Yocto I want to setup a CD system for a yocto project which is version controlled in a bitbucket repository. I want to write a bitbucket pipeline which triggers a jenkins job on a build server which starts a yocto build. The reason for this is that yocto builds can take up anywhere to 6h for complete rebuilds and thats pretty costintensive with bitbucket pipelines. My idea to test this setup was to use docker containers so I can test the whole setup locally with ngrok. There are some questions I have regarding this:

does the containerized setup for this make sense?
can I mark a bitbucket pipeline build from the outside as in progress/completed/failed?
what is the nicest way to trigger a jenkins job from bitbucket pipelines? Webhooks?
how can I make jenkins differentiate between two branches when triggered by bitbucket pipelines?

The setup I am pondering with atm are two docker containers. One is running the jenkins server and the other is the build server with a custom image, which I have verified working. I found that you can instrument multiple containers using docker-compose so I came up with this docker-compose.yml:

version: '3'
services:
    yocto-server:
        build: .
        container_name: yocto-server
        networks:
            - build-network

    jenkins-master:
        image: jenkins/jenkins
        privileged: true
        working_dir: /home/jenkins
        depends_on:
            - yocto-server
        ports:
            - 8080:8080
        networks:
            - build-network
        links:
            - yocto-server
networks:
    build-network:
        driver: bridge

Can jenkins invoke a command via ssh/telnet/docker magic to the second container with this setup?
Because as I see it I have to do the following steps:

mark the build in bbp as in progress
if repository is present in the container pull and checkout a specific branch
if repository is not pressent in the container clone it first
run the build with parameters provided by jenkins
depending on the exit code of the build flag the bbp build as completed/successfull
upload the artifacts to bitbucket

I am very thankful for any inside on this because I have very little knowledge about docker-compose and jenkins.

For clarification I am using Bitbucket Cloud.

I think you're asking too many questions in one post for the StackExchange QA format, but I can address the Bitbucket-related questions.

For automatically setting up jobs for each branch (and PR) in a Bitbucket project or repository, use the official Bitbucket Branch Source plugin.  This will also take care of the build status aspect of your question.
If you are using Bitbucket Cloud, the Bitbucket Branch Source plugin will also take care of creating webhooks in order to automatically trigger builds.
If you are using Bitbucket Server, you will need some other way of triggering builds.  I personally use the pull request notifier plugin for Bitbucket since our workflow is PR-based, but you can use whatever plugin or tool works for you - there are a ton of them out there.


How can successfully pre-verified changes cause regressions that should have been caught? In a CI context one of the commonly-used measures of increasing the quality levels of the integration branch is a mandatory set of pre-commit quality verifications (typically including building some artifacts, performing unit tests and even some feature/integration tests).

Yet some regressions (build breakages, various test failures) are detected by the CI system verifications in exactly the areas which were supposed to be covered by these mandatory pre-commit verifications.

During analysis of these regressions an argument often heard is that the developer who committed the change identified as root-cause of the regression has successfully passed all such verifications. And often the claim is supported by hard evidence indicating that:

after the final version of the change was reached it was ported to a fresh workspace based on the tip of the branch
the required artifacts were built from scratch (so the build was totally fine, no cache-related issues, etc.)
all mandatory tests passed, including those covering the area in question and should have detected the regression
no intermittent false-positives affected the respective verifications
no file merges were detected when committing the change to the branch
none of the files being modified was touched by any other change committed in the branch since the fresh workspace was pulled

Is it really possible for a software change to cause such a regression despite correctly following all the prescribed processes and practices? How?

There's one possibility I can think of, if when the dev are working on their own workstation, with sometimes images baked for virtual box to run on their workstation where your infrastructure doesn't use the exact same image.

The dev will need, while developing a feature, need to add a JVM parameter or whatever change to the middleware early in its work and forget it.

Before commiting, all unit/integration tests run on its workstation works great, as the baked image is shared, it works on every develloper system.

But when going through CI, it fails because the change to the middle-ware wasn't implemented, either because the dev forgot to ask for it, or just because the team in charge of updating the base images/provisioning system didn't had the time or did forget to update the system.

That's a good thing it break in CI, because it tells early before going into production that the system won't work as expected, but sometimes it becomes a hell to find the missing parameter.

This last point advocate to avoid rejecting commits, and just break on CI on a feature branch, thus it won't block anyone else, and let the dev fix the problem early, when the change is needed and prevent this change to be forget in the flow.

FWIW, we did exactly this here, developers had whole access to development machines and releases in Q/A were failing because a parameter change has been forget, we did move to chef to handle the configuration of the middleware (tomcat now) so each needed change to the infrastructure has to be coded somewhere and will be reproduced in all environment.


Monitoring checklist - What things should I be monitoring? We are building a (Zabbix-based) monitoring system for our applications; hovewer, I'm having difficulties in defining what to monitor?

I have so far come up with the following general categories:

hardware data: cpu, ram, swap, etc.
middleware data: perfomance/health for MySQL instantces, Tomcat instances, JVMs, etc.
logical or application data: the current status/health of the system, e.g. number of active users, page request, etc.
kpi data: data for business, e.g. user registration over time.
dashboard: quick overview of the system (e.g. microservices are running or not).

Are there any other fundamental categories for to monitor? Or is there another category system to use?

UPDATE: the purpose of the monitoring is

the see if the system functions correctly (at high-level, e.g. no services are down, etc. - much like a smoke-test)
see, if there are any indicators, that the system is likely to crash (e.g. historical data predicts that we will run out of disk space)
if any of these occur, send a warning to the appropriate staff (e.g. via e-mail)

UPDATE: the complexity of our system does not demand an extra application for reporting (e.g. monitoring KPIs); also, we are running in local/local cloud infrastructure, so the cost of the application is not (that)relevant - but it might be someday :-)

I like this video: GOTO 2016 • Monitoring Microservices • Tom Wilkie

One of the key ideas (for me at least) is to realize the difference between host monitoring and application monitoring. Basically host monitoring tells you that something is fatally wrong now, but application monitoring should be able to predict problems by detecting higher error rate or that requests are taking longer time so you can fix problems before your users notice them.

(I'm not affiliated with weaveworks or the goto conference in any way, I just like the content and think there are some interesting ideas. Use the downvote button to let me know that this answer is not good :) )


How to provide a user to access only a particular bucket in AWS S3? I have a list of buckets in AWS S3. I have created an IAM user. I have an option to provide S3 full or read only access for a user using groups. Is there any options to provide access only to a particular bucket?

Amazon's IAM roles generally grant a role access to a particular ARN (Amazon Resource Name). Amazon notes on their pages that for S3 a resource

  ...can be a bucket-name or a bucket-name/object-key.

They also provide a helpful example for doing just this which appears as follows:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["s3:ListBucket"],
      "Resource": ["arn:aws:s3:::test"]
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3:PutObject",
        "s3:GetObject",
        "s3:DeleteObject"
      ],
      "Resource": ["arn:aws:s3:::test/*"]
    }
  ]
}


Kubernetes image concept compared to Docker? Reading the K8S images article, I see there support for images from Azure, Google Cloud Registry and Amazon EC.

But, why is there no support for Docker Hub?
Is there any sane way to process/deploy Dockerfiles/Docker images in K8S ecosystem?

Given you can run K8S containers from your Docker images - are there any backdrawals? and if yes, what is the "Dockerfile" in the K8S world?

Side note. My overall impression is that Docker is very nice to adapt in development teams because of its easiness, but in terms of high-end scalability K8S is more powerful. So is this again a new gap between Dev and Ops?..

k8s is a container orchestration tool
a dockerfile is used to build an image
k8s is able to run docker containers

One could use yaml files to deploy containers:

apiVersion: apps/v1beta1 # for versions before 1.6.0 use extensions/v1beta1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  replicas: 3
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.7.9
        ports:
        - containerPort: 80


Jenkins transferring file using publish over SSH plugin but file not show in remote development server I have a running a jenkins job by jenkins pipeline. i wants to send an environmental (.env) file from my local system to my server using Publish Over SSH Plugin.

Here is my jenkins snippet:

stage('Pull In Development') {
        when { branch 'development' }
        steps {
            script {
                sshPublisher(publishers: [
                    sshPublisherDesc(configName: 'developmment-server',
                        transfers: [
                            sshTransfer(
                                cleanRemote: false,
                                excludes: '',
                                execCommand: '',
                                execTimeout: 120000,
                                flatten: false,
                                makeEmptyDirs: false,
                                noDefaultExcludes: false,
                                patternSeparator: '[, ]+',
                                remoteDirectory: '/var/www/sockets',
                                remoteDirectorySDF: false,
                                removePrefix: '',
                                sourceFiles: '**/*.env'
                            )
                        ],
                        usePromotionTimestamp: false,
                        useWorkspaceInPromotion: false,
                        verbose: false
                    )]
                )
            }
        }
    }

Then after transferring, it shows 1 file transferred.

SSH: Connecting from host [pasha-pc]
SSH: Connecting with configuration [stage-server-v1] ...
SSH: Disconnecting configuration [stage-server-v1] ...
SSH: Transferred 1 file(s)

But when i enter the server and go to /var/www/sockets no .env file here. SO am i make any mistake on ssh code.

NB: .env on git ignored file

Please try with double // before var in remoteDirectory: '/var/www/sockets'.


Gitlab runner fails with connection timeout I have gitlab runner with, docker as executor and docker:latest as default image.
Runner is connected to the project and runs pipeline, but it stuck at

Reinitialized existing Git repository in /builds/sachith/omy-project/.git/
fatal: unable to access 'https://gitlab.myhost.com:1234/sachith/omy-project.git/': Failed to connect to gitlab.myhost.com port 1234: Operation timed out

I logged to gitlab-runner running server and tried cloning project as gitlab-runner user and it worked.
Why am I getting this error?

This is likely because your requests were dropped before they landed on the GitLab host. Check your firewalls and other security rules after assured that the 1234 port is open


Istio Keeps On Showing TcpProxyValidationError Errors I initially created an EnvoyFilter to apply idle_timeout of 5s to outbound requests originating from workloads with label app: mecha-dev.
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: tcp-idle-timeout
spec:
  workloadSelector:
    labels:
      app: mecha-dev
  filters:
    - listenerMatch:
        listenerType: SIDECAR_OUTBOUND
        listenerProtocol: TCP
      filterName: envoy.tcp_proxy
      filterType: NETWORK
      filterConfig:
        idle_timeout: 5s

However, the filter did not work for me and I found out it was because my cluster is using istio v1.2.5 which is a very old version and does not support EnvoyFilters properly and the envoy API for the same have been deprecated.
Therefore, I deleted the EnvoyFilter using kubectl.
But, even after deletion of the filter the istio-proxy sidecar containers keep on showing me the following error again and again:
[2021-05-11 11:59:14.109][22][warning][config] [external/envoy/source/common/config/grpc_mux_subscription_impl.cc:73] gRPC config for type.googleapis.com/envoy.api.v2.Listener rejected: Error adding/updating listener(s) 10.155.0.113_443: Proto constraint validation failed (TcpProxyValidationError.StatPrefix: ["value length must be at least " '\x01' " bytes"]): idle_timeout {
  seconds: 5
}

I deleted the pods and re-created new ones but this error keeps on showing up. Is there anything else I need to clean up or delete to get rid of this error?

Restarting Istio controlplane should fix the issue on hand (pods in istio-system, most likely pilot).
More importantly do not run Istio 1.2 its very outdated version with many bugs and security issues.


KeyCloak, PrivacyIdea and Email Gateway I am doing a Micro-services application hosted in AWS to be accessed via Amazon API Gateway by end-users from several organisations worldwide. I am hoping to use KeyCloak as User Authentication Server (2FA). Keycloak will generate a JWT token for the authenticated user and pass the JWT token to the AWS API Gateway to authorise access to the micro services.

Keycloak will be hosted in an AWS Public VPC subnet. In addition, I will have PrivacyIdea as OTP code generator and dispatcher in another subnet or maybe the same subnet in the same VPC. I will also have an Email Gateway in same or another VPC subnet.

Privacy Idea will generate the OPC codes, probably using HOTP and dispatch them via the Email gateway to the users mailbox. The user's corporate email address will be setup in PrivacyIdea. These same users will be setup up in Keycloak. My idea is that somehow  the PrivacyIdea-generated and emailed OTP codes will be inputed by the users along with their login-ID and password into a Keycloak-generated login form, which will then authenticate the user.

Keycloak as authentication server and the use of Emailed OTP codes is an architectural choice that I have no room to change.

My problem is that Keycloak only directly supports Google Authenticator or FreeOTP mobile SMS or application based OTP; hence my having to introduce PrivacyIdea and an Email Gateway. Can anyone kindly provide me any technical guidance or experience on how the OTP seeding and communication of OTP codes from PrivacyIdea or OTP database sharing can be done.

Thank you.

There is no need to push any OTP values into the keycloak database.
You can use the new privacyidea keycloak provider.

The user logs in with his username/password and the privacyidea-provider will take care, that the users get the email and ask in a 2nd dialog for the OTP.

See:
https://github.com/privacyidea/keycloak-provider


How to deterministic and reproducibly assign increasing port numbers in Ansible? I am new to Ansible, but I have to maintain a set of playbooks, which correspond to services to be setup in a given environment. They need to be assigned a port, certificates, etc. This results in many files with lists of essentially always the same names and an assignment to them.

In many cases I think I can easily reuse the service_name as variable, but when mapping to IPs, ports or other numeric identifiers I have not yet figured out a way to deterministically assign them different numbers in a way that is reproducible, and preferably remains the same even when new services are added. I have considered using a SQLite database to store the services from and to generate the values from their ids, but I have no idea how to integrate that with Ansible.

I assume that assigning increasing port numbers is not something entirely new; it is something a lot of sysadmins to on a day-to-day basis so there has to be some way to do that.

Edit: We directly add the port numbers etc. in group_vars/all.yml like this:

ports:
    service1:1024
    service2:1025
    service3:1026

The inventory is generated automatically, since we create additional jails (BSD) and depends on the roles that will be executed.

Disclaimer: I'm not using Ansible.

What I would do is use a random "predictable" number.
According to Ansible doc you can seed the random number generator:

  As of Ansible version 2.3, it’s also possible to initialize the random
  number generator from a seed. This way, you can create
  random-but-idempotent numbers:

  "{{ 59 |random(seed=inventory_hostname) }} * * * * root
  /script/from/cron"

So in your case for a port number (I assume unpriviledged) I would go for a variable with something like:

port="{{ 32767 |random(start=1024,seed=service_name) }}"

Max at 32767 to avoid clashing with any client initiated port (See Ephemeral port for the reason).


Jenkins Pipeline Docker Agents with official Jenkins Docker image I'm using the official jenkins:lts image to host a Jenkins instance, with the Docker control socket forwarded inside the container to allow Jenkins to start new containers, and now I'm trying to run a pipeline that starts with

pipeline {
    agent {
        docker {
            image 'toolchain:2011'
            args '-v ${WORKSPACE}:${WORKSPACE} -w ${WORKSPACE}'
…

Running this job tries to execute the docker command inside the Jenkins container, which doesn't work because the package isn't installed. The regular Docker plugin works fine, as it uses the control socket directly instead of trying to run external programs.

I suspect I'm not the first person to do this -- how would I set up a Jenkins instance inside a container that can execute a pipeline with a "docker" agent?

You have to extend the jenkins/jenkins:lts to install the Docker client:

FROM jenkins/jenkins:lts

# Switch to root
USER root

ENV DOCKER_VERSION      docker-18.06.3-ce
ARG DOCKER_GID=993 # put the correct docker gid

# Download and install docker client
RUN wget --quiet -O- \
  https://download.docker.com/linux/static/stable/x86_64/${DOCKER_VERSION}.tgz | \
  tar zx --strip-components=1 -C /usr/local/bin docker/docker \
  && groupadd -g ${DOCKER_GID} docker \
  && usermod -aG docker jenkins

# Switch back to jenkins user
USER jenkins

You can now declare pipelines with docker agents.


Artifactory vs Git for Infrastructure Version Control / Configuration Management I'm trying to understand the difference between managing configuration files for the purposes of simplifying automated deployment.

We have 5 environments to build. So far 2 have been built with no versioning at all. Right now we're using Artifactory to "manage" the rpm files. I've been tasked to "version control the other stuff". Right now, other stuff means a couple of properties files.

My question is, if I were to simply "version control the other stuff", I would store the properties files in Github, I'd then have Jenkins "build" then "deploy" it to Artifactory? Isn't it extremely difficult to manage infrastructure versioning without infrastructure as code?

My second question is, since our goals are to simplify automated deployment, shouldn't we be using Ansible (or puppet or chef but I like ansible). We could then store Ansible playbooks (and cloudformation templates, etc.) in Github. In this case, what binaries would there be to version?

This is a simple thumb rule one could follow

Use version control (git, svn, cvs) for the work product created by humans
Use artifact management tool (artifactory, nexus, apache archiva) for the software bundle (artifacts) created by the system thru build or packaging process

HUMAN           ==>            System
GIT/SVN  (build/packaging)     artifactory/archiva


Migrate Rancher host from server to server I have 4 Rancher servers: one for each environment: dev, sandbox, staging & prod.

3 of them are of an older Rancher 1.4 version (sandbox, staging, prod) & are hosted on OVH hosting, the one left (dev) is on thr latest 1.6.14 & AWS.

I need to move 3 hosts from OVH to AWS & attach all of them to the only one aws Rancher server which already has 1 dev host.

If I stop rancher-agent on one of OVH rancher hosts & reconfigure it to point to aws, then it appears there in the console, but the config for stacks & etc is lost, as it’s, probably, stored in mysql of the original rancher server.

If I deploy a host from each of the OVH rancher servers to AWS, then I’m stuck with the same problem, after I switch the new hosts agents to the only aws rancher server the configurations for stacks & etc will be lost.

Please advise on the strategy to merge all that into single aws rancher server + 4 hosts: one for each environment. I’m trying not to recreate all 3 environments on aws manually, to avoid downtime & a lot of manual work.

Thanks!

First of all you need to support HA mode. You will need:

external MySQL DB (e.g. AWS RDS Aurora)
external load balancer (e.g. AWS ELB)
2 additional Rancher Server Nodes (to support quorum of 3)

Once you will be ready with external DB Rancher would ask you to backup & restore his database in it. After that you'll be asked to redeploy Rancher Server with additional params for external DB connection. Then you'll be able to run two more Rancher Server Nodes (e.g. using AWS ELB).

When previous steps have finished you would be able to add more Rancher Agent Nodes in AWS directly from Rancher Hosts tab. After that you can just switch off OVH nodes - payload would be transferred to AWS automagically.

Make sure to backup & restore all databases or data-sensitive instances on OVH hosts.

More information such as HA Requirements etc. can be found in documentation.


Github Actions Last Build API I'm looking for an option similar to Jenkins where I can view the last build, last successful build, last failed build, etc. with a simple api call. The Jenkins example would be:

jenkins_url + /job/${job_name}/lastBuild/

Does Github Actions offer anything comparable to this?

List workflow runs would be the equivalent API endpoint in Github Actions:
GET /repos/:owner/:repo/actions/workflows/:workflow_id/runs

You can use the status query parameter e.g. to filter for successful runs:
GET /repos/:owner/:repo/actions/workflows/:workflow_id/runs?status=success


How to login using serverspec using ssh key? The serverspec doc indicates that it is possible to login using ssh and serverspec:

require 'highline/import'

if ENV['ASK_LOGIN_PASSWORD']
  options[:password] = ask("\nEnter login password: ") { |q| q.echo = false }
else
  options[:password] = ENV['LOGIN_PASSWORD']
end

set :ssh_options, options

How to login using an ssh key?

serverspec is using this Ruby class to ssh to servers

One should set the path to the key into the ~/.ssh/config file:

Host X
    HostName X
    User Y
    PreferredAuthentications publickey
    IdentityFile ~/.ssh/Z

Now rake spec runs the test on the server.


Fastlane Screenshots with Authentication I'm currently using Fastlane to capture screenshots of my application to be used in both the iOS and Android app store. My problem is that Fastlane is only able to capture the loading screen and the username/password authentication screen. Is there a way to pass credentials (possibly test creds with limited access) into Fastlane to allow it to capture other screenshots of my application? If not, what other methods could I use?

Please check in Fastlane docs Launch Arguments under Advanced Snapshot section https://docs.fastlane.tools/getting-started/ios/screenshots/ I think it can help

  Launch Arguments

  You can provide additional arguments to your app on launch. These
  strings will be available in your app (eg. not in the testing target)
  through NSProcessInfo.processInfo().arguments. Alternatively, use
  user-default syntax (-key value) and they will be available as
  key-value pairs in NSUserDefaults.standardUserDefaults().

launch_arguments([
  "-firstName Felix -lastName Krause"
])

name.text = NSUserDefaults.standardUserDefaults().stringForKey("firstName")
// name.text = "Felix"

  snapshot includes -FASTLANE_SNAPSHOT YES, which will set a temporary user default for the key FASTLANE_SNAPSHOT, you may use
  this to detect when the app is run by snapshot.

if NSUserDefaults.standardUserDefaults().boolForKey("FASTLANE_SNAPSHOT")

  {
          // runtime check that we are in snapshot mode
      }

  Specify multiple argument strings and snapshot will generate
  screenshots for each combination of arguments, devices, and languages.
  This is useful for comparing the same screenshots with different
  feature flags, dynamic text sizes, and different data sets.

# Snapfile for A/B Test Comparison
launch_arguments([
  "-secretFeatureEnabled YES",
  "-secretFeatureEnabled NO"
])


Unable to use the class defined in src in vars of jenkins shared library Here is the folder structure in my shared library repo:

$ tree src vars
src
├── org
    └── common.groovy

vars
├── testSrc.groovy

src has:

$ cat src/org/common.groovy
package org.common

class CommonFuncs {
    def sayHi() {
        echo "Hi from CommonFuncs!"
    }

    def sayHello(str) {
        echo "Hello ${str}"
    }
}

and vars:

$ cat vars/testSrc.groovy
def call() {
    def tst = new org.common.CommonFuncs()
    return tst.sayHi()
}

pipeline script in Jenkins has:

@Library('jenkins-library') _
pipeline {
    agent { label 'my_servers' }

    stages {
        stage('test') {
            steps {
                testSrc()
            }
        }
    }
}

When I run job, I get below error:

org.jenkinsci.plugins.workflow.cps.CpsCompilationErrorsException: startup failed:
/var/jenkins_home/jobs/Test/jobs/mave/jobs/test_src/builds/8/libs/jenkins-library/vars/testSrc.groovy: 2: unable to resolve class org.common.CommonFuncs
 @ line 2, column 15.
       def tst = new org.common.CommonFuncs()
                 ^

What is wrong in here? I am new to groovy, and this is my first attempt to create a class in shared library src/, any help is appreciated.

It looks like Groovy can't find your class at the path specified.

Try renaming the file common.groovy to match the class name and move it to the common folder:

src/org/common/CommonFuncs.groovy