If we are using Kubernetes do we need to implement HA on both the infrastructure and application level? Is this something that's going to be covered at the application level with pod autoscaling? Or should we implement both horizontal pod autoscaling and infrastructure high availability within our cloud provider? The goal is for us to see minimal downtime.

Your application requirements really do influence your options here, maybe add more detail about the app because this subject is very much about your application architecture too.

The application state is fixed, pod's are rebuilt as per the spec or source container that defines them and the replication sets take care of the deployment so I'd say the application from a Kubernetes perspective is already highly available by design and the reason why the documentation contains little coverage if any with regard to application level HA.

It's worth noting too that Kubernetes High Availability with kubeadm is still in its experimental stage and your clusters need to be running Kubernetes version 1.12 or later.

Google's Kubernetes Engine for instance; offers 'Regional Clusters' and these are spread over many zones in a region. This certainly increases your cluster availability. More here in GKE regions

The official word on the current state of Kubernetes HA cluster options are well documented here. https://kubernetes.io/docs/setup/independent/ha-topology/


What do we know about how open source software becomes eligible for the Docker Hub library? Official community distros of well established open source software can be found in Docker Hub at the "special" place, the library, explicitely tagged as "official".

https://hub.docker.com/u/library/

How does it get there - by decision of Docker Inc., the company behind Docker, or is there some community application process?

For example, there is no such image for LaTeX.

Answers count referencing facts (no opinions).

From https://docs.docker.com/docker-hub/official_repos/

  Docker, Inc. sponsors a dedicated team that is responsible for reviewing and publishing all content in the official repositories. This team works in collaboration with upstream software maintainers, security experts, and the broader Docker community. While it is preferable to have upstream software authors maintaining their corresponding Official Repositories, this is not a strict requirement.


Docker Persisting Data During Build I am trying to create a custom container which also contains MongoDB (hence not using the official MongoDB container). I am trying to add authentication to the database, and I think I am having an issue in persisting authentication data, i.e. in dockerfile I run the following command at some point:

RUN (nohup mongod &) && mongo < /data/mongo.js

and mongo.js is:

use admin
db.createUser(
  {
    user: "admin",
    pwd: "abcd",
    roles: [ "root" ]
  }
)

which yields below during build:

Successfully added user: { "user" : "admin", "roles" : [ "root" ] }
Removing intermediate container...

So after build, when I login to the container and start mongo with --auth flag, still can't login:

# mongo -u admin -p abcd --authenticationDatabase "admin"
MongoDB shell version: 3.2.11
Enter password:
connecting to: abcd
exception: login failed

What is missing?

Instead of writing your own Dockerfile, you can simply pass auth through environment variables to official images, I use compose file something like this:

version: '3.1'

services:
  mongo:
    image: mongo
    restart: always
    container_name: mongo
    environment:
      MONGO_INITDB_ROOT_USERNAME: root
      MONGO_INITDB_ROOT_PASSWORD: abcd
    ports:
      - "27017:27017"
    volumes:
      - ./databases:/data/db

  mongo-express:
    image: mongo-express
    restart: always
    container_name: mongo-express
    ports:
      - 8081:8081
    environment:
      ME_CONFIG_MONGODB_ADMINUSERNAME: root
      ME_CONFIG_MONGODB_ADMINPASSWORD: abcd

you can have your database persisted in ./databases folder and nice in browser ui at port :8081


Cant ssh into running container containing portainer through docker exec I cant connect to my running container with portainer.
I tried already executing docker exec -it <CONTAINER-ID> /bin/bash. Also changed the shell path many times (/bin/sh, /usr/bin/sh, bash). Every time I get message:

OCI runtime exec failed: exec failed: container_linux.go:344: starting container process caused "exec: "/bin/bash": stat /bin/bash: no such file or directory": unknown

Container was created with command:
docker run -d -p 8000:8000 -p 9000:9000 --name=portainer --restart=always -v /var/run/docker.sock:/var/run/docker.sock -v portainer_data:/data portainer/portainer-ce

Can you tell me why I can't connect to this container only? Portainer web interface is working fine... I thought each container had access to shell, and basic tools such as ls

Containers may or may not include a shell, depending on how the container was built. Looking at docker hub it looks like the base image may be based on a "scratch" image, meaning that there may not be anything in the container other than the code needed to run the portainer service. The image layers here also suggest this is the case.
If you want a shell, you may want to use the portainer/portainer-ca:alpine tag. That should give you a container with a full alpine os (and the shell you are looking for). As another note, you may want to try using /bin/sh instead as not all images provide a bash shell.


Understand GitLab CI/CD better Did I understand this correctly, that stages will run in parallel if they have the same name?

stages:
  - build

rust-latest:
  stage: build
  image: rust:latest
  script:
    - apt-get update && apt-get -y install libgccjit-6-dev
    - cargo build --verbose
    - cargo test --verbose -- --nocapture

rust-nightly:
  stage: build
  image: rustlang/rust:nightly
  script:
    - apt-get update && apt-get -y install libgccjit-6-dev
    - cargo build --verbose
    - cargo test --verbose
  allow_failure: true

And that the stages should be the same if they are independent? Will this make deployments quicker compared to:

stages:
  - build
  - build2

rust-latest:
  stage: build
  image: rust:latest
  script:
    - apt-get update && apt-get -y install libgccjit-6-dev
    - cargo build --verbose
    - cargo test --verbose -- --nocapture

rust-nightly:
  stage: build2
  image: rustlang/rust:nightly
  script:
    - apt-get update && apt-get -y install libgccjit-6-dev
    - cargo build --verbose
    - cargo test --verbose
  allow_failure: true

No, jobs (which rust-latest and rust-nightly are examples of) will run in parallel if they are in the same stage (of which you only have one, build).


Unable to understand the logic behind this terraform function So i was going through a code of some one else and just could not make sense of it,, if may be some can of this,

"${replace(jsonencode(list(local.cadvisor_container_definition)), "/\"(\\d{2,5}|true|false)\"/", "$1")}"

locals {
  cadvisor_container_definition = {
    name         = "${local.cadvisor_container}"
    image        = "google/cadvisor:latest"
    portMappings = [
      {
        containerPort = "8080",
        hostPort = "8080"
      }
    ]
    mountPoints = [
      {
        sourceVolume  = "root",
        containerPath = "/rootfs",
        readOnly      = "true"
      },
      {
        sourceVolume  = "var_run",
        containerPath = "/var/run",
        readOnly      = "false"
      },
      {
        sourceVolume  = "sys",
        containerPath = "/sys",
        readOnly      = "true"
      },
      {
        sourceVolume  = "var_lib_docker",
        containerPath = "/var/lib/docker",
        readOnly      = "true"
      },
      {
        sourceVolume  = "cgroup",
        containerPath = "/sys/fs/cgroup",
        readOnly      = "true"
      }
    ]
    memory            = "${lookup(var.cadvisor_memory, var.env)}"
    memoryReservation = "${lookup(var.cadvisor_memory_reservation, var.env)}"
  }
}```

It seems like the intent of this expression is to take the result of encoding that data structure which contains numbers and boolean values in quotes and produce a new version with those values unquoted.

For example, anywhere the result contains the sequence "true" it would be replaced with true, quotes removed to produce a real JSON boolean value rather than a string.

A more direct way to achieve that result would be to set the values in cadvisor_container_definition to be of the intended type immediately:

locals {
  cadvisor_container_definition = {
    name         = local.cadvisor_container
    image        = "google/cadvisor:latest"
    portMappings = [
      {
        containerPort = 8080
        hostPort      = 8080
      },
    ]
    mountPoints = [
      {
        sourceVolume  = "root"
        containerPath = "/rootfs"
        readOnly      = true
      },
      {
        sourceVolume  = "var_run"
        containerPath = "/var/run"
        readOnly      = false
      },
      {
        sourceVolume  = "sys"
        containerPath = "/sys"
        readOnly      = true
      },
      {
        sourceVolume  = "var_lib_docker"
        containerPath = "/var/lib/docker"
        readOnly      = true
      },
      {
        sourceVolume  = "cgroup",
        containerPath = "/sys/fs/cgroup"
        readOnly      = true
      }
    ]
    memory            = var.cadvisor_memory[var.env]
    memoryReservation = var.cadvisor_memory_reservation[var.env]
  }
}

I changed the literal values in the above to their numeric or boolean equivalents. I can't see from what you shared whether memory and memoryReservation are also supposed to be numbers, but if they are then that can be arranged by writing tonumber(var.cadvisor_memory[var.env]) and tonumber(var.cadvisor_memory_reservation[var.env]).

The above examples use Terraform 0.12 syntax. It's possible that the configuration you are looking at here is written for Terraform 0.11 instead, which would explain this unusual string substitution technique because the Terraform 0.11 version of jsonencode suffered from Terraform 0.11's tendency to convert all primitive-typed values to strings. If you are still using Terraform 0.11 then the strange expression you found is likely still needed, but if you are using Terraform 0.12 then a change like I described above should work to simplify the confusing expression to just a straightforward jsonencode:

jsonencode([local.cadvisor_container_definition])


Jenkins Pipeline downstream job; exit unless it hasn't run in the last x hours Is there a way in Groovy directly or by integrating Jenkins plugins to have a downstream job complete only if it hasn't been run in the last X hours already?

Let's say I have unit tests, when green, kick off a job to create an image. I want to put an intermediary step between the two that will check to see when the last iteration of the image creation job ran, and if it was less than say 3 hours ago, NOT run the image creation job.

There isn't a way to make this work with cron that I can see, since the upstream job runs at random whenever commits are made, obfuscating any timing control. I can't just schedule the image creation because sometimes we get 10+ commits in an hour, and sometimes there's barely one a day. When that single one of the day does turn green I'd rather immediately create an image for use downstream, rather than waiting as much as 3 hours if I schedule it.

I'd prefer not to make it a 1:1 ratio commit to image only because storing them gets tricky in large numbers, and downstream from the image you'd never need them as frequently as the aforementioned 10 per hour.

To synthesize Alex and Steve Johnson's answers, you could put this at the top of your downstream job:

lastBuild = currentBuild.getPreviousBuild()
delay = 2

if Date(lastBuild.timestamp) >= new Date().minusHours(delay) {
    currentBuild.result = 'SUCCESS'
    echo "Less than ${delay} hours since the last build. Job will not run."
    return
}


Reaching a Docker environment running in VMware from LAN Here is my setup:
I have a Ubuntu 20.04.2 LTS machine running in VMware (vSphere client).
On this machine, I have a local Drupal website running in Docker using Wodby Docker4Drupal https://github.com/wodby/docker4drupal
The website can be accessed on the Ubuntu machine on http://drupal.localhost:8000.
The Ubuntu machine has an IP of 192.168.10.100 which I can ping from my local Windows machine which is on the same LAN (Windows computer IP address is 192.168.10.97).
Is there any way that I could access the Drupal website directly from the browser on a computer that is on the same LAN (ie. my Windows machine)?
I have tried to access http://192.168.10.100:8000 but I get a 404 page not found message. On port 80 I get ERR_CONNECTION_REFUSED. Any idea?
I shall also mention that on the Ubuntu VM, I also get the same 404 page not found if I try to reach http://0.0.0.0:8000 or http://127.0.0.1:8000 or http://localhost:8000. Only the hostname configured within the project resolves to the Drupal website (http://drupal.localhost:8000).

Here is the docker-compose.yml I am using.
version: "3.7"

services:
  mariadb:
    image: wodby/mariadb:$MARIADB_TAG
    container_name: "${PROJECT_NAME}_mariadb"
    stop_grace_period: 30s
    environment:
      MYSQL_ROOT_PASSWORD: $DB_ROOT_PASSWORD
      MYSQL_DATABASE: $DB_NAME
      MYSQL_USER: $DB_USER
      MYSQL_PASSWORD: $DB_PASSWORD
    volumes:
      - d4d-db-data:/var/lib/mysql # Use bind mount

  php:
    image: wodby/drupal-php:$PHP_TAG
    container_name: "${PROJECT_NAME}_php"
    environment:
      PHP_SENDMAIL_PATH: /usr/sbin/sendmail -t -i -S mailhog:1025
    volumes:
    - ./web:/var/www/html:cached

  crond:
    image: wodby/drupal-php:$PHP_TAG
    container_name: "${PROJECT_NAME}_crond"
    environment:
      CRONTAB: "0 * * * * drush -r /var/www/html/web cron"
    command: sudo -E LD_PRELOAD=/usr/lib/preloadable_libiconv.so crond -f -d 0
    volumes:
    - ./web:/var/www/html:cached

  nginx:
    image: wodby/nginx:$NGINX_TAG
    container_name: "${PROJECT_NAME}_nginx"
    depends_on:
    - php
    environment:
      NGINX_STATIC_OPEN_FILE_CACHE: "off"
      NGINX_ERROR_LOG_LEVEL: debug
      NGINX_BACKEND_HOST: php
      NGINX_SERVER_ROOT: /var/www/html/web
      NGINX_VHOST_PRESET: $NGINX_VHOST_PRESET
    #      NGINX_DRUPAL_FILE_PROXY_URL: http://example.com
    volumes:
    - ./web:/var/www/html:cached
    labels:
    - "traefik.http.routers.${PROJECT_NAME}_nginx.rule=Host(`${PROJECT_BASE_URL}`)"

  mailhog:
    image: mailhog/mailhog
    container_name: "${PROJECT_NAME}_mailhog"
    labels:
    - "traefik.http.services.${PROJECT_NAME}_mailhog.loadbalancer.server.port=8025"
    - "traefik.http.routers.${PROJECT_NAME}_mailhog.rule=Host(`mailhog.${PROJECT_BASE_URL}`)"

  traefik:
    image: traefik:v2.0
    container_name: "${PROJECT_NAME}_traefik"
    command: --api.insecure=true --providers.docker
    ports:
    - '8000:80'
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

volumes:
  d4d-db-data:

Docker container port needs to be be mapped to host port to be reachable in VM!!!


How can I monitor docker volume usage? I'm planning to automate docker activities using python. However, I couldn't find any efficient command for monitoring docker volume. So now I wish to use shell commands for that and later it can be converted using subprocess in python.

My requirement is when I give a volume name then I need the used space, free space, and total space of that volume for finding the storage threshold of the same.

I tried using the docker system df -v But it showing all details.

For example, I give the volume name as my_volume then I need storage details of the same only. Hope everyone understood my requirement and excepting a helping hand from someone.

I'd recommend a temporary container for this. It will work on just about any version of docker, and supports named volumes with non-default settings, e.g. that store data on remote systems (e.g. NFS) or modify the bind mound settings to use another folder.

A simple example is:

$ docker container run --rm -v "${volume_name}:/volume" -w /volume busybox df -P .
Filesystem           1024-blocks    Used Available Capacity Mounted on
/dev/..........              898086280 350169736 502226484  41% /volume

You can parse that output or reformat with the df command as appropriate for your use case. The busybox version of df will have fewer options than you may be used to:

$ df --help
BusyBox v1.31.1 (2019-12-23 19:20:27 UTC) multi-call binary.

Usage: df [-PkmhTai] [-B SIZE] [FILESYSTEM]...

Print filesystem usage statistics

        -P      POSIX output format
        -k      1024-byte blocks (default)
        -m      1M-byte blocks
        -h      Human readable (e.g. 1K 243M 2G)
        -T      Print filesystem type
        -a      Show all filesystems
        -i      Inodes
        -B SIZE Blocksize


How to get the AWS Access key of an already created Access ID Inside IAM management, I navigate to Users - User and got to the Security Credentials tab, It looks like this:

I see no link to show the access key of the already generated access ID, how to get it?

Access Key is that AKIA... one. What you need is a Secret Key - that can't be recovered after the initial setup.

Since you have never used this particular access key (it says "Last used: N/A") it's safe to Make inactive and then Delete and then create a new one through Create access key button above the form.

Hope that helps :)


Prometheus alert CPUThrottlingHigh raised but monitoring does not show it I have installed Prometheus to monitor my installation and it is frequently raising alerts about CPU throttling.

The Prometheus alert rules to identify this alert is :

alert: CPUThrottlingHigh
expr: 100
  * sum by(container_name, pod_name, namespace) (increase(container_cpu_cfs_throttled_periods_total{container_name!=""}[5m]))
  / sum by(container_name, pod_name, namespace) (increase(container_cpu_cfs_periods_total[5m]))
  > 25
for: 15m

If I look at one of the pods identified by this alert, it does not seem to have any reason to throttle :

$ kubectl top pod -n monitoring my-pod
NAME            CPU(cores)   MEMORY(bytes)
my-pod          0m           6Mi

This pod has one container with these resources setup :

Limits:
  cpu:     100m
  memory:  128Mi
Requests:
  cpu:     25m
  memory:  64Mi

And the node that is hosting this pod is not under any heavy cpu use :

$ kubectl -n monitoring top node aks-agentpool-node-1
NAME                       CPU(cores)   CPU%      MEMORY(bytes)   MEMORY%
aks-agentpool-node-1       853m         21%       11668Mi         101%

On grafana, if I look at the chart for this pod, it never goes above 0,000022 of cpu usage

Why is it throttling ?

The CPUThrottlingHigh is an alert created by the kubernetes-mixin project. There is an open issue (#108) to discuss this alert. I suggest that you read all the comments on this issue to better understand the problem.
In short, the problem is: When working with low CPU limits, spiky workloads can have low averages and still be being throttled.
Also, take a look at this issue (#67577) from Kubernetes project, which addresses a Kernel bug in CFS quotas that may cause unnecessary CPU throttling. The discussion is still open, and the Kubernetes project are even considering disabling CFS quotas for pods in the Guaranteed QoS (see #70585 for reference).
Consider the following options:

Increase (or even remove) your container CPU limits
Disable Kubernetes CFS quotas entirely
(kubelet's flag --cpu-cfs-quota=false)
Use a Kernel version that contains this fix (torvalds/linux 512ac99)


How to perform low-level inspecting of a Docker network traffic? I'm currently in a scenario that requires Docker container C to be able to intercept communication between docker containers A and B, without their knowledge. I am basically attempting to do a standard man in the middle attack on my own containers. A and B are bridged right now. I'm using tshark in container C to inspect incoming traffic, but C is not recieving anything, except for initial broadcasting requests.

I've tried different com.docker.network.bridge options but none seem to work well. This is what my network looks like at the moment:

[
    {
        "Name": "PKKJKO6WJEVH936CQ12H",
        "Id": "5033f8958b845153f94beaa4b3bdbe17879ee331ad9441737687eac70b37fbc4",
        "Scope": "local",
        "Driver": "bridge",
        "IPAM": {
            "Driver": "default",
            "Options": {},
            "Config": [
                {
                    "Subnet": "172.18.0.0/16",
                    "Gateway": "172.18.0.1/16"
                }
            ]
        },
        "Containers": {
            "41411a497edf7a632b042d71879d1c0b5941af0361325d2dbe4fefa9bbb30727": {
                "Name": "insane_goldstine",
                "EndpointID": "5c9ffaa6270e8a1823464fa1cdd9f64868083d32cfc7576a5843b1c76944353f",
                "MacAddress": "02:42:ac:12:00:03",
                "IPv4Address": "172.18.0.3/16",
                "IPv6Address": ""
            },
            "849135f83dca0d7261f03952b1648851dfb4c85a0cda0a7d1f5ab5b24e31a014": {
                "Name": "berserk_ramanujan",
                "EndpointID": "68a1ac542dea9fcfa1e35a209f571fc8c670f8d8ec33cf126c4dcde1bca1729d",
                "MacAddress": "02:42:ac:12:00:02",
                "IPv4Address": "172.18.0.2/16",
                "IPv6Address": ""
            },
            "af80dbfaf424fb4b8ce192a8d2e0e946d97ae1abb731bf3fc75918d31c084625": {
                "Name": "clever_goldwasser",
                "EndpointID": "0b2b3a4d24b63168abfbd0d32138ead9f1e053de60a1ac4487b2af87938e7cbd",
                "MacAddress": "02:42:ac:12:00:04",
                "IPv4Address": "172.18.0.4/16",
                "IPv6Address": ""
            }
        },
        "Options": {}
    }
]

What I believe is happenning is, that Docker bridge behaves like a standard networking bridge - it learns which MAC addresses are where and does not send traffic elsewhere. (At least if this page https://en.wikipedia.org/wiki/Bridging_(networking) is correct) Can Docker Bridge be configured to behave more like a networking hub? Possibly by keeping it's MAC memory empty? Or by continually erasing it?

So in conclusion, is there any way I configure docker, or container C, for C to listen to the traffic?

Turns out the answer is iptables, by adding a simple rule:

sudo iptables -I FORWARD -s <source ip> -j TEE --gateway <destination ip>

e.g.:

sudo iptables -I FORWARD -s 172.19.0.3 -j TEE --gateway 172.19.0.4

I can clone the traffic, so if I do this:

sudo iptables -I FORWARD -s <Container1 IP> -j TEE --gateway <Wireshark Container IP>
sudo iptables -I FORWARD -s <Container2 IP> -j TEE --gateway <Wireshark Container IP>

I will clone all traffic going between Containers1 and 2 to Wireshark container.

(Note that although this solution is a proof of concept, at the time of writing this answer)


How to protect secrets whilst enabling the ability to amend a pipeline I'm writing a CI pipeline using GitHub Actions.

The pipeline will build a Docker image, which it will then push to our Docker repository (AWS ECR).

In order to talk to ECR, we'll need to provide a secret (and some other details).

That secret we'll be pulling from Hashicorp Vault... though that itself requires a secret in order to access it, so to some extent we're just offsetting the problem.

The pipeline's code is in the same repository as the code for which it is run (to which our developers have access); though we can hold some actions called by this code in a separate repository (to which only our DevOps team have access) if needed.

Whilst we trust our developers, it's generally good practice to keep things locked down where possible.  As such, is there any way we can set things up such that developers can amend the pipeline without being able to (deliberately or otherwise) expose these secrets?  Are there any best practices around this sort of thing?

Developers or even admins of your GitHub repository wouldn't be able to see secrets configured there by default. Even if you debug GitHub Actions workflows and accidentally log secrets, GitHub would mark them with ***.

So the only attack vector I can think of is highly deliberate and it would require modifying GitHub Actions workflow code and egressing the secret outside to some pre-configured recipient on the Internet.

If you're worried about such attack vector beyond regular code reviews, you may try exploring CODEOWNERS as a way to protect .github directory - https://help.github.com/en/github/creating-cloning-and-archiving-repositories/about-code-owners.


How to run multiple containers with port expose I have 2 images on my docker toolbox, for jenkins and for artifactory.
I can use the command "run" for each to create a running container, jenkins on port 8080 and artifactory on oprt 8081, using the -p flag.
When I use the command "docker ps" I can see both containers.

My problem: I can only open one application at a time, for example:
Open jenkins on URL 192.168.99.100:8080 (docker toolbox use this IP by default), then, open artifactory on URL 192.168.99.100:8081, the page "jFrog Artifactory is getting ready to work..." will be displayed and then the connection to the artifactory will be lost.
But if I stop the jenkins container I could run the artifactory application properly.

I read that docker compose can help to run multiple containers but I didn't understand if it can help my case too.

Any help?

Notes:

My "docker ps" output (plus another command for better display)
I found out about nginx that can be used as reverse proxy but I managed to set only the Jenkins to work, the artifactory does not work. Seems like I am not configuring the .conf file properly. Is it possible to run artifactory via nginx without configuring SSL? Most of the examples I found had SSL in the .conf file.

My problem was that the docker-machine didn't have enough memory to run both services at the same time.

Using those commands fixed it:

docker-machine stop
"C:\Program Files\Oracle\VirtualBox\VBoxManage.exe" modifyvm default --cpus 2
"C:\Program Files\Oracle\VirtualBox\VBoxManage.exe" modifyvm default --memory 4096
docker-machine start

(The path to the VBoxManage.exe maybe be different for other users and it can be change to just "VBoxManage" if adding it to the environment paths)


Maven command for Sonarqube analysis I am planning to build a Maven project in Jenkins. I tried to do some research how to give goals for Maven for doing few steps in one command but cant get understood how to do.

What I mean is how can I give a goal where Maven can build Maven project like creating artifact, skip tests and analyse Sonarqube analysis with particular URL like (sonar:sonar -Dsonar.host.url=http://url)?

I tried but I am unable to give all goals at same time. I am planning to give in these three steps in three different goals. Any way that I can give in single step?

Well, you can pass multiple goals to maven in command line simply by passing them as arguments.
Skipping test require a java system property that you can also pass as an argument to the command line.

For packaging a maven artifact, executing a sonar analysis with the maven goal but skipping executing the tests you should use :

mvn package sonar:sonar -DskipTests

Before that, you need to decide if you want to put sonar settings in the maven configuration (i.e. pom.xml or .m2/settings.xml) or in the Jenkins server configuration.

With maven only, the sonar:sonar goal needs some requirements and maven configuration that you can find here in the documentation. Jenkins use will be reduced to execute maven, you won't need to install a Jenkins plugin.
Using Jenkins with sonar also needs some requirements and Jenkins configuration that you can find here in the documentation. Be sure to install the Sonarqube scanner Jenkins plugin. This plugin enable some Jenkins integration features.


Is it possible to autodeploy on every commit in a specific branch using Travis CI and GitHub? I'd like to deploy files using Travis CI and it only works for a tagged commit. When committing to the branch, there is a warning:

  Skipping a deployment with the releases provider because this is not a tagged commit.

Is there any way to deploy on branched commit using Travis CI?

To clarify, it works when I tag the commit, but I'd like to deploy files on each commit for the given branch.

Instead of using Travis' deployment features, build the deployment logic in to your normal "test" script(s). For a PaaS this might be as simple as pushing to another git remote if all tests pass, but the sky is the limit.


Dynamic service redirections in Docker Imagine you have decided to abstract from exact service location.

Normally, you have a cody world with the service name=DNS.

Say, you have prototyped something with S3 compatible API like exoscale/pithos.

But then you think okay this was actually a mockup and now you go for real S3 in production.

Would you rewrite the stack, trick around with ${ENV_VARIABLES} or is there some more elegant way to proceed?

As per https://12factor.net/config, configuration like this should be stored as environment variables.

  The twelve-factor app stores config in environment variables (often shortened to env vars or env). Env vars are easy to change between deploys without changing any code; unlike config files, there is little chance of them being checked into the code repo accidentally; and unlike custom config files, or other config mechanisms such as Java System Properties, they are a language- and OS-agnostic standard.

A service that you connect to is suitable to be defined as configuration, so the use of env vars is appropriate here.


multiple projects per EC2 with Load Balancers I have 2 EC2 instances and an Application Load Balancer.

The Load is not too high, but I'm keeping both instances working just in case.
Plus, it saves the time on deployment with (one at a time) and still provides "no downtime".

Now I need to add another project.

I'd like to use my EC2 instances and only create a new Load Balancer for it.

Can I do it?

I mean, I can place projects in folders in EC2, but I cannot find a way to set Load Balancer to use a specific folder instead of the root.

My best guess now is to set Nginx to use port 80 for first project and 81 for the second one. But I haven't try it yet. Is there a better way? Are there any hidden problems?

You can archive that, but you will need to use the Application Load Balancer (ALB) which allows you to create rules (like HTTP URL paths, host header conditionals, query strings, HTTP method) to forward the traffic to a specific pool of servers that on your case would be those 2 EC2 instances.

e.g.:

/app1 --> :/app1
/app2 --> :/app2

Notice that you should have the ALB rule like this: /app1/* and /app2/*


how to run multiple environments in single job in Jenkins I have two environments - front-end and back-end. My front-end deployment has  one job and my back-end deployment has another job. My team wants to deploy both front-end and back-end together. Can combine them into single job?

Ideally, they can choose whatever environment and respective branches they required to deploy for their testings in this one job.

A pretty standard way of doing this would be to chain your jobs or your pipelines.  This lets you keep separate jobs (which can be useful), but lets you trigger one job from the other and visualize the end-to-end flow.

So, you still just tell one job to run, but it runs both properly.  Their config and logs are still well separated and you get all the other isolation related benefits.

Example: https://www.jrebel.com/blog/using-jenkins-build-pipelines

You can also do this in normal Jenkins jobs not using a pipeline; there is an option to trigger other jobs near the bottom usually.


Make a job easily searchable in Azure Runbook I have a number of different jobs in my Azure Runbook. However, I don't have an easy way of differentiating between jobs when I look at the Overview of the Runbook. I am forced to click on each individual job to see which one it is. Is there a way I can "label" my job so that it is easily identifiable from the Overview screen?
Thanks!

AFAIK currently there is no way to label job so that it is easily identifiable from the Overview screen. If interested, you may raise feature request in this UserVoice / feedback forum. I see this related feature request already raised so you may even upvote it. In general, Azure feature team would check feasibility of a feature request, triage it, prioritize against existing feature backlog, add in roadmap as appropriate and would announce and/or update the related Azure document once a feature request is addressed.


Gitlab CI\CD Error: Failed to create container volume for /builds/root Unable to load image I've installed the Gitlab local server and also install the Gitlab-runner and everything needs to use the CI/CD tools. After that When I run the CI/CD tools to build and deploy the project, it makes build failed.

Error detail:

Using Docker executor with image alpine:latest ...

ERROR: Failed to
create container volume for /builds/root Unable to load image:
gitlab-runner-prebuilt: "open
/var/lib/gitlab-runner/gitlab-runner-prebuilt.tar.xz: no such file or
directory"

ERROR: Preparation failed: Unable to load image:
gitlab-runner-prebuilt: "open
/var/lib/gitlab-runner/gitlab-runner-prebuilt.tar.xz: no such file or
directory"

What is the problem and how can I resolve it?

Do you have a volume mounted with gitlab-runner-prebuilt.tar.xz? Your error log makes it seem like it can't find the file when it executes.


How to secure persistent user data with docker on client location? I'm working on a system where a dockerized application must run on client side machines (Ubuntu PC), with some sensitive data that must persist between runs that we want to limit access to. In prototyping I just stored the data in the host file system and used a bind mount to map that folder into the application container.

I want to avoid situtations where someone with access to the machine can just copy out some user data, or even look at it.

What would be a good solution in my case?

What I have read from google this sounds to be a bit of a different use case as to where user data/docker volumes are securely hidden away between the (fire)walls of server rooms and cloud instances and I haven't therefore been able to find a good solution on how to limit access to said user data so that no one (no proccess, nothing) except the running container has access to that data, even with physical access to the machine.

I am also very new to DevOps and deploying systems so I would appreciate  any insight on how to solve that matter!

I want to avoid situtations where someone with access to the machine can just copy out some user data, or even look at it.

You cannot stop someone with local access to the machine from reading data on the machine.  Full stop.


Jenkins Dockerfile declarative pipelines: getting the built image tag I'm trying to get the image tag generated with the help of the dockerfile agent in Jenkins declarative pipelines.

This Jenkinsfile is incomplete though, because I still need the deployment steps.

Normally, people would then want a step to push to Dockerhub. Although I don't even know how to do that, I want something different: I want to re-tag the built image locally to a name of my choice. This is because this is a hobby project, and my Jenkins and production servers are the same. The problem is that I don't know how to get the dynamically generated image ID for the docker tag command.

(Then what would come next is to swap out the old running image with the new one, but I could probably take care of that myself.)

Here is my Jenkinsfile:

pipeline {
    agent {
        dockerfile {
            args '--network szeremi -e DB_HOST=mysql -e APP_ENV=testing'
        }
    }

    stages {
        stage('Run tests') {
            steps {
                sh 'cd /var/www && php artisan migrate:fresh && vendor/bin/phpunit'
            }
        }
    }
}

Please help me use Jenkins declarative pipelines to re-tag my built-by-dockerfile image.

As I understand you want to add a variable to the docker image tag,

in this case, you have to get an environment variable from Jenkins and use it in your image label.

    pipeline {
  environment {
    registry = "docker_hub_account/repository_name"
    registryCredential = 'dockerhub'
  }
  agent any
  stages {
    stage('Building image') {
      steps{
        script {
          docker.build registry + ":$BUILD_NUMBER"
        }
      }
    }
  }
}

pleae use this URL for more info


Set a Jenkins job to not to clone the repo in SCM I have integrated Jenkins with Bitbucket using the Bitbucket Plugin. As per the plugin's Wiki, a given job will be triggered if the repository is set in the SCM of the job. As you know, if one set SCM in a Jenkins job, this is cloned in pre-build stage.

So far so good. However, the main purpose of the job I'm setting has nothing to do with the repository's content; instead, I just want the job to process the payload sent by Bitbucket. One could say it's not a big deal in terms of storage to clone a repository despite you really don't need it. I don't think so, adding unnecessary steps, consuming time and resources is not a good practice.

So, the question is: Does anyone know how to set an SCM in a Jenkins job but prevent it to clone the repository?

Yes, definitely. I do this all the time. You can specify configuration options for your pipeline and one of them is skipDefaultCheckout, which causes pipeline to skip the default "Declarative: Checkout SCM" stage.

The skipDefaultCheckout option is documented in Pipeline Syntax and here's an example Jenkinsfile showing how to use it:

pipeline {
  agent { label 'docker' }
  options {
    skipDefaultCheckout true
  }
  stages {
    stage('commit_stage') {
      steps {
        echo 'sweet stuff here'
      }
    }
  }
}


How to persuade developers to start using feature flag toggles? Assuming that feature flag toggles are a good idea, and should be implemented into code that developers write. For example Etsy swear by them as a major part of their culture.

What is a good way to persuade (and enforce) developers to start using feature flag toggles?

More information about feature flag toggles is explained in Q: How to use feature flag toggles, Q: What are feature flag toggles and very extensively in Pete Hodgson's article on the subject on Martin Fowler's blog.

Feature toggles are a common practice in high-velocity development because they de-couple development from release.  Dev teams can "soft-release" a new feature to production, in a disabled state.  This allows the feature to be released any time.  If the feature is dependent on other work or preparation, it doesn't have to wait for a major release to go to production.

As far as "convincing" developers to use them, that's an exercise in making the case for the freedom it offers. My experience is that it's not a tough sell to developers.  It's management that tends to be reluctant to try new things.  Try this:

Find a feature toggle framework.  Management/the business may be more
amenable to trying something supported by a common system
Start small.  Introduce the toggle system on a trial basis for a feature that will demonstrate its utility.
Demonstrate the toggle's ability to do A/B testing.  Enable the toggle for a subset of your web farm, then gather metrics on behavior.  Tiny differences in page layout have been demonstrated to have huge effects on revenue for retail applications (e.g. Ebay, Amazon)


Ansible: Exclude a parameter if variable is not set Is there a way to exclude a parameter entirely if a variable is not set?  I do not want to use a default value for this, I simply do not way the parameter referenced at all.

For example,

---
- name: manage_aix_lvm | creating new VG
  # https://docs.ansible.com/ansible/latest/modules/aix_lvg_module.html#aix-lvg-module
  aix_lvg:
    force: "{{ item.force | default('no') }}"
    pp_size: "{{ item.ppsize }}"
    pvs: "{{ item.disks | join(',') }}"
    vg: "{{ item.vgname }}"
    state: present
  become: true
  loop: "{{ lvm_groups }}"

In the event that variable item.ppsize is not set I want to exclude that entire parameter from being included.  The reason for this is that this is an optional parameter, and if not set, the Operating System will automatically calculate the value.

Is this possible within a single task?  Or will I need to create multiple Tasks/Resources to check if item.ppsize is set and redirect to different tasks/resources?

Thanks.

Q: "In the event that variable item.ppsize is not set I want to exclude that entire parameter from being included."
A: Use the default filter to omit module parameters using the special omit variable
    pp_size: "{{ item.ppsize|default(omit) }}"


Is running docker-compose not supported in BitBucket? Aim

The aim is to run docker-compose in BitBucket pipelines

Attempt

bitbucket-pipelines.yml

image: docker/compose:1.12.0

pipelines:
  default:
    - step:
        script:
          - docker-compose up --build

Outcome

The Docker Engine version is less than the minimum required by Compose.
Your current project requires a Docker Engine of version 1.13.0 or greater.

Discussion

According to this post it is not supported?

Here is my solution: (bitbucket-pipelines.yml)

pipelines:
  default:
    - step:
        script:
          - export DOCKER_COMPOSE_VERSION=1.18.0
          - export DOCKER_COMPOSE_URL=https://github.com/docker/compose/releases/download/${DOCKER_COMPOSE_VERSION}/docker-compose-$(uname -s)-$(uname -m)
          - curl -L $DOCKER_COMPOSE_URL > docker-compose
          - chmod +x docker-compose
          - mv docker-compose /usr/local/bin
          - docker-compose build --force-rm --no-cache --pull


How to identify yourself on a CV So here is your job role:

You help in designing common modules that allow other teams to easily deploy their code onto production. This includes Java code, dockerfiles, ansible modules, build pipeline configurations, etc.
You help promote DevOps practices for both other developer teams and the operation teams
You are partially responsible for the staging environments
You are responsible for investigating technologies that help the applications scale better and be more reliable

How would you identify your team? How would you identify yourself? You know that you could cheat and include DevOps in your name, but that would be wrong. On the other hand you want to be more specific, so a simple Software Engineer won't cut it, especially since you want the annoying recruiters to contact you for jobs related to this practice, not just because you like it, but because you know the average salary is also higher.

Technically mentioning DevOps might be wrong (the question you referenced is also subject to opinions).

But it can also be seen as clarifying/disambiguating - the term is widely used presently on the job market, not only according to your salary reference but also from the job ads themselves - see What is the difference between Sysadmin and DevOps Engineer?.

You'd only speak the recruiters' language - which is IMHO more important from the communication perspective than potentially being technically correct.

Language and culture are fluid - something wrong may become de-facto correct and vice-versa.


Passing variables between scripts in a Jenkins pipeline I have a declarative Jenkins pipeline that looks like the following:

pipeline{
    agent {
        ...
    }
    stages {
        stage("Provision EC2 Instance and Set ENV Variable"){
            steps {
                dir("docker-image-builder"){
                    sh 'python ./provisonInstance.py'
                }
            }
        }
        stage("Configure EC2 Instance with Ansible"){
            steps {
                ansiblePlaybook('./ansible.playbook.yml') {
                    ansibleName('1.9.4')
                }
            }
        }
    }
}

In my first stage, I have a Python script that provisions an Amazon EC2 instance that will be used to create an AMI. In the second stage, I will then configure configure my instance before finally creating an AMI and terminating the instance. In order to do this, I need the instance-id of the EC2 instance created in the first stage. Is there an elegant way to pass values generated by a script between stages, and if so how?

The most obvious method that I can think of is to write the value to a file on the agent in the first stage, and read it in the second stage, but this does not seem like an elegant solution.

For Declarative Pipeline:

This question already has a highly-upvoted answer on StackOverflow:

  If you want to use a file (since a script is the thing generating the value you need):

pipeline {
  agent { label 'docker' }
  stages {
    stage('one') {
      steps {
        sh 'echo hotness > myfile.txt'
        script {
          // trim removes leading and trailing whitespace from the string
          myVar = readFile('myfile.txt').trim()
        }
        echo "${myVar}" // prints 'hotness'
      }
    }
    stage('two') {
      steps {
        echo "${myVar}" // prints 'hotness'
      }
    }
    // this stage is skipped due to the when expression, so nothing is printed
    stage('three') {
      when {
        expression { myVar != 'hotness' }
      }
      steps {
        echo "three: ${myVar}"
      }
    }
  }
}

A second solution would be to write out output from your script directly or with writeFile and then later read it back in with readFile.  However, this comes with the significant caveat that both the reading and writing must take place on the same node, so you can't use this to send data across nodes.

You can also can archive files centrally with archiveArtifact and retrieve them with copyArtifact (and then read in the file with readFile).

Finally, there are some answers to a near-identical question on StackOverflow that offer alternative solutions.

However, if you can, I would suggest using Scripted Pipeline instead, which allows a much simpler solution:

def String myVar

stage('my-first-stage') {
  myVar = sh(script: 'my-command', returnStdout: true)
}

stage('my-second-stage') {
  sh("my-other-command --var='${myVar}'")
}

(I also already answered this question for Scripted Pipeline over on ServerFault.)

It looks like some of these solutions have already been proposed in the comments, but you are concerned that they are not elegant enough.  I'm not really sure what kind of elegance you're looking for, because I can't think of any way to simplify these answers in Jenkins-Pipeline-land.


DC Jenkins server to AWS servers connection We are trying to automate the deployment process for our application, our application has two components (one hosted on DC, one hosted on AWS).
Now, we are not sure how do we connect the jenkins from our DC to the AWS environment hosted servers using VPN or is there any way by which the connection can be established.

P.S: We cannot install a new jenkins in the EC2 instance directly and do the deployment there, as we would need to pull the source code for the build to the cloud layer which is not acceptable by our compliance team.

Depending on your use-case, you should look either:

Establishing a persistent VPN connection to the AWS VPC using the Site-To-Site VPN as described here: https://docs.aws.amazon.com/vpn/latest/s2svpn/VPC_VPN.html
Using an OpenVPN client via the build jobs using the client VPN service as  described here https://docs.aws.amazon.com/vpn/latest/clientvpn-admin/what-is.html


How to handle large deployments using tools like gitlab and docker How to deploy multiple instances (40+) of the same application with gitlab? (it is a simple tomcat web app)

I want to be able to individually upgrade/downgrade them using web interface.

We already have a working gitlab+docker pipeline that deploys one instance of an app. We are not using k8s yet.

Should we create gitlab environment for each instance? Do we have better options besides gitlab environments?

Update:
Instances (websites) do differ, they have the same codebase but different configuration. Right now the scaling is not a priority, what we really want is to have an interface to add, deploy, upgrade and downgrade websites. I've tried gitlab environmets on a smaller project and it's really awesome when you have like one or two websites, but when you have 10 or more and you have one environment for each website .gitlab-ci.yml grows exponentially.

The documentation shows that you can have 40+ branches with different configurations. You express concern that the configuration becomes large. With a good editor that isn't really a problem. Yet that feels wrong, right?

With git, in general, a repository per independent deployable works better than a single repository holding many independent deployables. So an alternative layout is not 1 repo with 40 branches, but 1+40 repos. That would be 1 repo that builds the latest code and creates releases. Then 40 repos that represent the configuration to deploy a particular release of the code configured for one website. The 40 repos that deploy the code can hold the 40 per-website configurations and a standardised build script that simply downloads the right release of the code to deploy along with the website configuration into the environment.

We use 1+N+M git repos. We have 1 repo that holds the deploy scripts. They are versioned and released. We have N repos that build the different apps into releases. Currently, two php apps and two react apps that work together but are deployed and upgraded independently. Then M repos that represent the environments/sites we deploy into which is currently only staging and live.

We are using k8s (OKD) and the deployment scripts are using helm. That just means that our deployment scripts and environment setup are simpler and rollback/downgrade is fully automated. We don't actually download the app to deploy during our build. Instead, we simply push out the docker registry tag of the version of the code we want to deploy into a given environment.

I started a project to opensource our approach to deploying apps from git which is called OCD. It is still a work in progress but it demonstrates some of the above ideas.


Which password database system work with ansible? Which password database system work with ansible to manage password of network devices and servers?

Any system would do. You might need to write a wrapper. For example:

> cat get-passwd-01.yml
- hosts: localhost
  gather_facts: no
  tasks:
    - command: ~/bin/get-passwd user9
      register: password
    - debug:
        msg: "passwd:{{ password.stdout }}"

.

> ansible-playbook get-passwd-01.yml | grep msg
    "msg": "passwd: T4gpJtZ69R5cZD9zQh"


Which libraries generate observability metrics for applications in the cluster to prometheus? I recently read "Cloud Native DevOps with Kubernetes" by O'reily and it mentions the RED and USE metrics which are a good starting point for gaining better observability into the system, however I'm having a hard time finding any information online about how to actually go about generating those metrics outside of re-inventing the wheel for each service.

Is there a set of libraries to actually generate RED and USE metrics within the application?  Or do we have to create a set of utilities for each language and each service independently?

Also, are there known solutions for sending CrashLoopBack or Pending pod metrics to Prometheus or do these also need to be reinvented in house?

For Application specific calls you need to use a Prometheus client and create the metrics yourself. The Prometheus client will allow you to create a /metrics endpoint where you can output the current metric values which Prometheus will scrape.

For metrics that are not unique to your application, you can use Prometheus exporters that will run on either the node, container, or runtime environment, or third-party software. These will also create targets which you can tell Prometheus to scrape to gather the metric data.

Once the metric data is collected you'll need to write your own RED and USE queries in graphana to display those metrics.


How to avoid a regular user can only test in production? When configuring test environments, the issue often comes up where I ask the question to the customer like in my question title here:

  How to avoid a regular user can only test in production?

There are typically the environments like unit test, system test, integration test, volume test, user acceptance test, etc (I bet there are yet another dozen of names like those). But they all relate to environments in which something differs from how things currently work in the real production environments. So none of these is what my question is about.

To make this a less abstract question, consider this sample below:

  Can non-SE employees only test SE sites in production?

I run into this scenario over and over (earlier today it happened again ...):

I discover some facility on an SE site that gets my attention (that I want to start using).
I start looking around in help pages, etc, and either don't find what I'm looking for, or what I find is not 100 accurate or clear to me.
I just try to start to use it, see what happens, hoping I will find the answer to what I couldn't find in the help pages (by just using that facility in some SE site ... read: by testing in production).
I then just wait and see what happens, which may be things like:

I learn about something new/interesting and it works great.
I get accused of trying to "fuzzle the system".
I get corrected by a mod because of something I did (which I shouldn't have done).
I discover a bug, which I either report (on meta.SE), or which I decide to consider as "a bug becomes a feature" (read: I don't want to disclose this handy feature).

However, doing some of those "experiments in production" may damage your reputation (not really your "points", but your real reputation).

So as a good SE citizens, I would be happy to do those experiments in an environment that is designed just for that: a none-production environment designed for users like me who want to do such experiments.

Voilà, if you were asked to configure test environments for SE sites, what would you do (or recommend SE management) about this DevOps issue?

PS: to my knowledge, at this very moment, there is no such SE environment that I can use right now to go check if a user that I "invite right now" for DevOps (while it is in private beta), and which I know did NOT commit to DevOps, will actually be able to login to the DevOps site. So instead of discussing it with mods on some other site, I already ... tested it in production. And despite what some moderator try to make me believe ... it did work (and the invited user now also has an account on DevOps.SE).

A Sandbox could be part of the solution

To bring material to the subject you can check the questions with tag sandbox on MetaSE, there are sanboxes for:

Q/A formatting.
Comments formatting.
Chat.

There's a feature request about "Can we get a Stack Exchange sandbox?" which has no status (neither declined, pending, review nor in-progress meta-tag).

So I assume the answer (to the specific sample in the question) is that there's no way to test out of real SE sites for now and my recommendation to SE would be to accept this feature request.

When is a sandbox appropriate?

I don't think it's always needed to give users a sandbox, and it can be overkill sometimes. Defects should be spotted on Q/A environments, and when they're not, well, either you can know there's a glitch by a fine monitoring of your logs, or you will just never know until someone tells you.

Sanboxes come at an additional cost

Even if you deploy a sandbox for your users to try and confirm a bug, nothing tells you they will inform you of a bug/problem and you had to go through adding a public environment with dummy or anonymised data with no idea of it's ROI.

So It boils down to: "You can create a sandbox for your users but it adds complexity to your infrastructure for a hardly measured return"

Recommendation

My personal opinion would be to put the efforts in log analysis to spot unwanted behaviors instead.


Application calling AWS internal load balancer in same subnet is timing out Some background:

I've created a moderately complex network using Amazon's vpc. It's a three-tiered network across two availability zones. Each layer has a subnet in zone-a and zone-b. The presentation layer is at the top, there’s an application layer in the middle, and a core layer at the bottom.

All of the security groups and ACL's for the subnets are currently allowing ALL inbound and outbound traffic to help me reduce surface area of the issue.

The presentation layer’s routing table is pointing all traffic to an internet gateway. The NAT gateway is in a segregated subnet also pointing all traffic to the internet gateway.

My application has two components, a UI (React.js) and an API (Node/Express). They are deployed as docker images. In front of each is a classic load balancer.

The UI-ELB is internet facing and resides in the presentation layer, routing traffic from 80/443 to port 8080 and is associated with my app-ec2 that is placed in the application-layer subnet.

My API has an internal load balancer in front of it. The API-ELB is in the application layer (in the same subnet as the app-ec2), and takes traffic on port 80/443 and routes it down to the api-ec2 in the core on port 3000.

Both load balancers are offloading the certificate before passing traffic to their instances.

I have both my load balancers associated as alias's in Route53, and referenced in the applications by their pretty url (https://app.website.com). Each load balancer passes the defined healthchecks and reports all ec2 instances in use.

Lastly, on the API I have enabled cors using the cors nodejs package.

Here's a quick and dirty diagram of my network.

The problem:

The APP-ELB successfully routes me to the application. However, when the app tries to send a GET request to the API-ELB, it first sends an OPTIONS request that tims out with the error code 408.

Where it gets weird

Some of the weirdest things I've encountered while debugging are:

I can SSH into the app-ec2 instance and can run a successful curl against the API-ELB. I’ve tried many, and they all work. A few examples are: curl -L https://api.website.com/system/healthcheck and curl -L -X OPTIONS https://api.website.com/system/healthcheck. It always returns the desired information.
I've moved the entire application out of my network into a public default vpc and it works as it's supposed to.
I have the api-ec2 writing all network requests to the console. While it shows the healthcheck requests, it does not show any requests from the app-ec2. This leads me to believe traffic is not even reaching the api.

Really the biggest thing that has me at a complete loss is that curling the internal api elb works, but the axios request to the same exact url does not. This doesn't make sense to me at all.

What I've tried

I originally spent a lot of time playing with ACL rules and security groups thinking I did something wrong. Eventually I just said, "screw it", and opened everything up to try and take that piece out of the equation.

I've spent way to much time playing with Cors on my api. Eventually landing on the configuration I have now, that is the default app.use(cors()) callback provided by the cors node package. I've also included the app.options('*', cors()) that is recommended in the documentation.

I've google everything under the sun, but specifically whether I need to define some special custom headers with the elbs? But can't seem to find anything. Plus, when I moved my app out of the network it worked just fine.

I'm sure I've tried many other things, but these seem to be the most pertinent. What am I missing? I realize this is potentially a very vague and broad issue, and an enormous post, but I appreciate any insight and your time in reading in it!

So what you have actually is this:

As your API ELB is in a private zone it can't be accessed from the internet.
Your frontend in React.js just run in User's browser and not on the UI servers, those server just serve static files.

You have two options, configure your frontend servers to redirect API calls to the API ELB or just update the API ELB to be internet facing.

The usual pitfall of JavaScript apps is forgetting they run inside user's browser and not on the frontend servers as a JEE application would.


How can I use Docker Hub in Terraform? The instructions in Terraform Docker Provider give the example:

# Configure the Docker provider
provider "docker" {
  host = "tcp://127.0.0.1:2376/"
}

# Create a container
resource "docker_container" "foo" {
  image = "${docker_image.ubuntu.latest}"
  name  = "foo"
}

resource "docker_image" "ubuntu" {
  name = "ubuntu:latest"
}

I'm assuming this is pointing to an on-prem docker source. How do I used Docker Hub directly?

This can be done with docker_registry_image:

data "docker_registry_image" "ubuntu" {
  name = "ubuntu:latest"
}

resource "docker_image" "ubuntu" {
  name          = "${data.docker_registry_image.ubuntu.name}"
  pull_triggers = ["${data.docker_registry_image.ubuntu.sha256_digest}"]
}


AWS EKS CronJob resource not found I have an Amazon EKS Kubernetes cluster and have the need to run a CronJob on it.
However, when I published the manifest to k8s, the object is there, I can list it, get it, but I cannot describe it and see its events. Also, it doesn't seem to start at all. According to my cron pattern, it should have started on the first hour.

What I have done so far:

I have created the manifest for the CronJob:

apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: aws-ecr-token
  namespace: default
  labels:
    app: aws-ecr-token
    component: infrastructure
spec:
  schedule: "* */11 * * *"
  jobTemplate:
    metadata:
      name: aws-ecr-token
      namespace: default
      labels:
        app: aws-ecr-token
        component: infrastructure
    spec:
      completions: 1
      backoffLimit: 3
      activeDeadlineSeconds: 60
      parallelism: 1
      template:
        metadata:
          name: aws-ecr-token
          namespace: default
          labels:
            app: aws-ecr-token
            component: infrastructure
        spec:
          containers:
          - name: aws-ecr-token
            image: docker pull mesosphere/aws-cli
            command:
            - "sh"
            - "-c"
            - "$(aws ecr get-login --no-include-email)"
            env:
            - name: AWS_ACCESS_KEY_ID
              valueFrom:
                secretKeyRef:
                  name: aws-ecr
                  key: accessKey
            - name: AWS_SECRET_ACCESS_KEY
              valueFrom:
                secretKeyRef:
                  name: aws-ecr
                  key: secretKey
            - name: AWS_DEFAULT_REGION
              valueFrom:
                secretKeyRef:
                  name: aws-ecr
                  key: defaultRegion
            volumeMounts:
            - name: aws-ecr-config
              mountPath: /root/.docker
          restartPolicy: OnFailure
          volumes:
          - name: aws-ecr-config
            secret:
              secretName: aws-ecr-config

Deployed it using:

$ kubectl apply -f aws-ecr-token-cron-job.yaml

I can get it:

$ kubectl get cronjobs
NAME            SCHEDULE      SUSPEND   ACTIVE    LAST SCHEDULE   AGE
aws-ecr-token   * */11 * * *   False     0         <none>          26m

But it never gets scheduled.

When I try to describe it, it shows:

$ kubectl describe cronjobs
Error from server (NotFound): the server could not find the requested resource

Any ideas why?

My version is:

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.0", GitCommit:"fc32d2f3698e36b93322a3465f63a14e9f0eaead", GitTreeState:"clean", BuildDate:"2018-03-26T16:55:54Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"10+", GitVersion:"v1.10.11-eks", GitCommit:"6bf27214b7e3e1e47dce27dcbd73ee1b27adadd0", GitTreeState:"clean", BuildDate:"2018-12-04T13:33:10Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}

Thanks in advance :)

There was an error in Kubernetes that causes failure when using kubectl describe with the cronjob resource. It was fixed at some point between the v1.10.x and v1.12.x versions (more info in this issue #45418).

You said:

  According to my cron pattern, it should have started on the first
  hour.

But your schedule is * */11 * * *, and it means: “At every minute past every 11th hour.”

I did not understand what you exactly meant with "on the first hour", but if you are talking about running the job on every new hour, the schedule should look like this: 0 * * * *


What are best practices for docker tag versioning? I've recently hooked our CI servers up to build docker images upon git commit.

We have about 8 different containers that get built, each with their own language / frameworks. Some are node and have a package.json, others are python services that contain no semantic version information.

My question is not about how to create tags, its about creating the values for the tag.

How to ensure that each tag has a unique semantic version number for the specific images? Who should be the authority on tracking / incrementing a build version?

I would direct you to my post Coupling docker registry and source control where dmaze answered from the official forums.docker.com.  Commit hash and branch name or tags suffices.

In your Dockerfile, use a LABEL to record the source of the build. That probably includes the commit hash from distributed source control (git, Mercurial), the branch name if relevant, any release tags if present, and possibly details like the timestamp of the last commit. docker history and docker inspect should be able to show these.
When you docker push your images, push them at least twice, with the commit hash and with the branch name as the “version” part (quay.io/mycorp/imagename:123abc7, quay.io/mycorp/imagename:dmaze-test). If release tags are readily available, the CI system should push images with these tags too.

We are currently using a combination of branch name/commit hash.  For us that seems to be enough.  timestamps while they are useful IMO just add clutter as they don't provide anything the commit hash doesn't.
I agree with 030 regarding:

who should be the authority on tracking / incrementing a build version

100% is responsibility of the CI to maintain such things, with proper communication between other teams.


SaltStack and Vagrant adding files I use file.managed to add a file:

/home/vagrant/.bash_profile:
  file.append:
    - name: /home/vagrant/.bash_profile
    - source: salt://config/user/.bash_profile
    - user: vagrant
    - group: vagrant
    - template: jinja
    - mode: 0775
    - replace: true

The file is added; however the file is owned by root/root and does not change to by owned by vagrant.

Using Vagrant 1.8.5.
Salt is configured to be masterless and file client is local.

You aren't actually using file.managed, you are using file.append - so salt is expecting to add onto the end of an existing file. This may be the cause of your issue. Instead try:

/home/vagrant/.bash_profile:
  file.managed:
    - name: /home/vagrant/.bash_profile
    - source: salt://config/user/.bash_profile
    - mode: 0775
    - user: vagrant
    - group: vagrant
    - template: jinja
    - replace: true

Also, be sure you are using the latest version of Salt Stack, 2016.11


Looking for ways to host application in AWS and have it monitored using opensource I am a newbie in the world of DevOps and have been assigned a task to implement web applications for the customers in AWS and that application monitored (security, metrics and logging) using any of the open source tools. I dabbled with Nagios a little and have gone through the AWS course on security and networking before.

Has anyone done a similar kind of project before? Could anyone share their approach so that it will help me prepare a design document?

You can try built-in tools that provide by AWS.
AWS calls them Management Tools.

Amazon Inspector

Automated security assessment service to help improve the security and compliance of applications deployed on AWS.
Amazon Inspector is an automated security assessment service that helps improve the security and compliance of applications deployed on AWS. Amazon Inspector automatically assesses applications for vulnerabilities or deviations from best practices. After performing an assessment, Amazon Inspector produces a detailed list of security findings prioritized by level of severity. These findings can be reviewed directly or as part of detailed assessment reports which are available via the Amazon Inspector console or API.
— AWS Inspector

Amazon Web Services (AWS) Monitoring With Nagios

Implementing effective Amazon Web Services monitoring with Nagios
offers the following benefits:
Increased server, services, and application availability Fast
detection of network outages Fast detection of cloud computing
environment problems
— Nagios

Some other tools:

Scalr

Management Tools by AWS


Is Azure WebDeploy safe for production Web Apps? We use Azure Web Apps in production. We deploy using WebDeploy either by right-clicking the project in Visual Studio and choosing "Publish..." or via CI in AppVeyor, which is WebDeploy ready.

My question is: Is WebDeploy safe for production environments? Will it interrupt running processes, or does it work some magic to make sure it doesn't? Can we publish to production using WebDeploy during business hours (when our customers are active) and not worry about it? A good answer will provide some documentation on this, as I have not been able to find this yet.

WebDeploy by itself cannot guarantee if your application pool will not be recycled during deployment. To be on the safe side, I would use deployment slots.
Also I would recommend to tag question with azure-web-sites so right people will help you.


How to change existing policies in an organisation? I assume that an organization wishing to do a DevOps transformation has some problems and policies it is interested in changing. This interest can come from top managers, middle managers, or even from bottom up. One of the biggest factors impeding this change is making other people buy into doing the change.

For example, in many cases pushing "new" ideas such as Agile, often fails. People resist change, and it seems like a wall that stops good things from happening. Yet there is a mandate for good things to happen.

What methods can be used to affect employees in an organization starting its DevOps transformation? Especially specific techniques and ways that are found to work. Specific as-in more Engineering, less hand waving.

You have to understand that processes change people who follow them. As people learn, internalize and get better at a process, it changes the way they learn how to solve a particular problem. A set of similar processes reinforce each other into a mindset the person uses to solve a category of problems and eventually form a set of values that guides decisions and new solutions to new problems.

Even if you change the process, without the change to the mindset and even more crucial to the values, the person will simply adapt the new process to conform to the same values, same mindset or even same solution as in the original process. At a certain point, it is not possible to divorce this person in this position from the mindset acquired or change the underlying values.

To institute a change, you have the following two options:

Bring in a person who will already have the right values and mindset and in best case scenario, understand the process that needs to be followed without your help.
Bring in and empower new employee, either recently hired, a new hire or a transfer from a different team in the organization and train him in the new process, hoping to instill the new mindset, hoping the new set of values will emerge.

If the change is local, you might prefer an internal transfer as that person would already share the global company wide values which you want to retain. In case of a larger change, you need to bring in someone from outside to have a fresh perspective and to not share the company wide values which you might be trying to change.

The important part is to empower the person, team or business unit to follow the processes and insulate them from the old team, other teams or the rest of the company, respectively, which might still follow the old set of processes. As it is very hard to insulate such change agent from management above, if the change is to be larger, it often needs to follow all the way up the management chain or come all the way from the top of it.

Note: It is hard to bring change to more than just your team without the support of management. Even inside your team it is difficult if others are already set in their ways. For a new team in a new company a successful evangelist can often affect the forming policies even without the support of management simply by being a leader or creating the path of least resistance for others to follow. But in established company, see above.


Deriving an item value from a structured file in Zabbix Zabbix's builtin items for monitoring system parameters are reasonably rich, but there are occasionally things it doesn't seem to make trivial to obtain, such as how many dirty cache pages there are.

For example, consider /proc/meminfo on your typical Linux system:

Inactive(file):  1348288 kB
Unevictable:       18876 kB
Mlocked:           18876 kB
SwapTotal:       8388604 kB
SwapFree:        8388604 kB
Dirty:             39620 kB
Writeback:           948 kB
AnonPages:       6184596 kB
Mapped:          5319216 kB
Shmem:            427020 kB
Slab:             741476 kB
SReclaimable:     633088 kB
SUnreclaim:       108388 kB
KernelStack:       14528 kB
PageTables:       118816 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    16575460 kB
Committed_AS:   19184632 kB
VmallocTotal:   34359738367 kB
VmallocUsed:           0 kB
VmallocChunk:          0 kB
AnonHugePages:         0 kB
ShmemHugePages:        0 kB
ShmemPmdMapped:        0 kB
CmaTotal:              0 kB
CmaFree:               0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:     9692684 kB
DirectMap2M:     7049216 kB

What I want is 39620, from the Dirty line. With a combination of sed and grep, this would be as simple as export field=Dirty; grep -e "$field:" /proc/meminfo  | sed -e "s/^$field: \+\([0-9]\)\+.*$/\1/"; fairly straightforward.

I could do this with a UserParameter:

UserParameter=proc.meminfo[*], grep -e "$1:" /proc/meminfo | sed -e "s/^$1: \+\([0-9]\)\+.*$/\1/"

and then access the value with the item key proc.meminfo[Dirty].

I'd prefer not to use a UserParameter for this, though, as UserParameter require me to deploy additional configuration to each node, and also the process spawns consume system entropy. Both are things I generally prefer to avoid.

If your Zabbix agent is 2.2 or later, you could use Zabbix agent item vfs.file.regexp like so:

vfs.file.regexp[/proc/meminfo,^Dirty.*([0-9]+),,,,\1]

If your Zabbix is 3.4 or later, you can use a master item to collect the data blob (perhaps with vfs.file.contents) and then extract and parse the desired values with item value preprocessing.


Git Release branching strategy for Continuous integration and deployment We have tried multiple branching strategies to adopt CI/CD.

Developers develop on the feature branch and then merging to master which get deployed to QA for testing and then master is deployed to production, so in short whatever goes out to QA/UAT goes out to production unless you put a create a PR/patch to fix the bug - Least hassle of merging, syncing etc.
Developers develop on the feature branch and then merge to master which gets deployed to QA/UAT for testing and based on the testing outcome, spin out a release branch for things which needs to be deployed to Production.

What are the recommendations in case of organizations with 50 odd devs and  spread across ~8 scrum teams?

Remember, we always have feature flags to toggle the releases and features pushed out.

You need to separate the 2 concepts a bit:

the integration part - how are the changes integrated into your master branch (or some other integration branch, that's also possible), which may be continuous or not. Technically neither approach you describe is actually continuous integration unless your feature branches have a very short lifespan - typically less than 1 day. But this part doesn't seem to be the main focus of your post.
the delivery/deployment part - how are the integrated changes from an integration branch (master in your case) being delivered/deployed. In other words your release branch strategy. This appears to be the point of your post, I'll focus on it below.

Both approaches you describe are valid release branch strategies, I've seen both being used. Each with their own advantages and disadvantages.

This approach is inline with the continuous delivery/deployment strategy. The CI/CD pipeline will have the commits to the master branch (either feature branch merges or point fixes) as entry point: each such commit will trigger the pipeline verification, attempting to progress into it as far as possible. The QA/UAT is one of your pipeline steps, it that passes then it goes into production. Effectively your releases are simply selected (typically tagged) commits from the master branch, they aren't really branches.

Advantages:

as you noticed - the least amount of branch activities
everyone is on the same page - developers, QA, etc all focus on the same branch context
lowest cost (but only if the project scale allows it to be effective, see also the disadvantages section)

Disadvantages:

you can't have actively maintained releases - you only have one "current" release out there, each additional fix will become another release. Which isn't necessarily a bad thing, such release model can actually be desired (for example in the case of SaaS products).
the release cadence is not actually predictable. It may be difficult to get releases out - if a problem is detected, by the time the fix for it is merged into the master branch some other changes may be merged in and other problems detected. This gets progressively worse as the rate of branch commits goes higher (larger scale projects) and may actually be non-convergent (the project may not reach stability at all). Doing CI/CD at scale is not a trivial matter.

This approach is a more traditional release strategy, but technically it's not really continuous delivery/deployment.

Advantages:

more predictable release cadence. The spinned-off release (aka throttle) branches allow reducing the rate of commits for a particular release. Typically a release branch would only receive fixes necessary for that particular release, fixes for future releases (or release trains) would go into the master branch instead. The reduced commit rate allows reducing the risks of new problems being introduced that would require additional commits, the branch is more stable. Typically the release branches are pulled from the master/integration branch commits that passed all (or most) of the CI/CD pipeline verification stages (for the master branch). In your particular example those branches wouldn't expect subsequent commits (they passed QA/UAT), but if say a problem is detected after deployment in production it would be possible to just commit that particular hotfix into the branch and re-deploy, without having to pick up any other commit that went into master since the branch was pulled, reducing the risk of additional regressions.
it supports multiple simultaneous/overlapping actively maintained releases, with potentially complex release policies in place (long-term/short term/rolling/etc, like OpenSUSE's Lifetime for example).
it supports multi-level releases (like major.minor.increment, for example) - aka release "trains": a major release branch could itself have multiple minor release children branches, etc.
each release branch could actually use the CI/CD methodology at its own level, independently of the other release branches - if it never gets upstream/bulk merges from its parent (i.e. effectively becoming its own "master")!

Disadvantages:

everyone is no longer on the same page. Each release branch has its own context which may come with its own policies, processes, tools, necessary resources, etc. Each requires its own development/testing/validation. This drives development/maintenance costs up, often significantly.
tracking issues and propagating fixes across multiple releases may not be trivial. Merging back release branches into master (or their parent branches) appears an inviting approach, but IMHO it's a bad practice, see How to get rid of develop branch for simplified Git flow .
context switching becomes a major problem.


Re-initialize etcd-cluster completely (on coreos) Over the past updates (espcially the update to etcd-wrapper: https://github.com/coreos/coreos-overlay/tree/master/app-admin/etcd-wrapper) my etcd-cluster with five machines broke. Now I'm trying to recover from that.

Currently I only use it for the etcd-lock while updating the machines. So there is basically no important information in there (yet).

I tried a lot of stuff and since, in my opinion, the documentation is not that structured, I mixed a lot of infos form different parts together. Currently I'm in a state, where one node wants to elect a leader with another node. But that other node thinks it is part of another cluster. And I cannot do anything on the first one anymore. etcdctl just revokes all actions with the info, that the cluster is not healthy...

So I want to do a fresh start again!

Could someone point me to some part of the documentation, that explains that? It seems, that the disaster recovery part (https://coreos.com/etcd/docs/latest/op-guide/recovery.html), that allegedly contained the info, has changed (and is also depreacted).

Additionally it would be really helpful, if someone could point me to the part of the docu, that explains how to setup a etcd-cluster without re-installing the whole system.

Thanks in advance!

PS: I would have gladly selected some better tags (like coreos, etcd, etcd-cluster), but I am not allowed to create them now :)

I managed to fix it. First I disabled and deleted all etcd2-related units. Then I created a new dropin to the etcd-member.service with the following content:

[Service]
Environment="ETCD_OPTS=--name cluster00 \
  --discovery https://discovery.etcd.io/<MY_ID> \
  --listen-client-urls http://<NODE_IP>:2379,http://127.0.0.1:2379 \
  --listen-peer-urls http://<NODE_IP>:2380 \
  --advertise-client-urls http://<NODE_IP>:2379 \
  --initial-advertise-peer-urls http://<NODE_IP>:2380"

I generated a new discovery id for the cluster.

Then I teared down the old cluster completely, by deleting all content of /var/lib/etcd/* and the directory /var/lib/etcd2 completely and (re-)started the etcd-member service.

Be aware, that I don't have ssl enabled at this point!


Service for hosting native JavaScript ES modules We will be serving static javascript ES modules and looking for an all-in-one hosting platform that is easy to use.

It must provide CORS headers (ES modules enable CORS by default)
HTTP/2 support is mandatory
Ability to set very long caching headers (content never changes)
Mirrored all across the world for fast transfers with low latency.
Pushing a new version should be a simple oneliner.

Have you considered CDNjs.com:

It supports CORS headers, it is dedicated to JavaScript after all.
It supports HTTP/2 as well as SPDY.
The cache duration is 355 days, which is probably plenty long enough.
Cloudflare is used as the CDN, they provide it to cdnjs.org for free.

Adding your library is a bit more than a one-liner unfortunately as you need to submit a pull request to their GitHub repository.  You will also need to have a certain popularity requirement:

  For the new library request issue, please make sure it's not a personal project, we have a basic requirement for the popularity, like 200 stars on GitHub or 800 downloads/month on npm registry.


how to set automatic git pull from bitbucket and complie project on ec2 instance create event with auto scaling I have just started reading about AWS services. I have a project where the backend is created with Spring-Boot, front-end is created with React and PostgreSQL is used as database. Both backend and frontend is different bitbucket repository. I want to use two different EC2 instance for front & back-end and RDS for PostgreSQL. Now I also want to use the autoscaling feature. The problem is how can I set the autoscaling feature in such a fashion where it automatically clone the app from bitbucket and then run a shell script file to compile and run the application.

You want to make use of user_data scripts.
It's basically a bootstrap shell script that gets executed on an instance during launch.
You can embed your app configuration script in there (either directly via bash, or have it call your configuration management software like Ansible or Chef).
Then, if you attach this script to a Launch Template or a Launch Configuration (which is an instance template required as part of an AWS ASG), it will be applied to the whole autoscaling group.
Then, when you have a scaling event, any new instances that come up in the ASG will run the script and deploy your code.
However, I would recommend you look into immutable infrastructure instead.  I.e. set up automation (i.e. via Packer) that builds your application code into your AMI, with user_data only used for actually starting the app.
Then, when you have a new build, have the same automation build a new AMI and replace your ASG (note you'll need to have your ASG created/deployed in a programmatic fashion.. I personally recommend Terraform).


Automate creation of branch + pull-request I have a repository (call it repo-X) of shell.nix files that define development environments for a number of separate projects (call them project-A, project-B).

When one of the projects releases a new version, I run a suite of automated tests in repo-X for the relevant shell.nix file, and if all goes well, I increment the version number in that file to match the new version of the project it supports.

I'd like to automate this process. I'm using Jenkins and GitHub to build PRs, but I'm not sure how I'd use CI to author a feature branch and create a corresponding PR.

I'm having a hard time even finding references/ideas, but maybe I'm searching for the wrong terms. What tooling should I look into for this?

You want a RCS/VCS that handles the version incrementally for you. I'm not sure how this relates to CI as the idea is to Continuously Integrate instead of spawning more branches. I'm not sure what you're using for your PR's, so I'll use Atlassian Bitbucket as an example. In the response to pushing up a new branch you get back the url for creating a PR which has an API which you can call to create your PR.
Git commands will do the branching for you:

# Get the latest description, usually based on tags : https://git-scm.com/book/en/v2/Git-Basics-Tagging
$branchName = git describe
# Checkout the new branch
git checkout -b $branchName
# Push up the branch to origin
git push --set-upstream origin feature/$branchName

It's best to have a firm grasp of git when dealing with VCS.
* https://try.github.io/
* https://git-scm.com/book/en/v2/Getting-Started-Git-Basics


Jenkins Promotions vs Pipelines We've been starting to link build&test&deployment jobs together into sequential tasks (for now as simple as: build -> run tests -> if tests pass -> deploy) and had some experiments with Promoted Builds and Pipelines.

Is our understanding correct that Promoted builds and Pipelines are two different separate strategies to Continuous delivery? Or, can they still be used together?

I just configured promoted builds plugin and the pipelines plugin and found at least one difference between the two plugins, i.e. the first plugin seems to be able to allow manually deploy of certain stages, while the pipelines plugins seems to deploy everything automatically.

According to this answer, manual deploy of a certain stage seems to be missing.

In summary, it seems that it would be nice to combine the two plugins as it could be handy to have a manual check before deploying to production, i.e. a human has to press the button in order to deploy to production.

According to this answer it is not possible to combine both plugins at the moment.


What are the measures of size and complexity of IT infrastructure How can I compare different infrastructure in regards to size and complexity. What can I measure and compare e.g. number of nodes, number of servers, architectures etc.

How do those measures and variables differ in what they compare? Which of them make sense to compare and what is the cardinality at which there is actual difference in the kind of work done, rather than just a bit of extra work.

What will let me decide that "This infrastructure is very large" or "This infrastructure is very complex" and what is the difference?

TL;DR: Measures of size can be split roughly in three different categories that I would define as depth in what you
  manage vs. what you outsource or consume as a service, breadth of
  services supported and height in numbers of instances,servers and
  customers. Measures of complexity largely depend on selected
  systems architecture, organizational structure of people supporting it and skill sets required. Where depth and skill sets
  go hand in hand adding both to size and complexity.

Size mesures

Note: Most of the following size requirements can also add to complexity through systems architecture, skill sets and organization structure needs.

Depth of Infrastructure

How much of the infrastructure you are outsourcing to others in increased levels of depth:

Do you simple use Software as a Service for all that you do?
Do you operate fully in public cloud, private or hybrid cloud or
using some PaaS?
Do you use Infrastructure as a Service?
Do you use Hosted and Managed Infrastructure in rented DC space?

Do you own or rent the hardware?
Does the provider manage infrastructure monitoring?
Does the provider manage basic system administration?
Does the provider manage hardware failures and maintenance?
Does the provider manage racking and server installation?
Does the provider manage internal networking?
Does the provider manage internet connectivity and routing?

Do you have just data centers with contracted remote hands?
Do you host everything on premises or in your own datacenters?

Breadth of Infrastructure

What are the different types of services you support?

Compute Resources

Bare metal servers
Virtualization layer (VMWare)
Container layer (docker, k8, mesos)
Serverless layer (lambda, functions)

Storage Resources

Standalone storage appliances
RAID in servers
Large standalone relational database clusters
Time series databases
Object store clusters

Network Resources
Observability Resources

System Log servers
Metrics and Graph systems
Search clusters

Automation Resources
Backup/Recovery Resources
Complex compounded services

ELK, Hadoop, etc.

Height of Infrastructure

What is the scale for each resources that you need? Do you operate the services on single server/instance or do you need use clusters of machines?
What is the level of redundancy you need?
What are your availability requirements?
What are your requirements for latency and throughput of services?
Do you need geographically distributed infrastructure? (international business, latency requirements or regulation compliance like GDPR, data localization laws, etc)
Do you need multiple datacenters in each geography?

Complexity Measures

Just very briefly...

System Architecture

When it comes to Infrastructure complexity, it quite closely follows the complexity of the distributed systems supported by the infrastructure. You have to take into account two types of systems:

Distributed systems supporting individual services.
Distributed system created by interdependencies of the services.

Distributed Systems Complexity

Every service your infrastructure supports can have in and of itself a different level of complexity with varying levels of requirements on the infrastructure. Systems supporting services can range through:

Single threaded.
Multi-threaded (shared memory, shared disk)
Parallel systems with data sharding
HA Failover (Primary / Standby) (Cold, Warm, Hot)
HA Cluster (N+M)
Real-time Clusters

Interdependence of Services

Let me start with an example. Let's say your infrastructure reports test results into ElasticSearch cluster. Your pager depends on monitoring and test data provided by ElasticSearch. Geographical distribution of ElasticSearch cluster makes it depend on your datacenter networking connectivity. Now one of your internet providers decided to make an unannounced maintenance on Saturday night, throughput drops, your traffic is rerouted to backup provider, monitoring traffic is de-prioritized to customer data traffic, ingest of monitoring events slows and your pager goes crazy.

Every time two services, two parts of the infrastructure depend on each other, they create a new single distributed system, whose complexity should be judged independently. Such dependency can be either removed or reduced. Remember that system is only as redundant and as available as the intersection of all services that it depends on.

Further examples of complexity increasing factors:

Dependencies on external services.
Attempts to mitigate failure from service dependency.

Multiple providers
Caching of data

Organizational structure

This is a chapter in and of itself ... people are often overlooked part of the overall system of IT infrastructure. We rarely think about redundancy, availability and latency factors when it comes to people, but just as with computers, these same issues affect organizations maintaining infrastructure and the complexity of it can sometimes easily outweigh the complexity of the computer systems. People involved in maintaining your infrastructure can span multiple time zones, languages, geographic locations, companies, pay scales and legal codes. Any of those factors are signs of increase in complexity.


Why Docker cant launch container? I have docker-compose file with one image (httpd) and one volume that look into ./ folder (It contains only one index.html static file). When I start my computer I launch this image with

docker-compose up

And it works, I see message

Starting mysiteloc_app_1        ... done

And can access it via localhost.

But when I wait several hours (seriously, I do nothing, for experiment I just wait 3 hours without touching keyboard or mouse) and try to use

docker-compose up

I get message

mysiteloc_redis_1 is up-to-date

But if I restart docker daemon all works nice. What the problem? Docker reference say that if i get up-to date message I changed volume or docker-compose.yml and need to rebuild, but firstly I dont changed anything, secondly why after reboot it works without rebuild?

P.S: When I write this message I realised that use httpd:latest image. Problem may be related with minor httpd updates? But then why I dont need rebuild after restarting daemon?

After getting your site up with docker-compose up there is No practical reason for running docker-compose up again without making any changes, because the container is running.

So docker deamon informs you there is nothing New to do.