Is Docker compose installation required everytime I start a new session on docker CLI I have downloaded docker toolbox to learn docker and I have started with docker compose topic. I had run few docker-compose command and next time when I restart the docker CLI(here docker quickstart terminal) everytime I have to run below docker-compose install commands

sudo curl -L https://github.com/docker/compose/releases/download/1.21.2/docker-compose-$(uname -s)-$(uname -m) -o /usr/local/bin/docker-compose

sudo chmod +x /usr/local/bin/docker-compose

Is there any way to permanently install docker compose.

When installing docker on windows or mac, there is a virtual machine called 'docker machine' that runs Linux OS. Docker has a command docker-machine used to start this virtual machine using docker-machine start. It is likely that every time you restart the toolbox, you start a new VM with new fresh copy of the OS. If you do docker-machine ls you should see multiple docker machines, one of them running. If instead on restart you always start the same docker-machine, it will have the previous changes, like previously installed packages, including the docker compose. You can give docker-machine start <arg> with the name of the docker machine you want to start, which should be always the same one.


What's the best complete DevOps setup? I'm curious what everyone recommends.

Vagrant is for setting up test environments right?
Ansible/Chef etc. seem good for servers you own, but what about cloud?
Docker... great for standardizing app, so why use Ansible/Chef?
Kubernentes.. good for deploying to multiple environments and scaling

I'm just trying to understand what the need for ansible or chef are? And how about Vagrant?

Also for the build cycle CI/CD pipeline with Git and artifacts.

Cloud providers:

AWS, GoogleCloud, Azure ? Which is best

Any recommendations about the best overall Devops setup?

Diagram would be best.

Stack is:
Node, React, MongoDb, Postgres, etc.

Vagrant is for setting up test environments right? Ansible/Chef etc. seem good for servers you own, but what about cloud? Docker... great for standardizing app, so why use Ansible/Chef? Kubernentes.. good for deploying to multiple environments and scaling

Just a personal note: the way you formulate these things makes me assume that you are diving freshly into these tools, and trying to figure out what they are for. And your reasoning is "tool X is for purpose Y". I know the feeling, I started out like that as well, trying to wrap my mind around what all this stuff is for. The trap is that the real reasoning should be "tool X does Z", instead. Few of these are really limited to a particular purpose.

For example. You say "Vagrant is for setting up test environments". This is wrong. Vagrant is for spinning up VMs in an automated way, including a zoo of pre-made images. "Ansible is good for servers you own" is wrong. Ansible is a tool to execute commands on a remote server where you have ssh access, with a zoo of recipes/playbooks to do the most common tasks. And so on.

This is not to berate you, but to make it easier for you. Try to be a little less abstract, and just look at what the tool actually does. Then it gets relatively easy to find proper uses, and you also can use the same too for different purposes.

For example:

Vagrant is great for also setting up development environments, not just testing. Or, really, any VM at all where you want the actual content to be mostly configuration-as-code and want to avoid clicking around in a VM GUI.

  I'm just trying to understand what the need for ansible or chef are? And how about Vagrant?

Now you know what the need for Vagrant is.  Vagrant literally solves the need to spin up VMs repeatably, automated, and have the configuration managed as files (i.e., checked into VCS together with the source code of your application, and so on).

  Also for the build cycle CI/CD pipeline with Git and artifacts.

Read the documentation of git & co and actually look at what it does. That, then, is its purpose, nothing more and nothing less.

Same for CI/CD. Continuous integration means literally that. To continuously (i.e., once or more a day, not just every few weeks or months, as it was done in the past) integrate (i.e., merge all the branches of your application that are in development currently). There is no hidden purpose there (or if there is, it should be obvious - doing it more often means every single time is more easy, since there's not such a long time inbetween). Obviously the details of how it works exactly without turning into a nightmare could be more involved, but the purpose is just what it says on the can.

  AWS, GoogleCloud, Azure ? Which is best

It does not make sense to think of "best" here. All three are extremely complex, and the fact that they are all still around means they all serve a purpose. You have to look at each of them, try them out, and compare them to your needs (and your personal likes/dislikes, sometimes).

  Any recommendations about the best overall Devops setup?

Clearly the best overall DevOps setup is the one that solves your problem and fits your overall needs and wishes. There are a vast amount of possibilities of good DevOps setups, and you have to choose for yourself.


Is Google Kubernetes (GKE) worth it for a program that runs once a week? My team is on the process of migrating all our cloud applications from AWS to GCP, and we were considering using GKE for our main program, until we got the news that Google will now charge an hourly management fee of $0.10 for each cluster in GKE.

We still haven't started using GKE, and are now considering if it's really the best option for us. Our workload consists of two process-heavy data-analyses pipelines, which we use to run up to 20 analyses simultaneously in different AWS instances, running those once a week, for around 15min to 3h at most (and perhaps will start running it twice a week on the nearby future). On AWS, we currently spawn several instances of EC2 machines for each pipeline, one for each analyses, so we were considering GKE as we could run several analyses on the same cluster, and let GKE do the scaling for us.

I'm also under the impression that this cost is on top of the cost to have one pod running constantly for every nodepool, is that correct? For every nodepool we create inside our cluster, we'd have to pay for a pod to be running in there 24/7 as well, so that's another 'idle cost' to be considered...

We'd only need to run workloads on the cluster for around 15min to 6h each week, so paying for nodes to be up and running 24/7 seems a bit of a waste, but perhaps I'm not seeing the big picture here, as analyses could be run on the same cluster and save us money? Is there an efficient way to shut GKE down while we're not using it - either the pods or, better yet, the whole cluster itself? That'd save us at the very least 690h of running idle every month.

Would killing and deleting all our clusters as soon as the last analyses is done, and recreating it every week, once new analyses are needed, be something feasible? Would there be a cost involved with that? Is that feasible to be done 'automatically'?

If your workloads are exclusively ephemeral, the answer is probably not worth the infra management effort of deploying K8S.

Kubernetes is best suited for types of workloads that are continuous, with regular incoming traffic and that require zero downtime. That's why it enables you to do things like rolling updates and implementing advanced traffic management patterns like Canary releases.

While you could have your cluster autoscale down to 0 worker nodes you would still incur in the cost of running the control plane node which requires running pods for the apiserver, the controller-manager, the scheduler and the kube-proxy in addition to running the autoscaler pods and log/metric collectors.

Deploying and destroying the clusters on demand is definitely feasible if you manage the cluster with Infrastructure as Code and design your pipelines so that they can setup and teardown clusters with minimal human intervention (waiting for apiserver to come up, waiting for workers to register and come up, etc). An even more elegant way of approaching it would be with using the Cluster API although this would require a management cluster.

Now, is it worth dealing with all the complexity inherent to K8S only to benefit from the downscaling and only to run occasional workloads? I'd say is probably best to manage your instances as regular autoscaling groups.


Is there a CI tool that guarantees no regressions in the branch quality level? Traditionally CI systems only perform monitoring of the quality levels in an integration branch, by performing QA verifications on the codebase where the changes are already committed, watching for regressions and sending notifications for human intervention.

But when these regressions are detected the branch has already been in trouble at least since the respective QA verification started and will remain in such state (or even get worse!) until all the culprits are identified, repairs for them committed and a new QA verification confirms the branch quality level was restored. The branch can be considered blocked for normal development during all this time.

Is there a CI tool capable of actually preventing such regressions from happening, which would perform pre-commit QA verifications and allowing commits only when the codebase updated with the respective commits would be passing those pre-commit QA verifications as well, thus guaranteeing a minimum branch quality level?

Update: the assumption is that suitable automated QA verifications with appropriate coverage to be able to detect the respective regressions are available for invocation by the CI tool(s).

As far as I can tell, you're looking for a tool that will reject commits that break the build—a CI tool probably won't be able to prevent regressions by actually fixing your code, but it can stop you from adding bad code to the repository.

Atlassian has a few interesting applications of Git hooks:

  Server-side pre-receive hooks are an especially powerful compliment to continuous integration because they can prevent developers from pushing code to master, unless the code meets certain conditions – like elite ninja guardians, protecting it from bad code.

If you're using Git, the hooks are very powerful (and there are similar hooks for SVN, Mercurial and other version control systems), and you might find it useful to use them to run pre-commit checks.

The Git documentation has a page on creating a hook to reject pushes if they don't meet a certain criteria which could easily be adapted to this use-case.

However, a lot of people would argue that rejecting commits is a bad idea on a feature branch—you'll just be wasting time fighting against your CI system when the build breaks for some reason, instead of actually fixing the bug.

On the master branch, it could make sense to reject broken merges, because you might want to ensure it always builds. For a feature branch, you will inevitably break things, and since the code isn't going into production now, it makes more sense just to warn you than actually reject your commit altogether.


How to manage Free and Open Source Software dependencies? We exist in an increasingly complex ecosystem of Free and Open Source Software, FOSS, and it's dependencies.  Having done a bit of analysis on one medium size project there are over 1,500 dependent software packages, not counting different versions of the same package or any packages developed internally for reuse.

How do we manage this complexity in an enterprise that needs to be able to answer the following two questions categorically

Have we met the licence terms of all of the software we are using?
What is our exposure to vulnerabilities in free or open source software?

I can think of two architectures that would support the answering of these questions, however, the enormity of the problem could well be clouding my judgement:

Approach #1: Walled Garden

Effectively firewall off the sources of these open source packages, i.e. npm, Docker Registry, nuget, etc., then create an internal repository of approved packages, implementing some process to whitelist packages.

Approach #2: Audit

Allow packages to be downloaded from the internet freely, however, perform source code analysis as part of the build pipeline to report on the packages currently being utilised.

Practically an organisation could use either solution or if necessary both solution to provide a degree of checks and balances.

Are their other architectures or tools that support the management of Free and Open Source dependencies?

Any organization that needs the level of control you've specified is essentially forced into the preapproval process ("walled garden"). It's a pain for developers, but it's a necessary process to keep the company safe. I've even heard of companies with programs that scan all their workstations regularly to find and remove unapproved software.

It's worth noting that there are a few ways to implement it. As you note, you can use an internal mirror. Another popular option is to vendorize dependencies into your application's repo.

Security is one of the main issues with a frozen version approach, as you've taken on responsibility for maintenance. This generally means having someone whose responsibility it is to follow the release discussions of every package you've frozen. It also requires allotting continuous time for updating dependencies, because otherwise it's very easy to get caught with a security vulnerability that's only been fixed in a version that's incompatible with your app.

In general, I find that companies often underestimate how much additional work they're taking on with this change, and strict compliance is actually much more costly than occasionally ending up with licensing or "disappearing dependency" problems.


What is Ansible's config equivalent of `--vault-password-file`? According the help of ansible-playbook one could use --user=REMOTE_USER to define the ssh user, but one could also define ansible_ssh_user: REMOTE_USER in either the host- or group_vars.

Question

What variable need to be defined in either the group- or host_vars directory to prevent that --vault-password-file has to be defined while running ansible-playbook?

Attempts

When ansible_vault_password_file: ~/.vault_pass.txt is defined in the config the decryption fails:

ERROR! Decryption failed on /path/to/vault

No associated vault variables was found in this documentation

Here is the definition:

DEFAULT_VAULT_PASSWORD_FILE = get_config(p, DEFAULTS, 'vault_password_file', \
'ANSIBLE_VAULT_PASSWORD_FILE', None, value_type='path')

This means that you either put in ansible.cfg or playbook:

vault_password_file: ~/.vault_pass.txt

Or in your shell defined this variable:

export ANSIBLE_VAULT_PASSWORD_FILE=~/.vault_pass.txt


Gitlab CI - Deploy to an active docker container I am running production code within an active docker container.
I would like to use the Gitlab CI tool to auto deploy changes to the production environment WITHIN the docker container.
I am having trouble finding a clean way to do this.
One idea I have is to simply log into the docker container as part of the gitlab-ci automation script

docker exec -it 82jf72h8dj2 bash
then git pull from the appropriate directory

In theory, this will work... but what happens when the docker ID changes? Is there any way to detect this as part of the script? I would hate to have to keep updating the CICD variable since the docker ID seems to change fairly often...

The image acts as a foundation for each container that you start from it.
You might do the update in your 'fashion' way, but containers are ephemeral - let's say it got restarted, you updates are lost and you are back to the previous version.
If you want to handle this process by yourself, how would you track your versions ? Essentially you would have a root docker image and after that it would be so hard to know which version is currently running ? Could you also safely roll-back version to version ? How about if somebody hijacks you container, due to some Unix Socket permissions with the Docker daemon ...
What I am trying to say, do not rediscover the wheel :)
Take advantage of modern technologies like Kubernetes, deploy each image for a new version there, you have a roll - out deployment, readiness probes and so on ... You would be safe that your software is always working, track the versions and have H/A.


Consul with Nginx, not using docker I am from development background and totally new to DevOps, I want to setup consul for service discovery for my micro-services, services are reverse proxied by Nginx. Now I want to configure Nginx with the consul, we are not using dockers as of now.

I have gone through many tutorials but I didn't get much about that how can I configure Nginx with the consul.

Any lead would be appreciated.

I have done this in the past using https://github.com/hashicorp/consul-template.

What consul-template does it generate a configuration file (for nginx) based on a certain template you provide. And the values that it fills into this template are coming from the configuration stored in consul.

Each time your micro-services register themselves in consul and need to be addressed in the nginx configuration file, consul-template does it for you by changing the file and sending a SIGHUP (kill -1) to the nginx process that makes it reload nginx configuration.

The repository for consule-template contains an example of managing nginx https://github.com/hashicorp/consul-template/blob/master/examples/nginx.md

If you are using Nginx Plus, which is the commercial version of Nginx. Then it has support for Consul built-in. https://www.nginx.com/blog/service-discovery-with-nginx-plus-and-consul/ and https://www.nginx.com/blog/service-discovery-nginx-plus-srv-records-consul-dns/ go into depth on this subject.


Zip/tar a folder using chef resource Can someone help me how to take backup for a specific folder and also zip/tar those folders. I am using this for taking a backup actually, but I would like  a compressed output:

bash "backup #{dir_name}" do
    user 'user'
    cwd "#{path}"
    code <<-EOH
      cp -rp #{dir_name} #{dest}/#{dir_name}_bkp_#{timestmp}
    EOH
end

Is there any chef resource to zip/tar a specific folder ?

Well that's not exactly about chef, you're basically asking how to use tar.

The command to create an archive of a folder would be
tar -zcvf <backup_thing>.tgz <path_to_backup>

In the tar options:

z pass the resulting tar archive to gzip for compression
c instruct tar to compress (create the archive)
v tells tar to be verbose, so you know at which point it is working, it has a performance impact
f <path> tells tar to output in a file or device

A correct chef resource for this would be as follow:

execute 'Backup #{dir_name}' do
  user 'user'
  command "tar -zcvf #{dest}/#{dir_name}_bkp_#{timestmp}.tgz #{dir_name}"
  live_stream :true # to see the progress during chef run, remove it and v in tar option if you don't care of it.
end

Using an execute resource avoid spawning a bash process to run tar, it is not needed, the shell_out class used by execute under the hood handles it already.

Be warned this is not idempotent, each run of chef will redo this backup command, that's maybe not what you're after.
For an update of jenkins (or something else) out of usual maintenance I would do a manual backup, check it is ok, and then run the upgrade (via chef eventually).

I'd recomend taking the tutorial at https://learn.chef.io and reading through the documentation at https://docs.chef.io


Easiest way to manage an elasticsearch cluster in GCP? I'm trying to create my own Elasticsearch cluster to use for security analytics. I'm familiar with the function and setup of a single instance deployment, but I'm having trouble figuring out the simplest strategy to set up and manage the cluster of any sizable scale in GCP. The two ways I've played around with are:

Building the ES cluster using Compute Engine VM's in GCP.
Managing the cluster using the Kubernetes Engine and the official Elastic docker image.

Since I don't have much experience with containerization, I'm trying to figure out if learning and managing a kubernetes deployment would save me some headache in the long run.

One could use helm to install elastic on managed k8s in GCP.


Secrets in Docker without Swarm I am using Docker, but not Swarm (or any other orchestration) for a project.
Is there a way to leverage Docker Secrets without Swarm?

You can use docker secrets a bit differently using docker-compose without having to use swarm. See this for the official documentation.
Example:

Create a simple compose file like so,

version: "3.7"

services:

  db:
    image: mariadb:10.5.2
    env_file:
      - ./db.env
    secrets:
      - rootpass
      - dbpass
      - mysqldb
      - mysqluser
    restart: always

Now add the following in the end

secrets:
  rootpass:
    file: /tmp/root_pass
  dbpass:
    file: /tmp/db_pass
  mysqldb:
    file: /tmp/mysql_db
  mysqluser:
    file: /tmp/mysql_user

Inside those files, keep your password, username, database name etc. in plain text. Then simply deploy the containers docker-compose up -d.

It's similar to how you define volumes and networks in a compose file.
Keep in mind that this isn't true secret implementation. Here's the github PR that added this feature, along with the main file if you're interested.


Stop Multiple Windows Service Simultaneously I have multiple windows service and there are no dependencies between them. I want to stop all the windows services at the same time. I don't want to stop all the services one by one. How can I achieve this?

The -name parameter of the Stop-Service command is an [array strings][1]. You can pass multiple services to it.

Stop-Service -name service1,service2,service3

Edit

[string[]]$services = @('service1', 'service2')
Stop-Service -Name $services -NoWait
while ((Get-Service -Name $services | Where-Object { $_.status -ne 'Stopped'}).Count -ne 0)
{
    Start-Sleep -Seconds 5
}


How to solve 'upstream connect error or disconnect/reset before headers. reset reason: connection termination' in Istio? I try to setup an aws load balancer (ELB) with SSL follow the instruction at #6566

Certificate was attached on ELB.

However, I got the issue "upstream connect error or disconnect/reset before headers. reset reason: connection termination" when trying to access our web on browser.

Our setup without SSL has been worked before.

I use the custom values.yaml to install the istio (helm template):

helm template ./istio/install/kubernetes/helm/istio --name istio \
--namespace istio-system --values ./mesh/values.yaml | kubectl apply -f -

I have inserted below annotations to the gateways tag:

istio-ingressgateway:
    serviceAnnotations:
      service.beta.kubernetes.io/aws-load-balancer-ssl-cert: "arn:aws:acm:ap-southeast-1:xxxxx:certificate/my-crt"
      service.beta.kubernetes.io/aws-load-balancer-backend-protocol: "http"
      service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "https"

Here is my gateway.yaml:

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: istio-gateway
spec:
  selector:
    istio: ingressgateway #default istio ingressgateway
  servers:
  - port:
      number: 80
      name: http-istio-gateway
      protocol: HTTP
    hosts:
    - "*"
    tls:
      httpsRedirect: true
  - port:
      number: 443
      name: https-istio-gateway
      protocol: HTTP
    hosts:
    - "*"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: api-gateway
spec:
  gateways:
  - istio-gateway
  hosts:
  - "*"
  http:
  - match:
    - uri:
        prefix: /socket.io/
    route:
    - destination:
        host: api-gateway-ws.default.svc.cluster.local
        port:
          number: 5001
  - match:
    - uri:
        prefix: /
    route:
    - destination:
        host: api-gateway.default.svc.cluster.local
        port:
          number: 5000

I encountered the same error but with a different issue. The Service port needed a name added to it. https://github.com/istio/istio/issues/19966. And they need to follow the format (protocol-suffix) https://istio.io/docs/ops/deployment/requirements/

ports:
    - name: https # Use http or https
       protocol: TCP
       port: 8080
       targetPort: 8080


Devops CICD run ReactJS application while testing with jest + puppeteer Scenario
I'm working on getting my end-to-end tests to work in a GitLab pipeline. Currently all the tests run locally but I'm having an issue getting them to run using GitLab's continuous integration feature.
When running the tests locally puppeteer will use a headless version of chrome for end-to-end tests but the ReactJS' CRA server has to be running for this to work. To get the CRA server running locally I use npm start in one terminal and once it is fully running I can run npm test in a different terminal. During testing Puppeteer uses headless Chromium to run e2e tests on the site.
The Issue
npm start starts my CRA server but that process doesn't stop as I'm not sure how to stop it unless I manually quit it. This hangs docker and blocks me from ever starting my tests.
Question
How can I start my application with npm start and once it starts then run npm run test:e2e ? Ultimately I'd like to quit CRA once the tests are completed.
Current Pipeline yml
image: node:10.19.0
stages:
  - test
  - build
  - deploy

cache:
  # untracked: true
  key:
    files:
      - package-lock.json
  paths:
    - node_modules
#test_async:
#  script:
#    - npm install
#    - node ./specs/start.js ./specs/async.spec.js

before_script:
  - apt-get update && apt-get install -y lftp gconf-service libasound2 libatk1.0-0 libatk-bridge2.0-0 libc6 libcairo2 libcups2 libdbus-1-3 libexpat1 libfontconfig1 libgcc1 libgconf-2-4 libgdk-pixbuf2.0-0 libglib2.0-0 libgtk-3-0 libnspr4 libpango-1.0-0 libpangocairo-1.0-0 libstdc++6 libx11-6 libx11-xcb1 libxcb1 libxcomposite1 libxcursor1 libxdamage1 libxext6 libxfixes3 libxi6 libxrandr2 libxrender1 libxss1 libxtst6 ca-certificates fonts-liberation libappindicator1 libnss3 lsb-release xdg-utils wget
  - if [ ! -d "node_modules" ]; then npm install; fi

test:
  stage: test
  script:
    - npm start
    - npm run test:e2e

build:
  stage: build
  script:
    - npm run build

# deploy-to-staging:
#   stage: deploy
#   script:
#     - lftp -e "open ftp.mediajackagency.com; user $MJA_FTP_USER $MJA_FTP_PASS; mirror --reverse --verbose build/ /var/www/domains/projects/my-project/build/; bye"
#   environment:
#     name: staging
#     url: http://dev.mediajackagency.com/projects/my-project/build
#   when: manual
#   only:
#     - dev

I switched to using a production build for testing by use of a local webserver. I leveraging the [jest-puppeteer][1] library which allows you to start a webserver with your tests.
jest-puppeteer.config.js
/**
 * find all flags at the following site @jkr
 * https://jestjs.io/docs/en/cli
 */

module.exports = {
    launch: {
        devtools: true, // allows for use of 'debugger;' in tests
        // executablePath: '/usr/bin/chromium-browser',
        headless: true,
        defaultViewport: {
            width: 1024,
            height: 768,
            //isMobile: true,
            //hasTouch: true,
        },
        ignoreDefaultArgs: ['--disable-extensions'],
        args: [
            '--enable-font-antialiasing',
            '--font-render-hinting=medium',
            '--disable-gpu',
            '--disable-dev-shm-usage',
            '--disable-setuid-sandbox',
            '--no-first-run',
            '--no-sandbox', // GOOD
            '--no-zygote',
            '--single-process', // GOOD

            "--renderer",
            "--no-service-autorun",
            "--no-experiments",
            "--no-default-browser-check",
            "--disable-extensions",
        ]
    },
    server: { // launches webserver just for tests @jkr
        command: 'npm run test:webserver'
        // cd build && ../node_modules/local-web-server/bin/cli.js --port 3000 --spa index.html
    },
    browser: 'chromium',
    browserContext: 'default'
};

.gitlab-ci.yml
image: node:10.19.0 # https://hub.docker.com/_/node/
# image: node:latest

cache:
  # untracked: true
  key: my-project-name
  # key: ${CI_COMMIT_REF_SLUG} # per branch
  # key:
  #   files:
  #     - package-lock.json # only update cache when this file changes (not working) @jkr
  paths:
    - .npm/
    - node_modules
    - build

stages:
  - prepare # can install ci-deps here ? @jkr
  - test # uses test:build specifically @jkr
  - build
  - deploy

# before_install:

before_script:
  - npm ci --cache .npm --prefer-offline #
#   # - apt-get update && apt-get install -y lftp gconf-service libasound2 libatk1.0-0 libatk-bridge2.0-0 libc6 libcairo2 libcups2 libdbus-1-3 libexpat1 libfontconfig1 libgcc1 libgconf-2-4 libgdk-pixbuf2.0-0 libglib2.0-0 libgtk-3-0 libnspr4 libpango-1.0-0 libpangocairo-1.0-0 libstdc++6 libx11-6 libx11-xcb1 libxcb1 libxcomposite1 libxcursor1 libxdamage1 libxext6 libxfixes3 libxi6 libxrandr2 libxrender1 libxss1 libxtst6 ca-certificates fonts-liberation libappindicator1 libnss3 lsb-release xdg-utils wget
#   - if [ ! -d "node_modules" ]; then set NODE_ENV=dev && npm install; fi # make sure dev dependencies are installed todo: 'if' needs to be verified working @jkr
#   # - npm install
#   # - if [ ! -d "build" ]; then npm run build; fi

prepare: # could this job be in a before_script for test ? @jkr
  stage: prepare
  script:
    - npm install
    - npm run build:test
  # cache:
  #   paths:
  #     - node_modules
  #   policy: push

test:
  stage: test
  script:
    - npm run test:ci-deps
    - npm run test:e2e # runs puppeteer tests @jkr

build:
  stage: build
  script:
    - npm run build
  # artifacts:
  #   paths:
  #     - build

deploy-ftp:
  stage: deploy
  script:
    - lftp -e "open ftp.my.domain.com; user $MJA_FTP_USER $MJA_FTP_PASS; mirror --reverse --verbose build/ /var/www/domains/dev/projects/my-project/build/; bye"
  environment:
    name: staging
    url: http://my.domain.com/my-project/build
  when: manual
  only:
    - dev


How to implement the immutable server pattern without loosing the ability to do post-mortems? The immutable server pattern is a deployment discipline favouring the reproducibility of deployments.  It is characterised by the fact that “a server that once deployed, is never modified, merely replaced with a new updated instance” and implementing this discipline requires an automation of server deployment. This automation has numerous operational advantages, one of the most important is allowing the quick and reliable replacement of failing instances in an infrastructure. This automation also implies that server deployment is described by versioned software artefacts and is subject to iterative improvements.

A popular aspect of implementations of this discipline is the removal of remote access methods to the server once it has been launched (esp. removing SSH access).  Removing remote access is an easy way to ensure that the configuration of the server matches the configuration prepared by the deployment automation.

However, when investigating the causes of a software failure in a post-mortem, relying on structured monitoring is not always enough and remote access to the machine could be necessary.  It is a common practical situation that server monitoring does not cover all of the failure sources, or that monitoring can be impaired by the server failure itself, which would likely be the case if the server runs out of memory or reaches its process limit.

How to implement the immutable server pattern without loosing the ability to do post-mortems?

First of all, removing ssh on an immutable server doesn't guarantee there'll be no change, it's more that as there should be no need to change something you reduce the attack surface by removing a remote access channel.

One way to keep a sort of post-mortem is log centralisation. There's a myriad of methods to achieve it, ELK stack, Splunk, syslog...

Another more crude way to keep a post mortem for an immutable server is to have a script on the shutdown process (an immutable server failing would be shutdown and a new one spin up to replace it) to gather a core dump of the program, a memory dump and send them to a remote system for analysis along with most of the logs.

Main advantage of this solution is that you get back only failing system information at time of problem, allowing to gather larger informations than getting them periodically.

It's hard to be more specific on how to achieve this, each distribution has some way to get things and I've no generic example.


problems running Ansible with become on domain-connected Windows Server I am running several tasks on a Windows 2016 server with Ansible including:

Joining it to a domain SOMEDOMAIN
SOMEDOMAIN\SOMEUSER has been added to the local administrators group on the machine with Ansible.
Running several tasks on the domain-connected machine as a domain-user SOMEDOMAIN\SOMEUSER

The last bit requires me to run Ansible tasks using become and runas (as per the Ansible Documentation on the topic).

Problem

Unfortunately, even a trivial example like the below fails:

---
- name: add to local admin
  hosts: all
  vars:
    ansible_become_password: `some_password`
  tasks:
  - win_whoami:
    become: yes
    become_user: ''SOMEDOMAIN\SOMEUSER''
    become_method: runas

The error message reads

The full traceback is:
Exception calling "RunAsUser" with "7" argument(s): "LogonUser failed (Logon failure: the user has not been granted
the requested logon type at this computer, Win32ErrorCode 1385)"
At line:1087 char:9
+         $result = [Ansible.BecomeUtil]::RunAsUser($username, $passwor ...
+         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [], MethodInvocationException
    + FullyQualifiedErrorId : Win32Exception

fatal: [SERVERNAME]: FAILED! => {
    "changed": false,
    "msg": "Failed to become user SOMEDOMAIN\\SOMEUSER: Exception calling \"RunAsUser\" with \"7\" argument(s): \"LogonUser failed (Logon failure: the user has not been granted the requested logon type at this computer, Win32ErrorCode 1385)\""
}

Attempts

I do understand this is related to a privilege missing on the machine.
I also know that group policies get applied on the machine when it joins the domain, so its possibly something related to the group policy. (I don't have access to the domain controller to validate this)

I checked the Group Policy "Access this computer from the Network". It is set to Everyone, Administrators, Backup Operators, Users which is the default setting. My account is part of the Administrators group, so this doesn't look like the cause.

The Question

What other possible causes are there? Or in other words, what other privileges (or modifications to the ansible task) are needed to make Ansible become with the runas method work on domain-connected Windows machines?

My problem resolves down to an issue with group policy, and is likely to be solved with the same answer as this question:

https://serverfault.com/questions/695839/error-1385-when-executing-runas-on-windows-enterprise-7

i.e. the required permissions for become to work on Windows, seem to be:

Allow Logon as a Service
Allow Logon Locally

Alternate Solution

reference

  batch: Runs the process under a batch context that is similar to a
  scheduled task with a password set. This should bypass most WinRM
  restrictions and is useful if the become_user is not allowed to log on
  interactively.

There is another permission called 'Allow Logon as a Batch Job". If the domain-user has this permission, the user can run scripts as though they were scheduled tasks.

You can run your scripts using Ansible become and change the logon type to batch as shown below:

win_whoami:
    become: yes
    become_user: ''SOMEDOMAIN\SOMEUSER''
    become_method: runas
    become_flags: logon_type=batch


Understanding and tweaking AWS policies for job functions There are quite many possibilities to configure AWS policies; Amazon has therefore provided policies for job functions.

Now is it anyhow possible to sneak into JSON code of these policies to learn in detauls what they do and use them as foundation for own policies along their lines?

Sure you can have a look at the JSON policy documents. On that page that you link to (AWS Managed Policies for Job Functions) click the Policy you're interested in while being logged to your AWS account and it will take you to the policy definition.

E.g. if you click to AdministratorAccess it will open the IAM Policies page and show you the JSON:

Hope that helps :)


Ansible dynamic inventory using AWS plugin -- how to obtain internal DNS records or private IP? Seeking help from those who've worked with dynamic inventory in Ansible.
I have Ansible set up with the aws_ec2 plugin (not using the python script as it's been too problematic). Currently, I'm only using an aws_ec2.yaml file as my inventory and variable source of truth.
When I run ansible-inventory --graph, Ansible returns my instances as public AWS DNS hostnames. I want either private DNS hostname or IP address so I SSH will against the hosts (we have firewall rule blocking ssh against the public AWS hostname).
How can I instruct the aws_ec2 module to do this via the plugin? Thanks.

The hostnames parameter controls what the plugin uses as the host name. It accepts a list of attributes, and will use the first one that the instance has defined:
hostnames:
  - private-dns-name
  - private-ip-address

You can also set the ansible_host variable using compose. This changes the connection target without modifying the inventory hostname:
compose:
  # Note the underscores; this functionality uses the Ansible variable names instead of the raw AWS attributes.
  ansible_host: private_ip_address


Running consul in the background using Chef I am trying to create a Chef cookbook that will run Hashicorp Consul in the background and will continue with the rest of the recipe and not get hung on starting Consul.

I have tried the & method at the end of the command but when chef-client exits it kills the process and will not let me run any checks / testing against Consul.

Anyone have a suggestion on how to resolve this? I know I have seen a few implementations where they use ruby scripts but I would like to avoid that as I suck at ruby. Also do not want to use another cookbook from online, rather make my own to learn how as this is a learning project.

The usual way to do that is to use a service manager (systemd or supervisord) to handle the service lifecycle (keeping the consul program running and restarting it if it fails for exemple) there's cookbooks existing to help creating those service units.

The cheap alternative if you count on chef running periodically and handling that is to use nohup which is a linux program made exactly for this.


Adam Smith vs. fullstack developers - and productivity in DevOps? By Adam Smith, labour division can make you by 240 times more effective (on example of a pin factory producing pins in 18 steps).

Why then are multi-skilled roles are so in demand if this actually reduces productivity - or was Smith just wrong, why then?

Searches for "fullstack developer" still trend on Google, however apparently slower than two years ago:

=====

To sum up, a full stack developer would be able do virtually all the value chain (correct me if I am wrong):

Discuss with customers and refine workable agile requirements for his part of the job
Decide which architecture, tooling and components pick up - just give him a notebook
Write code for frontend, backend, ingration, which is cross device compatile and does not requrire much testing, or includes it
Profile and scape data, use Cloud AI/ML APIs for advanced features
Write up required IaC code and rollout
Be on call in case of error or sales processes
Be aware of security relevant design, overall patching, migration and modernization
Account time table in a scroutinized way to ease invoicing of the employer
... did I forget anything?

UPD - "we need the productivity of specialisation but we don’t want the insular worldview of “extreme division of labour”. (DevOps Guys, "DevOps, Adam Smith and the legend of the Generalist", 2013-2016)

There are two types of work:

Exploitation - Well defined work that can be easily divided into well-defined stages, where each stage can be learned and mastered on its own and handover between stages does not require communication.
Exploration - Undefined work, which requires learning and experimentation to accomplish each stage and handover between stages requires massive amounts of communication of all learning and status of the project.

Adam Smith concerns himself wholly with exploitation and not at all with exploration. The work done in Research & Development departments of the industry is by its definition mostly exploration and so it is not covered in any way by Adam Smith.

But we have seen that in later continuous improvement stages, which are partly exploitative work, the application of CI/CD can bring similar gains in productivity, which could in some way be probably traced down to Adam Smith by someone very imaginative.


Why `docker stack` needs Docker to run in swarm mode? What are the implications when working on a single node? I am experimenting with docker stack as a replacement to docker-compose. But I had the following message on my first deploy attempt:

sudo docker stack deploy --compose-file stack/app-stack.yml app-stack

this node is not a swarm manager. Use "docker swarm init" or "docker swarm join" to connect this node to swarm and try again

Once issued the docker swarm init command, docker stack deploy works as expected.

Why it is mandatory to switch to swarm mode to run docker stack? And most important, what are the implications of switching to swarm mode when working on a single node?

Why it is mandatory to switch to swarm mode to run docker stack?

A stack and service are swarm mode objects by their very definition. The stack is a collection of services, and each service is a definition of a target state. Swarm mode takes those target states and manages containers to achieve that target state.

  And most important, what are the implications of switching to swarm mode when working on a single node?

Pretty minimal from the user perspective. There's another few ports opened and listening. If you try to directly manage containers (stop, rm, restart) that are managed by a service, you'll find that swarm mode will take steps to get you back to the service definition, which confuses some that keep finding new containers recreated when they delete all running containers directly. You just need to run the docker service commands to manage them instead.


git - develop base and extended feature - commits in base should reflect in extended in pull Hie I am working around in git to achieve something like mentioned below.

Imagine there is a Big feature which is going to develop - since it's very big they have split into core feature and an extended feature.

Now they have divided into teams for development.

Now how to make branches in a below-mentioned way

the core will be developing in core_branch extended will be developing in extended_branch containing core development it should be like that when we perform git pull in extended_branch it has to pull if any commits made to core_branch how to achieve this??

I am aware of git rebase / git merge. In either of the cases, extended team/developers have to check any new commits in the core branch and merge/rebase their extended_branch

Because you have a large project with multiple teams using multiple development branches will give you merge headaches, especially if the core and extended features have a significant overlap in the codebase.

The extended feature team(s) will spend a lot of time periodically merging the core feature changes and dealing with the fallout from each such merge, which can not be properly/realistically estimated ahead of time. If there are multiple teams working on the extended feature they might decide to go on separate branches to further isolate and protect themselves against these periodic merge instabilities. It's just an illusion, that'll make things even worse at the overall project level - it can lead to Branch Merge Hell.

You may want to take a look at Trunk-Based Development (TBD), with feature-flags and/or branch by abstraction. Most effective when using a gated-commit CI system.


How to calculate disk space required by Prometheus v2.2? We are trying to calculate the storage requirements but is unable to find the values needed to do the calculation for our version of Prometheus (v2.2).

Prometheus(v2.2) storage documentation gives this simple formula:

needed_disk_space = retention_time_seconds * ingested_samples_per_second * bytes_per_sample

retention_time_seconds: We took our retention time of 720 hours and converted to 2 592 000seconds.
bytes_per_sample: We used rate(prometheus_tsdb_compaction_chunk_size_bytes_sum[1d]) / rate(prometheus_tsdb_compaction_chunk_samples_sum[1d]) based on the formula from this site, except we did not have prometheus_tsdb_compaction_chunk_size_bytes_sum so used prometheus_tsdb_compaction_chunk_size_sum instead. This gives us ~1.3, which sounds about right.
ingested_samples_per_second: We cannot find a way to get this value. We tried the following:

This site suggest prometheus_local_storage_chunk_ops_total, but we do not have this metric.

  Another useful metric to query and visualize is the prometheus_local_storage_chunk_ops_total metric that reports the per-second rate of all storage chunk operations taking place in Prometheus.

This uses rate(prometheus_tsdb_head_samples_appended_total[1h]), but the result is ~18600 suggesting 68TiB storage.

  Combined with rate(prometheus_tsdb_head_samples_appended_total[1h]) for the samples ingested per second, you should have a good idea of how much disk space you need given your retention window.

As above, this issue mentions rate(tsdb_samples_appended_total[5m]), but probably meant (or what we got working is) rate(prometheus_tsdb_head_samples_appended_total[5m])

  You can still get samples per second via rate(tsdb_samples_appended_total[5m]).

This uses prometheus_local_storage_ingested_samples_total, but again, it is not available to us.

  sample rate = rate(prometheus_local_storage_ingested_samples_total{job="prometheus",instance="$Prometheus:9090"}[1m])

Finally this issue also says to use rate(prometheus_local_storage_ingested_samples_total[2m]), and once again, it is not available to us.

  The other side is ingestion, which is way easier to reason with capacity-wise. You can find out how many samples your server is ingesting with this query:
  rate(prometheus_local_storage_ingested_samples_total[2m])

Any assitance or nudge in the right direction would be much appreciated.

>>> What worked in the end <<<

As was pointed out, our math was broken, so the below metrics worked for the calculation.

ingestion_samples_per_second: rate(prometheus_tsdb_head_samples_appended_total[2h])
bytes_per_sample: rate(prometheus_tsdb_compaction_chunk_size_sum[2h]) / rate(prometheus_tsdb_compaction_chunk_samples_sum[2h])

I think your math may be off.  If you use the ~18600 number you found I get a very different result from 68TiB:

2,592,000 (seconds) * 18600 (samples/second) * 1.3 (bytes/sample) = 62,674,560,000 (bytes).  This is ~62.675 Gigabytes (divide the digital storage value by 1e+9
) and I'm guessing a more reasonable number for your infrastructure.


Are there any guidelines for when to use gitlfs and will it break the CI? Git large file system git-lfs could be used to store large files and at a txt file to the git repository. The question is when to decide whether a file should go to git LFS and if that is done whether the existing CI configuration will break.

Sometimes jars are added to a repository like the gradlew.jar. According to me, binaries like jars should not be stored in git as these are not text files that change, but a binary does not have to be large.

Currently we have multiple git repositories that contain jar files. I wonder whether all these jars should be moved to git-lfs.

When to decide whether a file should go to git LFS

Your files can go to git-lfs at any time, but it would be inneficient to implement this solution for few files. Specially if it's basic gradlew jars.

I have used the gradlew strategy previously. The idea is to push this "wrapper" along your code to make the build process portable. Any platform with the proper JRE will be able to build the application in a similar fashion. This removes the headache of enforcing teams to install specific versions of gradle.

The purpose of gi-lfs is to leverage the benefits of git along with the ability to store large files remotely by caching them and indexing them. It reduces the burden of downloading a big repository when binaries are stored.

In my opinion you should start consider this solution when:

Your git repository contains large files (anything in the MB magnitude)
They represent the main content of your repository
These files are updated regularly
It is taking your CI tool minutes to pull/clone the repository

If you consider this solution to track binary versions, I would suggest you go for binary repositories instead such as:

Jfrog Artifactory
Archiva
Nexus

Does the existing CI configuration will break

If you decide to go git-lfs, the normal git commands won't work. You will need to either create symbolic links toward git-lfs after doing the setup or add lfs after git in your CI tool. i.e. git clone becomes git lfs clone.


How do you manage your inventory of spot instances on AWS with Ansible I am new to Ansible, with some experience in Chef. I am considering learn and use Ansible to manage environments.

I was wondering, what would be the best practice to manage inventory of AWS spot instances?

For example, when a spot instance is shut down the old IP is not relevant anymore in the inventory of hosts.

Are there any other alternative approaches for the use case of elastic environments?

You are looking for Dynamic Inventory

Please look into ec2.py and ec2.ini of Ansible, and also the Dynamic Inventory docs of Ansible are very good for understanding how to manage your AWS inventory.


How to create GitHub Actions for unit testing in .NET projects? Could You help me? I use NUnit framework to test my .NET project. And I want to run my tests via GitHub Actions.

What should be included in the assembly of my project? Maybe there are some standard examples?

You do not need to include anything into your assembly to run your tests using GitHub Actions. Just create workflow file in .github/workflows folder with the following content (assuming that you have .NET Core project):

---
name: Tests

on: push

jobs:
  tests:
    name: Unit Testing
    runs-on: windows-latest
    steps:
      - uses: actions/checkout@v2.1.0
      - run: dotnet test

  dotnet is pre-installed on windows machine but is not pre-installed on macos and ubuntu. So, you have to install dotnet by adding an extra step in case you want to run it on one of these machines. You can use actions/setup-dotnet action for this purpose.


Docker and Kubernetes integrations compatibility matrix? Two example integrations of Docker and Kubernetes are OpenShift and Rancher Labs AFAIK.

Sadly but in fact we have not completely escaped us from the dependency hell.

Question: is there an established source of information which distributions here package which versions (like Ubutu/Debian version chronicles on Wikipedia)?

Background.

Not-so-obvious facts for newbies (judged by my learnings so far) are (defining acceptable technical usability to a level that you do no need hours to debug even 101 tutorials):

Kubernetes is very dynamic project and it is not so easy to deploy it as-is (but seem to get better with each version, also thanks to kubeadm I would say)
Kubernetes does not support any Docker versions coming after the community fork to moby and Docker CE.

See also: https://stackoverflow.com/questions/44657320/which-docker-versions-will-k8s-1-7-support

Every Kubernetes release has an External Dependencies section in the respective Changelog. E.g.:

CHANGELOG-1.14:

  The list of validated docker versions has changed. 1.11.1 and 1.12.1
  have been removed. The current list is 1.13.1, 17.03, 17.06, 17.09,
  18.06, 18.09.

CHANGELOG-1.13:

  The list of validated docker versions remain unchanged at 1.11.1,
  1.12.1, 1.13.1, 17.03, 17.06, 17.09, 18.06 since Kubernetes 1.12.

CHANGELOG-1.12:

  The list of validated docker versions was updated to 1.11.1, 1.12.1,
  1.13.1, 17.03, 17.06, 17.09, 18.06.

and so on...


Is there an established scheme for creating diagrams about Containers, Services etc.? We have UML to describe classes, components, workflows. We have BPMN to describe business processes. We have more old-fashioned forms used in times past for structural diagrams.

Is there anything established or semi-established in the DevOps world? The information contained in a simple diagram should be something like:

Which containers are there in a containerized app, including their associations (i.e. which statically depends on which for building) and the (micro)services they provide?
How do they interact dynamically (networking etc.)?
What responsibility does each of the containers fulfill?
What requirements does each container have on the host, if relevant?
... and so on.

I'm neither asking for a list of any private schemes somebody may fancy, nor about opinions on which is the most beautiful.

I'm strictly asking about whether there exist any that either by purpose are industry standards or have turned out to just be used in many places by chance/convention.

The purpose of using such a scheme would be to have something that would instantly be recognized by fellow DevOps people; or which could easily be shown in contexts where it is not appropriate to look into Dockerfiles or other Infrastructure-as-code sources (e.g., when scribbling on a board, or holding a more abstract presentation).

The established method is infrastructure as code.

Want to find out what processes run in a container? Check the Dockerfile. Infrastructure dependencies? Check the Cloudformation template/Terraform blueprint. Interactions between containers? Check the docker-compose.yml or Kubernetes yaml file.


How can I best deliver read only data assets to a Kubernetes application? I'm migrating an HPC app to Kubernetes, and am trying to determine the best way to provide read only data assets as a configuration managed snapshot.

Previously, my team had delivered our application as a set of RPMs, but as we move to Kubernetes, we're delivering Docker images. This works fine for our application binary, as instead of a pile of RPMs that all have to agree, we can just deliver a known working image.

The problem, however, comes with our read-only data assets (similar to a game's asset files). Several different Docker images might rely on a single set of data assets, and so we'd prefer not to bake them into the actual Docker image itself (plus we want the ability to change the assets without having to recompile the application images).

We're unsure as to the best approach for this. The first idea would be to create a "data container" that simply runs NFS and serves out the data. This successfully isolates the data from the application and allows us to collapse a set of data RPMs into a single tagged docker image, but to me it seems like it might be overkill.

I know that we're essentially looking for a persistent volume for Kubernetes, but the problem for us is bundling all of the data into a single package that has the same delivery convenience as a Docker image.

Is there a better way to provide this read-only data as a version controlled snapshot?

we'd prefer not to bake them into the actual Docker image itself (plus we want the ability to change the assets without having to recompile the application images).

This is indeed good properties to have. You are on the right track.

  The problem, however, comes with our read-only data assets (similar to a game's asset files).

This can be solved with multiple solutions, it's hard to tell what is the best for you. Here are some alternatives.

Persistent Volume - ReadOnlyMany

  I know that we're essentially looking for a persistent volume for Kubernetes, but the problem for us is bundling all of the data into a single package that has the same delivery convenience as a Docker image.

Yes, you could create a Persistent Volume with access modes:

accessModes:
- ReadWriteOnce
- ReadOnlyMany

then (by an automatic process)

Load your data onto the volume with a Job using access mode ReadWriteOnce
Load the volumes to your pods/services using access mode ReadOnlyMany

and keep the volume immutable, you create a new when you need to change content. You can also use multiple volumes this way.

CDN

  our read-only data assets (similar to a game's asset files).

This inherently also sounds what typically is served by Content Deliver Networks. But it depends on your use case if this is a good solution or not.

Buckets

Data for e.g. CDN is usually stored by "buckets" at cloud providers, that may also be an option, then your applications load data using HTTP typically. Minio may be an alternative if you want to do this in your cluster.


Where can I find professional insights into the impact of DevOps upon SecOps? I'm writing about the arrival of DevOps practices such as continuous integration (CI) and continuous delivery (CD) and their impact upon SecOps (or whatever the equivalent team was called). My audiences are SecOps and DevOps professionals who are familiar with industry tools and terminology.

I'm looking for a resource, such as interviews with Security professionals, that describe the impact this had upon security practices. Specifically, the context would be small to medium-sized organizations (approximately 25 to 5000 FTEs) with one or more dedicated software development teams that reorganized as DevOps and implemented CI/CD.

Also, if you would like to share your experiences, I would appreciate hearing about them here. Please describe the organization at a high level; which DevOps practices were adopted/discontinued; which SecOps practices were adopted/discontinued; and particularly any key challenges and how they were overcome.

I know some SE groups are sensitive about receiving open-ended questions like this. However, I think it would be valuable for other visitors to read about and generalize from your professional experiences. If you feel like this question should be placed in a different forum, please let me know and I'll be glad to move it there.

As alluded to by Tensibai ♦, I think the answers you are looking for start from my answer to "What is SecOps?", specifically from the "Further Reading" section:

DevOpsSec by Jim Bird
DevSecCon: A DevSecOps Conference
Velocity EU: Continuous Security

Of the above I am currently reading Jim Bird's book, have had the pleasure of working with the organiser of DevSecCon in a previous role and had the privilege of attending the Velocity Session in Amsterdam last year.

I would advise you to reach out to Francois Raynaud and ask the question of him, he has worked in the space for a long time and is really helping the community evolve as part of DevOps.

Beyond my links, the O'Reilly Security Conference has published videos on Safari Books Online:

2016

New York, NY
Amsterdam, Netherlands
San Francisco, California


Do Docker containers take up space? When I list my Docker images, I can see their sizes:
$ sudo docker image ls

REPOSITORY                TAG                          IMAGE ID       CREATED         SIZE
travisci/ci-ubuntu-1804   packer-1606831264-7957c7a9   0a7a71407638   8 days ago      15.6GB
travisci/ci-ubuntu-1804   packer-1593521720-ca42795e   6093ba41f031   5 months ago    12.3GB
travisci/ci-sardonyx      packer-1547455648-2c98a19    d1b323af5b2a   23 months ago   9.02GB

However, docker containers do not display the space used:
$ sudo docker container ls

CONTAINER ID  IMAGE         COMMAND       CREATED         STATUS         PORTS  NAMES
5794ae898b68  6093ba41f031  "/sbin/init"  10 minutes ago  Up 10 minutes         travis
762e3f8df2da  0a7a71407638  "/sbin/init"  4 hours ago     Up 4 hours            distracted_noyce

Of course, I understand that if you're writing to containers, some space will be used on the hard drive.
But when you create a container, does Docker actually clone the image before booting up the container? Or does it work with diffs, only storing the differences between the image and the current container disk state?
And, how can I see the actual disk space used by a container?

Yes, docker containers when they are running can take up some space as per your workload and application but as mentioned in the other answers - containers are built over layered filesystem therefore only extra space they consume is from some file/object they create/download during runtime which is not the part of base image like some debug log files, etc.
For ex. - When you run 100 containers of a base image of size 1GB, docker will not consume 100 GB of disk space as it leverages layered filesystem and top immutable layer is shared across all containers of same image.
When you stop and remove (yes!, remove explicitly or use --rm flag with docker run command) a container then it's space is reclaimed.
To check space used by containers (and other objects), you can use:
$ docker system df and for detailed info. use verbose flag $ docker system df -v
Initially when you start a container the SIZE column will show 0B but as soon as you start writing to the disk SIZE will change.
EDIT: added more info. as suggested by @Benjamin in comments


Is there a way to export details of all EC2s in an AWS account? I am aware there's a Tag Editor in AWS Console which allows us to export details of all EC2s in an AWS account.
However that list seems to be missing ami and operating system of the EC2.
Is there a way to export details including ami and operating system of each EC2 in an AWS account?

You can use AWS CLI and run the below command to get details of all ec2 instances in your account.
aws ec2 describe-instances

You can also use filters, if required. Please refer the documentation.


Microsoft DevOps: build triggering vs release I have this setup:

Build 1 builds and trigger Release 1 when finished
Build 2 is triggered by Build 1 and release Release 2

If I have one parallel job, can I be sure that:

Release 1 is always run before the Build 2 is triggered?

I have tested this, and it does the above, but I can't find any documentation on the flow anywhere.

I need to release Release 1 before build 2 starts.

Do not trigger Build 2 automatically from Build 1. Instead add a last step to Release 1 to queue Build 2. There are a few ways to do it, e.g. https://stackoverflow.com/questions/52821204/how-to-queue-build-pipeline-as-task-from-release-pipeline


CentOS 7 SSH login with Keys Authentication Is there any possibility to create one master account (for one master laptop for example), from which I can log in to all accounts available on the server ?

My server has Keys Authentication enabled and PasswordAuthentication disabled.

Now, after reboot I can login to server from my (master laptop) as root, but I can not login as a regular user, I get „access denied".

Yes, but you will first need to log in as the master or root account, and then sudo su otheruser in order to log in as that user. There is no way to log in as any user without first going through a privleged user's account.

An additional caveat is that the user you are trying to log in as needs to have a shell of /bin/bash or similar valid shell in /etc/passwd. If the user has a shell like /bin/fase or /bin/nologin you will be unable to change to this user's account (but neither will anyone else for that matter).


Is artifact promotion in this pipeline correct? I have been tasked with rebuilding my DevOps pipeline and I need some help with a few parts.

We will have our code in BitBucket and have something trigger a server to spin up and validate our PR's.  I don't fully understand how to do this at the moment, but its a pretty common pattern so it shouldn't be too bad.  We will run our Unit Tests, do some static code analysis (test coverage, complexity, code quality, etc.).

Next we will have a Jenkins job monitoring master.  It will run the unit test suite again (I am thinking that this is necessary because when merging the PR's we could introduce some errors possibly).  Next it will build the bits and push it to an artifact repository.  This will then start the staging test job.

The staging test job will pick up the bits built by the previous job, or allow any user to start it with a valid staging artifact. It will deploy it to a staging environment and then Run the automated tests here.  If they pass it will somehow mark this artifact as acceptable.  This is my first big question, is it common to move a compiled artifact between repositories to let the team know when it has passed certain quality thresholds?

Next we will have a job that allows anyone to take an artifact that has passed the automated tests and deploy it to some UAT environment where they can do any manual testing or demo's as required.  My big question here is how can we mark this artifact as passed/failed?

Finally we have a job that can push an artifact that has passed manual testing out to production.  My thinking is that this would allow us to push anything deemed acceptable to go to the customer, and would give us rollback if we build it correctly.

To summarize my questions:

How do we mark artifacts that have passed certain stages in the pipeline in an automated fashion?
How do we mark artifacts as passing final stages in a manual way?
Is this missing anything major or have you run in to issues with a similar pattern?

It depends on the artifact repository you are using but in general, you tag the artifacts in the repository to indicate their state.  This can specify it has passed some level of quality gate, some approval, or what ever is required to move through the stages.

I would say you tag the artifact as to it's state, not move between artifact repositories.  A single artifact repository can be used to hold the artifacts at multiple versions, and multiple states.  This provides a single audit point and a single way of controlling the deployment which can be very helpful.

With tagging of the artifacts you can be clear if they are ready for deployment, and if so to which environments.  The tags can be updated in multiple ways, but I would recommend automated processes, such as when an artifact is deployed, the artifact is tagged.  When it set of tests pass the end of that marks the artifact as ready.  If you have manual testing, that testing will be tracked in some tool or dashboard or approval process, when it is approved, that process should tag the artifact as ready to move on.


Using Red Hat Docker image as base image I am considering using https://access.redhat.com/containers/#/registry.access.redhat.com/rhel7 as a base image for my Dockerfile instead of getting some random image in DockerHub. The main reason is the vetting process the images undergo.

Once any additional dependencies are installed, the resulting image will be pushed to a private Docker Hub repository. Is this a valid process or is there a disadvantage that I've missed?

As long as you have access to the FROM image registry, you are totally free to choose where to pull from and where to push to.


Common config in Kubernetes ConfigMap Kubernetes already provides a way to manage configuration with ConfigMap.

However, I have a question/problem here.

If I have multiple applications with different needs deployed in Kubernetes, all these deployments might share and access some common config variables. Is it possible for ConfigMap to use a common config variable?

It is not possible for a ConfigMap to reuse or import other ConfigMaps.

You can, however, have many deployments reuse the same ConfigMap. In this case, you should extract out the common configuration to a single ConfigMap and reuse it across deployments while still allowing each individual deployment to have their own specific config.


Open Ports on Google Cloud Load Balancer It seems that by default, Google Cloud load balancers expose a number of ports unecessarily.  I haven't found a way to expose only 80/443 and every time I make one of their load balancers, the following ports are seen in an nmap:

PORT     STATE  SERVICE
25/tcp   open   smtp
80/tcp   open   http
110/tcp  open   pop3
143/tcp  open   imap
443/tcp  open   https
465/tcp  open   smtps
587/tcp  open   submission
993/tcp  open   imaps
995/tcp  open   pop3s
1720/tcp open   H.323/Q.931
8080/tcp open   http-proxy

Is there a way to block 25, 465, 587, 993 & 995?  Note that this question is aboout GCP load balancers, not firewalls.

You can't add deny rules to GC firewall. The default policy is Deny. You can only add allow rules - allow everything you need and let everything else get rejected.

Since the ports you need to block are allowed by default, you simply need to remove them. Check the name of the default rule:

gcloud compute firewall-rules list [NAME …] [--regexp=REGEXP, -r REGEXP] [--filter=EXPRESSION] [--limit=LIMIT] [--page-size=PAGE_SIZE] [--sort-by=[FIELD,…]] [--uri] [GLOBAL-FLAG …]

and delete it with:

gcloud compute firewall-rules delete NAME [NAME …] [GLOBAL-FLAG …]

You may check here for more detailed explanation on how to handle Google Cloud firewall.


How to order physical backup of S3 data? I'd like to order backup of data to be copied out from S3 to a physical drive (SATA HD) and to be shipped to the address. I know it's possible using Amazon Glacier, however I cannot find such option anywhere. What would be the easiest way to do that?

AWS Import/Export is a service that accelerates transferring data into and out of AWS using physical storage appliances, bypassing the Internet. AWS Import/Export Disk was originally the only service offered by AWS for data transfer by mail. Disk supports transfers data directly onto and off of storage devices you own using the Amazon high-speed internal network.

Import/Export was released on May 2009, and described in this blog post - https://aws.amazon.com/blogs/aws/send-us-that-data/

Released on October of 2016 there is also Amazon Snowball, a service for customers who want to transport terabytes or petabytes of data to and from AWS, or who want to access the storage and compute power of the AWS cloud locally and cost effectively in places where connecting to the internet may not be an option.

Blog post describing the snowball service -
https://aws.amazon.com/blogs/aws/aws-importexport-snowball-transfer-1-petabyte-per-week-using-amazon-owned-storage-appliances/

The documentation explains how to prepare a device for export, and how to create the Amazon S3 Export process.

Here are some older docs:

http://docs.aws.amazon.com/AWSImportExport/latest/DG/PackingS3Jobs.html
http://docs.aws.amazon.com/AWSImportExport/latest/DG/createanexportjob.html


How to gain console access to a MySQL server running on a docker container from the host? I thought I had this working a few minutes ago:
root $
root $ docker container ls
CONTAINER ID        IMAGE                       COMMAND                  CREATED             STATUS                   PORTS                               NAMES
9eb4a18def01        mysql/mysql-server:latest   "/entrypoint.sh mysq…"   6 minutes ago       Up 6 minutes (healthy)   0.0.0.0:3306->3306/tcp, 33060/tcp   mysql
root $
root $ mysql -u root -p
Enter password:
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)
root $
root $ docker exec -ti --user root mysql bash
bash-4.2#
bash-4.2# mysql -u root -p
Enter password:
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 16
Server version: 8.0.21 MySQL Community Server - GPL

Copyright (c) 2000, 2020, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql>
mysql> SELECT host, user FROM mysql.user;
+-----------+------------------+
| host      | user             |
+-----------+------------------+
| localhost | healthchecker    |
| localhost | mysql.infoschema |
| localhost | mysql.session    |
| localhost | mysql.sys        |
| localhost | root             |
+-----------+------------------+
5 rows in set (0.00 sec)

mysql>
mysql> CREATE USER 'user'@'%' IDENTIFIED BY 'password';
Query OK, 0 rows affected (0.09 sec)

mysql>
mysql> GRANT ALL PRIVILEGES ON *.* TO 'user'@'%' WITH GRANT OPTION;
Query OK, 0 rows affected (0.13 sec)

mysql>
mysql> FLUSH PRIVILEGES;
Query OK, 0 rows affected (0.05 sec)

mysql>
mysql> SELECT host, user FROM mysql.user;
+-----------+------------------+
| host      | user             |
+-----------+------------------+
| %         | user             |
| localhost | healthchecker    |
| localhost | mysql.infoschema |
| localhost | mysql.session    |
| localhost | mysql.sys        |
| localhost | root             |
+-----------+------------------+
6 rows in set (0.00 sec)

mysql>
mysql> exit
Bye
bash-4.2#
bash-4.2# mysql -u user -p
Enter password:
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 21
Server version: 8.0.21 MySQL Community Server - GPL

Copyright (c) 2000, 2020, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql>
mysql> exit
Bye
bash-4.2#
bash-4.2# exit
exit
root $
root $ mysql -u user -p
Enter password:
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)
root $

but clearly not.  How do I check that the container connection is accessible?  I'm trying:
root $
root $ telnet 127.0.0.1 3306
Trying 127.0.0.1...
Connected to 127.0.0.1.
Escape character is '^]'.
J
8.0.21%#%T:/]T�e-r&UR([/caching_sha2_password
^CConnection closed by foreign host.
root $
root $
root $ telnet 127.0.0.1 33060
Trying 127.0.0.1...
telnet: Unable to connect to remote host: Connection refused
root $

but that seems a bit odd.

Solution:
root $
root $ mysql -h localhost -P 3306 --protocol=tcp -u user -p
Enter password:
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 17
Server version: 8.0.21 MySQL Community Server - GPL

Copyright (c) 2000, 2020, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql>
mysql> SELECT host, user FROM mysql.user;
+-----------+------------------+
| host      | user             |
+-----------+------------------+
| %         | user             |
| localhost | healthchecker    |
| localhost | mysql.infoschema |
| localhost | mysql.session    |
| localhost | mysql.sys        |
| localhost | root             |
+-----------+------------------+
6 rows in set (0.00 sec)

mysql>

per:
https://stackoverflow.com/q/33001750/4531180


Make container start if docker daemon or host system is restarted First off, I am new to Docker. I was able to create a container based off an image I had previously pulled, start it, but the container is stopped if the docker daemon or the host system are restarted.

So the question is, what is the next step (after creating a container) to make it persistent so that it is started automatically when the daemon is started or restarted?

Thank you in advance.

You need to add a restart policy to your container.

example:

docker run -dit --restart=always hello-world

In this example, I am telling docker to always restart the hello-world container no matter what happens (and also run it in the background interactively). If you do a docker ps, you will see it keeps restarting. To revert this, you need to update the container restart policy like this:

docker update --restart=no <CONTAINER_ID>


How to keep previous containers on docker-compose up? I am running a CI/CD pipeline that builds my service and ends with docker-compose up -d which brings up the service with a new docker image tag (my-service:tag1).

But when I look at docker ps -a the previous container (that was running image service:tag0) does not appear anymore, and I cannot access its logs.

Why does docker-compose remove the previous container?

Is there a way to see the logs of the previous containers?

docker-compose up will remove containers when:

The service name is the same, but has a change, in which case the old container is replaced with the new state of the service
You include the --remove-orphans option when running docker-compose

Otherwise, if you change the name of the service, and do not specify --remove-orphans, you will see "orphaned" containers still running.


What is the meaning of source artifacts that must be in a scripted form? Quote from rom https://www.ibm.com/developerworks/library/a-devops6/index.html

  The key prerequisite to versioning everything is that all source artifacts must be in a scripted form. This goes for the infrastructure, the data, configuration, and the application code. The only exception is for libraries and packages — JAR files and RPM packages, for example — that you use but never modify. After all source artifacts are scripted, you can easily version them.

What do they mean by "source artifacts must be in a scripted form"?

IMO these could be possible synonyms for scripted, in the context of your question here:

in a textual format, which you can edit in some editor, like YAML, XML, JSON, PHP, etc.
NOT in a binary format, the result of some build process (like a .EXE file, etc).

And I bet the reason for the "key prerequisite" (as in your quopted text), is that in the end you want to be able to perform some sort of comparison between 2 versions of such files. E.g. like a patch file (if usig GIT). Think about how a compare of 2 versions of a binary looks like, pretty sure every line is updated (or: all old lines are deleted, and all new lines are inserted).

This applies to pretty much any operating system, be it Linux, Windows, ..., or even zOS (aka good old mainframes).


Check connectivity of 5000 + servers from ansible and get list of hosts which are connectable and which are not I need to check connectivity of hosts on port 43. I can manually login and check connectivity by:

nc -v `hostname` 43
nc -v `hostname_1` 43

Is there any ansible script that goes through all the servers and saves the output of connectable hosts in one file and not connectable in another file?

Maybe a simple shell script would be more useful. If must to use ansible, it could be something like that (assuming you have all servers in group testall):

- hosts: testall
  connection: local
  gather_facts: false
  tasks:

    - wait_for:
        timeout: 2
        port: 43
        host: '{{ (ansible_ssh_host|default(ansible_host))|default(inventory_hostname) }}'
      register: waitfor
      ignore_errors: true

    - copy:
        content: |
          {% for host in groups['testall'] %}
          {% if hostvars[host].waitfor.failed %}{{ hostvars[host].inventory_hostname }} {% endif %}
          {% endfor %}
        dest: /tmp/bad_file
      delegate_to: localhost

    - copy:
        content: |
          {% for host in groups['testall'] %}
          {% if not hostvars[host].waitfor.failed %}{{ hostvars[host].inventory_hostname }} {% endif %}
          {% endfor %}
        dest: /tmp/good_file
      delegate_to: localhost

hosts that have this port opened will be in /tmp/good_file, others are in /tmp/bad_file


How to redirect the request to a different node when there is an unexpected exception and one of the pods gets restarted We are deploying the application in a Kubernetes cluster and when we face an unexpected error and the pod gets restarted or goes down we are getting a 502 Gateway error even though there is another pod that should start accepting the requests without getting the 502 gateway error.
We are making sure that the application is cluster enabled and the request is redirected to the another pod automatically when we stop/kill the pod gracefully but doesnot happen on a exception.

You should configure a Readiness Probe in the Deployment for your app. When using a Readiness Probe, no traffic will be sent to instances that are not prepared to receive traffic.


Where do I use Ansible and where is Terraform used? ( Strengths & Weaknesses) What is Ansible mainly used for?

Where do I use Terraform?

This question focusses on where the tools have relative strengths or weaknesses.

For example:

Ansible for VM build and Software installation.
Terraform for Configuration (Apply and Enforce) and Compliance Enforcements.
Terraform is used to Setup Cloud Environments mainly  etc.

Hope this helps the question be more precisely answered

I think you have to consider the history of the two tools to answer this question properly:

Ansible was created to manage the installation and maintenance of virtual machines, thus it originally only had the capability to install software into an operating system, perform systems administration tasks within that machine.  Subsequently, it has been extended to be able to create cloud resources in AWS, Azure, GCP and others.
Terraform was created to manage cloud resources, it's effectively a facade layer over the various APIs in AWS, Azure, GCP, etc.  It has subsequently been extended to be able to cover many of the functions that Ansible also covers such as installing software.

My personal workflow is to use both applications for what they are really good at, thus I create infrastructure (Virtual Machines, Clusters, Storage, Networks, etc) in Terraform and then use Ansible to install and configure software as required.

I can, and have in some situations, just used Terraform or Ansible, it's just a little harder as you have to learn how to use one tool in a different way.

Bruce Becker pointed out in a comment that one of the main differences is the fact that Terraform is "eventually consistent", in that it does a good job of understanding the current state and then making changes that converge on a "desired state". This workflow is better for the majority of tasks than Ansible. In Ansible you can do a test run with --check, but this doesn't always take into account cascading changes.


Where is the documentation of Kubernetes API server configurations? Questions

Where is the explanations or documentations of the Kubernetes API Server configuration parameters?

Background

There are parameters to configure for Kubernetes API server e.g. KUBE_API_ADDRESS (which seems needs to be changed from 127.0.0.1) for multi node cluster to work.

Looking for a definite configuration documentation of how to configure API server but so far could not find one.

Setup CentOS (Kubernetes.io) says below.

# The address on the local server to listen to.
KUBE_API_ADDRESS="--address=0.0.0.0"

Kubernetes GitHub says below.

# --insecure-bind-address=127.0.0.1: The IP address on which to serve the --insecure-port.
KUBE_API_ADDRESS="--insecure-bind-address=0.0.0.0"

Connecting to Kubernetes API server from outside of Vagrant box #250 says below.

  By default kube-apiserver listens only on 127.0.0.1.
  Without reconfiguring it it is not possible to connect to Kubernetes using kubectl from another machine.

Kubernetes 1.7 /etc/kubernetes/apiserver is below.

###
# kubernetes system config
#
# The following values are used to configure the kube-apiserver
#

# The address on the local server to listen to.
KUBE_API_ADDRESS="--insecure-bind-address=127.0.0.1"

# The port on the local server to listen on.
# KUBE_API_PORT="--port=8080"

# Port minions listen on
# KUBELET_PORT="--kubelet-port=10250"

# Comma separated list of nodes in the etcd cluster
KUBE_ETCD_SERVERS="--etcd-servers=http://127.0.0.1:2379"

# Address range to use for services
KUBE_SERVICE_ADDRESSES="--service-cluster-ip-range=10.254.0.0/16"

# default admission control policies
KUBE_ADMISSION_CONTROL="--admission-control=NamespaceLifecycle,NamespaceExists,LimitRanger,SecurityContextDeny,ServiceAccount,ResourceQuota"

# Add your own!
KUBE_API_ARGS=""

https://github.com/kubernetes/kubernetes/blob/master/cluster/centos/master/scripts/apiserver.sh

# Insecure kube configuration parameters go under here when node['kubernetes']['secure']['enabled'] == 'false'
<% if node['kubernetes']['secure']['enabled'] == 'false' -%>
KUBE_API_ADDRESS="--insecure-bind-address=0.0.0.0"
...

# Secure kube configuration parameters go under here when node['kubernetes']['secure']['enabled'] == 'true'
<% if node['kubernetes']['secure']['enabled'] == 'true' -%>
KUBE_API_ADDRESS="--bind-address=0.0.0.0 --insecure-bind-address=127.0.0.1 "
...

It seems that the KUBE_API_ADDRESS will only listen to 0.0.0.0 if kubernetes has been secured.


In kubernetes, using an nginx ingress controller what is recommended way of adding back end services? I'm running on a simple KVM host and have poor man's load balancing with round robin DNS to and nginx ingress based on this chart which is documented here

So to direct traffic to backend pods it seems from those docs that all you  would need to do is apply appropriate labels and magic will ensue.

It's a bit too magical for me and I'd rather just edit the nginx config and put reverse proxy entries in for the service urls for my microservices.

What's the typical way of doing this?

Ingress controllers allow you to not manually wire stuff up, that's the whole point.

If you'd rather do it manually You could have an nginx array (either inside Kubernetes our outside of it) and just point to the ip/port of the services that expose your microservices. If you use an nginx array external to Kubernetes you'd have to use the externalIP service type. If you run the Nginx array on Kubernetes you could store your "manual proxy config" in a configmap so that all your nginx proxies get the same config.

I would definetely advise to not do this tho. Ingress controllers are built to simplify traffic routing, and as your complexity grows you will definetely want to take advantage of them. A good labeling scheme is absolutely paramount in Kubernetes, and that goes for ingresses aswell.


How to execute shell command or call URL on new gitlab issue? I want to execute a shell command or call a URL when a user creates a new issue. The reason behind this is to synchronize our Gitlab issues with our CalDav calendar infrastructure. We are using Gitlab community.

That's mostly what gitlab webhooks are for I think.

For your case that would be an issue event and you'll have to create something to parse the posted json and do whatever is needed on your CalDav infra to match your needs.


What are ChatOps approach use cases of production infrastructure management? Many "simple" use cases shown in examples of ChatOps approach, creation of cloud instances, deployments to test environment etc., are there any use cases for production infrastructure management?

ChatOps is an absorbing topic; practically it means something very different to each team using it.  So much so that Atlassian have put together what could be considered to be a Chat maturity model.

Sputnik: Trying chat out to see how it works.
Mercury: Starting to move to chat.
Gemini: Chat as the primary source of communication, experimenting with bots.
Apollo: Chat as the nexus of communication and work, including bots.
Elon Musk: Chat is the primary centre of mission critical work.

Important: Chat and ChatOps is very specific to an organisation, think of Chat as the Fabric for your team's bots to existing within it's the bots that provide the functionality that enables business and technical processes.

With the brief lesson in ChatOps Theory over I can talk about the experiences that I have had with ChatOps:

DevOps Support

Slack and PagerDuty has fantastic integration allowing any newly raised incidents in PagerDuty then posted in one or more Slack Channels with information about the incident and buttons for acting upon the knowledge:

Beyond connecting PagerDuty and Slack we also connected all of our monitoring and alerting tools into Slack so that it was the central point for all real and potential problems, including the ability to query New Relic directly from Chat.

Chat as the Modern Command Line

With another client we built custom bots that executed scripts on remote machines, we did this using a combination of calling the management APIs in Azure and running Cluster Fork commands:

  @checky scale serviceA 30: Scale ServiceA to 30 machines.

  @checky kill serviceA#4 : Kill instance number for on ServiceA.

  @checky exec serviceA#1,5 ps -awx : run the ps command on serviceA instances 1 and 5 and post the output to chat

StackOverflow Close Vote Reviewers

We have some chatbots running in the SOCVR channels that help us to:

Find questions that need close votes.
Identify comments that contain "Heat" - i.e. might be offensive or lead to escalations.
Identify questions that are of poor quality that might need review.

Further Reading

So, What is ChatOps? And How do I Get Started?
VictorOps / O'Reilly ChatOps Book
ChatOps Reddit
The Cog Book


How to skip rest of hosts when item found in host - Ansible I have a list of hosts using aws_ec2 and I want to find the first host where the ec2 instance has a specific docker container.
I created a play book with a list of tasks, which ultimately executes a curl command.  I only want to execute it on one host but first I have to find that host based on if the host is running a specific docker image.  I'm using a shell task to execute docker ps | grep my-filter to see if the host has the docker container named "my-filter", once that container is found on that particular host, I'd like to execute the rest of my tasks on that particular host.
I was looking at with_first_found but that seems to only apply to files.
Any help/pointers would be appreciated.

Let's test the use-case with 10 remote hosts (test_01 - test_10) in the group test and user my-user in the hosts test_03 and test_07. To test the existence of the user, use the shell command
shell> cat /etc/passwd|grep my-user

For example the playbook
shell> cat playbook.yml
- hosts: test
  tasks:
    - shell: cat /etc/passwd|grep my-user
      register: result
      ignore_errors: true
    - set_fact:
        results: "{{ results|default({})|
                     combine({item: hostvars[item].result.rc}) }}"
      loop: "{{ ansible_play_hosts }}"
      run_once: true
    - debug:
        var: results
      run_once: true

gives
ok: [test_01] =>
  results:
    test_01: 1
    test_02: 1
    test_03: 0
    test_04: 1
    test_05: 1
    test_06: 1
    test_07: 0
    test_08: 1
    test_09: 1
    test_10: 1

To skip the rest of the hosts, when the first host is found, iterate the list of the hosts and test rc. Then add the host to a new group and run the next play with this group. For example,
shell> cat playbook.yml
- hosts: test
  vars:
    dresult:
      rc: 1
  tasks:
    - block:
        - shell: cat /etc/passwd|grep my-user
          register: result
          loop: "{{ ansible_play_hosts }}"
          delegate_to: "{{ item }}"
          ignore_errors: true
          when: (result|default(dresult)).rc == 1
        - set_fact:
            my_host: "{{ result.results|json_query('[?rc == `0`].item')|first }}"
        - debug:
            var: my_host
        - add_host:
            name: "{{ my_host }}"
            groups: my_group
      run_once: true

- hosts: my_group
  tasks:
    - debug:
        var: inventory_hostname

gives
shell> ansible-playbook playbook.yml

PLAY [test] ****

TASK [shell] ****
failed: [test_01 -> test_01] (item=test_01) => changed=true
  ansible_loop_var: item
  cmd: cat /etc/passwd|grep my-user
  delta: '0:00:00.023271'
  end: '2020-09-25 08:20:31.823168'
  item: test_01
  msg: non-zero return code
  rc: 1
  start: '2020-09-25 08:20:31.799897'
  stderr: ''
  stderr_lines: <omitted>
  stdout: ''
  stdout_lines: <omitted>
failed: [test_01 -> test_02] (item=test_02) => changed=true
  ansible_loop_var: item
  cmd: cat /etc/passwd|grep my-user
  delta: '0:00:00.017642'
  end: '2020-09-25 08:20:33.386657'
  item: test_02
  msg: non-zero return code
  rc: 1
  start: '2020-09-25 08:20:33.369015'
  stderr: ''
  stderr_lines: <omitted>
  stdout: ''
  stdout_lines: <omitted>
changed: [test_01 -> test_03] => (item=test_03)
skipping: [test_01] => (item=test_04)
skipping: [test_01] => (item=test_05)
skipping: [test_01] => (item=test_06)
skipping: [test_01] => (item=test_07)
skipping: [test_01] => (item=test_08)
skipping: [test_01] => (item=test_09)
skipping: [test_01] => (item=test_10)
...ignoring

TASK [set_fact] ****
ok: [test_01]

TASK [debug] ****
ok: [test_01] =>
  my_host: test_03

TASK [add_host] ****
changed: [test_01]

PLAY [my_group] ****

TASK [debug] ****
ok: [test_03] =>
  inventory_hostname: test_03

PLAY RECAP *****
test_01: ok=4 changed=2 unreachable=0 failed=0 skipped=0 rescued=0 ignored=1
test_03: ok=1 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0


Dockerfile COPY command with three input parameters I've encountered a Multi-stage Build example, where the first stage builds a ngnix server, and the second stage creates a ngnix service from that image.

The second stage looks like this:

...
# second stage
FROM scratch
COPY --from=build /etc/passwd /etc/group /etc/
COPY --from=build /usr/local/nginx /usr/local/nginx
COPY index.html /usr/local/nginx/html
...

This is a working example.

Now, notice the first COPY command. It received three parameters(!)

What is the meaning of 3 parameters in the COPY command? do you have a reference to documentation regarding this input?

Notice that the second COPY command uses 2 parameters, which I assumed are src and dest folders.

As per this docker documentation the copy instruction may have multiple sources but only one destination.
Your first copy is coping two file /etc/passwd and /etc/group to /etc.


Internet connection inside Docker container in Kubernetes I have a Kubernetes pod based on a Jenkins/slave container to which I mount a Docker socket and a Docker binary file with necessary kernel modules in privileged mode.

Inside that pod I build a Docker image, in which I run Docker container. Inside that container I don't have Internet connection at all because the pod container uses a flannel network (198.x.x.x) and that container uses bridged docker network (172.x.x.x), which is not available inside the pod container.

How can I make the Internet available inside the second container which is being created inside the Kubernetes pod container?

Using Docker API in Jenkins pipeline is not a solution for me as long as it limits output of error logs and I can not commit changes made in the second container because of immediate removing of that container after build.

In my opinion k8s should not be used to build docker images as it is an orchestration platform, but there is an open issue in the GitHub k8s repository. In conclusion, building docker images on a k8s cluster seems not to be possible at the moment.


Is it a good idea to have a secondary person on call for production operations? The primary person on call could need help in situations such as:

There are too many issues goes on
Primary needs to step out for a short amount of time
There are certain activities that are better taken up by a secondary person so that the primary can focus on more critical issues

So, is it a better practice to have a secondary on call for production operations?  If so, does it make sense for alerts to automatically go to the secondary person in case the primary person doesn't acknowledge them on time?

I can't imagine having only one person on-call at a given time. There are several reasons that the primary person could be unable to handle an issue - perhaps they are unable to receive alerts, they have limited Internet access for a period of time, they or a family member could be ill and they are unable to work yet the on-call rotation hasn't been updated yet. If you have agreements in which you need to start and/or resolve issues in a specified amount of time, having only one person to get notified and start the work seems very risky.

Having a clearly-defined second person is also useful if the first person gets stuck. There's someone who knows that they are on-call and is hopefully doing the best they can to remain in communication and have access to the resources they need. It makes it easy to know who that person is and get help when it's needed outside.

However, if you are having "too many issues going on" or have so many on-call issues that the primary needs to "focus on more critical issues", that would seem to be indicative of poor product or process quality. Situations that require someone being called off-hours should be rare and not the norm, and simultaneous issues should be very rare.

Of course, this is likely to vary by system. Companies that operate at scale of Amazon and Google are vastly different than a B2B service with half a million total users globally who use the system during working hours. Larger, more complex systems with a need for greater up-time are going to have more people on call and shorter thresholds before proceeding down a call tree than smaller systems.