Data Science pipelines and monolithic model blobs Normally, one important topic in DevOps is how we take care of automated creation and delivery of software artefacts.
With the rise of data science there is a new type of artefact - monolithic binary blobs representing a trained neural net for example or other machine learning models. Such a blob can be many GB in size and its creation is not yet standardized AFAIK which brings organizations back to the pre-CI age. Nevertheless, such blobs have their version and associated collections of training data (corpora) which tend to grow rapidly as well.
What are best practices to address this new challenge using DevOps methods - if even possible?

Personally I don't see any reason for which an Artefact Repository - the recommeneded DevOps tool of managing artefacts - wouldn't be applicable to trained neural nets or other artefacts.

The artefact size might have some upper limit for a particular artefact repository, but in such case it would be a technical or policy limitation one, not a fundamental/principial one.

As for applying DevOps methodologies for the process producing these artefacts, I think most if not all of them can be applied equally well, as long as the artefacts:

are produced from some sort of specification which supports change versioning (equivalent to software source code)
are built via a repeatable and automatable process
are validated using some sort of repeatable and automatable verification (similar to QA), eventually using some supporting data (training data in this case, equivalent to DB snapshots, for example)

Side note: monolithic software code delivery is still a big deal and is perfectly maintainable with DevOps methodologies (with a bit of care), not everything can be split in microservices. Size doesn't matter enough to make DevOps not applicable.


Can I remove a docker image and still keep the container? Trying to save space on the server, and I've noticed that the running containers are much smaller in size than the images I've had to pull to create the containers.

I was wondering about the dependency of the running containers on the images  and  if it it's possible to remove the images and only keep the containers.

There are 2 really major points for Docker containers.

You make sure the app dependencies are versioned and encapsulated
inside the container. Container runs smoothly eveywhere, as long as you have the Docker daemon.
You have the capability to create/destroy a
container on any time, also known as ephemeral property. You could test easily, maintain N of them if needed ( Replica Sets in K8S for example are doing just that, Docker Swarm uses services to maintain this number as well ).

The official Docker docs defines an image as

  An image is a read-only template with instructions for creating a
  Docker container

You can't remove an image, because the container relies on it.

docker image rm ubuntu:20.04
Error response from daemon: conflict: unable to remove repository reference "ubuntu:20.04" (must force) - container ecd06b4eb4bf is using its referenced image

Then you might force the process

 docker image rm -f ubuntu:20.04
Untagged: ubuntu:20.04
Untagged: ubuntu@sha256:747d2dbbaaee995098c9792d99bd333c6783ce56150d1b11e333bbceed5c54d7
Untagged: ubuntu@sha256:8bce67040cd0ae39e0beb55bcb976a824d9966d2ac8d2e4bf6119b45505cee64

But that just untags the image, it's still here.

docker image ls
REPOSITORY             TAG                                        IMAGE ID            CREATED             SIZE
<none>                 <none>                                     1d622ef86b13        3 weeks ago         73.9MB

So to your question, the best practice is to only remove images that are not being used by any containers. If you maintain 2-5 versions of your app, you could safely remove the others. Also you schould strive for minimum size of each image, even in total that schould not bother your available space that much.


ECK - Install Plugins: Error privileges My setup:
AWS EKS
* Platform: eks.3
* K8s: 1.18

ECK (Elastic Cloud Kubernetes):
* Deployment via: Operator
* Elasticsearch version: 7.10.1

Summary
I'm trying to install the repository-s3 plugin at Elasticsearch pod.
Here's my YAML:
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: elastic-test
spec:
  version: 7.10.1
  http:
    tls:
      selfSignedCertificate:
        disabled: true
  nodeSets:
  - name: default
    count: 1
    config:
      node.store.allow_mmap: false
    podTemplate:
      metadata:
        annotations:
          traffic.sidecar.istio.io/includeInboundPorts: "*"
          traffic.sidecar.istio.io/excludeOutboundPorts: "9300,443"
          traffic.sidecar.istio.io/excludeInboundPorts: "9300"
      spec:
        automountServiceAccountToken: true
        initContainers:
          - name: install-plugins
            securityContext:
              privileged: true
            command: ['sh', '-c', 'bin/elasticsearch-plugin install --batch repository-s3']

And when I'm applying the deployment crashes:
Exception in thread "main" org.elasticsearch.bootstrap.BootstrapException: java.nio.file.AccessDeniedException: /usr/share/elasticsearch/config/repository-s3
Likely root cause: java.nio.file.AccessDeniedException: /usr/share/elasticsearch/config/repository-s3
    at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:90)
    at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)
    at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
    at java.base/sun.nio.fs.UnixFileSystemProvider.newDirectoryStream(UnixFileSystemProvider.java:431)
    at java.base/java.nio.file.Files.newDirectoryStream(Files.java:476)
    at java.base/java.nio.file.FileTreeWalker.visit(FileTreeWalker.java:300)
    at java.base/java.nio.file.FileTreeWalker.next(FileTreeWalker.java:373)
    at java.base/java.nio.file.Files.walkFileTree(Files.java:2840)
    at org.elasticsearch.common.logging.LogConfigurator.configure(LogConfigurator.java:220)
    at org.elasticsearch.common.logging.LogConfigurator.configure(LogConfigurator.java:129)
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:354)
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170)
    at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:161)
    at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86)
    at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:127)
    at org.elasticsearch.cli.Command.main(Command.java:90)
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126)
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92)
For complete error details, refer to the log at /usr/share/elasticsearch/logs/elastic-test.log

Has someone managed to get around this?

I discovered that it's a know bug of Elasticsearch version (7.10.x), after look this issue and I set to the version 7.9.3 and works fine!


Ansible fetch all vars files from a directory to generate a combined vars file What I'm looking for is a task (or tasks) to read whatever .yml files exist in a specified folder and combine them into a file containing all the variables. For example:

pets/steves_pets.yml:

cats:
  - spot
  - snowy
hamsters:
  - mephistopheles

And

pets/jacks-pets.yml:

dogs:
  - rex
  - woof
hamsters:
  - prometheus

Becomes

wherever/all-pets.yml:

cats:
  - spot
  - snowy
dogs:
  - rex
  - woof
hamsters:
  - mephistopheles
  - prometheus

In fact it's not even necessary to generate an all-pets.yml - as long as the full combined list is available to Ansible for use in roles / tasks. But a combined list file approach would be fine if easier.

Any pointers would be greatly appreciated!

Sounds like you want the combine filter.

You could read each file into a specific variable, then use the combine filter to create all-pets. At that point, it would be in memory of course, so you might want to use the answer to this question to write it to a file:

- name: Read vars
  include_vars:
    file: "pets/{{ item }}_pets.yml"
    var: "{{ item }}"
  loop:
    - steves
    - jacks

- name: Combine vars
  set_fact:
    all_pets: {{ steves | combine(jacks) }}

- name: Write the file
  copy:
    content: "{{ all_pets | to_yaml }}"
    dest: pets/all_pets.yml


How to trigger Jenkins job by deploying git pipline using webhook? I have a Jenkins job which I want to trigger after deploying pipeline in Gitlab, Is this possible with webhooks or some other way?

I just played with it locally on a "safe network", but it may give you an idea.

I put in .git/hooks/post-commit something like:

wget --auth-no-challenge --http-user=$USER --http-password=$APITOKEN --secure-protocol=TLSv1 $JENKINS/job/$JOB/build?token=$JOBTOKEN -O /dev/null -o /dev/null

You need to create API token in jenkins and grab the webhook url for a given job.


How to manage the versioning of Azure SQL Databases? I am looking for a tool that will manage the versioning of Azure SQL Databases, and check for missing schema or migration scripts and apply those changes based on what is missing.

Any other solutions or tips would be great. I just want to make sure the database schema is flat across the board regardless, and apply changes based on version number.

Any suggestions?

I assume from the ansible tag in your question that you need something that can run from a command line, so that rules out plugins to SSMS like dbForge.

Try Flyway (has a free version) or Red Gate (commercial).


Switching between blue / green services in Kubernetes using UI page The project is going with a new technology stack and I need to do a new CI / CD implementation. I started to play with Kubernetes and how to do blue / green deployment via command line. I managed to do switch manually via command line changing label selector in the service to filter green or blue pods. Now, what exactly I am looking for is kind of UI page where QA guys could do a switch between blue and green instances once they finished to test the application on staging server.

So:

Web site is deployed to staging environment (i.e. green);
QA test it and confirm that everything is fine;
QA navigate to some UI to make green instance live and blue to staging;

Are there any tools on the market for this?

Thank you.

Jenkins allows you to construct a CI/CD pipeline that meets your requirements. Here's an example of a pipeline using an input step to decide which environment to deploy to. All you would need to do is wire the stages together.

Here's an example of the pipeline code (declarative) used to create the example:

pipeline {
  agent any
  stages {
    stage('Deploy Staging') {
      steps {
        echo 'deploying to staging'
      }
    }
    stage('Test Staging') {
      steps {
        echo 'testing staging env'
      }
    }
    stage('Blue/Green?') {
      steps {
        input(message: 'Blue/Green?', id: 'deploy', ok: 'deploy?', parameters: [choice(choices: 'blue\ngreen', description: 'Select an environment', name: 'PROD_ENV')])
      }
    }
  }
}

Other tools that provide similar capabilities are Rundeck, Stackstorm or Ansible Tower.


Copy multiple files using * wildcard from kubernetes container I have several files named out1, out2, ... in my Kubernetes container. I want to copy them to my local computer. I am using:

$ kubectl cp pod:/path/out* .

But I am getting an error:

  tar: Removing leading `/' from member names
      tar: /path/out*: Cannot stat: No such file or directory
      tar: Exiting with failure status due to previous errors
      error: path/out* no such file or directory

How can I use kubectl cp with wildcards?

Wildcard support was added to kubectl cp in the #72641 pull request.
It is merged but only available in Kubernetes v1.14.0. Take a look at the ChangeLog:

Notable Features:
[...]
• kubectl supports copying files with wild card
(#72641, @dixudx)

However, as stated on the issue #78854 the PR was made without proper tests and the feature is actually broken. The PR to fix the problem (#78928) is open since June, and it's not merged yet.
Although the PR to fix the feature exists, they are also considering removing the support to wildcard copy.


How can I unify my distributed build environments using Jenkins and Docker? I have recently taken over the CI management for a medium-sized team of developers. Currently, we have one Jenkins server instance as well as six other instances of Jenkins agents to build upon. We have been using labels to determine which builds to run. For example, some instances are setup to run under environments A or B. In the future, we would like to add more instances of A and B, as well as add new environments, e.g., C, D.

My goal now is to unify these build environments and define them in one place to allow for better scale-ability and ease of management of these instances. Docker seems to be a solution to this approach. I know that I can create a Docker image for each of my individual environments. What I don't know is how to distribute the image into containers across my six machine instances. The solution I am looking for involves allowing multiple docker containers to be spun up on a single instance (docker host?), but only if needed (i.e., the scenario when all other hosts are currently hosting a container). Additionally, I need a solution that supports Windows and Linux based hosts.

I know that Docker support in Jenkins seems to be based largely via plugins, so I was hoping for some direction in which plugin to research first.

It sounds like you have a main Jenkins orchestration server and six additional nodes added for build capacity. If you are attempting to improve scalability, there are quite a few options.
If you have containers representing the build environments (A, B, C D), an option would be docker swarm across the existing machines, allowing all of the hosts to be managed as a group and your containers to be dropped on them.
https://github.com/jenkinsci/docker-swarm-plugin
Similarly, but more complicated, create a kubernetes cluster (Openshift, Rancher, etc.) with your current six machines and allow the main Jenkins to spin up and use containers on the clustered kubernetes as needed. This would allow you to create, add additional machines to the cluster, if scaling is needed. You could even add your main Jenkins server to the cluster and run Jenkins as a container.
https://github.com/jenkinsci/kubernetes-plugin
Another option is to add cloud (GCP, Azure, AWS) capacity as a node(s) to push overflow to pay as you need capacity in the cloud.
https://github.com/jenkinsci/google-compute-engine-plugin
Lastly, you could also use your current solution, as is, while developing a cloud native CI/CD pipeline and slowly migrate to a more modular, scaling infrastructure. There are so many options in this space, some not even needing Jenkins, solutions could be tailored to specific needs.
I hope this gives you some ideas and hints!


i have questions about docker-compose postgres:

image: healthcheck/postgres:alpine
environment:
  POSTGRES_USER: postgres

What is the default user of postgresql?
How does hardcoding the postgres user improve security?
What special rights does the postgres user have?
What is the upside and downside of using an unprivileged user in this scenario?
Is it necessary in our scenario to have an unprivileged user?

I searched for the healthcheck/postgres:alpine and I found the following repository with the Dockerfile:

https://github.com/docker-library/healthcheck/blob/master/postgres/Dockerfile.alpine

Assuming this is the correct repo for your docker image, the dockerfile specifies:

FROM postgres

COPY docker-healthcheck /usr/local/bin/

HEALTHCHECK CMD ["docker-healthcheck"]

So it extends the postgres docker image which is documented here:

https://hub.docker.com/_/postgres

Specifically for the password they mention the following:

  POSTGRES_PASSWORD This environment variable is recommended for you to
  use the PostgreSQL image. This environment variable sets the superuser
  password for PostgreSQL. The default superuser is defined by the
  POSTGRES_USER environment variable.

  Note 1: The PostgreSQL image sets up trust authentication locally so
  you may notice a password is not required when connecting from
  localhost (inside the same container). However, a password will be
  required if connecting from a different host/container...

Read the whole documentation at the image homepage.


Ansible: Detecting in parent role when meta dependency role had change I would like to make an ansible role that drops some credentials into /etc for a few services that are running.  Each service is slightly different and requires different mechanisms for restarting to pick up the new credentials.
Is it possible to have a meta dependency somehow communicate back up to the consumer/parent to trigger a handler to restart the services that depend on the file?

I think you can use listen as explained here: https://medium.com/@george.shuklin/listen-feature-for-handlers-in-ansible-29183524c7e1


golang and circle CI setup Go and devops are both new things for me. I'd like to setup a CI that tests and builds my module, then stores artifacts for download. In addition it'd be good to cache dependencies for next builds.
I don't get how to do it using subdirectory for the go product. I want my git repository to look like that:
+ repository_root/
  + go/
    - go.mod
    - go.sum
    + hello/
      - main.go
  + cpp/
  + python/
  + .circleci/

go.mod
module github.com/{USER}/{REPOSITORY_ROOT}/go

go 1.14

Of course {USER} is replace with my github user name and {REPOSITORY_ROOT} is the repo name.
For now I have a following config.yml that doesn't work.
version: 2
jobs:
  build:
    working_directory: ~/project/go
    docker:
      - image: circleci/golang:1.9

    steps:
      - checkout:
          path: ~/project/

      - run: mkdir bin
      - run: go get -v -t -d ./...
      - run: go test -v ./...
      - run: go build -o bin/hello.exe github.com/{USER}/{REPOSITORY_ROOT}/go/hello

      - store_artifacts:
          path: bin/
          destination: hello-app

Error on build command:
#!/bin/bash -eo pipefail
go build -o ./bin/hello.exe github.com/{USER}/{REPO_ROOT}/go/hello
can't load package: package github.com/{USER}/{REPO_ROOT}/go/hello: cannot find package "github.com/{USER}/{REPO_ROOT}/go/hello" in any of:
    /usr/local/go/src/github.com/{USER}/{REPO_ROOT}/go/hello (from $GOROOT)
    /go/src/github.com/{USER}/{REPO_ROOT}/go/hello (from $GOPATH)

Exited with code exit status 1
CircleCI received exit code 1

I know that the module is outside of GOPATH but I have the same setup on my local machine and the same build command works just fine.

My approach nowadays, always package your build process into Dockerfile. I.e. here is Dockerfile for one of my open-sourced Go projects:
https://github.com/relizaio/relizaGoClient/blob/master/Dockerfile
Note: if you're using go modules (which seems to be the case), replace RUN go get -d -v ./... with RUN go mod download in the build-stage. BTW, missing go mod instruction might be the reason for your build failure in the current setup.
Then in your CI, simply do docker build instruction, and then copy your artifact from the container if you need it outside.
Key benefit: you can verify build process against same Dockerfile locally and you can port it to any CI tool whatsoever.


AWS Spot instances price While trying to create spot instances I notices that the Price for almost all instance types (from m3 , c3, c4) are always higher than the On-Demand price of corresponding variants in us-east-1c and us-east-1f regions (for almost 3 months)

Can someone explain why? Is AWS falling short of VMs in those AZs?

You notice that you now have access to 6 availability zones in us-east-1, where you previously only had 4 or 5.

Adding capacity is the reason that new availability zone is now available.

Because of the way the spot market works, there is no way to know what the highest winning bid was, at any given time... because the current market price is based on the lowest winning big for each round.  That sounds counter-intuitive, but here's the logic: given n available instances in the spot market, the system identifies the highest n bids.  These bids are the winning bids.  The lowest of these winning bids is the price all of the winners will pay, and becomes the new current market price.

Each time the market shifts (new bids arrive, or active apot requests are canceled, or on-demand instances are started/stopped, presumably resulting in the available spot pool growing or shrinking), the system re-evaluates all the bids.  If you are no longer a winning bidder, your instance will be terminated, but if you're a new winner, you'll get the instance you've been waiting for.

Based on what's publicly known, we can conclude these things:

the current market price is the lowest bid among all the current winning bidders, assuming there are any winning bidders
you will not get a spot instance right now if you bid below the current market price, but you'll get one if/when the price drops below your current bid.

Therefore, when the spot price is at maximum, there are no unsold spot instances.  We can logically conclude that this is necessarily true, because:

a single winning bid that is lower than the maximum would cause the market price to be lower than thr maximum; therefore, it defies logic that there could be any spare capacity, because this implies that all current winning spot instance bidders made outrageous bids when they could have obtained the instances for substantially less money either by placing a smaller spot bid or by using an on-demand instance
it also defies logic that AWS would artificially inflate the price above the on-demand price, because there is no rational reason for someone to pay more in the spot market when the could use an on-demand instance.

We can further conclude that one of two things has happened to cause this, both of which have the same net result:

there is no spot capacity available at all -- if this is true, it makes sense that the indicated price would stick to the upper maximum value, because you must bid at or above the current price to get an instance now.  The actual market price in a situation where there is no capacity at all would be more properly called NULL because there is no price at which you can obtain a spot instance.  Or, you might say the current market price is infinite, because regardless of how high you bid, you can't have a machine that doesn't exist.  Some might naïvely say the current price is $0 but that is definitely incorrect and would send the wrong market signal.  Where this leaves us is that the most logical representation for the price in a zero-capacity condition would be the maximum/ceiling price.  This gives a clear market signal to any sensible algorithm -- don't bid in this market, the value is not there; or
there is some capacity, but it's all claimed by people who bid the maximum possible.

Which of these two conditions is actually true doesn't matter, because the net impact is the same -- no available capacity.

If that second possibility doesn't make sense to you, that's because you are a rational person.  There are always people who try to game the system.

Instances in the spot market are ephemeral, since the market shifts are outside your control.  I've had spot instances run uninterrupted for many months on end, and I've had them cycle up and down many times over the course of one day.  This is part of the market.

It is possible to improve the chance of a spot instance persisting by placing a high bid... but it is not possible to guarantee that a spot instance will keep running, regardless of how high you bid... because even if all the high bidders colluded with each other and bid the exact same high price, the capacity is finite, so if all current spot instance capacity is sold, not all of the high bidders would have machines allocated to them.

However, there's always that guy who thinks he's smarter than the rest of us, bidding far too high, presumably on the assumption that his instances are likely to continue running, and that since price spikes are transient, he expects only to pay the high price for only a short time.  If there are such gamblers in the market, then they would have the same effect of pushing the price to its limit when capacity is constrained.  This becomes clearest when you consider the fact that if there were only 1 available machine, then only 1 person needs to place an outrageous bid, to set the market price unreasonably high.

The maximum spot price bid is now 10x the on-demand price.  This has not always been the case.  The logical conclusion is that there are indeed those who bid outrageously high, because they are trying to game the system and use spot instances for a persistent workload, so they bid high to try to keep those machines running.  It's a flawed strategy, but it does explain the prices hitting the ceiling and sticking there when capacity is limited.


How to avoid interactive dialogs when running "apt-get upgrade -y" in Ubuntu 16.04 when packaging with Packer? I'm using Packer to create an AWS AMI based on an Ubuntu 16.04 image. In the beginning, I'm doing an upgrade:

sudo apt-get update
sudo apt-get upgrade -y

Here is the relevant part of my provisioners section:

"provisioners": [
  {
    "type": "shell",
    "inline": [
      "sudo apt-get update",
      "sudo apt-get upgrade -y"
    ]
  }
]

This breaks the automatization, however, as an interactive dialog pops up:

amazon-ebs: Found kernel: /boot/vmlinuz-4.4.0-72-generic
amazon-ebs: A new version of /boot/grub/menu.lst is available, but the version installed
amazon-ebs: currently has been locally modified.
amazon-ebs:
amazon-ebs: 1. install the package maintainer's version
amazon-ebs: 2. keep the local version currently installed
amazon-ebs: 3. show the differences between the versions
amazon-ebs: 4. show a side-by-side difference between the versions
amazon-ebs: 5. show a 3-way difference between available versions
amazon-ebs: 6. do a 3-way merge between available versions (experimental)
amazon-ebs: 7. start a new shell to examine the situation

I also tried to set export DEBIAN_FRONTEND=noninteractive before (as recommended in this answer). Unfortunately, it makes no difference.

Questions:

Is there a way to get past the iteractive dialog (selecting option 1 would be fine)?
Is it instead better to avoid upgrades and instead trust that the AMIs are up to date and contain the critical security patches?

Background: This is the relevant part of my "builders" section, where I configured it to use the latest available AMI:

"builders": [{
  "type": "amazon-ebs",
  "region": "eu-central-1",
    ...
    "source_ami_filter": {
        "filters": {
            "virtualization-type": "hvm",
            "name": "*ubuntu-xenial-16.04-amd64-server-*",
            "root-device-type": "ebs"
        },
        "owners": ["099720109477"],
        "most_recent": true
    },
  ...
}]

Note: Turns out that the noniteractive mode works if you run apt-get update with both the -y and the -q flag.

This sequence of commands works for me:

apt-get update
DEBIAN_FRONTEND=noninteractive apt-get upgrade -yq

So, DEBIAN_FRONTEND=noninteractive is correct but you also need the -q flag.

Source: https://github.com/moby/moby/issues/4032


Is Vagrant an operating system by itself? If Vagrant (and similar software) uses to orchestrate, install and continuously upgrade one or more OSs on a single machine, isn't it some kind of an operating system by itself?

Or it actually virtualizes an OS on top of an existing OS and doesn't operate as an OS by itself?

No, Vagrant is a tool for creating virtual environments, which it then can run inside a virtual machine (originally only using Virtualbox, but nowadays also KVM and VMware among others).

The environment created by Vagrant is, however, a fully virtualized OS (e.g. Ubuntu).


Unable to build Image as it is asking some input I am trying to create an Image which will run ftp server on centos. Dockerfile content is as below:
FROM centos

RUN \
  yum update -y && \
  yum install vsftpd -y

EXPOSE 21
CMD vsftpd -DFOREGROUN

I am running below command :
docker build -f webserver .

which is asking me input as below:
Please select the geographic area in which you live. Subsequent configuration
questions will narrow this down by presenting a list of cities, representing
the time zones in which they are located.

  1. Africa      4. Australia  7. Atlantic  10. Pacific  13. Etc
  2. America     5. Arctic     8. Europe    11. SystemV
  3. Antarctica  6. Asia       9. Indian    12. US
Geographic area: 9

once I enter 9 and press enter button, it stuck at the same window. Can you please help me, how should I process this?

The docker build process is non-interactive. You have to find a way to make it not ask for anything. The way to do that is by creating the necessary config files and placing them in the correct locations through the Dockerfile before installing vsftpd. The questions are there simply for vsftpd to generate a/some configuration file[s] for itself. You create one beforehand and place it correctly using the copy command before run yum install. If vsftpd finds the valid config file[s], it won't ask for anything anymore.


Docker machine connection with https I am new to Docker. I installed Docker tool box on my Windows machine.

Using pom.xml I want to generate docker image .
pom.xml:-

<plugin>
    <groupId>com.spotify</groupId>
    <artifactId>docker-maven-plugin</artifactId>
    <version>0.4.11</version>
    <configuration>
        <repository>${docker.image.prefix}/${project.artifactId}</repository>
        <imageName>${docker.image.prefix}/${project.artifactId}</imageName>
        <dockerDirectory>${project.basedir}/src/docker</dockerDirectory>
        <dockerHost>https://192.168.99.102:2376</dockerHost>
        <dockerCertPath>C:/Users/shridhar/.docker/machine/machines/default</dockerCertPath>
        <forceTags>true</forceTags>
        <imageTags>
            <imageTag>${projectVersion}</imageTag>
            <imageTag>latest</imageTag>
        </imageTags>

        <!-- push the image on docker hub -->
        <serverId>docker-hub</serverId>
        <registryUrl>https://hub.docker.com</registryUrl>
        <resources>
            <resource>
                <!-- war file locaion wheere war should be present (copied) -->
                <targetPath>/   </targetPath>
                <directory>${project.build.directory}</directory>
                <include>${project.build.finalName}.war</include>
            </resource>
        </resources>
    </configuration>
</plugin>

Docker machine details

$ env | grep 'DOCKER'

DOCKER_HOST=tcp://192.168.99.102:2376
DOCKER_MACHINE_NAME=default
DOCKER_TLS_VERIFY=1
DOCKER_TOOLBOX_INSTALL_PATH=C:\Program Files\Docker Toolbox
DOCKER_CERT_PATH=C:\Users\shridhar\.docker\machine\machines\default

I get the below exception:

Caused by: com.spotify.docker.client.exceptions.DockerException: java.util.concurrent.ExecutionException: com.spotify.docker.client.shaded.javax.ws.rs.ProcessingException: org.apache.http.client.ClientProtocolException

If I make tcp connection

<dockerHost>tcp://192.168.99.102:2376</dockerHost>

then I get below exception:

Caused by: java.lang.IllegalArgumentException: An HTTPS URI for DOCKER_HOST must be provided to use Docker client certificates

What changes do I have to make to build image sucessfully?

Considering you say you want to build an image (and not push it to Docker hub or wherever):

docker-maven-plugin project recommends using dockerfile-maven for new projects, as the former

  [...] ultimately led to a lot of unnecessary confusion with our users that stemmed from introducing extra abstractions and a need for configuration on top of what Docker is providing.

The dockerfile-maven README should get you started, with both a sample Dockerfile and  the relevant plugin section for your pom.xml


How do -dev apk packages differ on Alpine for the purposes of creating docker images? I recently started a new position at a company that is transitioning over to docker for local and prod, and inherited a few 'base image' repositories for php, node and a few others.

Unfortunately the creator is no longer at the company, so I'm going through the system and trying to piece it together. None of the images are used in production yet, and it's fair to say they are probably not road ready yet.

The images are based off alpine and I noticed particularly in the php image that most of the apk dependencies are using -dev distros:

apk add --no-cache $PHPIZE_DEPS bzip2-dev freetype-dev gmp-dev icu-dev ...

I couldn't easily find out the difference between, for example, bzip2 and bzip2-dev but my assumption would be that these packages are aimed at people who develop those packages. Next best guess would be they have some advantage for general developers (php in this example) but if so I'm not sure what those benefits would be.

What is the purpose of the -dev packages, and perhaps some guidance on where they would be appropriate (e.g. local only or fine for prod?)

dev packages are typically intended for development and build purposes. They usually contain header files and other resources for providing programmer APIs. For example, python-dev contains C headers for C/C++ applications that integrate python functionalities (which of course isn't necessarily python itself).

The "non-dev" packages provide the actual application or library: for example, python provides the python interpreter binary, the libpython3.so shared library, and compiled python modules.

For inspecting the contents of specific packages, you could use Alpine's excellent package explorer: locate the package of interest (for example, python3-dev), and click the "Contents of package " link at the bottom, to see the complete list of files installed by the package.


ansible does not play roles and tasks I'm learning ansible and I'm having a problem, when I run my playbook, it only runs the gather_fact task and does not play my roles.  On the other hand when I run just a module with the ansible command -m or playing the playbook locally, there my roles are played. Do you have an idea ? I posted the question on the google group, but for the moment no answers. thank you in advance

If instead of using the command line:
ansible-playbook -u root --private -key ~ / .ssh / id_rsa centos7.yml -i xxx.xxx.xxx.xxx,

I use:

ansible-playbook -u root centos7.yml -i xxx.xxx.xxx.xxx,

It works.


Porting a docker container to a different Linux host OS I'm new to Docker. I have looked at many tutorials, however, this is still not clear to me :

For example, If I create a docker image on a system which is using Ubuntu as the host OS. Can I then later port and use this same docker image on a different computer using a different flavour of Linux for the host OS, for example CentOS or any other?

Yes, as long as you have the Docker daemon running on the CentOS. Docker heavily utilizes a set of kernel features - namespaces and cgroups. That set comes native for all Linux derivatives.

In terms of 'porting' the image, you have 2 widely used options:

docker save on your Ubuntu host. Get the saved archive to your CentOS and load the image with docker load
The above option is not so convenient for the standard CI/CD stack, so you might just push your image from your Ubuntu host to a so called registry. Once the image is in your registry, you could pull it from your CentOS host.


OMD vs Ubuntu install issues When installing OMD 1.3 on Ubuntu, I get the following output:

dpkg: dependency problems prevent configuration of omd-1.31.20160708:
omd-1.31.20160708 depends on libapache2-mod-proxy-html; however: Package libapache2-mod-proxy-html is not installed.
omd-1.31.20160708 depends on apache2-mpm-prefork; however: Package apache2-mpm-prefork is not installed.
omd-1.31.20160708 depends on libboost-program-options1.54.0; however: Package libboost-program-options1.54.0 is not installed.
omd-1.31.20160708 depends on libboost-system1.54.0; however: Package libboost-system1.54.0 is not installed.
omd-1.31.20160708 depends on libperl5.18; however: Package libperl5.18 is not installed.
omd-1.31.20160708 depends on php5-cli; however: Package php5-cli is not installed.
omd-1.31.20160708 depends on php5-cgi; however: Package php5-cgi is not installed.
omd-1.31.20160708 depends on php5-gd; however: Package php5-gd is not installed.
omd-1.31.20160708 depends on php5-mcrypt; however: Package php5-mcrypt is not installed.
omd-1.31.20160708 depends on php5-sqlite; howeve
dpkg: error processing package omd-1.31.20160708 (--install): dependency problems - leaving unconfigured

I have updated and upgraded my system quite a few times and still issue persists and has confirmed that the dependencies exist.

Does anybody know if the above packages and their versions are required? I have the dependencies installed, but on the latest version and I am assuming that OMD and the packages that comes with it requires the older version.

Any advise or solutions to this issue?

System:Ubuntu 16.04.3 LTS OMD Version: 1.31.20160708

According to How to let dpkg -i install dependencies for me? it sounds like dpkg doesn't have support for automatically installing a package's dependencies.

Personally I'd switch to a package manager which has such support, like apt-get. You'll like it ;)

Alternatively you could:

use a "helper" application around dpkg as suggested in the above-mentioned answer
manually collect the entire list of missing dependencies and install them as well. But it can be very tedious.

  I have the dependencies installed, but on the latest version

The version is part of the dependency, so technically the above means you don't actually have the dependencies installed. At least not for that particular version of OMG.

  and assuming that OMD and the packages that comes with it requires the
  older version?

Bingo. You have 2 options:

re-create an older version of your system which is compatible with your particular version of OMD. It could be based on Docker or some VM technology (or some plain old hw you may have lying around). I won't consider downgrading your system a solution :)
look for a newer version of OMD, hopefully one that's happy with what you have installed and on which its dependencies can be installed (some packages suport multiple versions installed on the same system). Also note from omdistro.org, you may be looking for another package:

  With the growing popularity of OMD, we came to a point were it makes
  sense to develop two specialized versions of OMD.

  Check_MK Raw: Check_MK Raw Edition focuses on Check_MK. Use this if you mainly use Check_MK. This is a smaller package including
  all components to run Check_MK.
  OMD-Labs: OMD-Labs contains all components of the original omd and some more addons like Grafana and Influxdb as well as Naemon and
  Icinga 2.


Monitoring immutable infrastructure virtual machines Currently I am using the HashiCorp stack (Packer, Terraform) and Ansible, to generate Immutable Infrastructure and Infrastructure as Code.

This builds a Virtual Machine from a base image, provisions it and applies it to some VSphere servers.

One of the issues I'm having is knowing when a VM is down or unresponsive.

What would be a good way of monitoring VMs?

I've looked a bit at Consul and Prometheus and am thinking that that is the way forward?

I've looked a bit at Consul and Prometheus and am thinking that that is the way forward?

In short, yes.

Consul is designed exactly for this - it will run health checks against all of your services that you define and can detect when things go down. However, what it does not do it alert when things do go down, it is more designed for automatically taking things out of circulation when they are unhealthy. But it does give you a single point you can query for the health of your infrastructure which you can then alert on manually somehow or combine it with consul-alerts to do this for you.

Prometheus is geared towards metrics and can monitor the health of your applications in ways that consul is not really well suited. It has its only able to alert when defined events occur and you can tune it to monitor exactly what you require.

You might also want to consider external checks like what services such as panopta offer  (this is just an example, there are many alternatives out there) to check public endpoints to make sure they are still working from an external perspective.


How to add a new column with custom value extracted from the job? The current list of Jenkins jobs has 4 columns:

Name
Last Success
Last Failure
Last Duration

Is there any way to add a new column which would have some custom value extracted from the job?

From what I can tell you can definitely customize the columns for a custom view (I can't find where to change it for the "All" view, but it's probably possible).  Unfortunately it doesn't look trivial to add a custom column without writing a plugin, for example the jacoco plugin:

https://github.com/jenkinsci/jacoco-plugin/blob/master/src/main/java/hudson/plugins/jacococoveragecolumn/JaCoCoColumn.java

Jenkins Java Doc:

http://javadoc.jenkins.io/hudson/views/ListViewColumn.html

Edit: This plugin might get you closer to what you're looking for:

https://wiki.jenkins-ci.org/display/JENKINS/Extra+Columns+Plugin


Can a kubernetes pod be forced to stay alive after its main command fails? After starting a long running kubernetes job we've found that the final result will fail to upload to its final location. We would like to force the pod to stay open after the main process fails so we can exec in and manually process the final results. If the main process fails and the pod exits before uploading the final result we will lose a large amount of time to re-process the job.

Is there a way to ensure the pod stays alive manually?

Keeping your pod alive after its main process exited is an anti-pattern and not possible in K8s.

The entire concept of a K8s Job is to run till completion. So what you are looking to do is neither ideal nor recommended.

It would be better to change the entrypoint command of your container to include a retry/error handling mechanism in case of failure.

With that said, you do have ways in which you can debug the failed pod.

If your Pod's container process exits due to error, the Pod will be marked with Status: Failed

But you can still debug the failed pod by attaching an Ephemeral container: https://kubernetes.io/docs/tasks/debug-application-cluster/debug-running-pod/#debugging-with-ephemeral-debug-container

Note: Ephemeral containers are available as K8s alpha feature only from v1.16


REST API: create project with powershell returns unspecified internal error I'm creating a a project with the code below.
The response is always an unspecified error.

Invoke-RestMethod :
{"$id":"1","innerException":null,"message":"TF400898: An Internal
Error Occurred. Activity Id:
3fbed4b2-9751-41ba-acbe-4d20f02ae556.","typeName":"System.Web.Http.HttpResponseException,
System.Web.Http","typeKey":"HttpResponseException","errorCode":0,"eventId":0}
At line:1 char:1
Invoke-RestMethod -Uri $adoTaskUri -ContentType "application/json-pat ...
~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~
CategoryInfo          : InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Invoke-RestMethod], WebException
FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeRestMethodCommand

I tried the body in postman and the projects are created just fine.
Can someone see what I'm doing wrong?
Listing of the projects works; the body works in postman; I can create tasks with a similar script.
$pacToken = ""
$organizationName = "BLECUST"

$adoHeader = @{"Accept"="application/json";Authorization=("Basic {0}" -f [Convert]::ToBase64String([Text.Encoding]::ASCII.GetBytes(("{0}:{1}" -f "",$pacToken))))}
#$adoHeader #accept about anything and add security in the header

#api to create
$adoTaskUri = "https://dev.azure.com/BLECUST/_apis/projects?api-version=5.1"

#list all projects
$response = Invoke-RestMethod -Uri $adoTaskUri -ContentType "application/json-patch+json"  -headers $adoHeader -Method GET
$response.value | ft name, visibility, url

$body='{
  "name": "FabrikamTravel18",
  "description": "Frabrikam travel app for Windows Phone",
  "capabilities": {
    "versioncontrol": {
      "sourceControlType": "Git"
    },
    "processTemplate": {
      "templateTypeId": "6b724908-ef14-45cf-84f8-768b5384da45"
    }
  }
}'

Invoke-RestMethod -Uri $adoTaskUri -ContentType "application/json-patch+json" -Body $body-headers $adoHeader -Method POST

Your script works, just update the last content type to application/json:
Invoke-RestMethod -Uri $adoTaskUri -ContentType "application/json" -Body $body -headers $adoHeader -Method POST

PATCH is used to update a team project: https://docs.microsoft.com/en-us/rest/api/azure/devops/core/projects/update?view=azure-devops-rest-6.1


What options are available to get a value that is associated to a key in yaml? I often used jq to lookup a value in json. How to do this in yaml? I do not want to use something like sed as too much characters have to be removed then.

One could use go-yq if one wants to lookup a value by applying jq style, i.e. if firefox_version then one could look it up by adding a dot:

docker run -v ${PWD}:/ansible-firefox utrecht/go-yq:2.1.0 \
       .firefox_version /ansible-firefox/defaults/main.yml


How to optimize blue/green AWS deployments? I got the following setup with AWS code deploy:

However it takes about 7 minutes from the moment I push a git branch remotely to the moment the deploy is completed. The time is broken down like so:

bitbucket deploy: 45 seconds

provisioning a new EC2 instance: 3:40 minutes

installing application on replacement instance: 40 seconds

2 minutes rerouting traffic from old instance to new one *this one is extremely long, if it were done manually it would literally take 10 seconds)
17 seconds terminating second instance

All together it's around 7 minutes.

How can I optimize this? I'm looking to shrink it to below 1 minute.

bitbucket deploy: 45 seconds

This seems correct. Not sure of your repo size, but if it is hosted outside of your region, that's not out of the ordinary.

  bitbucket deploy: 45 seconds

Again, seems correct. Not much room for optimization here.

  provisioning a new ec2 instance: 3:40 minutes

What size of instance are you provisioning? Are you provisioning one or more instances? The obvious solution would be to use in place deployment vs blue-green, but I think the extra time is well worth it. If this is for production/staging, I think you are fine. If this is in a testing environment, and you are aiming for quick feed back, you could consider provisioning new instances after a successful blue-green so that, if there is no AMI change, they are ready to go for installation immediately.

  installing application on replacement instance: 40 seconds

Nothing to optimize here

  2 minutes rerouting traffic from old instance to new one *this one is extremely long, if it were done manually it would literally take 10 seconds)

My guess is that step is performing extra steps that you do not perform when you manually reroute traffic. It's possible that CodeDeploy is performing extra checks to make sure that connections are draining, new connections are established before old ones are closed, and other test/checks. This is probably your best area to optimize, but not at the sacrifice of checks and balances.

  17 seconds terminating second instance

Nothing to optimize here.

Summary:
I don't think it is feasible or worthwhile to attempt to optimize your deployment further. The amount of effort it would take and sacrifices you would need to make in your test and checks far out ways the potential minute or two you would save in my opinion. I also don't think it is possible to achieve a <1 minute deployment. That would average ~15-20 seconds a step, which would require a lot of sacrifices in your deployment strategy.

If your goal is for quick iterations for QA, code testing, integration testing, etc., and you absolutely want sub 1 minute deployments, I recommend creating a Cloud Formation stack with your exact deployment except skipping the blue/green deployment model for an in place deployment model. This way, you can quickly see the results of your code. You should then later run your full suite blue/green deployment after your testing in either another testing environment or your staging environment.


How to deploy a custom build of a Jenkins core plugin? How to deploy a custom build of a core plugin bundled in Jenkins?

This section of the official Jenkins Plugin Tutorial will probably be of use.

According to the writers of that documentation, you need to copy your .hpi file to $JENKINS_HOME/plugins, remove the plugin's development directory, and finally create an empty file called .hpi.pinned for maven to use. Then you should be ready to restart Jenkins and use your custom build of a core plugin.

They also suggest running mvn clean when you're ready to update Jenkins to remove old files (after changing core plugins, old .jar files might still be around after updating).


How to get information about resources in an Azure Pipelines run? I work in a platform team that builds and supports Azure Pipelines YAML templates and various custom scripting for these pipelines. People in my organisation use these YAML templates like:
resources:
  repositories:
    - repository: templates
      type: git
      name: PROJECT/cicd_stuff
      ref: refs/heads/releases/v0.21
extends:
  template: our_awesome_template.yml@templates
  parameters:
    ...

In order to reliably match the version of our scripts to the version of the YAML templates, we now include an extra build stage in our pipeline templates that checks out the repo templates repo and puts all our scripts in an artifact to be used in that run. Using "release branches" allows us to safely put out and test new features and fixes to our pipelines: Teams can upgrade their pipeline on their own pace, with older version remaining supported for a while.
I'd like to start using Azure Artifacts for our script artifacts, but now I'm wondering "how can I determine which version of my scripts I should be downloading from Azure Artifacts?" The information included in the resources block would work for me, but I can't seem to access that with an expression or with a predefined variable. The only solution I can currently think of is to use the az pipelines cli. Any thoughts?

You can do:
variables:
  templates.ref: $[ resources.repositories['templates'].ref ]

See Repository Details docs.


Container Image Promotion across environments in ECR/ECS We have started doing some initial work on deploying containers on ECS Fargate service.
However we are not yet completely figured out how to manage our image promotion across different environments.
In our case, developers managed their own environment so they can deploy the docker image specifying just the tag of something they have built.
We only have UAT and PROD environments , As of now whatever we have in develop branch got baked in as image and pushed to ECR as develop tag and that gets deployed to UAT.
we release the code from develop to master as part of our release process.
During the release process ( from develop to master merge ) we create another container image this time on merge to master and that gets pushed to ECR as master tag which gets deployed to PROD.
I think this process is not ideal and it can be improved by a lot margin.

We shouldn't need to create image on each stage and should promote the same image tag in across environments ( i.e whatever we have in UAT will get QA/TESTED and released to PROD as artifact ), how are you managing this image promotion across environment ? what are other release process look like ?

We are currently tagging our image as branch tag i.e develop or master , How are you managing these tags creation ? Are you using GIT SHA ? or Sem ver ?

Any other improvements or general comments can be really helpful as well.

Although difficult to give a canonical answer, I think I understand your frustration. My opinion is  that you are focussing on promoting the wrong thing - the image.
I would start with the concept of "Dev-Prod parity" (or in your case, UAT/Prod parity). This concept says:

Keep development, staging, and production as similar as possible

The simplest way to do this is to use the same image in the two environments. The difference between them comes from the configuration and backing services. These are set at run time, not at build time - ie they are changed when you start the container not when you build it.
So, your process should look like:

Build the image, tag it with uat
UAT environment discovers new image, new image gets deployed
Run UAT against new deploy, they pass
tag that same image with latest
Prod discovers new image, gets deployed.

This process is extremely lightweight and you will have certainty in what is deployed.
TL;DR instead of promoting images and having to rebuild them, just shift the tag used in prod.


How can I check for an installed packaged version on multiple linux hosts? i need to check more than 100 servers for the snapd package, but I don't like the output at all. I searched all day for different options without luck. Attached is the output of ansible-playbook. Any hints guys?
---
 - hosts: centos
   become: true
   tasks:
    - name: Check Hostname
      command: /usr/bin/hostname

    - name: Check for package if is installed
      yum:
        list: snapd
      register: result

For checking the package version I tried with the following playbook but Ansible doesn't like the syntax:
---
 - hosts: test2
   become: true
   tasks:
    - name: Check Hostname
      command: /usr/bin/hostname

    - name: Check for package if is installed
      yum:
        list: snapd
      register: package_name_version

    - name: set package version
      set_fact:
        package_name_version: "{{ package_name_version.results|selectattr('yumstate','equalto','installed')|map(attribute='version')|list|first }}"

Thanks,
Juls

A simple way would be to do:

ansible -i <inventory> <target group, lets' say some CentOS systems> -m shell -a "rpm -qa | grep <package name>"

That would display a list consisting of server name and package version.


How frequently do PaaS:VPS agencies upgrade OS_release in comparison to PaaS:shared-webserver? I know that in a In PaaS:shared-webserver service a user could have the OS of its server environment release_upgraded in 8 or 12 years or so ! For example, I host a few sites on a particular host that uses a very old version of CentOS (from about 2007).

I never used a PaaS:VPS service (like AWS/MSA/GCP) and I wonder if they have a different standard of doing OS release_upgrades; for example, I wonder if due to utilization of OS deployment tools like Vagrant, it might it be say per each new release (say, each 2 years) instead?

How frequently do PaaS:VPS agencies upgrade OS_release in comparison to PaaS:shared-webserver?

From my experience this is very provider specific and requires evaluating the SLA offered for a particular service.

For example the siteground cloud server offerings shown at
https://www.siteground.com/cloud-hosting.htm state: We preinstall and maintain all the software you need on your server.

While offerings from liquidweb they make it clear they will customize the support level to fit your requirements (the custom levels of management are detailed here). You can chose the support level for the product you purchase, whether it's a cloud VPS or dedicated server hardware.

IMHO most cloud products will probably be vendor managed and provide you a way to perform needed customization. But general statements are not useful when it comes to spending money. You have to confirm what your vendor is providing.


What is the proper term for lead time from commit to deployment? I know for example following DevOps relevant concepts related to time:

MTTR - mean time to recovery
Sprint length - fixed agreed time to hand over features to customer (what this specificall implies is part of DoD I think)

Now my question - how do you name the lead time from a code commit hitting the repository to a first running deployment where you could do a first acceptance test to the feature? To a production system?

From code repository to any environment, as a running software would be "Time to Deploy" ( which is different from deployment time, btw ).

From code repository all the way to production environment, the best aproximation is time to market (TTM), taking aside that time to market does involve the conceptual design of a product.

But as you said you are tagging an pipeline, I should advise:

Use the tags that makes sense: to you, your team and to your product.

A common pipeline could be : BUILD > TEST > DEPLOY

But we could tag the same pipeline, making more steps visible, ie :
SOURCE CHECKOUT > CODE CHECK > UNITARY TESTS > BUILD > INTEGRATED TESTS > DEPLOY > ACCEPTANCE TESTS

As a rule, if you have trouble on one step and must check the output to understand whats going on, this step could be divided in one or more tags.

If makes sense to you, you could even tag something like SOURCE CHECKOUT IDLE TIME ( the time took for the build machine to connect to the repository server, before downloading the source )

Hope this could be useful, happy tagging!


Jenkins multi branch pipeline configuration for bitbucket How to configure multi branch  pipeline job for bitbucket ....i have to clone from different branches and build the jobs.... I  found multi branch is the option... How to configure for bit bucket

I recommend using the BitBucket Branch Source Plugin when using BitBucket. Documentation can be found here


Convert AnsibleUnsafeText to String I am using the ansible shell module to compare the chain of 2 SSL certificate with openssl (they have the same chain):
openssl x509 -issuer -noout -in Certiticate1
openssl x509 -issuer -noout -in Certiticate2

I register the output in variables chain_from_certificate1 and chain_from_certificate2 then i compares there standard output:
- name: Compare the Chain
    fail:
      msg: Error The chains aren't the same
    when: chain_from_certificate1.stdout !=  chain_from_certificate2.stdout

Ansible tell me that the chains are not similar (and it's false because they are).
I think that the issue came from the comparison operator because the type of the variable are "AnsibleUnsafeText" and not string.
My question how to compare 2 AnsibleUnsafeText ? and it's necessary to convert them to string before comparison ?
Thank you in advance.

I believe this is due to non existing variable attribute.
Most likely chain_from_certificate1 and chain_from_certificate2 don't have stdout
Did you try debug by printing the variables?
With stdout:
Did this throw similar failure?
    - name: print variable
      debug:
        msg: "{{ chain_from_certificate1.stdout }}"

Without stdout:
What is output of this?
Will this print same output as of manually execute openssl x509 -issuer -noout -in Certiticate1?
    - name: print variable
      debug:
        msg: "{{ chain_from_certificate1 }}"

My assumption is that the script set the value of stdout in chain_from_certificate1 and chain_from_certificate2 rather than the stdout: <value>. Therefor stdout do not exist as attribute.
Or chain_from_certificate1 and chain_from_certificate2 contain another data structure.
The key is to print them out.


How to get into an App Container Manually with Garden-RunC Backend? So I have been trying this-tutorial to access the runC container from the diego-cell VM. However, when I try to run

/var/vcap/packages/runc/bin/runc exec -t <instance_id> /bin/bash

I get the error

exec failed: container "<id>" does not exist

when I run cfdot for the given application it does list this instance-id,
also when navigating into the location

/var/vcap/data/garden/depot/<container_id> it exists.

But runc in unable to open the container, also when I run

runc list

the list of running containers in the vm is empty

diego-cell/49dbeca6-36eb-4910-94b9-48960891a0b1:~# /var/vcap/packages/runc/bin/runc list
ID          PID         STATUS      BUNDLE      CREATED     OWNER
diego-cell/49dbeca6-36eb-4910-94b9-48960891a0b1:~#

I've run all as root as mentioned in the link.

Why am I not able to access the runC containers, What am I missing?

P.S: Although the intent is same, I'm not looking for cf ssh

I suspect this may be something coming from containerd changes - running PCF 2.5 the runc list works fine, 2.6 runc list shows nothing in the inventory.  I have not yet found the equivalent command that will allow me to find the container id and how to execute a shell entry to that container instance yet, but I'm hot on the trail.


How to structure a IaC for production, and development? We have a typical multi layered web application:  a balancer, 2 web server 2 db server.

I feel we strongly need to start using a kind of Infrastructure As Code in order to replicate easily environments, testing infrastructure updates, control production servers.

I've never done that and I'm struggling to understand the workflow.
I'm thinking something like this:

create locally my machines with docker and Dockerfiles.
use terraform to setup locally the infrastructure provided by docker (or docker-compose?).
reuse in some way the Dockerfiles to provide the production or staging machine (we don't use Docker in production, but rather AWS with EC2/AMI and RDS).
use ansible to deploy the application.

I'm not sure this is the way. Or maybe it's too complex

Ok so I understand that you are in AWS and Docker already and Terraform is something you want to bring into picture for IaaC. and Yes, docker for production. Now, let's start with environment:
Assume you have three environments, Dev, Test and Prod. So you create three folders, Dev, Test and Prod which contain infra code for three environments. Now, I don't understand what you mean by infrastructure provided by docker but docker will contain your web servers and web application. Number of containers will depend on how large or small your application is and are you in micro services architecture. So what Terraform does for you is creating almost everything: Right from network structure(VPC, subnets, NACLs, gateways etc), autoscaling groups(I prefer this as you may want to use docker orchestration), security groups, IAM roles, policies to RDS and ECS etc.

So you can tag them in your code as per their environment and you create each of these components using Terraform modules.

So now you have Terraform code which creates infrastructure for you. Store your state file in S3 because you want consistency in your infrastructure state. Keep your database state file separate from other state files. You may want to have separate state files for your network, security groups, RDS, IAM, Cloudwatch etc.

Now comes the deployment part: Since you are in Docker, why don't you use AWS ECS? It will be easy deployment (Almost zero downtime) and orchestration will be managed by ECS itself. So now even deployment you can do using Terraform. Maintain a separate state file for this too.

You can create separate docker images for dev, test and production during your CI. If you do not have any docker tagging strategy, you may use $BUILD_NUMBER:$ENV for now. One of the purpose of using docker is that you don't need to care about where you want to run the container so identical docker images for Dev, Test and Production is almost always ok with some configuration changes.

You do not create anything locally. Use CI/CD tools such as combo of Jenkins and Git which triggers everything.


localhost Pod to Pod communication k8s I am trying to transfer an old legacy system to k8s.

I have a custom server that client apps connect to in order to communicate with one another. Both the server and clients were originally created to run on the same system and communicate using localhost.

I Dockerized both server and client apps. I tested that they work well in the same pod (everything runs well using localhost). Now I would like to separate the server and apps into their own pods. I would like the apps to automatically connect to the server.

Question: How do I connect the client apps to the server without changing the source code of the client apps or server?

Are you basically saying that it’s hard coded to “localhost” and you need to somehow change the meaning of “localhost” because you cannot change the source code?

If so, it sounds fairly hard to achieve... but maybe you can override the meaning of localhost in the hosts file of the pod (the right way).

https://kubernetes.io/docs/concepts/services-networking/add-entries-to-pod-etc-hosts-with-host-aliases/

I feel like this will probably have unintended side effects and may affect kubernetes, but I could be wrong.  It’s definitely worth a try.

Note: the overrides can point to services created over the pod deployments if it works :).  They can have reliable internal domain names that don’t move around.


OpenShift fails to deploy jenkins - connection refused I am using the GitHub project elio/openshift-vagrant. vagrant up and then the script oc-up.sh gives me a master and two nodes. Ansible provision runs without problems.

After provision I can login to https://master.example.com:8443 as admin. I can create a project. When I try to deploy jenkins-ephemeral this fails with message

  error: couldn't get deployment jenkins-1: Get
  https://172.30.0.1:443/api/v1/namespaces/test2/replicationcontrollers/jenkins-1:  dial tcp 172.30.0.1:443: connect: connection refused

Ansible installer status shows no problems in the cluster. Why is this not working? How can I troubleshoot this problem?

Have you set Jenkins ssh key to have access to source?
Another point is to validate if the SSH port is open. According to error can be one of these things


How can I download an artifact from Jenkins using the Google Login Plugin? I recently spun up a new Jenkins instance and installed the Google Login Plugin for authentication via Google. I would like to download build artifacts to servers using a shell script, basically curl or wget, but cannot authenticate properly from command line.

I tried the following wget command without success.

wget --auth-no-challenge --http-user=XXXX --http-password=XXXX https://____/job/_____/lastSuccessfulBuild/artifact/target/ROOT.war

I also tried the same without http-:

wget --auth-no-challenge --user=XXXX --password=XXXX https://____/job/_____/lastSuccessfulBuild/artifact/target/ROOT.war

The response is always

401 Invalid password/token for user: XXXX

I attempted to follow this post but ran into an issue.

I have seen this workaround, but it seems like I'd be adding a security hole.

Does anyone have experience with this plugin or in general just downloading from an OAuth-protected endpoint?

Thank you to the comment from user54 for the answer.

Both of my pasted wget URLs work as long as I'm using the correct value for password. In this case, I needed to use my user's API Token as described in the answer to another question:
https://stackoverflow.com/a/34634247/317951


How to run multiple commands with command module in Ansible I want to run couple of commands in ansible playbook I am trying below:

 - name: Running multiple cmds
   command:
    - |
      cmd2
      cmd2
      cmd3
      cmd4

But i am getting an error on this task while running above code.

Update: I am trying to use below commands using command module but getting an error:

- name: install pexpect using pip
  shell: /bin/bash -c "pip install pexpect"

- name: Join system to AD
  expect:
    command: "{{ item }}"
    loop:
      - source /etc/os-release
      - realm join --membership-software=adcli --user=username@EXAMPLE.COM --computer-ou="OU={{ env }},OU={{ account }},OU=XXXX,OU=XXXXXX,DC=XXXXXXX,DC=XXXXXXX" --os-name="$PRETTY_NAME" --os-version="$VERSION" <DCNAME>
    responses:
        Password for *: "{{ username | b64decode }}"

Error:

TASK [adjoin : Join system to AD] ***********************************************************************************************************************************************************
fatal: [localhost]: FAILED! => {"msg": "The task includes an option with an undefined variable. The error was: 'item' is undefined\n\nThe error appears to be in '/tmp/ansiblepull/playbooks/roles/adjoin/tasks/main.yml': line 58, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n- name: Join system to AD\n  ^ here\n"}

Can anyone help me in solving this.

I am getting an error while syntax-check as well.

The Ansible documentation states that the command module doesn't get a shell

  The command(s) will not be processed through the shell

The command module takes the command as an argument, so you can't have  list as you've written there. You could do the same thing in a loop: with command: {{ item }} :

- name: "Run {{ item }}"
  command: "{{ item }}"
  loop:
    - cmd2
    - cmd2
    - cmd3

You could also have several command: tasks instead.

However, this is generally considered bad practice (E.g. E303 from Ansible Lint rules) because it tends to break idempotency. It would be better to decompose those commands into something that uses Ansible modules.


if the bottle neck is the number of connections, is it better to scale redis by sharding or by replication? Background

last week we had a major outage, we identified the bottle neck to be redis number of connections:

Our quick fix was to scale redis from cache.m4.4xlarge to chache.m4.10xlarge (vertical scaling)

However this scaling is quite expensive:

we would like to scale horizontally:

Unfortunately, the application code we are using now only allows us to either create a redis cluster with multiple shards, or using a single shard with multiple replicas (see discussion here).

Question

considering that our bottle neck is the connection limit, would it be getter to scale by clustering and increasing the shard number, or by having a single shard and replicating it?

Redis Cluster could be considered as an OOTB solution for the Scalability requirements. There is another interesting open source library known as Redis Shard for sharding implementations on Redis. Please find the GitHub repository here.


TravisCI build continues to fail even after changing alpine version So I ran my Docker build with TravisCI and I got a failure like so:

FAIL src/App.test.js
  ● Test suite failed to run
    TypeError: Cannot assign to read only property 'Symbol(Symbol.toStringTag)' of object '#<process>'
      at exports.default (node_modules/jest-util/build/create_process_object.js:15:34)

I did not think that this App.test.js file would give me problems after I removed the logic from within the it block like so:

import React from "react";
import ReactDOM from "react-dom";
import App from "./App";

it("renders without crashing", () => {});

I thought the issue may have been the latest release of node because I saw others have similar error messages, so I changed my version like so:

FROM node:10.15-alpine as builder OR FROM node:10.15-alpine as I have various Dockerfiles for each service I am building as this is a multi-container project.

That change I made did not resolve the issue, TravisCI continues to fail my build with the same exact error, what gives here?

I thought perhaps the issue would be a version of Jest mismatch, but I have not installed any version of Jest, this is my package.json file inside my client/src/App.test.js:

{
  "name": "client",
  "version": "0.1.0",
  "private": true,
  "dependencies": {
    "react": "^16.8.3",
    "react-dom": "^16.8.3",
    "react-scripts": "2.1.5",
    "react-router-dom": "4.3.1",
    "axios": "0.18.0"
  },
  "scripts": {
    "start": "react-scripts start",
    "build": "react-scripts build",
    "test": "react-scripts test",
    "eject": "react-scripts eject"
  },
  "eslintConfig": {
    "extends": "react-app"
  },
  "browserslist": [
    ">0.2%",
    "not dead",
    "not ie <= 11",
    "not op_mini all"
  ]
}

None of the solutions above have helped me. You cannot upgrade Jest because react-scripts comes with version 23.x so you are just adding on top of what react-scripts would already have. You cannot upgrade react-scripts so that it will have jest@24 because the latest version has jest@23.

Changing to v11.10.1 does not help either and within my Dockerfile I have tried changing the 'FROM' in my Dockerfile to FROM node:10.15-alpine and that has not helped.

When I tested it in this manner:

docker run -it --rm -v $(PWD):$(PWD) -w $(PWD) node:10.15-alpine npm test

it passed!

 PASS  src/App.test.js (10.163s)

  ✓ renders without crashing (5ms)

But it does not pass in TravisCI

This is my Dockerfile for my client/src directory:

FROM node:10.15-alpine as builder
WORKDIR '/app'
COPY package*.json ./
RUN npm install
COPY . .
RUN npm run build

FROM nginx
EXPOSE 3000
COPY --from=builder /app/build /usr/share/nginx/html

I even tried changing my .travis.yml file to this:

- docker run -it --rm -v myproject/react-test node:10.15-alpine npm test -- --coverage,

when it was originally this:

script:
  - docker run myproject/react-test npm test -- --coverage

Absolutely nothing appears to be working here.

You cannot rely on FROM node:10.15-alpine alone to fix this. I had to go with the fix from the facebook team. So if anyone else did not get a fix from modifying your Dockerfile to node:10.15-alpine, go to your package.json file and change react-scripts to the following facebook fix:

"react-scripts": "3.0.0-next.b0cbf2ca"

Then go ahead and remove your node_modules/ folder via rm -rf node_modules and then run npm install, check your jest-util file, it should now have an output like this:

└─┬ react-scripts@3.0.0-next.b0cbf2ca
  ├─┬ babel-jest@24.5.0
  │ └─┬ @jest/transform@24.5.0
  │   ├─┬ jest-haste-map@24.5.0
  │   │ └── jest-util@24.5.0  deduped
  │   └── jest-util@24.5.0
  └─┬ jest@24.5.0
    └─┬ jest-cli@24.5.0
      ├─┬ @jest/core@24.5.0
      │ ├─┬ @jest/reporters@24.5.0
      │ │ └── jest-util@24.5.0  deduped
      │ ├─┬ jest-runner@24.5.0
      │ │ └── jest-util@24.5.0  deduped
      │ ├─┬ jest-runtime@24.5.0
      │ │ └── jest-util@24.5.0  deduped
      │ ├── jest-util@24.5.0  deduped
      │ └─┬ jest-watcher@24.5.0
      │   └── jest-util@24.5.0  deduped
      ├─┬ jest-config@24.5.0
      │ ├─┬ jest-environment-jsdom@24.5.0
      │ │ └── jest-util@24.5.0  deduped
      │ ├─┬ jest-environment-node@24.5.0
      │ │ └── jest-util@24.5.0  deduped
      │ ├─┬ jest-jasmine2@24.5.0
      │ │ ├─┬ jest-each@24.5.0
      │ │ │ └── jest-util@24.5.0  deduped
      │ │ └── jest-util@24.5.0  deduped
      │ └── jest-util@24.5.0  deduped
      └── jest-util@24.5.0  deduped

You should be good to run TravisCI build now. The issue is the react-scripts version 2.1.8 is conflicting with new version of Node. Suffered through this all day long, grateful to the facebook team for getting back to me.

For reference: https://github.com/facebook/create-react-app/issues/6475


What does "Does not have minimum availability" in k8s mean? What does "Does not have minimum availability" mean?

A GitHub discussion was found, but it is not clear to me what the error message means.

As @Tensibai indicated in one the comments, this could be caused as there is insufficient CPU or memory, but that is not always the case.
For example, a helm chart was just deployed, it failed and the workload in GCP indicated that:

Pod errors: CrashLoopBackOff

Based on the comment of @Tensibai the first impression was that there were insufficient resources, but further analysis using kubectl describe pod <pod-name> indicated that in this case the livenessProbe check failed:
Liveness probe failed: Get http://10.16.0.13:80/: dial
tcp 10.16.0.13:80: getsockopt: connection refused

In summary, the Does not have minimum availability message is generic. Multiple issues could trigger this and more in dept analysis is required to find the actual error.


What is the origin of "Release early and often"? Where does the origin of the Release Early, Release Often (RERO) principle come from?

One of the earliest references to this term originate in Jim McCarthy's book Dynamics of Software Development. This term seems to also be embedded in the Lean Startup movement, which built upon ideas in Lean Manufacturing, TRIZ, and other sources explaining how to evaluate an experimental product and check if it fits the market before investing too much effort in its completion.


Cannot pull ansible docker image $ docker pull ansible/ansible
Using default tag: latest
Error response from daemon: manifest for ansible/ansible:latest not found

I try and get started with ansible but I get the above error message trying to pull the Docker image. Is it broken or am I doing something wrong?

If run you docker pull <image>, you are implicitly asking for a tag named latest.  There is no such tag available from the Ansible repository.  You can see a list of available tags at https://hub.docker.com/r/ansible/ansible/tags/.

For example, if you wanted the fedora27py3 version of the image, you would run:

docker pull ansible/ansible:fedora27py3

I note that the most recent of thoses images is 7 months old. If you're experimenting iwth Ansible you might want to just install it into a Python virtual environment.


What software ihandles shipping to ElasticSearch the standard output of a .NET Core web application running in a container? I have a ballpark idea of how the ELK stack works. I read that by default, standard output is captured and that output will make its way to the Elasticsearch repository. If I post something to stdout, that content piece will be when visualized with Kibana in one field called "message".

So, besides the title which was anyway getting too long, here's the additional ask. How do I send a structured message, JSON I guess? How do I configure it? I read that it is possible, but I have some difficulties piecing it together. There's something about a JSON template that needs to be registered somewhere, but I lack a clear example.

How do I send a structured message?

If you want structured messages check out some of the recommended third party logging options.  Some of these options have built-in support for Elasticsearch logging.  For instance, Serilog provides a "sink" for putting logs directly into Elasticsearch.

How do I configure it in code?

These options all tie into the ILogger interface, so testing different logging frameworks should require very little code changes.  However, the initial entry point of the application (e.g. Program.cs) will need to be configured to use the 3rd party logger.  The official .net core docs go over this in detail.

How do I configure it for Elasticsearch?

This will depend on the logger.  The appsettings.json will most likely be where you configure the .net core application's connection to Elasticsearch.


Does active choice reactive parameter support hide fields option? I have the below pipeline:

I want that once I choose in the InstllationType the value 'Recreate', some fields will disappear. is it possible to do it with an active choice parameter? if not, is there any other way for doing it?

No, this is not possible, because the number and type of parameters for a build are fixed and cannot be modified on-the-fly.  The only way to modify which parameters show up in the "build with parameters" UI is to change the job configuration, which alters the parameters permanently.

In my opinion, if you need to do several different tasks which require several different input parameters (such as "build", "test", "install", "deploy", etc.), you should have separate jobs in Jenkins for each task.  I personally think it is a good ideal in general to avoid complex Jenkins jobs which require a lot of input parameters and try instead to simplify job configuration in order to reduce the likelihood of subtle, difficult-to-debug errors when executing jobs.


How to get code onto aws ec2 instance using terraform and gitlabs I've just started learning Terraform (utilizing the AWS provider) and I have managed to replicate our infrastructure which includes quite a few amazon web services.
I can successfully deploy it and get a basic Apache website up and running on an EC2 instance with the following bash script (run from Terraform):
#!/bin/bash
sudo apt update -y
sudo apt install apache2 -y
sudo systemctl start apache2
sudo bash -c 'echo frontend-website > /var/www/html/index.html'

Very basic, but I'm having issues in finding some documentation on the best practices for getting our project code onto the AWS EC2 instance once provisioned, and how everything should be setup for Gitlab CI/CD to keep it all automated.
Especially for a bigger project (e.g., separate frontend and backend instances) where you may have a dev, test, and prod environment for the frontend and backend.

Setup a Jenkins server. Please do code checkout using Jenkins. Jenkins supports parameterized build. you can pass selected parameter from Jenkins UI to linux bash.
Then use linux's cli command to fire terraform.Terraform supports several workspace. You can use same code for dev, test and prod.
terraform workspace new dev
Use this command to create new workspace. After that , we can use this workspace name to your code. Below is the content of local.tf
locals {
  env = terraform.workspace
}

Like below use the same code for multiple env.
resource "aws_autoscaling_group" "asg" {
  name = "${local.env}-${local.project}-demo-asg"
}

Moreover to save your state file securely , you can use s3 as backend.
terraform {
  backend "s3" {
    bucket = "tfstate-demo"
    key = "keyfiles"
    region = "ap-southeast-1"
    dynamodb_table = "tfstate-demo"
  }
}

To separate backend and frontend service, use best practice of AWS. Create a public ALB for frontend and then move traffic to backend services via private ALB.
You can use Ansible as a configuration manager to build your server then create a AMI of that using packer. Later on, use the same AMI in ASG.


How to get VPC id in Terraform module? I have a module structure like

module "vpc" {
source = "./modules/vpc"
}

module "prod_subnets" {
source = "./modules/vpc/modules/subnets/production"
}

when running am getting an error like

[0m on modules/vpc/modules/subnets/production/production.tf line

    162, in resource "aws_subnet" "prod-pub-1b":
    14:22:55 162: vpc_id = [4maws_vpc.production_vpc[0m.id
    14:22:55 [0m
    14:22:55 A managed resource "aws_vpc" "production_vpc" has not been declared in
    14:22:55 prod_subnets.

can anyone please help ?

Without seeing the contents of your two modules I'm guessing a bit, but it looks like you have an AWS VPC declared in your vpc module and some subnets declared in your prod_subnets module and you are asking how the configuration of the subnets can get access to the VPC ID.

If so, the answer is that the vpc module must export the VPC ID as an output value and then the prod_subnets module must accept the VPC ID as an input variable.

In your vpc module, you can declare a vpc_id output value like this, for example in a file modules/vpc/outputs.tf:

output "vpc_id" {
  value = aws_vpc.production_vpc
}

In your prod_subnets module you can declare a vpc_id input variable, for example in a file modules/vpc/modules/subnets/production/outputs.tf:

variable "vpc_id" {
  type = string
}

Then in your existing modules/vpc/modules/subnets/production/production.tf file, on line 162, change the vpc_id argument for the subnet to refer to that variable:

  vpc_id = var.vpc_id

Finally, you must then edit the top-level file whose source code you shared in your question to pass the value between the two modules, like this:

module "vpc" {
  source = "./modules/vpc"
}

module "prod_subnets" {
  source = "./modules/vpc/modules/subnets/production"

  vpc_id = module.vpc.vpc_id
}

module.vpc.vpc_id means to take the value of the vpc_id output value from the vpc module.


Azure service tags REST API and subscription level rights Goal
I want to use the service tags REST API endpoint to get all the AzureCloud datacenter public IP addresses.
I am aware that I could manually download the generated json file from Microsoft.
My end goal is to use the endpoint to programmatically automate a task.
Problem
I can get a response with status code 200 of about 46k lines, but the AzureCloud tag is nowhere in the response body. It skips from AzureBotService to AzureCognitiveSearch. I believe AzureCloud should be between those two tags.
Steps
Firstly, I created an account using the AZ CLI tool with az ad sp create-for-rbac -n "MyApp".
With a tenant ID, app ID/client ID, password/client secret, subscription ID. I can use https://login.microsoftonline.com/{{ tenant_id }}/oauth2/token to get a bearer token.
Using the bearer token, I can query the service tags using https://management.azure.com/subscriptions/{{ subscription_id }}/providers/Microsoft.Network/locations/eastus/serviceTags?api-version=2020-05-01.
What I've done
I came across a GitHub issue that is somewhat related to my problem stating that by design, we cannot get all the datacenter public IPs if the user does not have subscription level rights.
I tried the following commands to try to give subscription scope rights to my app, but to no avail:

az ad sp create-for-rbac -n "MyApp2" --role reader --scopes "/subscriptions/{{ subscription_id }}" to create a new account while specifying reader role and scope of the subscription. (I have retried the API with the new IDs as well).

az role assignment create --assignee "{{ app_id }}" --role "Reader" --scope "/subscriptions/{{ subscription_id }}" to create a new role and specify scope.

az role assignment create --assignee "{{ app_id }}" --role "Owner" --subscription "{{ subscription_id }}" to create a new role and specify subscription.

My app still has the scope set to "This resource".
I suspect there is something I am misunderstanding about Azure, given I am completely new to Azure (just created a free trial account yesterday).
All help will be appreciated, thank you!

To use servicetags api you have to register Microsoft.Netowork resource provider at azure subscription level.

Instead you can use these api:
to get the ip address list of  a specified Azure Service in a specified Azure region:
https://azinfo.azurewebsites.net/api/servicetags/{regionName}/{serviceName}
to get valid region names
https://azinfo.azurewebsites.net/api/GetRegionNameList
to get valid service names
https://azinfo.azurewebsites.net/api/GetServiceNameList
to get the all the ip address list of all Azure Services in a specified Azure region
https://azinfo.azurewebsites.net/api/region/{regionName}


Parallelising Packer with Ansible Issue

I have an Ansible role which covers Windows, macOS and Ubuntu OS. I also have a Packer template file for each OS (due to small nuances and changes in the template file to handle provisioning) which handles standing up and provisioning the VM with Ansible onto VSphere.

So to build all three VMs, it goes something like:

  Windows build                Mac build                 Ubuntu build
---------------------     ---------------------      --------------------
| Packer -> Ansible | --> | Packer -> Ansible | ---> |Packer -> Ansible |
---------------------     ---------------------      --------------------

Which isn't very efficient and can take a while.

Solution

I'm thinking of parallelising this somehow, by potentially having Packer standup all three VMs, but not provision it - only get the IP address of the VMs, which then (with some manipulation), put the IP addresses into an Ansible inventory file, and the ansible playbook is then run against that.

Something like:

          --------------
Packer -> | Windows VM | <----
   |      --------------     |
   |      --------------     |
   -----> |  macOS VM  | <----
   |      --------------     |
   |      --------------     |
   -----> | Ubuntu VM  | <----
          --------------     |
                             |
                          Ansible parallel provisioning

Is this an optimal way of using Packer and Ansible together? or is there a better way of parallelising these builds?

What about using a CI tool? Create a parallel job and the jobs will run parallel. For example, one could use Jenkins:

https://jenkins.io/blog/2017/09/25/declarative-1/

stage('run-parallel-branches') {
  steps {
    parallel(
      a: {
        echo "This is branch a"
      },
      b: {
        echo "This is branch b"
      }
    )
  }
}

but there are other CI tools that could do the same.


What is the AWS equivalent of Azure Management Groups? Microsoft Azure has Management Groups. (They are a level above subscriptions that allow you to organize subscriptions into containers called "management groups" and apply your governance conditions to the management groups. )
When I studied for my AWS Associate Architect, I don't recall running across an equivalent concept. I'm trying to work out if Amazon Web Services has an equivalent.
My question is: What is the AWS equivalent of Azure Management Groups?

There is no direct equivalent to Management Groups in AWS.
You can achieve some of the same functionality by using AWS Control Tower, specifically around access controls.  I.e. you can set up groups and assign roles to any accounts in an AWS Organization on a per-account or per-group basis and have them deployed to child AWS accounts.
However, there is no such equivalent around, for example, resource allocation.
You can achieve some of the same functionality by using CloudFormation, or by using CT IAM policies policies with specific restrictions, but it's a much more convoluted process than with Azure.


Azure DevOps Pipeline Conditional Flow with Manual Intervention I am wondering if Azure DevOps has the ability to do conditional flow as part of a release definition.

My use case: I have a PowerShell script that deploys a SQL Server DACPAC to create or update a database. Sometimes this deployment will fail because a change was made that could potentially cause data loss. Instead of absolutely failing the deployment, I would like to trigger a manual intervention for a DBA to review the changes. If they resume, I would like to attempt the deployment again, but this time with a flag that will ignore potential data loss issues.

Finally, if that second db deployment attempt is successful, I would like the flow to continue back to the rest of the deployment tasks.

Here is a general flow diagram:

Other requirements:

If the db deployment fails, but the manual intervention resumes and succeeds, I want the entire deployment to be marked as succeeded.
I want the flow after resuming an intervention to merge back into existing tasks that would be run on db deployment success. Corollary: I do not want to maintain dual flows with identical tasks.

I am using the Designer (Web UI) and not YAML for all my current builds/releases, but I am also interested if YAML has a solution for this.

Azure DevOps has a section for Conditions, but I did not find that particularly helpful. The number of conditions seems limited. It also looks like variables can be set on jobs, but this appears to be only available with YAML.

Is there a way to set up the conditionals at the end of phases/jobs or tasks to achieve this branching behavior?

@Shayki's answer was a large portion of what I was looking for (thanks!), but I was eventually able to figure out a solution that met all of my requirements. It involves a less than ideal use of the Release Description to propagate state between jobs.

The updated flow diagram looks like this:

I was originally thinking of the conditional flow as an if else statement, but it is really more of multiple if statements where the condition is the same. If flag is set, do X; if flag is set, do Y; etc.

Details

Instead of running the agentless job when a previous condition fails (which would result in the entire deployment failing, a violation of one of my requirements), I created a custom condition that the agentless job runs as:

If all the previous tasks have succeeded, and if the Release.ReleaseDescription variable contains the string defined in the _dbDeployErrors variable, then this job will execute (and the manual intervention task inside of it).

The _dbDeployErrors variable isn't too important, I set it to ;DB_DEPLOY_ERRORS_$EnvironmentName. The main idea is that, at least with the Web Designer, there is no way to pass information from an agent job to an agentless job. So I wrote a powershell script which uses the Get Release and Update Release Azure DevOps REST API to update the Release Description parameter on the release. This allows me to pass information to the next job in the deployment.

The basic steps for setting this flag are:

Run the db deployment
If the deployment has errors, do not fail task, but instead set an agent variable
Next task determines if the variable is set, and if so calls the Update Release API to append the flag to the release description
Future jobs and tasks can now analyze the state of the release description variable

Note: I set an agent variable instead of making the API call from the db-deploy script so that I can keep my deploy script clean and maintainable. I move the responsibility of calling the API to a dedicated script instead.

If the Manual Intervention is resumed, then the third job executes as an agent job. It will attempt to deploy the db again, but this time it sets a -Force flag on the script to ignore possible data loss (it was approved by the one who performed the intervention). If this deployment succeeds, then a quick cleanup is run to call the Update Release API and clear the flag from the release description. Finally, the other tasks can resume as normal.

Here is what the tasks in my final release definition looks like:


How should I run a multicontainer app/platform with nginx behind traefik? I have Traefik reverse proxy running to allow a couple of docker containerized apps to run on the same domain just to learn about running multiple containers following https://www.smarthomebeginner.com/traefik-reverse-proxy-tutorial-for-docker. Everything works well.  I want to install Device Hive to test it as an IOT platform, but it has multiple containers and it's own nginx proxy and docker-compose file https://github.com/devicehive/devicehive-docker/blob/master/rdbms-image/docker-compose.yml.

Is there a way I can add the Device Hive platform to my existing Traefik build or is it more trouble than its worth.  I wondering, more or less while making sure I'm not having a port conflict, if there's an easy way to have Traefik point a subdomain(subdomains are used for the other apps) to the 80/443 ports of the nginx proxy, and just have everything work.  My other option is to create another VM for the IOT platform.

I may be searching with the wrong terms but I haven't really found an answer. Any help?

Note: I want to use Device Hive on my existing server which is using Traefik. When I look at the docker compose file for Device Hive I don't see how I could change it to leave Nginx out.

Although I agree with @BruceBecker's comment, i.e. running two different rproxies, it is possible to run traefik in front of nginx. At the previous job we configured several servers that were running on nginx behind traefik.

Ensure that each microservice has the correct labels. Note that Traefik 2.0 changed significantly.