Can I run an API gateway on the same instance as my API? Is there any reason why an API gateway should be on a separate server to the API? Other than ensuring enough resources for both. I am using krackend API gateway and hosting a python API.

It should be fine to run the API Gateway on the same server as API-apps - for smaller setups. For larger setups when you need more servers may make sense to use dedicated nodes for API Gateways. API Gateways are increasingly playing a dual role as an application load balancer, or Level 7 Proxy for your APIs - those have a specific role when it comes to network addresses and routing, this is the main reason to why it makes sense to use dedicated servers for this part of your stack.

why an API gateway should be on a separate server to the API?

what is more important for this part of the stack is that you would like to avoid single-point-of-failure, so you should use more than one instance for API Gateways, and also for the API apps. When using multiple instances, it is often easier to separate the load balancer/Gateway from your app workload due to the specific needs for network addresses for the gateway, but in the end this depends on how your whole architecture looks.


Why trigger something remotely (using Jenkins) and then just forget about it? When using Jenkins to trigger some process (eg a remote build) at a remote location, why would you ever want to "fire" something, and then just "forget" about the outcome of what got triggered remotely? At least that is what "fire and forget" seems to be about.

Such "fire and forget" sounds like not respecting some of the basic rules in IT: in whatever process (or program, etc) you write, try to always be prepared for unexpected conditions (return codes, etc) that may arise. So that you'll also have appropriate error handling in place.

To the extend it does make sense, in the (mainframe) SCM tool that I'm most familiar with, you'd start some FTP process from a dev environment to a remote target (eg to distribute and activate executables). And sooner or later (asynchronously) somebody gets some kind of acknowledgement back (from the remote target to the dev site) that indicates "success or failure". And if within a reasonable amount of time no feedback at all is returned, you know that something is not working as it should. Using such scenarios, you'll never forget what got fired.

Any examples for which using "fire and forget" in Jenkins does make sense?

I wouldn't describe Jenkins jobs as "fire and forget."  They are designed to run automatically without the need for human interaction, however, that doesn't mean the result of a job is not visible.  Quite the opposite- detailed logs of all activity are available for jobs ranging from command-line tools to custom deployment jobs.

The testing results from Jenkins jobs are routinely displayed on large-screen TVs all over the Development floor of many companies I've seen.  There are whole plugins devoted to making this type of information available to developers, Ops, and management in a dashboard format.


Disabling of docker build cache - common practice for large organizations? I've been working with a large company that currently disables the docker build cache for CI/CD.  Their justification is that they are managing many projects, so caching would be cumbersome.
This results in builds that are considerably slower than our local docker builds.  They also regularly hit limits in downloading from docker registries.  And the DevOp engineers have no incentive to optimize for caching, so the builds run poorly when running docker with a cache.
Is the disabling of the docker build cache considered to be a common (or best) practice for large organizations?  Are there resources that spell out the best practices in such situations?  And if this is not the best place to ask this question, where should I go?

I guess each organization does what is best for them, but one of Docker's best practices is to leverage build cache.
Applied to a large company, I'd say that if they would use Docker caching (correctly) and save a couple of seconds/minutes on every build. They would, given their size, save many hours/days of Engineering time, which amounts to a lot of money. Also, when a build takes too long, it breaks the flow of the individual, which means more context switching on the side of the engineer, which usually leads to less productivity.


Are package facts getting invalidated automatically when I install remove packages? In Ansible I can use a task such as:
- name: Gather the package facts
  package_facts:
    manager: auto

to gather the package facts and a condition such as:
when: "'foobar' in ansible_facts.packages"

to only execute a task when package foobar is installed. Gathering the package facts is a rather lengthy process compared to many of the other tasks in my playbooks.
For the sake of this question we can also assume that instead of manager: auto I'm using manager: apt.
Now I was unable to find any information regarding if or when the fact ansible_facts.packages gets invalidated after a subsequent package manager operation and a look at the fact via debug module doesn't give me any indication as to when it was gathered. Similarly I found no obvious way of checking against the cache time of the package manager. So in essence I was unable to come up with a means of only gathering (or re-gathering) the package facts when necessary.
So, questions:

does the ansible_facts.packages fact get invalidated after subsequent package manager operations?
if not: how can I get close to the desired "only gather package facts if something changed"?

NB: according to ansible --version I am running version 2.10.7.

Package facts (as any other facts like devices, disks and their partitions, IP addresses...) don't refresh automagically.
In your case, you can use a handler and notify it from any task that can possibly invalidate the cache. Here is a quickNdirty example:
- name: Pseudo code demo for package facts refresh
  hosts: localhost

  tasks:

    - name: Gather the package facts unconditionally at play start
      package_facts:

    - name: Make sure package X is installed
      package:
        name: package_X
      notify: refresh_package_facts

    # By default, handlers run at the end of the play.
    # If you want to force them to run at a certain point,
    # you need the following meta task.
    # Note => this will run !!ALL!! notified handlers
    - name: Run handlers if notified
      meta: flush_handlers

    - name: This task will run with an up-to-date cache
      debug:
        msg: "{{ ansible_facts.packages | length }} packages are installed"

  handlers:

    - name: Refresh the package facts
      package_facts:
      listen: refresh_package_facts


What is the exact use of DETACHED HEAD state in Git? When I checkout a specific commit hash, it goes to detached mode where HEAD POINTER can't point to a recent commit.

What are the things we can do and what is the use of going detached head state in Git?

The "normal" way of working with GIT is by checking out branches. HEAD is a "pointer" to the latest commit in that branch, and GIT will allow you to commit new changes after that pointer.
However, as you figured out, it is possible to check out a specific commit hash, if you want to. In this situation, if you make changes and try to commit them, you will be unable to do so, since you are not at HEAD, and GIT will not allow you to commit your changes.
Most of the time, detached HEAD (checking out an older commit) is used if you want to test how your project ran at a certain point in time.
If you need to make changes to the code or need to build it using Jenkins MultiBranch Pipelines, for example, it is also possible to create a new branch from that specific commit by running:
git branch my_new_fancy_branch 7y8u33ii
or
git checkout -b my_new_fancy_branch 7y8u33ii
... where 7y8u33ii is the hash you provided in the question. For this new branch, 7y8u33ii will be HEAD, so you will be able to commit new changes. If you need to get your code into the branch from which you created this new branch from, you will need to do a merge, but this is another discussion altogether.


Challenges of using AWS Lambda chain Context:
I have a pipeline of 6 lambda functions (chained together), triggered by an SNS notification which is generated whenever a file lands on S3. This pipeline essentially takes the file(few GBs), filters it (Spark cluster is created to run the job, then deleted at the end), and inserts it into a DB. Lambdas are orchestrating the flow.

Issues:
If one Lambda fails, the chain breaks hence no effective failure handling. Secondly, we experience timeouts if a polling/computation takes longer than 5 minutes, so no effective retry. It takes a long time to test/debug an issue if a lambda fails. Also there is no visibility, say for example how many jobs failed and how many passed? we dont know. Getting a bunch of SNS notifications on email is not very effective/helpful. If the chain breaks, we cannot perform cleanup operations like deleting SPark cluster or housekeeping steps.

My Questions:
Is AWS Step Functions a good choice for solving the above issues? When would you not use a Step Function service? If you cannot invoke Step Function through SNS, then what would be the best way to call it whenever a file lands on S3? Feel free to share any other approach to easily and effectively tackle this usecase.

I am using lambda in response to objects appearing on S3 buckets.  For me in my particular use case, there are around 600 triggers a day.  I have written our lambda functions to use their stdout as their log file, writing all key events to standard out in a semi-structured textual sentence mixed with key/value pairs for data values, and then feed the cloudwatch logs into splunk.

Here is what I have found...

Lambdas will (occasionally) be triggered multiple times for the same object.
Lambdas will (sometimes) take noticeably longer to run than normal.
To see / diagnose issues caused by concurrent invocations that interfere with each other, the AWS Web Console doesn't do it. You need a tool like splunk.

However, does the value of this work provide enough revenue to cover the costs of using ECS(Elastic Container Service) for your process management?  If you are able to run an ECS container, you could trigger the lambda on S3 object creation to create a message on an SQS message queue. Your ECS container is listening to your queue and when a message arrives, processes it just like within your lambda.  Its output to trigger the next stage is to pop a message onto the SQS queue handling the next task.

This will solve a few problems for you, such as escaping the 5 minute maximum for Lambdas, and creating an easy way to retry failures.

However, it will create a few new issues too.

Creating and deploying docker images is more work than editing lambda on the AWS console.
Infrastructure costs are more fixed in nature, rather than proportional to workload.


Can I bypass host key checking in Ansible? I'm running an Ansible playbook with a subtle hack to work on an arbitrary port forwarded port (so I can use one machine with no direct access to lots of machines).

I've got a pre-task to change the ansible_port variable, so then when I start running my real tasks and roles, I get prompted to accept the hostkeys for localhost on some random port.

Because I am naive and don't care about security, I would like ssh to auto-accept and redirect to /dev/null (or another file for logging).

Is this possible?

This is typically done by setting the following value in ansible.cfg:

[defaults]
host_key_checking = False

If you don't want to modify ansible.cfg you can set an environment variable like so:

export ANSIBLE_HOST_KEY_CHECKING=False

Source: http://docs.ansible.com/ansible/intro_getting_started.html#host-key-checking


Advantages of Linux Containers over Docker I'd like to understand the differences between Linux Containers and Docker, especially what kind of advantages Containers have in comparison to Docker.

Docker is a specific implementation of Linux containers, or if you want to be more precise Docker is a distribution of tools that includes runc which is an implementation of Linux containers. Other implementations include rkt, LXC, LXD, and (I think) Snappy from Ubuntu.


What is Systems Thinking? The "First Way" in Gene Kim's explanation of DevOps requires "Systems Thinking" https://itrevolution.com/the-three-ways-principles-underpinning-devops/

Where can one learn the ability for "Systems Thinking"? What are some of the resources that can take someone who has no familiarity with "Systems Thinking" and enable that person to start doing "Systems Thinking" in his daily work?

Googling "system thinking" returned a lot of resources that are about "Systems thinking" rather than "System thinking". This source indicates that "Systems thinking" is one of the three ways that could be applied in DevOps.

After reading several references "Systems thinking" is about interactions rather than silos. If one translates this to DevOps then "Systems thinking" is aligned with the theory and also important for DevOps as this is the interaction between QA, Dev and Ops.

In order to apply "Systems thinking" one should be able to communicate with people from Dev, Ops and QA, listen to them, understand their problems and find ways to let them work together instead of getting a "through it over the wall" mentality and unconnected departments in the company.

An element that really requires "Systems thinking" is Continuous Delivery (CD). Multiple departments are involved, while CI is mostly used by Dev, QA and Ops will be definitely need to be involved in this process as one does not want to release a newer version if integration or manual tests failed and Ops want to be informed if there are changes to the system as they monitor systems and try to prevent and solve issues on the platform.

In summary, "Systems thinking" is required for DevOps engineers. If one only sees Dev, Ops and QA as individual parts, but no interdependency then this is not aligned with the DevOps theory, i.e. intersection between Dev, Ops and QA.

References

https://medium.com/disruptive-design/tools-for-systems-thinkers-the-6-fundamental-concepts-of-systems-thinking-379cdac3dc6a

http://www.thwink.org/sustain/glossary/SystemsThinking.htm

  a discipline for seeing the 'structures' that underlie complex
  situations, and for discerning high from low leverage points.

https://thesystemsthinker.com/connecting-systems-thinking-and-action/

  The iceberg is one of the most frequently used models to explain
  systems thinking (see “The Iceberg”). Thanks to movies like “Titanic,”
  many people recognize that most of an iceberg sits hidden beneath the
  water; that is, out of sight.

https://www.accenture.com/us-en/blogs/blogs-devops-evan-erwee-system-thinking

  System Thinking allows people to make their understanding of social
  systems explicit and improve them in the same way that people can use
  engineering principles to improve their understanding of mechanical
  systems.

  ...

  We can therefore state that System Thinking for DevOps is required to
  understand Software Engineering as a social construct using
  engineering principles.

  From a System Thinking approach, DevOps thinking is fundamentally
  different from the traditional form of analysis. Traditional analysis
  focus on separating the individual software engineering pieces, e.g.,
  architecture, coding, testing, tooling etc. In contrast, the DevOps
  approach focuses on how all aspects of the software life cycle
  interact with other constituents of the system. Circling back to the
  definition of DevOps we can easily identify the ‘behavior traits’
  observed in the DevOps movement, namely: adaptable/flexible (improve
  relationship), cooperative (better communication), diplomatic
  (collaboration), etc.

http://www.systemicleadershipinstitute.org/systemic-leadership/theories/basic-principles-of-systems-thinking-as-applied-to-management-and-leadership-2/

  Systems thinking is a management discipline that concerns an
  understanding of a system by examining the linkages and interactions
  between the components that comprise the entirety of that defined
  system.

https://thesystemsthinker.com/introduction-to-systems-thinking/

  What exactly is systems thinking? In simplest terms, systems thinking
  is a way of seeing and talking about reality that helps us better
  understand and work with systems to influence the quality of our
  lives. In this sense, systems thinking can be seen as a perspective.
  It also involves a unique vocabulary for describing systemic behavior,
  and so can be thought of as a language as well. And, because it offers
  a range of techniques and devices for visually capturing and
  communicating about systems, it is a set of tools.

https://www.burgehugheswalsh.co.uk/Systems-Thinking/Approach.aspx

  Fundamentally, Systems Thinking is straightforward. We treat the
  situation of interest as a system and examine it using system
  properties and characteristics like purpose, context and structure to
  discover more.

http://study.com/academy/lesson/systems-thinking-in-management-definition-theory-model.html

  System thinking is a method of critical thinking by which you analyze
  the relationships between the system's parts in order to understand a
  situation for better decision-making. In simpler terms, you look at a
  lot of the trees, other plants and critters living around the trees,
  the weather, and how all these parts fit together in order to figure
  out the forest.


RDS IOPS utilizing fully I have upgraded RDS mysql to 1000 IOPS from 120 IOPS, But same database operations utilizing more than 1000 IOPS now. Is there any way to implement RDS cluster or Is it possible to make use of lambda to reduce the usage of IOPS and load?

Newbie here, If anything wrong please suggest the right way. Thank You!

First try to review current caching parameter values.
Then check list in AuroraMySQL.Reference.ParameterGroups
And try to tune/increase caching parameters like query_cache_size

Alternative solution is to use architecture redesign with AWS CloudFront to reduce load on primary system (probbly web?).


How do i connect docker pipeline on jenkins to kubernetes cluster I am trying to setup a pipeline on jenkins that uses docker containers to execute tasks like for example deploy an helm charts using a docker image with helm

here is what my Jenkinsfile looks like

pipeline {
    agent none
    stages {
        stage('Helm') {
            agent {
                docker { image 'dtzar/helm-kubectl' }
            }
            steps {
                sh '''
                helm version
                kubectl version
                helm ls
                kubectl get all -n test
                '''

            }
        }
    }
}

but it fails at the helm ls line because it can not connect to the kubernetes cluster.
How do i connect this pipeline to use the Kubernetes configuration(kubeconfig) i have setup

here is error am getting

+ helm ls
Error: Kubernetes cluster unreachable

I could not find any documentation on how to connect docker pipeline to connect to the kubernetes cluster
What do i add to the Jenkinsfile to be able to connect?

  UPDATE

i also tried this

pipeline {
    agent none
    stages {
        stage('Helm') {
            agent {
                docker { image 'dtzar/helm-kubectl' }
            }
            steps {
              withCredentials([kubeconfigContent(credentialsId: 'k8s-cluster', variable: 'KUBECONFIG_CONTENT')]) {
                sh '''
                helm version
                helm ls
                kubectl get all -n test
                '''
              }

            }
        }
    }
}

and got this error

+ helm ls
Error: Kubernetes cluster unreachable

I could not find any documentation on how to connect docker pipeline to connect to the kubernetes cluster

What do i add to the Jenkinsfile to be able to connect?

so seems the docker container is unable to resolve the dns of the kubernetes cluster

how do i resolve this?

Maybe it is strange solution, but I would like to provide it to you:

You can get actual kubeconfig file from your Kubernetes cluster or locally and change it to base64 format like in example below:

- cat /root/.kube/config | base 64

Then use your base64 configuration file in pipeline:

- echo $your_variable | base64 -d | tee /root/.kube/config

In such case you will be having actual kubeconfig file on runner machine.

In case of DNS problem - just need to specify IP address of your runner machine in /etc/resolv.conf file.


How to publish Jenkins job icons externally? I'm using Promoted Builds Plugin to assign certain icons to Jenkins jobs on certain conditions and I'd like to publish these promotion icons.

How can I generate external links to these icons per each job, so they're accessible externally?

—————

For example I'd like to create the table in README.md on GitHub like:

Job 1 | ![](link_to_icon_of_job_1)
Job 2 | ![](link_to_icon_of_job_2)
Job 3 | ![](link_to_icon_of_job_3)

The "Embeddable Build Status Plugin" will do exactly that.

Note: This requires your Jenkins server to be accessible from the internet if you're using GitHub because they cache all images in their CDN.


Running multiple websites on Docker in isolated or shared database containers? So I have been reading up on Docker, and I cannot quite get my head around this concept. There are literally dozens of tutorials on the web about spawning a Wordpress instance on Docker within 5 minutes. All these sites spawn their processes separately; a singe SQL container, a single webserver container and a PHP + filesystem container.

Apparently the Docker philosophy is to keep everything nice and small and separated. Proper isolation, distribution and scalability; all fine. But what if I need two Wordpress sites, or ten?

The advantage of separate containers is that every site can be instantiated with its own passwords and other secrets via a docker-compose file, but having 10 database containers running next to each other feels like wasting resources. Once single SQL container should be capable of running 10 databases.

I was wondering if running multiple instances is indeed resource heavier, and/or if the added value of strict isolation outweighs the resource usage?

(I have chosen SQL, but the question basically applies to all resources that can be shared across sites. Think of mailservers, virus scanners, etc.)

Docker and all the other fancy names are just tools. The examples are just examples.

Resource-wise, Docker changes nothing much beyond usual Linx/Unix multiprocessing. It does not have the overhead of VMs, and little overhead over just starting the process directly ( https://stackoverflow.com/questions/21889053/what-is-the-runtime-performance-cost-of-a-docker-container ).

So it is really up to you to decide how to use it. If you have closely related applications under your own command, maybe in your intranet, why not use a shared DB. If they are customer DBs, or you might need to run different versions of the software, or different incompatible configurations, then separate services might be fine.

For RDBMSses especially, they are usually very good at managing resources (RAM, HDD) and so for this case specifically, running only one may just be the best thing. It's really more about what your needs are than general guidelines.


Terraform: can't connect to AWS provider using shared config file or static variable I'm trying to use terraform to manage AWS resources and trying to set up the credentials configuration. I'm following the official documentation: https://www.terraform.io/docs/providers/aws/index.html

My first idea was set a shared credentials file so I configure it:

~.aws/credentials

[default]
aws_access_key_id=****
aws_secret_access_key=****

~.aws/config

[default]
region=us-east-1
output=json

app/main.tf

provider "aws" {
    region = "us-east-1"
    version = "~> 2.0"
    profile = "default"
}

terraform {
    backend "s3" {
        bucket = "example-bucket"
        key    = "terraform-test.tfstate"
        region = "us-east-1"
  }
}

When I run terraform init I receive the following message:

Error: No valid credential sources found for AWS Provider.
        Please see https://terraform.io/docs/providers/aws/index.html for more information on
        providing credentials for the AWS Provider

I have already tested the credentials using aws cli and it's working perfectly.

After that, I tried to configure static credentials in main.tf like this:

provider "aws" {
    region = "us-east-1"
    version = "~> 2.0"
    access_key = "****"
    secret_key = "****"
}

Same error...

I decided to test with environment variables and then it worked. But now I want to know why I could not configure with static variables or shared config file. All this cases was described in the official docs, what am I doing wrong?

The easiest way I've found to properly set credentials for my local runs and remote terraform cloud runs is to use environment variables. Before running set the environment variables for

AWS_ACCESS_KEY_ID
AWS_SECRET_KEY
AWS_DEFAULT_REGION

This should allow you to run in both Linux, PowerShell or even remote runs in Terraform cloud without issue.

I've gone between various approaches and this has provided me the easiest workflow. If you're doing this in AWS directly then of course you could use IAM instance role and avoid credentials altogether. This wouldn't allow you to keep the same workflow for local and remote though.


Running a background process in Pipeline job I am looking for a way to run a java process in background using Jenkins declarative pipeline. Below is the snippet of code

stage('Deploy'){
        steps{
            script{
                withEnv(['BUILD_ID=dontkill']) {
                    sh "nohup java -jar test-0.0.1-SNAPSHOT.war &"
                }
            }
        }
}

Already aware of ProcessTreeKiller of Jenkins and every answer suggest to do the same. I have tried using environment inside the steps block with no luck.

Jenkins version : 2.60.1

Pipeline plugin : 1.1.7

Any help is greatly appreciated.

Any one facing the same problem and using pipeline project, set JENKINS_NODE_COOKIE instead of BUILD_ID.

Dont waste your time setting HUDSON_COOKIE, HUDSON_SERVER_COOKIE, JENKINS_COOKIE or JENKINS_SERVER_COOKIE. None of them work for pipeline project.

Refer to https://issues.jenkins-ci.org/browse/JENKINS-28182 for more details.


az aks browse doesn't open browser automatically on WSL and only showing JSON instead of the dashboard az aks browse works fine on my Mac Mini, but I'm trying to work with it in WSL2 Ubuntu 18.04 with Azure CLI on a Windows 10 machine.

$ az version
This command is in preview. It may be changed/removed in a future release.
{
  "azure-cli": "2.1.0",
  "azure-cli-command-modules-nspkg": "2.0.3",
  "azure-cli-core": "2.1.0",
  "azure-cli-nspkg": "3.0.4",
  "azure-cli-telemetry": "1.0.4",
  "extensions": {}
}

Kubernetes 1.14.8 which shouldn't make a difference given it works fine in macOS. Appears to be an issue with WSL2.

The first issue is running:

az aks browse -g <groupname> -n <clustername>

Just brings up the following in az cli:

Proxy running on http://127.0.0.1:8001/
Press CTRL+C to close the tunnel..

It doesn't automatically open the browser from WSL2 like it does in macOS and Linux. So you have to CTRL + Click the http://127.0.0.1:8001 which will open the browser.

The second issue is it only brings up a JSON response with a list of API endpoints that looks like the following:

paths
0   "/api"
1   "/api/v1"
2   "/apis"
3   "/apis/"
4   "/apis/acme.cert-manager.io"
5   "/apis/acme.cert-manager.io/v1alpha2"
6   "/apis/admissionregistration.k8s.io"
7   "/apis/admissionregistration.k8s.io/v1beta1"
8   "/apis/apiextensions.k8s.io"
9   "/apis/apiextensions.k8s.io/v1beta1"
10  "/apis/apiregistration.k8s.io"
11  "/apis/apiregistration.k8s.io/v1"
12  "/apis/apiregistration.k8s.io/v1beta1"
13  "/apis/apps"
14  "/apis/apps/v1"
....

And doesn't do anything after that.

Followed all of the necessary steps for az login, az aks get-credentials, and RBAC.

What gives?

I would expect it to work like az login and open up a new tab in default browser. But somehow it doesn't work like that. I think the fix must come from azure CLI.

Temporarily, For az aks browse you can setup an environment variable on WSL.

export BROWSER="/mnt/c/Program Files (x86)/Google/Chrome/Application/chrome.exe"

In this case next when I run az aks browse it will open up the dashboard in new chrome tab.

You can use this variable to use your favorite browser. I would also add this variable to .bashrc or profile to be added on every session.


A name for testing software migrated to a new environment against original application This is more of an english language question - we have various types of tests - unit tests, A/B tests, etc...

Suppose I need to migrate code to a different language, machine, or any other different environment specification, with the definition that it reproduces the original application exactly (bugs included!). I am looking for a name for such a test.

Currently I am using functional testing, but I feel that's to general and am looking for something better. It is also somewhat misleading since the "function" is not the implemented algorithm but reproducing behavior of another application.

@Bruce suggested regression tests, which are a good option we considered, but dropped it since we (and I think most people) use those for actual changes in the original code base (in this question's setting) that may add features, fix bugs etc... Perhaps this is too nit-picky.

As mentioned in the question, there is a subtle difference between Regression testing and the situation described here.

To paraphrase Wikipedia:

  Regression testing ... is re-running functional and non-functional tests to ensure that previously developed and tested software still performs after a change. If not, that would be called a regression.

However, a situation is described here where the functionality of the code is the same -- since no new features or functionality is added, this is technically not regression testing.

However, what does need to be asserted is the behaviour of the code. If run in the same way after the re-implementation, does the code still behave in the same way, beyond still giving the same results (which would be covered in unit testing and functional test).

In "The Practical Test Pyramid", Martin Fowler describes the various types of testing which a well-maintained application could have. Near the apex are "Acceptance Tests", i.e

  Do Your Features Work Correctly?

After a re-write, I believe Acceptance Tests would be the answer here.

Note - Acceptance tests also include Operational Acceptance Tests, described by Wikipedia as:

  Operational acceptance test (OAT) criteria (regardless if using agile, iterative or sequential development) are defined in terms of functional and non-functional requirements; covering key quality attributes of functional stability, portability and reliability.


Permission denied within mounted volume inside Docker/Podman container I am starting to learn about containers using podman that came with RHEL8.1 (which AFAIK can be used in place of docker), and have the following baby Dockerfile as a learning exercise:

# Use Alpine Linux base image
FROM alpine:latest

# Install pacakges
RUN apk --no-cache add bash gcc make

# Make a directory for source code
RUN mkdir /src_dir

# Set working directory to the same directory
WORKDIR /src_dir

# Set this directory as a volume
VOLUME [ "/src_dir" ]

As you can see, I've installed the most basic gcc and make into this container with the goal of mounting a set of source files on my container host into the /src_dir directory within the container.

I next build the container image in the host directory containing the Dockerfile:

podman build -t my_image .

I then start the container with this command

podman run -it -v /host/foobar:/src_dir /bin/bash

Where /host/foobar/ on my host is an arbitrary directory containing some arbitrary source code, all of which my local user on the host has full read/write access to. For example, there is one file /host/foobar/test.c. This then brings me to a bash prompt inside the container. I can see that I'm at the correct place because:

bash-5.0# pwd
/src_dir

However, I have absolutely no read/write access to /src_dir. Both ls -lh and cat test.c gave me permission denied errors. If I change to the root directory (or any other directory) of the container, I can see and access other things. Strangely, if I run ls -lh / I can see /src_dir as being owned by root:root, so I don't understand why as the container's root user I can't access anything in it.

I also tried podman inspect [container ID], and in the output I can see:

...
"Mounts": [
            {
                "Type": "bind",
                "Name": "",
                "Source": "/host/foobar",
                "Destination": "/src_dir",
                "Driver": "",
                "Mode": "",
                "Options": [
                    "rbind"
                ],
                "RW": true,
                "Propagation": "rprivate"
            }
        ]
...

Which suggests that there is read/write permission?

Perhaps I'm missing something obvious as a beginner, but what do I have to do so that I can run the gcc and make inside this container on the source files mounted in /src_dir so that the container essentially acts as a complete development environment?

Thank you!

P.S. I've read that it's a good idea to use a separate non-root user in the container, but I haven't figured that out yet... Is this easy to achieve given my situation?

Thanks to the people here, the solution is quite simple (but not obvious):

My GNU/Linux container host has SELinux activated, and that's why I was having permissions problems. The solution is to simply append a :z to the podman run volume argument so that this:

podman run -it -v /host/foobar:/src_dir /bin/bash

becomes this:

podman run -it -v /host/foobar:/src_dir:z /bin/bash

That's it.


Route webhook to all replicas Normally a request gets routed to one of the replicas.  However, if each instance need to take some action when a webhook request comes in, how can I route the request to all replicas?

Obviously only one instance can reply, so I am skeptical if this is possible.

Alternatively, is there a way for when one of the instances receives the request, to communicate it to the others?

This is something you'd have to build. Kubernetes doesn't natively offer this as a pattern.

There are a few approached you can take, but a common on is to use a queue that has this functionality. I'd suggest starting with Nats because it's a CNCF project that is easy to run natively in Kubernetes. When one of your replicas receives the request, it can post a message to the queue that can deliver the message to every subscriber.


How to set 'image tag mutability' using CloudFormation? When one would like to create an Elastic Container Registry then it is possible to set 'image tag mutability':

Image tag mutability
The image tag mutability setting for the
repository. Select "immutable" to prevent image tags from being
overwritten by subsequent image pushes using the same tag. Mutable
Immutable

According to this user guide it does not seem to be possible to configure this using AWS CloudFormation.

The ImageTagMutability property in ECR is not currently supported in CloudFormation, as verified by experimentation (nor is it part of the LifecyclePolicy property). There is sometimes lag between feature availability in CloudFormation, and the mutability flag was only introduced in late July.

This comes up elsewhere in the corners of AWS, including:

AWS Global Accelerator - this entire service is not provisionable using CloudFormation.
AWS WAF geo match rules cannot be created in CFN.


Can I run multiple pipelines in a single GitLab repo using GitLab CI/CD? I know that I can't have multiple gitlab-ci.yml files in one repo, but it still seems fairly limited. Say for example I have one set of tests I want to run whenever a change is pushed or on PRs, and another set I want to run every 24 hours. Is there a way to do that, or do I only get one set of tests I have to use whenever I want to run CI?

Yes, you can use the rules syntax. You can use this in combination with regex for commit message, ci_pipeline_source or any other available CI variables.

job1:
  script:
    - do something on schedule only
  rules:
    - if: '"$CI_PIPELINE_SOURCE" == "schedule"'
      when: always

job2:
  script:
    - runs on Merge request pipeline
  rules:
    - if: $CI_MERGE_REQUEST_ID
      when: always
    when: never

job2:
  script:
    - runs on changes to anything in src directory or Dockerfile
  rules:
    - changes:
        - src/*
        - Dockerfile

Note you can also include multiple CI templates in your gitlab-ci.yml. Additonally, you can include a CI template with "hidden" job templates that you can reference with extends:
# Include ci files
include:
 - local: '/templates/.scheduled-job-template.yml' # CI template with hidden job (.scheduled_job) that runs on schedule
 - local: '/templates/.lint-and-validate.yml' # contains jobs for linting and validating project

job1:
  extends:
    - .scheduled_job
  rules:
    - if: '"$CI_PIPELINE_SOURCE" == "schedule"'
      when: always

job2:
  script:
    - runs on Merge request pipeline
  rules:
    - if: $CI_MERGE_REQUEST_ID
      when: always
    when: never

In doing this you can compose the jobs/pipelines you want in its own yml file and then define the jobs using those templates in the gitlab-ci.yml, which will help keep things maintainable and clear if you are running numerous different pipeline/pipeline configurations from the same project.


How to integrate docker-compose or docker run command arguments into SCM and Jenkins build? When I do docker run I have to give extensive argument options for my container setup.  To mitigate this and ensure all developers are using the same arguments I switched to docker-compose.

Now when I try to go run the same build from jenkins I can no longer use the docker plugin since no jenkins plugins seem to support docker-compose up, they all need to use the docker commands for the remote API instead of the docker-compose commands.

Unless there are reasons not to, I would prefer docker-compose up.  I am aware I could create a bash script the does docker run <arguments> and let my jenkinsfile parse the script and load in the proper args but this feels ultra hacky.

How can I use one or the other and keep them both in SCM and ensure that if a change is made to docker source (either the docker-compose.yml or possible a bash script running docker run with the proper arguments), it will work in both places?  Are there tools for this?  a plugin I am missing?

Also I desire to use CoreOs which doesn't natively support jenkins (no java and other things).

If you have docker compose installed on the Jenkins Build machine you can use sh docker-compose up to run docker-compose from a new shell.

In general, you can run any command from your JenkinsFile using the sh command.


Parameterized Kubernetes manifest files? I'm learning Kubernetes, coming from an AWS/CloudFormation background to set up my company's infrastructure. One of the strengths of CloudFormation is the use a template that can be parameterized for re-used. So I can use the same template to deploy all our microservices in Fargate, giving it the Docker image, version, CPU/memory counts, etc for each microservice.
When using Kubernetes manifest files, is there good way to "templatize" manifest files? I've read  kustomize, but as the title says it's "template-free," and to use it for all our microservices, I'll need multiple /someapp/... directory trees, which is NOT what I want.
Sure I can use sed but that seems archaic. I can use yq to massage my yaml files, but this also seems archaic. For either approach, changing a manifest file entails changing the corresponding sed and yp commands.
Any suggestions on how to approach this? How about 3rd party tools like Codefresh? How about helm? (I haven't looked at it yet.).

I educated myself more about helm and it looks like it has these features that I need

packaging
templating
release management, including rollbacks

so I'll study it more.


VSTS deployment variable So I'm automating a process we currently have where we deploy a database and then have an exe that 'activates' the database.  The default values are different for each product and is determined by a cmd line arg.

I've created 3 VSTS Release environments based on the hardware IO board for the environment.  How can I pass in this product flag at deploy time as it isn't dependent on the environment or release itself.

Edit:
The product flag value isn't known until the database is deployed into an environment, and value can not be set in the release definition creation time or at release time.  Only when deployed into a particular environemnt

While you can use the Release Description field the moment you Start a Release

which fills in the $(Release.ReleaseDescription) variable, I discourage this approach.

The whole point of a Release Pipeline is to automate and having users inputting data is against this approach.
If you do not trust VSTS for Secrets, you can store them in Azure KeyVault or similar and pull them down while releasing. It is not a chance that there are Tasks to facilitate this.


Proper mapping for service architectures, Docker and BitBucket mapping - how do you do it? With Atlassian BitBucket, you can have:

Projects, which incorporate..
Source code repositories.

Now, you could have a microservices with more than just one codebase and need therefore more than just repository. What is the best practice to reflect this if you have many different projects?

My idea now is to encode microservice prefixes to repository names, but is this really the way to go?

Like:

Project A

service1.repo1
service1.repo2
service2.repo1

Project B

serviceN.repoX

One could also create libraries and call them using a dependency manager like a package.json if node is used or ivy if the code is written in java.


What Chief is accountable for DevOps? CIO

  Chief information officer (CIO), chief digital information officer
  (CDIO) or information technology (IT) director, is a job title
  commonly given to the most senior executive in an enterprise
  responsible for the information technology and computer systems that
  support enterprise goals.

and

  CIOs manage a business’s IT systems and functions, creates and
  delivers strategies and policies, and places great emphasis on
  internal customers.

CTO

  A chief technology officer (CTO), sometimes known as a chief technical
  officer, is an executive-level position in a company or other entity
  whose occupation is focused on scientific and technological issues
  within an organization

and

  CTOs place emphasis on the external customers to the organization and
  focus on how different technology can make the company more
  profitable.

COO

  The COO is responsible for the daily operation of the company, and
  routinely reports to the highest ranking executive, usually the chief
  executive officer (CEO). The COO is usually the second in command at
  the firm, especially if the highest ranking executive is the Chairman
  and CEO.

Discussion

As DevOps is closing the gap between several disciplines, e.g. Development, Operations, Quality Assurance one could argue that the CIO is accountable for DevOps as all these parties are internal customers. However, as a CTO is also focused on the outside world, one could also say that the CTO is accountable as he or she could implement DevOps because he or she heard in the outside world that this discipline was beneficial for some companies.

From my experience (where I currently work and another company where a friend works at) the DevOps team's report to the CIO.

Also to note that our DevOps team works a lot with, mostly alongside the CTO on a "day to day" basis compared to the CIO.


What's the difference between "Ops Dev" and "DevOps"? I've heard both terms, but I can't differentiate between the two.

I've never seen ops dev used before this post.  Searching around I see things like this article that has it in the title but nowhere in the article.  Then there's this article from 2015:

  So in this way, when “OpsDev” happens, IT ops teams are helping developers do their jobs, but developers are also architecting applications with IT ops in mind, helping them utilize resources better. The days of developers simply creating applications and “throwing them over the fence” to testers and IT ops teams are over.

which seems to define OpsDev as "development in a company that practices DevOps" (or more cynically, "development done correctly").

Finally, there's https://dzone.com/articles/opsdev-is-coming :

  What Is OpsDev?
  OpsDev means that the dependencies of the various application components must be understood and modeled first before the development process begins. In addition, the consideration for infrastructure stability, environment modeling, security and audit/compliance measures are first and foremost. Application components are stubs and they do not need to be in their final forms. Secondly, the environments in which the components will be deployed for production must be modeled. Thirdly, the processes to deploy the various components to the target environments must be automated as much as possible.

Again, this sounds to me just like normal, modern development.

DevOps is a problematic term because it has many different definitions.  OpsDev, on the other hand, appears to have essentially no definitions.  From a personal perspective, I would mentally substitute "development" for every instance of "OpsDev" and ignore the term; it appears to be just another instance of people inventing new terminology to describe old concepts so that they sound more exciting.


What is the difference between php-cli and php-fpm, why 2 php variants and why choose one or the other? Looking at https://hub.docker.com/_/php I can see reference to php-cli and php-fpm.

Now, I am aware of the concepts of PHP FPM - Fast CGI Process Manager. I'm also aware the PHP can be run in the command line and for php composer and Symfony console.

Does the Docker php-fpm not have PHP CLI tools like php-fpm. Or is php-fpm a superset of php-cli ?

Why would one choose one or the other?

Thank you.

They are different tools for different purposes:

php-cli is the standalone tool for running a PHP scripts in the same way you would run it from the command line: php somescript.php.  It seems you can also use it to base other images from.

This variant contains the PHP CLI tool with default mods. If you need a web server, this is probably not the image you are looking for. It is designed to be used both as a throw away container (mount your source code and start the container to start your app), as well as a base from which to build other images.

php-fpm is the FastCGI server implementation of PHP which you would use with a FastCGI compliant web server such as Apache or Nginx.

This variant contains PHP-FPM, which is a FastCGI implementation for PHP. See the PHP-FPM website for more information about PHP-FPM.

You also have two other variants:

php-apache which is just Apache configured with mod_php
php-alpine is the smallest possible Docker image based upon Alpine Linux, albeit with some compatibility issues.

You can see all of the Dockerfiles on GitHub and whilst it doesn't appear that php-fpm is based upon php-cli practically they are likely to be very similar and php-fpm is probably a superset of php-cli.


Docker Security Scanning: How to integrate Clair with a docker private registry? According to the following documentation it should be possible to integrate clair with a docker private registry. How to do that?

  Registry Integration

  Clair can be integrated directly into a container registry such that
  the registry is responsible for interacting with Clair on behalf of
  the user. This type of setup avoids the manual scanning of images and
  creates a sensible location to which Clair's vulnerability
  notifications can be propagated. The registry can also be used for
  authorization to avoid sharing vulnerability information about images
  to which one might not have access.

Both Docker Registry and Docker Engine have API interfaces.
I consider it should be possible to implement a Docker plugin for this integration to have images scanned for example as somebody pushes them. (side note - Docker registry is an image registry, not container registry).

This scanner functionality is also possible through the commercial Docker Enterprise edition.
https://docs.docker.com/docker-cloud/builds/image-scan/

UPD here an example with clair/docker, looks quite simple
http://blog.xebia.com/docker-containers-vulnerability-scan-clair/


Jenkinsfile script to waitUntil a folder is created With this Jenkinsfile code, I am trying to test A/B folder creation, and also testing if waitUntil works till the folder is created, I expect it to wait for 0 secs as the folder is created before waitUntil gets called.

stage('other job') {
                    steps {
                        script {
                            timeout(10) {
                                def folder = new File( 'A/B' )
                                println "Waiting for " + folder
                                println "fe==" + folder.exists()
                                waitUntil {
                                    if(folder.exists()) {
                                        return(true)
                                    }
                                }
                            }
                        }
                    }
                }

It fails with below error, what's wrong here? the folder is not gerrit created in workspace.

Timeout set to expire in 10 min
[Pipeline] {
[Pipeline] // stage
[Pipeline] }
[Pipeline] echo
Waiting for A/B
[Pipeline] echo
fe==false
[Pipeline] waitUntil
[Pipeline] {
[Pipeline] }
[Pipeline] // waitUntil
[Pipeline] }
[Pipeline] // timeout
...
Failed in branch other job
...
[Pipeline] End of Pipeline
java.lang.ClassCastException: body return value null is not boolean
    at org.jenkinsci.plugins.workflow.steps.WaitForConditionStep$Callback.onSuccess(WaitForConditionStep.java:167)
    at org.jenkinsci.plugins.workflow.cps.CpsBodyExecution$SuccessAdapter.receive(CpsBodyExecution.java:377)

Your block expression doesn't return a value in all cases.

Try this instead:

waitUntil { folder.exists() }


Make use of CIs as Maven Repository in order to use it for building Docker images without including dependencies I am trying to make images from our company Maven based applications. These apps are using some libraries which are included in a Maven repository, but fills a lot of disk space. They also have some dependencies which should be downloaded. I have seen this question which is talking about downloading dependencies (and unfortunately has no approved answer), but for me it is more important to use my local repository without adding it to the image (which increases image size from 300M to 4G) and caching downloading dependencies is in the second place.

If I wanted to ADD my local repository to the image, I would have make this Dockerfile:

FROM java:8.0_31
MAINTAINER Zeinab Abbasimazar (zeinab.abbasi@peykasa.ir)
ADD .m2/ $HOME
WORKDIR .
RUN apt-get update;apt-get install -y maven;cd Development/;mvn clean install -P test
CMD ["./Component.sh &"]

Which makes this image:

sudo docker images
REPOSITORY                                   TAG                 IMAGE ID            CREATED             SIZE
maven                                        3.3.9               44ab79a0bd9e        2 hours ago         4.678 GB

This size for an image is not affordable for me; could anyone please help  me on this? Is there any workaround using gitlab or Jenkins to play Maven Repository role?

Docker images are after all VM templates, i.e. they have to be more or less self-contained: you get the image, you run the environment with all the dependencies.

To approach the challenge operationally, Docker supports reuse of environments through the FROM statement i.e. you could maintain a base image and on top a much smaller image with the app itself.

But, from the strategic point of view, I would say your architects might want consider splitting the whole thing into smaller microservices - that would allow for 5 smaller images. Of course I cannot judge whether it's really possible.


How to apply DRY to k8s files? At the moment there are more than 100 different deployment files. The majority of the config is identical, but there are some differences and therefore every docker image has its own deployment template. If a change has to be made to one of the deployment.yamls, most of the time this has to be copied to other files as well and that is a little bit tedious. How to apply Don't Repeat Yourself (DRY) while constructing k8s files without creating spaghetti?

This is a similar problem I've been facing. And we're not the only ones it seems. GitLab just added a very interesting feature in their recent 10.5 release where you can pull external files into the pipeline.
In their blog post, they had this to say:

For many of our largest customers, the DevOps team is responsible for providing CI/CD pipelines to a large number of development teams throughout the organization. Previously, this was a painful process to manage. There wasn’t a scalable way to distribute reusable pipeline configuration which meant code needed to be manually copied between multiple .gitlab-ci.yml files in multiple projects. This was a labor-intensive and error-prone process. Additionally, it didn't provide adequate controls to ensure testing and deployment is consistently enforced for each repo.
Starting with 10.5, you can now include external files in CI/CD pipeline definition. Included files can be either local (contained in the same repo) or remote (accessible via HTTP/HTTPS). Including local files allows a large and complex .gitlab-ci.yml to be broken up in to modular chunks that are easier to maintain.

https://about.gitlab.com/2018/02/22/gitlab-10-5-released/#include-external-files-in-cicd-pipeline-definition
This is probably the best way to approach it from a OOP perspective as well: modularize certain aspects of your builds and include it in downstream code.
My peers working at another company do a similar approach with their Jenkins builds. They've essentially build on a 'makefile' for their builds that is based on templates and evaluates certain variables set in the Jenkins UI. Based on those variables, the code will then fetch (curl or git) 'module' files from a central 'builds' repository. It then uses all the modules to compile a single pipeline bash script and runs it.
This allows central changes to be done to a single file, that all the builds inherit.


Prometheus - join series based on the key part of a label In 1.16 k8s deprecated the label kubernetes.io/role: worker
I had previously been using as a filter to target graphs to specific node role using this in my grafana variables declaration

label_values(kube_node_labels, label_kubernetes_io_role)
#join series based on the the label value
  sum(kube_pod_container_resource_requests_cpu_cores
  * on (node) group_left (label_kubernetes_io_role) kube_node_labels{label_kubernetes_io_role="$node_role"}

In this scary new world of using the label node-role.kubernetes.io/worker: "" this no longer works, nor can I necessarily regex it when the part I care about is in the key.

Is there a new way anybody knows of to filter queries by arbitrary labels on the node

label_values(kube_node_role, role)

sum(kube_pod_container_resource_requests_cpu_cores
  * on (node) group_left (role) kube_node_role{role="$node_role"})

kube_node_role out of kube-state-metrics can give you a relationship to join


What exactly are "Truly reproducible builds"? What exactly are they?  Why are they important, in the domain of Continuous Delivery?

Context: I have seen in one of (I guess reddit)'s comments that Truly Reproducible builds are still an under-research technology, and is very difficult to create.

So, I wanted to know why are they so difficult to create?

What exactly are they?

Here is a quote from reproducible-builds.org:

  Reproducible builds are a set of software development practices that create a verifiable path from human readable source code to the binary code used by computers.

Why are they important?

IMO the easiest way to explain their importance is to consider them as a variation of a backup procedure.

As an example:

Assume a business which uses (depends) on some software package licensed from some software vendor. Whereas the business only gets the executables, not the sources, etc that were used to create those executables.
Everything goes well, but at some point something goes wrong with the software vendor, e.g. they go out of business (eg bankrupcy).
This may expose a risk to the business (in the long run). I.e. if there is no procedure/agreement in place for the business to get (legal) access to all required sources, documentation, build procedures, etc related to anything from the software vendor used (back in the days) when the executables (used by the business) were created (and shipped to the business).
That's were "Software Escrow" comes to the rescue: if there is such agreement in place, one would think that via a 3rd party, it would still be possible for the business to get access to "whatever was used" to be able to reproduce the executables, so that from there on, the business might have a chance to continue to use that software, and where apropriate start maintaining it themselves (for only running their own business).
However, the "whatever was used" in the previous bullet is the toughest part to make this work. It requires that the 3rd party performs appropriate validations upfront. And trust me, it takes a while before you can recreate an executable for which you can prove that, apart from (eg) the link date, it is a perfect match with what the software vendor delivers to the software agent.

And why are they so difficult to create?

If the above sample is still not clear enough, imagine you're my software escrow agent, tell me what you need as input to recreate a copy of the software licensed by my customer. Get it? You didn't forget to check about which version of my compiler, maybe my OS, compile/link options, versions of reusable components (includes), libraries, etc?


How to determine if a Jenkins build is triggered by a Pull Request merged to Master I think i've scoured every post on this topic but haven't found my specific scenario. (Maybe that means I'm building the wrong thing OR i've built something unique and new? The former is far more likely)

I have a multibranch pipeline in Jenkins being triggered by webhooks from GitHub. I want to automatically tag the repo being built only when a PR from develop|release|hotfix branches are closed and merged to master.

What I want to drill in on is how to identify when a master branch build is triggered by the merge from a PR, and not from anything else. I don't want to tag if master builds due to a non-PR merge. (This would allow updating readmes and code comments without generating a new tag.)

I'm getting the tag string from a file in the repo, and I know how to use the PR builder to identify the source and target of a pull request for when conditions. I know how to get the JSON build information and pull data out, but what I want doesn't seem to be there.

when { changeRequest target: 'master' } only identifies the PR build, not the master build that results when the PR is closed and merged.

when { branch 'master' } obviously identifies a master branch build, but how can I limit it to only build when the build is triggered the merge from a closed PR?

You might try installing the GitHub Integration plugin. Reading their documentation, they provide a number of environment variables you can use for your purpose.

GITHUB_PR_STATE can be OPEN, CLOSE
GITHUB_PR_SOURCE_BRANCH for the source branch (e.g., hotfix/foo)
GITHUB_PR_TARGET_BRANCH or master

Using the when condition, you can run a job/stage when it's a closed PR merging into the master branch using:

when {
  allOf {
    expression { env.GITHUB_PR_STATE == "CLOSE" }
    expression { env.GITHUB_PR_TARGET_BRANCH == "master" }
    expression { env.GITHUB_PR_SOURCE_BRANCH == "hotfix/foo" }
  }
}


Windows network share application deployment automation We have a historically grown setup of interface applications that needs to be  inventorized and cleaned up. Afterwards deployment should be automated and application versions somehow documented.

My problem is that this setup is on a Windows network share only accessible from my work computer (Windows 7).
There is currently no way to change that setup.

Tools that I already looked at are Puppet, Capistrano, Octopus Deploy, RapidDeploy and Ansible. The last one seems to be exactly what I need, but is only available for Linux.

Which tool can be recommended to be used for this task?

Check out Otter it's basically the only Configuration management/Infrastructure as Code tool with native Windows support or Possibly BuildMaster as well which is more on the release coordination deployment automation side of things.

If you're a Windows heavy shop definitely take a look at the tools mentioned on the Winops wiki page - will probably help point you in the right direction.


Dockerized MySql slower in Wifi than ethernet When I try to connect to a MySQL server that runs in a docker container(in System A) through my application(Running in system B) via wifi, it takes at least 4X amount of time than it would when both systems are connected via ethernet. I tried adding skip-name-resolve in my.cnf and mysqld.cnf but in vain. I cannot find where the problem is and how the same can be solved!

I am attaching the dockerfile below. Thanks

Dockerfile

your mysql server running on system A and if you are trying to make connection vai wifi or sending request to make connection on mysql it is mya possible reason should be latency in network. if both system is different and you have exposed you mysql to internet then also it may happen.


TFS Service hook should not fire when work item is updated via REST API I wrote a service hook on work item update but I don't want to fire the service hook when work item is updated from REST API, becz its the same data that other system had and it post to ADO and now ADO is posting back that data again, and it is creating duplicates, so any workaround to not to fire service hook when work Item is updated using the REST API or can bypass integration user, ?

The service hook will fire as part of the notifications process.  You can suppress all notifications by setting the URI parameter suppressNotifications to true.

An example request URI might be:

PATCH https://dev.azure.com/fabrikam/_apis/wit/workitems/{id}?suppressNotifications=true&api-version=5.0

See the work item update API documentation for more information.


How to use feature flag toggles? What are the different ways to use feature flag toggles in applications?

If you were to explain to a developer the exact things that should be done in order to get from nothing to a full feature-flag-toggled application, what would those steps be?

Feature flags are an engineering device that can be used to avoid long-lived branch and conflicts in product development.  Here is how it can be used the context of an object-oriented language to help developers collaborate on a specific product feature while one handle a new version. This solution can also be used in non object-oriented contexts, provided a notion of “interface” exists. (cf. OCaml module system.)

For the purpose of illustration, we assume a tool presenting reports about data stored in a database. The code implements a DatabaseClient class used to perform requests. As the dataset grows, it becomes clear that some alternative data layout would improve the application performance. Therefore Alice will develop a new version of the DatabaseClient able to retrieve data from the structures with improved layout, while Bob will maintain the historical DatabaseClient.

With the following steps, Alice and Bob can collaborate on short-lived branches while minimising their conflicts.

Alice rename DatabaseClient to DatabaseClient_v1 and create a delegate class called DatabaseClient that uses an object DatabaseClient_v1 and implements an interface called DatabaseClientInterface. (If possible, this DatabaseClientInterface should be a code artefact but duck-typed languages not always support this.)
Bob reviews changes made by Alice in 1 and is aware that his maintenance job should happen on DatabaseClient_v1.
Alice introduces a new configuration flag in the application that governs the behaviour of the DatabaseClient delegate and implements a DatabaseClient_v2 placeholder, a class implementing the DatabaseClientInterface whose methods all throw a “Not implemented” exception.

After this, Alice and Bob can collaborate without explicit synchronisation, because code written in their respective iterations is subject to the DatabaseClientInterface. This minimises the risk of a conflict resulting from their concurrent work.

Iterations from Alice can be very short, like implementing a test, implementing a method, or even partially doing so, because in production, the code is not selected for use and does not need to be fully functional. The automated testsuite should be configured so that the DatabaseClientInterface always uses DatabaseClient_v1 while Alice can easily toggle to DatabaseClient_v2 when running the testsuite locally – or in a custom CI setup. Once everything is ready, a single commit can perform the change, by updating the configuration value governing the  DatabaseClient delegate.


Jenkins Exception that is caused by stdout redirection using windows cmd I'm trying to execute an exec file from a jenkins server, using windows cmd. This is what is run from jenkins Windows batch window:

C:\MyApp\MyApp.exe

When run regularly from command line, the app prints to the console reporting its progress.

When I run it from jenkins, I get the following exception (MyApp stands for the name of my actual application):

08:13:56 Started by user NO
08:13:56 Building remotely on SFM BSP builder (ComputerName) (projectName) in workspace C:\Jenkins\workspace\MyBuildMachine
08:13:56 Running Prebuild steps
08:13:56 [MyBuildMachine] $ cmd /c call C:\Windows\TEMP\jenkins3066529473565168228.bat
08:13:56
08:13:56 C:\Jenkins\workspace\MyBuildMachine> C:\Users\NO\Desktop\MyApp.exe
08:13:56
08:13:56 Unhandled Exception: System.IO.IOException: The handle is invalid.
08:13:56
08:13:56    at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
08:13:56    at System.Console.GetBufferInfo(Boolean throwOnNoConsole, Boolean& succeeded)
08:13:56    at MyApp.App.AppInit(String[] args)
08:13:56    at MyApp.Program.Main(String[] args)
08:13:56
08:13:56 C:\Jenkins\workspace\MyBuildMachine>exit 255
08:13:56 Build step 'Execute Windows batch command' marked build as failure
08:13:56 Email was triggered for: Failure - Any
08:13:56 Sending email for trigger: Failure - Any
08:13:56 Sending email to: myEmail@email.com
08:13:58 Finished: FAILURE

Needless to say, the app didn't do its job.
Trying to understand the problem, I found here an explanation about stdout redirection, and Jenkins actually does that automatically, which causes this exception. And indeed also when I run it in cmd with stdout redirection, i.e. C:\> MyApp.exe >> appOut.txt, it shows the same error.

I have no access to the source code of the app. I know that it could be solved from inside. My question is: Is there any way to prevent Jenkins from redirecting the stdout to a file, or wrapping the exec somehow such that will allow me to run it? (of course, running it from another exec and just catching the exception won't work. There won't be an exception but also the app won't run).

I found out how to do it without pipeline. It only has to do with command line script:

start "" C:\MyApp\MyApp.exe >> text.txt

This call is asynchronous (i.e., the command line won't wait for MyApp.exe to return). If you want it to wait, you can use the /w flag, like this:

start /w "" C:\MyApp\MyApp.exe >> text.txt


AWS ECS Update with new docker image I've setup a cluster on AWS ECS, it's hosting nginx and a webapp.
I've used these commands to create my resources
aws ecr create-repository --repository-name <repo-name>

ecs-cli configure --cluster <cluster-name> --region eu-west-3 --config-name <conf-name> --cfn-stack-name <stack-name> --default-launch-type EC2

ecs-cli up --keypair <keypair> --capability-iam --size 1 --instance-type t2.medium --cluster-config <conf-name>

docker-compose build

docker-compose push

ecs-cli compose up --cluster-config <conf-name>

I've also setup a service in that cluster connected to the task created above.
My goal is to be able to push the docker images (That already works) and then use some command to get ECS to fetch the new images and load them up (this is where I'm stuck). I have to do this through the CLI since I want to automate this later on.
My question is:
Should I just delete the running task so that the service will replace it? This works if I do it in the AWS Management console. But it causes downtime of my webservice.
Another options is to use the command below which I found here:
aws ecs update-service --cluster <cluster name> --service <service name> --force-new-deployment

However, when I run that command I get this error:
EVENTS  2021-04-21T05:32:45.842000-07:00        2c47e2c0-3a76-4dc1-bfef-21392b65871b    (service <service-name>) was unable to place a task because no container instance met all of its requirements. The closest matching (container-instance 25785de8f80041e0afd7b0adfda5f2fa) is already using a port required by your task. For more information, see the Troubleshooting section of the Amazon ECS Developer Guide.
EVENTS  2021-04-21T05:29:11.592000-07:00        4279fb7c-9c36-477b-acb3-173edb8e2c31    (service <service-name>) has reached a steady state.

I'm guessing it can't create another EC2 instance because I've set the number of instances to 1 on creation of my cluster. But I tried with 2 instances and it seems overkill to have two instances up and running all the time. How can I achieve this?

AWS container service team member here. IF you are still early on in your journey to automate this via a CLI I would suggest you consider moving past the ecs-cli. The follow-on version of this CLI is called AWS Copilot and has more sophistication to implement those workflows. Note that Copilot dropped support for docker compose because we have implemented a native integration with Docker and the new docker compose cli. You can read more here and here.


How can i check if all CustomResourceDefinitions have been created properly? I have installed with success cert-manager following the steps indicated here.

Now, i am doing an script that checks if every steps are done properly, so my question is: How can i check if all the CustomResourceDefinitions are created via kubectl, for example?

I use google kubernets engine v1.13

This command works: kubectl get CustomResourceDefinition | awk '{ print $1 }'

You can use -o yaml or -o json


How to dynamically create digital ocean droplets without password change? I am working on a web app where that I divided into two blocks:
1. One for front-end.
2. One for backend doingheavy computation which won't be used every time
Both parts of the web service are hosted on digital ocean.
In order to cut costs, I created manually a snapshot of the back. The idea is to create a droplet from this image using a rest API if required by the front. At the start of the backend droplet, two docker containers are created immediately.

My problem is that since the droplet is a fresh ubuntu instance, my cron jobs are not executed after the creation of the droplet.
I am suspecting the reason is because Ubuntu is asking for password change and blocking every thing.

For that, I need to rechange the password of the instance manually then reboot it. This is clearly not possible.
Is there a way to automate this part ?

After more tests, I found out that in order to do this, I need to use an ssh key with the API. The password change phase is bypassed after this.


JFrog Artifactory or JFrog Bintray , when to use one over the other? We are actually looking for a package management system for our project.
The target is simple, have a centralized system to keep our packages ( apps and middlewares ) and use them with our CI/CD tools (Jenkins, Ansible, Docker ...)

Looking for options I discovered JFrog Artifactory and JFrog Bintray. they both seems to do the same job but I don't believe JFrog is maintaining two concurrent products with the same options.

What is the difference between Artifactory and Bintray?
How do we choose one over the other and why?

JFrog Artifactory and JFrog Bintray both manage binaries (and any other file type you can think of). I'd like to see them as two different parts of your CI/CD pipeline.

Artifactory is mostly meant to be used inside the organization. For managing all binaries coming in as dependencies (like maven central jars) and being produced by your build process (like your pack.rpm or org/pack-docker docker image). These can later be used by others in the org or released to Bintray.

Bintray is meant to provide a universal distribution platform for your end deliverables. For example, the pack.rpm you want to be available for your customers to download. And also that org/pack-docker docker image.

Look into the features lists of each and see how they differ in order to best match their intended role.

Artifactory features
Bintray features

I hope this helps explaining the differences.

Disclosure: I work at JFrog


Multi-language Monorepo We currently have a "front-end" git repository that is all of our JavaScript client apps, and a "back-end" git repository containing all back end APIs written in .NET Core.  There is also a few other random repositories built on differing technologies. This is all managed via Azure DevOps, which I believe will let us trigger builds on particular paths.

front-end repo
-- applications
----app1
----app2
-- common-library

back-end repo
-- apis
----api1
----api2
--common-library

some .net-core-product repo

We were considering putting everything in a single mono-repo because it is easier for the developers to manage with git clients such as Visual Studio.  I cannot find an example of a multi-language monorepo - is there an example on github? What are the pros/cons?

Monorepos have been getting a lot of attention in recent years since google popularized them.  If you aren't on a similar scale as google it might not work out as well for you.  I've been in a startup that moved to monorepos and we quickly found that:

most tools are built with one repo leading to one artifact that gets deployed.  You will have to rework all of those tools to handle the monorepo.  There are tools that are designed for monorepos.
a monorepo puts all of your eggs in one basket.  If somebody steals a laptop would you rather them get half of your code or all of it?  Keeping different things in their own repos reduces exposure.
your CI for code validation is now much more complicated.  You need to support multiple languages in your code conformance tests.  Developers will get upset when you check backend stuff and they've only made frontend changes.  Now you're stuck trying to figure out how to disable some CI checks, but make sure they're enabled when needed.

All of this adds up to a load of work for release engineers.  Do you have people focused on release engineering with free time to tackle multiple complicated projects?  Google can make this work because they have hordes of engineers and they can dedicate ten folks to this sort of thing for months until it is smooth enough to develop on top of.  If you are not at the same scale as google you might not be able to follow their example.  The folks that have succeeded with this tend to be big organizations.  You may not have found an open source example because none exists.
These points are also noted in this article:

The frank reality is that, at scale, how well an organization does with code sharing, collaboration, tight coupling, etc. is a direct result of engineering culture and leadership, and has nothing to do with whether a monorepo or a polyrepo is used. The two solutions end up looking identical to the developer. In the face of this, why use a monorepo in the first place? Please don’t!

So focus on improving your engineering culture and don't get worried about not following the latest tech trends.


Docker swarm join linux container Error - remote CA does not match fingerprint Start docker swarm :
docker swarm init --advertise-addr MANAGER-IP

Join docker swarm:
docker swarm join --token  MANAGER-IP:2377

I am using Windows 10,
it is working fine with Windows container mode, but gives below error in Linux container mode.

Error:

Error response from daemon: remote CA does not match fingerprint. Expected: 91030413f17ec7c023a2a796ee05a024915080ca8dfd646a597c7e966f667df6

Docker swarm manager command: docker node ls

ID                            HOSTNAME            STATUS              AVAILABILITY        MANAGER STATUS
2zf1l2o7sl2a1qka55s2vi77x *   moby                Ready               Active              Leader

Host name is moby, when running in Windows container mode it gives machine host correctly.

If you configured and joined correctly you should see two hosts: one for Linux and one for Windows.

Windows hosts cannot natively run inside the Linux hosts and vice-versa. This means you have to have different kernels for Linux hosts and Windows hosts in the same swarm. This way they can communicate across the same Docker network.

Refer to Docker Engine on Windows for more info.


What are the best practices for anti-virus / anti-malware when using containers? One of the SOC 2 controls is apparently around ensuring antivirus/anti-spam/anti-malware is running on production servers. I've been googling to see if there are any recommendation/best practices for this in the container world. I'm not sure if it's a good practice to run AV on the host OS? Some materials I found in my Googling seem to indicate that if you do something like that, you would need to exclude the files used by the container to avoid locking up the container. Which sounds like it would defeat the purpose.

In our particular case, we're running on GKE using Google's pre-made container OS. Anyone have experience with this?

This is an area where you will need to talk directly to and potentially help educate your auditors.  What you have to remember is they are responsible for ensuring that adequate controls are applied, not specifically that "Production Servers have Endpoint Protection"; Anti-virus and Anti-malware are among some of the most basic controls applied to technology.

As a bit of a starter for ten, you could consider the following:

Secure Software Development Lifecycle (S-SDLC): In traditional environments, developer and administrator workstations would have Anti-virus and Anti-malware products installed, therefore any code that a developer creates and an administrator deploys will have been subject to routine automatic scans.  It's easy to forget that the same controls need to be applied to our automated pipelines, one option would be to scan both source code and artefacts with ClamAV or a commercial antivirus product.

Trivy - not anti-virus but does offer vulnerability scanning so controls against malware entering via an upstream 3rd party.
Anchore
Clair

One of the best practices for container development is reducing the attack surface area by maintaining a hardened and minimal base image.  For example, removing any non-essential components such as shells, network utilities, and text editors.  Utilising a pre-hardened image such as the CIS Images and applying
If you manage your own Kubernetes masters and nodes then you should consider the attack vectors against that host use endpoint protection to mitigate, although if you use Hosted Kubernetes, i.e. Google Kubernetes Engine or Azure Kubernetes Service, you need to jump through some hoops to get it working.  As you've probably already seen there is some basic advice on what to avoid, and you need to understand the limitations of the scanning engine - namely it can't see into containers.

Docker Anti-virus Exclusions - remember Kubernetes is using ContainerD
Kubernetes ClamAV
Sysdig Falco
Carbon Black

Equally, masters and nodes should be leveraging SELinux as a Mandatory Access Control security mechanism and generally have the relevant CIS or STIG benchmarks applied to them.
There are a small but growing number of "hot" container endpoint protection tools, it's fairly nascent so explore your options carefully:

Dagda
OpenSCAP
Sysdig Falco

Integrity Checking is very important as whilst it isn't anti-malware per se, it is a component part of most anti-malware packages so might just give you the signal you need to know that something is wrong:

images should be signed, and Kubernetes should be configured to only use images signed by a trusted party.
hosts should absolutely employ File Integrity Monitoring with something like twistlock.
equally, file integrity monitoring within the container itself can be valuable.

Finally, remember "Infrastructure as Code" if you can destroy all of your infrastructures and rebuild it from scratch automatically it makes it significantly easier to recover from a compromise.  This is a tough sell with an auditor.

If you implement everything above, you will quickly find you run into information overload - this is where a Security Information and Event Management (SIEM) tool can help bring everything together in a single pane of glass.


Troubles with mapping ports on a very simple Docker container I'm new to Docker and following a tutorial on Udemy.com. The turtorial has me setting and running a very simple container.

Docker File

# Specify a base image
FROM node:alpine

# Putting files into a specific directory
WORKDIR /usr/app

# Install some dependencies
COPY ./package.json ./
RUN npm install
COPY ./ ./

# Default command
CMD ["npm", "start"]

Index.js

const express = require('express');

const app = express();

app.get('/', (req, res) => {
    res.send('Hi there');
});

app.listen(5000, () => {
    console.log('Listening on port 5000');
});

package.json

{
    "dependencies": {
        "express": "*"
    },
    "scripts": {
        "start": "node index.js"
    }
}

I'm working on a laptop running Windows 10 Home. Using the Docker Quickstart Terminal and PowerShell.

I've edited my HOSTS file as to point localhost to 192.168.99.100, which is the ip the container gets.

I've tried hitting the server by both 192.168.99.100:5000 and localhost:5000. No matter what, when I try to access via browser(I.E. and Chrome) I get the following message:

This site can’t be reached

192.168.99.100 refused to connect.

  curl http://192.168.99.100:5000

returns

  curl: (7) Failed to connect to 192.168.99.100 port 5000: Connection refused

However, If I run 'docker-machine ls', I get:

  tcp://192.168.99.100:2376

Why port 2376? I cannot figure out where/how this port is being assigned.

Trying the browser on that port I get a different message:

This page isn’t working

192.168.99.100 sent an invalid response.

ERR_INVALID_HTTP_RESPONSE

This would suggest(at least to me) that the server is there on that port, but not doing something right.

curl http://192.168.99.100:2376 --output -

Gives me some garbled output in the terminal, and writing it to a file gives garbled output as well. Not much, just a few garbage characters.

I suspect this is the "Hi There" message that is not being formatted correctly.

So my questions would be, am I doing something wrong in my dockerfile? Why/how is the container getting this port 2376 when i'm specifically mapping to 5000?

Any insight would be appreciated.

May be you are using bridge network mode. In that case, you need to bind your host port to container port to use same port.
So when you execute docker run, you may need to add
-p 5000:5000 as a flag to docker run command. Also I don't see any expose in Dockerfile. So you may want to add EXPOSE 5000 in Dockerfile just above cmd.


How to deploy to Appengine standard from gitlab CI? How to deploy from Gitlab CI to AppEngine Standard environment?

AFAIK there are only 2 ways of deploying GAE apps:

using the gcloud app deploy command:

  gcloud app deploy - deploy the local code and/or configuration of your
  app to App Engine

using the Admin API. From Deploying Your Apps with the Admin API:

  To deploy a version of your app with the Admin API:

  Upload your app's resources to Cloud Storage.
  Create a configuration file that defines your deployment.
  Create and send the HTTP request for deploying your app.

Both gcloud (used in the 1st method) as well as gsutil (used in the 1st step of the 2nd method) are part of the Google Cloud SDK, so you'll need to have the SDK installed on your runner.

Summarizing the required steps from a GAE deployment example using the 1st approach that I found in Getting started with Angular 6, GitLab CI/CD and Google App Engine (probably a good read for details):

in gitlab set env vars with your GAE service account and project ID info
in your .gitlab-ci.yml file's deployment section add a before_script section for installing the SDK (update for current SDK version)

        before_script:
          - wget https://dl.google.com/dl/cloudsdk/channels/rapid/downloads/google-cloud-sdk-218.0.0-linux-x86.tar.gz
          - tar -xzf  google-cloud-sdk-218.0.0-linux-x86.tar.gz
          - ./google-cloud-sdk/install.sh

and a script section for the actual deployment, referencing those env vars:

        script:
          - echo $GAE_KEY_FILE > gae_auth.json
          - ./google-cloud-sdk/bin/gcloud auth activate-service-account --key-file gae_auth.json
          - ./google-cloud-sdk/bin/gcloud app deploy --project=$GAE_PROJECT_ID


Best way to create a Ci/CD pipeline to reduce bugs and facilitate refactor I started a project with a small team (3 people ) and we had to hack a lot and built a big application with no unit tests, relying only on manual. Now we have a huge technical debt and we started implementing CI/CD. Our goal is to start doing cypress tests everywhere, then some unit tests and refactoring the code as well as implementing static analysis tools on the Ci/CD pipeline:

Our idea is that we will not have any local environment but just server-based. So developer push code to Gitlab, Gitlab runner in Digital Ocean push the repository to staging server, Developer debugs in Digital Ocean using Local Cypress Runner and his own modifications, if he breaks something or forgets to do a full regression test a command-line test regression runner is triggered BEFORE the second deployment so it doesn't allow the code to pass to the deployment/production server.

Questions
1: do we need to separate deployment environment from staging in 2 droplets? can we achieve this configuration without major headaches in the same droplet?

2: do we need to use docker? if not how it should be done

3: what setup can we do to reduce manual work? our problem is not needing approvals but automating as much as we can since we don't have IT/DevOps guy and neither QA.

Note: the first stage is to implement full regression, the second stage is to implement code quality tools (static analyzers) such as sonarqube, phpStan, etc.

Stack: LAMP stack with PHP Codeigniter and vanilla JS.

It might not be obvious now, but usually as projects advance full regression costs (resources/time) grow much faster than static analysis ones.

You'll also find that static analysis alone isn't a sufficiently solid bug gate, it must be complemented with regression testing as well.

So I'd place the full regression after the static analysis in the CI/CD pipeline - I wouldn't want to waste full regression costs on code which doesn't even pass static analysis.

Down the road you'll probably find that running full regression and/or full static analysis in earlier pipeline stages can easily become bottlenecks:

if you block other commits while waiting for the run results you'd slow down the commit rate
if you allow other commits while a run is in progress they'll change the repository context potentially invalidating the earlier run results (for example you can't immediately fix a runtime bug uncovered by the run if the subsequent commit that happened in the meantime causes a build time breakage).

Addressing these bottlenecks usually means splitting such runs across 2 major  functional stages (each one can actually consist of multiple sub-stages and/or steps):

the (pre-)integration stage(s), which execute subsets of these runs in the earlier pipeline stages:

shorter - to maintaining a sufficiently high commit rate and avoiding bottlenecks
the target quality level is below deployment, but high enough to keep development flowing
fixing regressions detected in these stages has a high priority.
it's possible to use such runs to gate commits/merges in order to prevent rather than detect and fix regressions, in which case these would actually be pre-integration stages

the post-integration stages, executing the longer/full regression and/or static analysis runs:

the goal here would simply be increasing the quality to deployment levels (i.e. blocking the actual deployment if not met)
regressions detected at this stage aren't blocking for development, fixing them is a lower priority


How to use edited inventory file? I have workflow with 2 templates. First I add new host to my hosts.ini file:

- name: Add to host file
     blockinfile:
       path: /var/lib/awx/projects/_52__glusterfs/hosts.ini
       block: |
         gluster1 ansible_user=<user> ansible_host=<ip>

And second I wait for host up:

---
- name: Weit befor all hosts ssh up
  hosts: localhost
  tasks:
   - name: Test 1
     shell: cat hosts.ini
   - name: Test 2
     shell: cat /var/lib/awx/projects/_52__glusterfs/hosts.ini
   - name: Test 3
     shell: cat ansible.cfg
   - name: Wait for hosts up
     wait_for:
       host: gluster1
       port: 22

But i get eror:

 [WARNING]: provided hosts list is empty, only localhost is available. Note
that the implicit localhost does not match 'all'

So output from tests in template 2 is (pseudocode output):

shell: cat hosts.ini
>>>
[gluster]

shell: cat /var/lib/awx/projects/_52__glusterfs/hosts.ini
>>>
[gluster]
gluster1 ansible_user=<user> ansible_host=<ip>

shell: cat ansible.cfg
>>>
[defaults]
inventory = /var/lib/awx/projects/_52__glusterfs/hosts.ini
host_key_checking = false

[ssh_connection]
ssh_args = -o ControlMaster=auto -o ControlPersist=60s -o UserKnownHostsFile=/dev/null

[inventory]
enable_plugins = host_list, script, auto, yaml, ini, toml

Why I have empty hosts.ini in my execute directory and why ansible use it instead of /var/lib/awx/projects/_52__glusterfs/hosts.ini in my ansible.cfg file?

You have this in your config file:

[inventory]
enable_plugins = host_list, script, auto, yaml, ini, toml

Do you really need all these plugins?  If not, they'll just add time to your processing.  There is also a small chance that one of them is successfully parsing your inventory file as an empty inventory.  And the documentation states that

  Once an inventory plugin succeeds at parsing a source, any remaining inventory plugins will be skipped for that source.

If you do not need those plugins, I'd comment that line (or rework it to remove some of the items).

Also, are you sure that it is using the local hosts.ini?  The command ansible-config might help confirming that.

Regarding your first question ("Why I have empty hosts.ini in my execute directory"), there is nothing in your playbooks pointing to that file being created or modified.  Did you mean to  have it modified?  Or you don't know how it was created?  The question there is not clear.


How to get rid of develop branch for simplified Git flow In a continuously developed web project (not a product) we currently have the following branching strategy, roughly based on git flow:

develop branch: latest working version
master branch: version to be released / released version
feature branches: features in development
hotfix branches: urgent bugfixes in released version

Master is read only, updated via pull requests from develop or hotfix branches. Each update results in a release candidate being built and deployed to the staging system. Release candidates are deployed to production after manual approval.
Feature branches are created based off develop, or from the last commit that has been merged into master. A pull request from a feature branch to develop is built, deployed to a free test system where integration tests and acceptance tests (automatic & manual) are executed. When successfully tested and reviewed, the PR gets merged, so that it will become part of the next release (i.e. merge from develop to master).
My goal
I would like to simplify this and get rid of the develop branch. The develop branch has mostly historical reasons and since it is always a successfully tested version, I think it is unnecessary to keep it separated from master. Removing it will also simplify the release process because there is no additional merge anymore.
I have the following constraints:

releases are scheduled and should not be fully automated
while feature branches are typically short lived, some stay unmerged for several weeks (e.g. a redesign) but need to be tested as well (currently as open pull requests to develop)
sometimes a single feature should be released outside the regular release, effectively turning it into a hotfix. With the current strategy I can rebase a feature branch and merge it directly into master
it also happens that we need to hold back features after acceptance tests with external systems on staging failed

Where I am not sure about the transition:

currently I am building pull requests for testing and merge commits for releases. Can I unify this?
how to deal with hotfixes when the master is ahead of the latest release. Should I build and deploy releases directly from hotfix branches?
is there a sensible way to deal with features that should be excluded from a release after they have already been merged? Is a separate develop branch really an advantage for these cases? Most of the time I end up reverting and re-reverting commits manually anyways.

Let's say you take out the master branch  (you can rename develop to master to confuse your team if you like later) and simply use tags for releases either on develop or hotfix branches. You took out a branch, but the difference is just a change in syntax. Change for change sake.

Now let's say you actually take out develop with keeping the locked master branch. What will happen is that integration of code will slow, it will live longer separated in feature branches, especially close to release dates. This will Increase the difficulty of merging each time and slow the process down.

Within the constraints you have laid down I don't see any positive effects of making such change. It would require relaxing the constraints, especially the first one.


monitoring/alerting system which can support very arbitrary data source and logic I am looking for a monitoring/alerting tool/system which can support very arbitrary data source and logic.

(following is a made up example to more easily demonstrate my needs)

I have a database where I can find out the query execution times for my clients(which are individual companies).  The query times are expected to vary greatly between clients, because of the data of each company.  I want to have an alert when the query time for a client is ?% over some median value from the past, and where I am comparing Monday's data against previous Mondays' data(I am being difficult for demonstration purpose).

I want to avoid defining a separate alert for each company client.

I want to avoid defining a separate alert for each kind of query I want to monitor.

I want to avoid having each source pre-push data to one central place.

I would like to monitor some other things that has nothing to do with databases, and want to apply some entirely different logic for my alerts - e.g. some REST api call to some other system, or some ELK or time series data, or some hardware monitoring data.

I am contemplating a custom solution, but want to see if there is already a solution out there that can meet some or more of my needs.  I have seen/used some monitoring solutions out there such as Nagios, though not an expert on those.  Will appreciate any pointers or recommendation.

Consider using something like nagios or Zabbix. While it might be nice to avoid custom queries and alerts, this is impossible. The odds of someone using your specific suite of tools in your configuration are slim-to-none - and the odds of having pre-canned monitoring for that suite in that specific configuration are even slimmer. Simply put, there is no way to avoid this custom monitoring. While monitoring, alerting and reporting management and configuration can be tedious, there is no easy shortcut for doing it properly and it's probably going to take more time, dedication and resources to do it properly than you think.

Out of the 4 monitoring systems I have used extensively, I must recommend nagios - it seems to be the best suited for configuration management systems and most compatable with DevOps philosophy.

It may sometimes be useful to use a system such as elastalert or similar systems as middleware for alerting/monitoring/reporting, but you will have to judge each situation/monitor on a case-by-case basis, but you will find these tools (and most monitoring systems) have some SDK, API or tool capable of helping you with your monitoring customization. Just choose the easiest to configure system with the lowest ROI and support cost and be sure to stick with it and follow your monitoring migration project to completion. Be sure to consider dedicated resources/personnel for this task if possible.