What are best practices for over-committing resources in a staging environment? Suppose there is a staging environment very strapped for resources as new development, data workloads and other applications grow. Ideally one would like to maintain production parity, even at the resource allocation level per each deployed item in the staging environment. For example, one of the 12 Factor App principles describes this, taking great pains to emphasize not just a shallow kind of parity with mocking or lightweight spoofed resources, but rather really maintaining parity by running the same resources in staging.

Obviously for projects that scale up a lot, you may be budget constrained and simply have no choice but to spoof some resources in staging, use "sample" data or microservice stubs with the same shape as what real dependencies will look like in production, and other sacrifices.

But one type of sacrifice that is difficult to approach is reducing the basic allocated resources, like CPU, RAM, disk access, networking resources, logging, etc., so that a deployment in the staging environment becomes quite a different entity if you are interested in testing behavior of the whole stack of components together.

There are various tools that can help you automatically reduce resource usage, such as auto-scaling in many cloud vendor environment and in kubernetes, which in effect can lead to "overcommitting" your staging environment if deployments are asking for the same kinds of resources as in production but being tuned to use less as a function of real load or other dynamic conditions.

My question is: what is the state of best practices or understanding about this practice of overcommitting resources in staging. How often does this type of requirement become a necessity from budget and what are the major principles to think about when designing a solution in that case? What about the flip side of the issue: that "the cost of doing business" for a particular deployed entity should have the cost of significant production parity in staging impounded in as a basic need?

Very good question; I'm interested to see what comes up here.

In terms of cost, obviously each company and team is different.  Ideally, everyone should be resource and cost conscious... but most people really don't seem to penny pinch by heavily monitoring and tuning their workloads (probably as that takes a certain skill level and tool set).

My Thoughts / Habits:

Unlike most on-prem environments, it is very easy to spawn up dynamic environments in the cloud (if you're automating properly).
So, when cost is prohibitive, you can just spin up full-scale staging environments in order to do load testing and then shut them down.
This will work for most use cases.  Obviously some things do only break when systems are alive a certain period of time under a certain load.
We usually keep identical but smaller environments in staging.  E.g. linearly scaled down presto clusters/spark clusters, k8s micro-services with less copies, and the like.
We try to keep the actual memory/cpu requests (or config) the same on any tool.  This allows for proper load testing without config changes for anything that scales.

Additional Thought

I'm pretty fond of the DevOps Toolkit book series by Victor Farcic.  If you're using kubernetes, the 2.5 book (https://www.devopstoolkitseries.com/posts/devops-25/) goes into detail on using prometheus and alert manager to detect over-sized or undersized resource requests, among other things.

If you're really into saving resources and testing well simultaneously, it may make most sense to implement and use this in staging and prod.  That way you can tune your alerting and automate your response so you can keep both environments sensibly sized for themselves in a consistent and enforced manner.

I think you'd still want to ensure you could set the staging environment (or another dynamic environment) to the prod settings to load test effectively though.


Terraform: Delay the MySQL Provider I have a Terraform project which uses the MySQL provider.  Because of the way Terraform works, the MySQL provider will fail to load correctly because the AWS_RDS instance doesn't exist.  Is there a way for me to delay or split my project so that the MySQL provider doesn't load during the main "apply"?

I've examined workspaces, and I see that you can use a count to trigger a resource to load or not load based off of the workspace name.  But that didn't work anyway I thought.

I've also seen this issue mentioned on the Hashicorp GitHub, but on one seems to be offering a workaround

Any help is greatly appreciated.

Here is a snippet of my data-storage configuration thus far:

resource "aws_db_subnet_group" "default" {
  name       = "${var.database_subnet_group_name}"
  subnet_ids = ["${var.database_subnets}"]

  tags {
    Name = "${var.database_subnet_group_name}"
  }
}

resource "aws_rds_cluster_instance" "cluster_instances" {
  count              = 1
  identifier         = "app-aurora-cluster-${count.index}"
  cluster_identifier = "${aws_rds_cluster.some_cluster.id}"
  instance_class     = "db.t2.medium"
  db_subnet_group_name = "${var.database_subnet_group_name}"
  publicly_accessible = true
}

resource "aws_rds_cluster" "some_cluster" {
  cluster_identifier = "app-aurora-cluster"
  availability_zones = ["${var.database_azs}"]
  database_name      = "${var.database_name}"
  db_subnet_group_name = "${var.database_subnet_group_name}"
  master_username    = "auroradmin"
  master_password    = "%XwPn}gU6sX<y8Wx"
  skip_final_snapshot =  true
  vpc_security_group_ids = ["${var.database_security_groups}"]
}
/*
# This has to be commented out during the first pass
# because Terraform will try to connect to the
# MySQL Aurora server to refresh the state even
# though it currently doesn't exist.
# Configure the MySQL provider
provider "mysql" {

  endpoint = "${aws_rds_cluster.some_cluster.endpoint}"
  username = "auroradmin"
  password = "%XwPn}gU6sX<y8Wx"
}

I think you could achieve something similar by first running Terraform against your RDS resources, and then running Terraform again on all resources.

On your first pass, you can use -target option to target only a subset of the resources. (ie. RDS resources).

Be aware that using this option is not recommended in Terraform docs. If I were you, I would use it for time being and change it as soon as one of these issues below is solved:
https://github.com/hashicorp/terraform/issues/10462
https://github.com/hashicorp/terraform/issues/2430
https://github.com/hashicorp/terraform/issues/4149

For details on how to use -target and its caveats, see: https://www.terraform.io/docs/commands/plan.html#resource-targeting


Keep getting permissions error gcloud.container.clusters.get-credentials I am trying to integrate CircleCi with gcloud Kubernetes engine.

I created a service account with Kubernetes Engine Developer and Storage Admin roles.
Created CircleCi yaml file and configured CI.

Part of my yaml file includes:

docker:
            - image: google/cloud-sdk
        environment:
            - PROJECT_NAME: 'my-project'
            - GOOGLE_PROJECT_ID: 'my-project-112233'
            - GOOGLE_COMPUTE_ZONE: 'us-central1-a'
            - GOOGLE_CLUSTER_NAME: 'my-project-bed'
        steps:
            - checkout
            - run:
                  name: Setup Google Cloud SDK
                  command: |
                      apt-get install -qq -y gettext
                      echo $GCLOUD_SERVICE_KEY > ${HOME}/gcloud-service-key.json
                      gcloud auth activate-service-account --key-file=${HOME}/gcloud-service-key.json
                      gcloud --quiet config set project ${GOOGLE_PROJECT_ID}
                      gcloud --quiet config set compute/zone ${GOOGLE_COMPUTE_ZONE}
                      gcloud --quiet container clusters get-credentials ${GOOGLE_CLUSTER_NAME}

Everything runs perfectly except that the last command:

gcloud --quiet container clusters get-credentials ${GOOGLE_CLUSTER_NAME}

It keeps failing with the error:

ERROR: (gcloud.container.clusters.get-credentials) ResponseError: code=403, message=Required "container.clusters.get" permission(s) for "projects/my-project-112233/zones/us-central1-a/clusters/my-project-bed". See https://cloud.google.com/kubernetes-engine/docs/troubleshooting#gke_service_account_deleted for more info.

I tried to give the ci account the role of project owner but I still got that error.

I tried to disable and re-enable the Kubernetes Service but it didn't help.

Any idea how to solve this? I am trying to solve it for 4 days...

I removed the service account and recreated it (with same name and roles). Generated a new key and updated the CI and suddenly it worked. gcloud glitch.


How can I send monitoring notifications to Google Chat in Google Cloud Platform? I want to send monitoring notifications to Google Chat in Google Cloud Platform. Under Monitoring > Alerting > Edit notificaton channels, Google Chat is not listed (only the competitor Slack, interesting...).
If I try to use a Webhook, the message never arrives in Google Chat. This is because the webhook request sent by Google has the wrong format - it does not include a "text" key in the JSON body (see here for the format of the request sent by Google).
Is it impossible to send monitoring notifications to Google Chat, even though the competitor Slack is available?

I think the current solution is to implement some script to send notification. I followed this guide and wrote a Python Cloud Function to do the notifications.


Problem with logging in docker when using fluentd for logging of three flask apps running in a docker network behind an nginx I am facing a strange problem with logging when I set up a network of three docker containers each running a small flask app.

My python application looks as follows:

from flask import Flask, request
import sys
import logging
import logging.handlers
import os

app = Flask(__name__)

logger = logging.getLogger('Logger')
logger.setLevel(logging.DEBUG)
log_handler = logging.StreamHandler(sys.stdout)
logger.addHandler(log_handler)

service_name=""
if 'SERVICE_NAME' in os.environ:
    service_name = os.environ['SERVICE_NAME']

logger = logging.LoggerAdapter(logger, {"serviceName" : service_name})

@app.route("/ping")
def ping():
    logger.info(f'answering ping request')

    return f'{service_name} OK'

if __name__ == "__main__":
    logger.info("Started")

    app.run(debug=True, host='0.0.0.0')

I start three instances of this in a docker network behind nginx:

docker network create tNet

docker run -it -d -p 24224:24224 -v `pwd`:/fluentd/etc -e FLUENTD_CONF=fluentd.conf --name=fluentd_logging fluent/fluentd

docker run -d --log-driver=fluentd --name ms1 -e SERVICE_NAME="ms1" --network tNet testflask
docker run -d --log-driver=fluentd --name ms2 -e SERVICE_NAME="ms2" --network tNet testflask
docker run -d --log-driver=fluentd --name ms3 -e SERVICE_NAME="ms3" --network tNet testflask

docker run -d --name=nginx --network tNet --volume `pwd`/nginx.conf:/etc/nginx/conf.d/default.conf -p 2000:2000 -p 3000:3000 nginx

The nginx configuration is as follows:

upstream app {
    server ms1:5000;
    server ms2:5000;
    server ms3:5000;
}

server {
    listen 2000;

    location / {
        proxy_pass http://app;
    }
}

My fluent configuration looks as follows:

<source>
    @type forward
</source>

<match>
    @type stdout
</match>

When I use curl I can see that each instance of the service is responding:

curl localhost:2000/ping
ms1 OK
curl localhost:2000/ping
ms2 OK
curl localhost:2000/ping
ms3 OK

When I however inspect the logs I can only see logs from the 3rd instance of the service:

docker logs <fluent container ID> | grep -Eis "answering ping"

19-08-06 15:34:46.000000000 +0000 be338a3d5aa0: {"container_id":"be338a3d5aa0b8920366fdf1ba2a267082cee47e681adcea16b6c51c9c82cf0d","container_name":"/ms3","source":"stdout","log":"answering ping request"}

Has anyone an idea? Thank you in advance.

I finally found out the problem: the fluent container needs some time to start up; because it's started with --detach the subsequent containers try to connect before fluentd is actually running, only the 3rd instance of the flask application connects when fluentd is already up. So, delaying the startup of the remaining containers solves my problem.


Microservices terminology i.e. distinguishing between "core " and "other service instances" In a standard cloud based microservice application, there could be two main categories of microservices:

core services such as service registry & discovery, api-gateway, load-balancer etc.
service instances e.g customers, billing and postage services.

In this illustration, I used  core services and service instances to distinguish the two main groups. Are there standard naming conventions for these two or is my classification/thinking right?

Here are some demo projects about the subject and each of these consists of the mentioned services: Weaveworks Socks Shop, Spring PetClinic and PiggyMetrics. These applications have top ranking stars on GitHub.

The last couple of years I am/was part of couple of DevOps teams that are/were creating microservices. Some of the projects failed miserably.

Reasons:

architect that only read a blog about microservices and claims that he is the expert of microservices
team members that lack knowledge, but indicate that they know everything about it as the architect told them about it
lack of DevOps mindset
no support from management
thinking that microservices are the holy grail, i.e. Y=>X instead of X=>Y

According to me, a DevOps mindset is that one works together and that one focuses on deliverables, i.e. new software live within two weeks. If one wants to create all microservices in advance then a lot of assumptions have to be done and nothing will be put to production, i.e. available to the customer.

This happened in one of my projects. The architects was continuously talking about "what if this would happen and this" and so on. Outcome? After one year, nothing was running in production.

In my opinion, some companies start with microservices because they have heard about it, but do not know exactly what microservices are. Microservices are not the holy grail, e.g. one could create nanoservices, one should master automation.

My point of view about microservices is that one should start with a monolith first, e.g. if you want a webshop just start with it and ensures that it is running in production within two weeks. Secondly, start monitoring it, check if there are problems in certain areas, e.g. more bugs in component X or component Y is extremely slow. If it is clear that component Y is slow, create a plan of extracting it, rewrite it within two weeks, apply autoscaling.

I am in favor of the "cheese slicer method". As the cheese itself is a product that delivers value and is running in production and the X=>Y approach is used. Start with a problem and check what could solve this problem, instead of thinking about a solution, e.g. microservices (Y) that could solve a (hypothetical) problem.

For your information. I am currently busy with a personal microservices project that consists of two microservices, e.g. an api and a write microservice. I am first solving the impediments and cleaning up my backlog. Off course I have some ideas about next microservices, but I want to put first things first and add value to the customers instead of doing a lot without finishing this microservice project that I have already experienced a couple of times.

Note: once the current backlog is complete, an additional paragraph about

  If Microservices terminology i.e. distinguishing between core and
  other service instances

will follow.

April 26 2018

At the moment I have a monorepo that consists of 5 microservices, i.e. an API, writer, selector, insert and tableCreator. On one hand it is nice as the shared code could be used directly, but when a code change is pushed then I need to build everything.

The next steps is to create five repos and extract the code from the monorepo to each individual repo. Advantages of this is that a code change to a repo builds the microservice that has been changed, smaller code base (easier to read).

The common code will be moved to separate repos as well and published as artifacts so they could be used by the microservices.

One of the prerequisites is that CI and CD are enabled for each repo. Otherwise it will take a lot of time before everything has been deployed.

Microservices are not the holy grail

Automation is key

"Do one thing and do it well".


What is happening in an Azure Devops build and release? I'm finding the Azure devops documentation pretty poor - or maybe I'm just not familiar enough with it for it to make any kind of sense to me. Having said that, I've managed to create a build pipeline that runs tests and published my project, then I created a release pipeline and have successfully released to an Azure App Service.

However, I honestly don't know what is happening behind the scenes...

I have this build pipeline yaml:

# ASP.NET Core
# Build and test ASP.NET Core projects targeting .NET Core.
# Add steps that run tests, create a NuGet package, deploy, and more:
# https://docs.microsoft.com/azure/devops/pipelines/languages/dotnet-core

trigger:
- master

pool:
  vmImage: 'Ubuntu-16.04'

variables:
  buildConfiguration: 'Release'

steps:
- script: dotnet restore
  displayName: Restoring Dependencies

- script: dotnet build --configuration $(buildConfiguration)
  displayName: 'dotnet build $(buildConfiguration)'

- task: DotNetCoreCLI@2
  displayName: Unit Tests
  inputs:
    command: test
    projects: '**/*Tests/*.csproj'
    arguments: '--configuration $(buildConfiguration)'

- task: DotNetCoreCLI@2
  displayName: Publishing
  inputs:
    command: publish
    publishWebProjects: True
    arguments: '--configuration $(BuildConfiguration) --output $(Build.ArtifactStagingDirectory)'
    zipAfterPublish: True

- task: PublishBuildArtifacts@1

I'm not sure if I need both the DotNetCoreCLI@2 and PublishBuildArtifacts@1 publishing tasks. Are they doing the same thing? Where are the files being published to? Can I see them?

And I've created a Release Pipeline where artifacts points at the above release pipeline:

#Your build pipeline references an undefined variable named ‘Parameters.ConnectedServiceName’. Create or edit the build pipeline for this YAML file, define the variable on the Variables tab. See https://go.microsoft.com/fwlink/?linkid=865972
#Your build pipeline references an undefined variable named ‘Parameters.WebAppKind’. Create or edit the build pipeline for this YAML file, define the variable on the Variables tab. See https://go.microsoft.com/fwlink/?linkid=865972
#Your build pipeline references an undefined variable named ‘Parameters.WebAppName’. Create or edit the build pipeline for this YAML file, define the variable on the Variables tab. See https://go.microsoft.com/fwlink/?linkid=865972

steps:
- task: AzureRmWebAppDeployment@4
  displayName: 'Deploy Azure App Service'
  inputs:
    azureSubscription: '$(Parameters.ConnectedServiceName)'
    appType: '$(Parameters.WebAppKind)'
    WebAppName: '$(Parameters.WebAppName)'

How is the release pipeline moving the files to the Azure App Service, and can I look to see what is going to the App Service?

Bit of a noob with devops as you might have guessed

Lets take this line by line:

My additions are added as comments.

trigger:
- master
# The branch you are building.

pool:
  vmImage: 'Ubuntu-16.04'
# The type of VM your code will run on.

variables:
  buildConfiguration: 'Release'
#This ends up defining which webconfig file to use.

steps:
- script: dotnet restore
  displayName: Restoring Dependencies

- script: dotnet build --configuration $(buildConfiguration)
  displayName: 'dotnet build $(buildConfiguration)'
# Builds your application
- task: DotNetCoreCLI@2
  displayName: Unit Tests
  inputs:
    command: test
    projects: '**/*Tests/*.csproj'
    arguments: '--configuration $(buildConfiguration)'
#Runs unit tests for your application
- task: DotNetCoreCLI@2
  displayName: Publishing
  inputs:
    command: publish
    publishWebProjects: True
    arguments: '--configuration $(BuildConfiguration) --output $(Build.ArtifactStagingDirectory)'
    zipAfterPublish: True
#Pushes your application as an artifact (You'll be able to see it in your list of artifacts in Azure DevOps)
- task: PublishBuildArtifacts@1

As for your release script, you are forgetting to pass in the values so your release script isn't doing anything at the moment.
But when it does work, it will take the objects from your artifacts or container images and deploy them into your webapp the same way it would do it if you ran those commands from your local machine.


Adding a persistent volume to a Kubernetes deployment from Gitlab Auto Dev-ops Recently, GitLab partnered up with Google Cloud to provide automatic deployment to Kubernetes clusters directly from project repos.

I enabled Auto Dev-Ops on my project, which just contains some Node.js and a Dockerfile, and as if by magic, I have a working deployment (after figuring out that I needed to make sure the exposed port was 5000 rather than anything else!)

But now I need to add some persistent storage to this. I'm aware of how to do this with pure Kubernetes, but I'm struggling to find out how to do this as part of my continuous deployment.

GitLab uses this script for Auto Dev-Ops, and I presume I might need to fork this to make my own .gitlab-ci.yml to add the custom volume, but I just cannot see anywhere in which this could be done.

The only other option I can work out is to use the provided PostgreSQL installation which does have a persistent volume to store my data, but this would just be skirting around the basic issue.

Do I need a custom buildpack? Do I just do it via the Google Cloud Dashboard?

The answer is that all Auto Dev-Ops deployments at present use Helm to package and deploy your application.

In order to make changes to the deployment, such as adding a persistent volume claim, you should download and modify the Helm chart that GitLab uses.

The files in this repo should go into a new directory chart/, or you can link to another location by setting the AUTO_DEVOPS_CHART project variable to a URL.

This page has more details.


How to cache third party resources in a build pipeline? A build-pipeline often requires to access third party resources, aside from the source code of the artefacts it is meant to build. Each of these third party resources introduce a risk factor in builds, for instance:

Distribution repositories are temporarily unavailable, therefore installing distribution packages temporarily fails.
Distribution repositories are permanently unavailable, same and worse as before.
Some NPM free-software package stops to exist, because its author reclaimed that withdrawal.
Some source file or binary ad-hoc package for a third-party tool is not available anymore.

This list could be extended ad lib and while studying the various reasons leading to some third-party resource to disappear is much more entertaining as it sounds, it first of all has for us a disastrous consequence: the build-pipeline is broken.

What the is the best way to cover the two following cases when building docker images:

Caching files downloaded with a curl or wget.
Caching Debian packages.

There are several plausible strategies that can be considered here. For instance using a proxy and configuring the build system to route all its requests through this proxy . A second, cheap, possibility would be to wrap curl calls to store results in a local cache.

(In this question I would consider sharing the cache among several units of the build pipeline as a question of secondary importance.)

I'm not overly familiar with Docker particularly but it sounds a lot like issues one would get when building a normal .NET application or java app with the need to store their external dependencies (DLL or jar).

To solve this I'm quite fond of artifactories. Two of the most popular are sonatype's Nexus and JFrog's Artifactory.  Both offer a free Open Source version and both will have some way to store customised package or use it as a raw depot.  There are other options out there, I just listed these two because I've worked with them already and know they will do what you need.  As a bonus you might also be able to use them to manage the docker image directly as well.

There are others too worth mentioning, you can find a comparison table for them, you'll get a better view of their capabilities. (thanks to Karl Harnagy for the link)

Note that some of these options might require one or another of the paid versions so YMMV.

Some continuous integration systems will also support the concept of artifacts (from experience Jenkins and team-city do this, others might as well though) which should allow you do achieve similar results directly.

If none of these do it for you you can look into other technologies or roll your own.  I personally like to leverage whatever I already have this reduces the maintenance required in long term and makes passing the buck easier when comes time to look for new challenges elsewhere.

Hope this helps


Jenkins - sending email after build with the build version as a subject I would like to send an email at the end of my build to inform success or failure. I'm using the email ext-plugin for Jenkins. I want the version number of the build to be the subject of the email.

The problem is that the version is something that changes in each build, so I need to somehow edit an environmental variable to hold the version number each time (I have a script that saves it into a file). This is the default configuration:

I would like to have, for example, set the version number into a variable using windows batch script, like this:

rem # Loading version File number
set MY_BUILD_VERSION=<myVersionFile.txt
rem # Loading content
set MY_BUILD_CONTENT=<myContentFile.txt

And using it in the email plugin, like this:

The problem is that the environmental variables aren't recognized outside the .bat file they are being run from, although I'm using the envInject plugin, the documentation says that what's run from a script won't be recognized outside of it.

Is there any way to load data into the email plugin from a file, or any other dynamic way to load my version number dynamically?

I found a way to set environmental variables globally from a file!
I'm using envInject for loading variables from a file.

For example, lets say I have a file that is called "versionNumber.txt" and that it's created during the build process. Maybe I update it in every build, or any other way. then I create a prosFile that can be identified by the envInject, by running a batch script:

set /p VERSION_NUM=<versionNumber.txt

Let's assume my version number was 1.5.5.10. so now my VERSION_NUM equals that string.
now VERSION_NUM has the value that I want. But it's local, it won't help me. so I rewrite it to a file in the following format (also, using a batch script):

echo VERSION_NUM=%VERSION_NUM% > C:\Temp\prosFile

now I have a file that contains the following content:

VERSION_NUM=1.5.5.10

And now I just need to load the file using the envInject plugin:

And now I have an environmental variable named VERSION_NUM that can be used anywhere in Jenkins. But I wanted to make it the subject of my email, so I'll simply do (in windows, in linux it might be different):

That's it!


Launch jenkins agent from jenkins job? Is it possible to launch a jenkins agent from within a script in another job? I'm open to either shell or pipeline options.

To be clear I don't mean the agent the job is running on itself. I'm talking about spinning up a completely separate agent.

Sometimes our master gets slammed with unit test requests and will make jobs wait in the queue for 15-20+ minutes instead of just spinning up more agents. I want to see if there's a way to spin an agent up intentionally so I can tie it into a job that polls the queue for wait times.

What you are looking for is called dynamic provisioning of Jenkins' slaves, and yes this is doable. Setting it up depends upon your desired infrastructure provider/resource. The easiest example in my opinion is using AWS EC2 instances. Install the Amazon EC2 Plugin for Jenkins, and configure it with the appropriate access keys, IAM roles, etc. to allow it to provision EC2 instances in a specified VPC. Here is a short (but slightly outdated) guide.

If you are moving to K8's and Docker, you may want to consider looking into the Docker Plugin for Jenkins. If you are using bare bones infrastructure, this should allow you to more effectively utilize what current hardware you have.


What is a difference between a Walking Skeleton and an MVP? I've recently stumbled upon the "Walking skeleton" term and got a bit confused about the terminology.

Is it the same or a variation of a Minimum Viable Product, or how does it relate to MVP in general?

The walking skeleton reflects the rough structure/shape of the product, but it may be entirely stubbed out (initially), meaning it may not be actually functional. The skeleton cannot move without the muscles.

The MVP however must meet a certain minimum functionality level to be considered viable.

As development progresses the walking skeleton typically morphs into an MVP and eventually into a final product.


What are best practices of deploying docker-compose using Ansible? Ansible is used to deploy docker-compose on a system and subsequently start the container.

When ansible runs from the local system, the docker-compose file is deployed using Ansible.

When the code is pushed to bitbucket the file content is:

../../../../docker-compose.yml

and the CI deploys this on the system as well.

Structure

project/ansible

project/docker-compose.yml

project/ansible/roles/role/files/docker-compose.yml

Discussion

In order to prevent that the docker-compose.yml need to be changed on two places, a symlink has been created, but that seems to be ignored by bitbucket as the link instead of the content reside in this file.

This Q&A indicates that:

  The data referenced by the symlink is not stored in the repository.

and creating a symlink to avoid code duplication in docker-compose.yml seems to cause the problem.

Instead of using a symlink that will be stored as is in BitBucket one could use:

- name: Copy docker-compose.yml
  copy:
    src: ../../../../docker-compose.yml
    dest: /path/to/docker-compose.yml


Get vars in Jenkins Pipeline In FreeStyle Project, I have lots of env vars, which are available from any part of Jenkins' job. I also used ghprb plugin to trigger that job against GH PRs, which added some extra env vars.

Later, that job became more complex, so I've decided to put it on Jenkins Pipeline. But now, the issue is, I can't retrieve those env vars in a pipeline, which were available before, in the Freestyle Project.

I'm a newbie in writing Jenkins Pipeline, that's why I decided to use declarative approach for that purpose (but I don't think it's root cause). So, my question is, how to retrieve env vars in Jenkins Pipeline, such as: ${BUILD_CAUSE}, ${ghprbTargetBranch}, ${ghprbActualCommitAuthorEmail}?

If you want to use environment variables with the ${VAR} syntax in the declarative pipeline, then you need to define those variables in the environment { } block.

for example:

environment {
        DISABLE_AUTH = 'true'
        DB_ENGINE    = 'sqlite'
    }

    stages {
        stage('Build') {
            steps {
                echo "Database engine is ${DB_ENGINE}"
                echo "DISABLE_AUTH is ${DISABLE_AUTH}"
                sh 'printenv'
            }
        }
    }

If you want to use environment variables that come from other plugins etc, then you need to access them with the env.VAR syntax.  For example:

echo "Running ${env.BUILD_ID} on ${env.JENKINS_URL}"


Docker not exposing port on mojave mac I used brew to install docker, docker-machine and docker-compose. Everything is working, but when I use expose to map port on my host machine, it is not working.

When I ssh in docker machine service is running perfectly inside docker machine, but outside nothing coming up.

My software versions are

Docker version 18.09.5, build e8ff056

docker-machine version 0.16.1, build cce350d

docker-compose version 1.24.0

and I am using xhyve for virtualization

Here is my Dockerfile

FROM node
  LABEL maintainer xyz@gmail.com
  RUN git clone -q https://github.com/docker-in-practice/todo.git
  WORKDIR todo
  RUN npm install > /dev/null
  EXPOSE 8000
  CMD ["npm", "start"]

I use following command to run docker

docker run -i -t -p 8000:8000 --name example1 todoapp

I have read many article and noticed there is issues in docker and mac related, but not sure how can I solve it. Please any suggestion. I am not using mac docker app due to system resources limit.

I also read somewhere that I should use sudo than it should work, but I tried and get following error

docker: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

I think sudo will require when port is below 1000 but my port is 8000 so I don't think I need sudo here.

Most likely the port is published on the VM and you are trying to connect to your laptop's IP and not seeing it. Newer versions of docker that use xhyve include some networking tricks to publish the port all the way to your laptop, but docker-machine with VirtualBox does not do this.

You can lookup the IP of your VM with docker-machine ls or see it in the variable echo $DOCKER_HOST and connect to port 8000 on that IP to reach your application.


Automated tag versioning on unchanged docker images I am starting to use Continuous Integration on my microservices solution (monorepo approach), and I am referencing this post for versioning approach.

But, let's say I make some change in one of the microservice and commit that change, but all the others are exactly the same. How the process would know that those microservices (docker images) weren't changed so don't need to increase the build number? I feel like I would end-up with many docker images with different build numbers but same content.

Am I misunderstanding the concept? Sorry, this whole microservices and docker thing are pretty new to me.

How it should look like:

Commit changes in some microservice and push it to VCS origin repo
CI system sees new commit
CI starts build and tests
If everything's good, it pushes image to docker registry with tags

So, the answer is: CI should know, what service has changes.


How to have jenkins add a review to a pull request I have a Jenkins instance, with credentials for a "robot" github account, which we use to interact with the GitHub API as a team. Typically, jobs are defined as declarative pipelines. I am looking for the correct syntax to perform the following steps:

At the start of the run, create a review
At the end of each stage, post a review comment, with the status of the stage, e.g. "build passed".
If all of the stages pass, approve the pull request.

I can allow scripted steps, but the pipeline as a whole should be declarative. No shared libraries, and preferably only using commonly-used plugins.
(update with what I have tried)
I have tried using the pipeline github plugin's review method:
pullRequest.review('APPROVE')

naively, but this throws errors.
What would be the correct syntax here?

I guess the simplest and direct solution would be to use GitHub API: https://docs.github.com/en/rest/reference/pulls#create-a-review-for-a-pull-request
But for the requirement you have, PR checks is usually the way to go and you have a plugin that does the API communication for you: https://www.jenkins.io/doc/pipeline/steps/pipeline-githubnotify-step/


Deploying different versions of the same application stack in the same system We are a small team with few resources and we have a product which is near completion. The stack comprises of a few Java Spring Applications, a Python Application and a Windows Application and using a Postgresql DB. We have a single physical system where till now, we used to deploy all of these application and test them.

Now we are in a situation where we would need to have multiple versions of this stack running at the same time. For eg. there should be a stable running version and at least 2 versions under testing. We want to achieve this using the single Windows system we have. Is there a way to achieve this? How viable is using multiple Docker images?

First of all, I would not recommend to use a single system for running stable and testing version both on the same system. There are many reasons for this such as:

Your testing version can cause cpu and memory spikes on system which will ultimately affect your stable version and end users.
Secondly, it will be difficult to manage the dependencies and resources for multiple versions in a single system.
It can also take a lot of debugging time if the monitoring is not properly setup.

Ideally, you should use 2 different system. One for running the stable version and other testing versions. It will be easy to manage and always a good practice to keep it on different stack.

Still, if you want to use a single system, then yes, you can use docker which will avoid a lot of conflict issues which can arise. Also, the dependencies and shared libraries can be managed easily with different docker images and running docker containers.

If the team is proficient in docker, you can also use docker swarm to setup clusters. You can setup different clusters for stable and testing versions.

I hope you chose the best possible option and implement it.


Run docker inside a VirtualBox of Windows Server 2016 (GUI) on Mac OS I have downloaded the Windows Server 2016 iso from Microsoft for evaluation.

I then installed it via VirtualBox on my Macbook Pro running the latest Mac OS (High Sierra).

I downloaded Docker for Windows but when I try to load it, it either fails with an error saying it cannot write to database, or it fails because it says virtualization / Hyper-V has to be enabled in BIOS.

I have Hyper-V selected in the VirtualBox settings so that should work (and restarted the virtual machine).

However it still fails.

Is it possible to run Docker for Windows on a Windows Virtual Machine, all hosted on Mac OS?

According to me this is not possible as the hyper-v should be enabled on the host system. If this is enabled in a VirtualBox, but not on the host system, then this will not work according to what I have experienced.


Practical reasons for 200 limit of AWS resources in CF templates Not sure I get it. If my service layer is completely maintainable, indepedent services and very easy to maintain taking only a few minutes to deploy and has no bugs, AWS still won't let me deploy it if it uses 201 resources. This seems like dogma or am I wrong?

The 200 limit can be raised, however this is a grinding process that can take months. You work through customer support to get "elevated" as a client to more AWS privileges.

Resources:
- Worked under a company that specifically ran into this problem, took 8 months before that limit was lifted.


Grant access permission to file when mounting from host Below is except of my docker-compose.yml file

I am trying to mount from host file inside the container, which works i.e. able to mount
but then I need to give it permission (my clumsy chown command, below).

I am not sure where to run it as wherever I place my (command: chown,-R,elasticsearch,/usr/share/elasticsearch/config/)

It gives me error that 'no such file or directory' which I think: my code run before volume is attached.

I tried placing it after container_name as well, same 'no such file or directory'

Any advise how to do it?

image: docker.elastic.co/elasticsearch/elasticsearch:7.6.2
container_name: elasticsearch

volumes:
  - ./elastic-stack-ca.p12:/usr/share/elasticsearch/config/elastic-stack-ca.p12
  - esdata1:/usr/share/elasticsearch/data
*command: chown,-R,elasticsearch,/usr/share/elasticsearch/config/*

Here is full yml, if it is helpful

    version: '3'

services:

  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.6.2
    container_name: elasticsearch
    command:
     chown,-R,elasticsearch,/usr/share/elasticsearch/config/
    environment:
      - node.name=elasticsearch
      - discovery.seed_hosts=elasticsearch
      - cluster.initial_master_nodes=elasticsearch
      - cluster.name=docker-cluster
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"

    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - ./elastic-stack-ca.p12:/usr/share/elasticsearch/config/elastic-stack-ca.p12
      - esdata1:/usr/share/elasticsearch/data
    ports:
      - 9200:9200

  kibana:
    image: docker.elastic.co/kibana/kibana:7.6.2
    container_name: kibana
    environment:
      ELASTICSEARCH_URL: "http://elasticsearch:9200"
      ELASTICSEARCH_USERNAME: "kibana"
      ELASTICSEARCH_PASSWORD: "kibana"
    ports:
      - 5601:5601
    depends_on:
      - elasticsearch

volumes:
  esdata1:
    driver: local

I don't think volumes can be specified with relative paths.
Put an absolute path instead of ./elastic-stack-ca.p12 and you should be fine.


How do I enter a multi line variable in Terraform? It's for the public key variable for testing lab enviroments in my variables.tf file
I tried

<<EOF
the string
EOF

but terraform dosn't seem to like that

Terraform variable type constraints separate "primitive" types from "complex" types. These primitive types include strings:

A primitive type is a simple type that isn't made from any other types. All primitive types in Terraform are represented by a type keyword. The available primitive types are:

string: a sequence of Unicode characters representing some text, such as "hello".
number: a numeric value. The number type can represent both whole numbers like 15 and fractional values such as 6.283185.
bool: either true or false. bool values can be used in conditional logic.

So, simply declaring the variable as a string, with the linebreaks in it would suffice.


S3 bucket policy errors I have the following bucket policy for Amazon S3, which is giving the following error:

  Error: Invalid bucket policy syntax.

Either it is a JSON syntax error or a Policy error.

This is the policy I am trying and failing with:

{
    "Version": "2012-10-17",
    "Statement": [{
        "Effect": "Allow",
        "Action": [
            "s3:ListBucket",
            "s3:GetBucketLocation",
            "s3:ListBucketMultipartUploads"
        ],
        "Resource": "arn:aws:s3:::<my bucket>",
        "Principal": {
            "AWS": [
                "arn:aws:iam::560184616970:user/<my username>"
            ]
        },
        "Statement": [{
            "Effect": "Allow",
            "Action": [
                "s3:AbortMultipartUpload",
                "s3:DeleteObject",
                "s3:DeleteObjectVersion",
                "s3:GetObject",
                "s3:GetObjectAcl",
                "s3:GetObjectVersion",
                "s3:GetObjectVersionAcl",
                "s3:PutObject",
                "s3:PutObjectAcl",
                "s3:PutObjectAclVersion"
            ],
            "Resource": "arn:aws:s3:::<my-bucket>/*",
            "Principal": {
                "AWS": [
                    "arn:aws:iam::560184616970:user/<my-username>"
                ]
            },
            "Statement": [{
                "Effect": "Allow",
                "Action": "s3:ListAllMyBuckets",
                "Resource": "*"
            }]
        }]
    }]
}

Can anyone give me guidance here?

It looks like you tried to compile multiple policies into one.
Each original had a statement with its array of statements and you nested them.

This (untested) should work with each statement block within the array of Statement of the policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:ListBucket",
        "s3:GetBucketLocation",
        "s3:ListBucketMultipartUploads"
      ],
      "Resource": "arn:aws:s3:::<my bucket>",
      "Principal": {
        "AWS": [
          "arn:aws:iam::560184616970:user/<my username>"
        ]
      }
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3:AbortMultipartUpload",
        "s3:DeleteObject",
        "s3:DeleteObjectVersion",
        "s3:GetObject",
        "s3:GetObjectAcl",
        "s3:GetObjectVersion",
        "s3:GetObjectVersionAcl",
        "s3:PutObject",
        "s3:PutObjectAcl",
        "s3:PutObjectAclVersion"
      ],
      "Resource": "arn:aws:s3:::<my-bucket>/*",
      "Principal": {
        "AWS": [
          "arn:aws:iam::560184616970:user/<my-username>"
        ]
      }
    },
    {
      "Effect": "Allow",
      "Action": "s3:ListAllMyBuckets",
      "Resource": "*"
    }
  ]
}

An alternative could be to use aws bucket policy generator


How can I autoscale instances based on Kafka topic lag? I have autoscaling groups that I want to dynamically scale up/down based on the amount of lag showing in certain Kafka topics. There's a 1:1 relationship between topic and autoscaling group that needs to be scaled. I'm fairly sure I can't use CloudWatch metrics to approach this.

Is there a way I could accomplish this using Jenkins without creating race conditions and other issues?

Note: My application is not constrained by CPU, Disk I/O, or any other metric provided by CloudWatch; the constraint is the application's ability to process from incoming Kafka topics.

CloudWatch is absolutely the way to go on this front, you can surface Kafka Metrics in CloudWatch either by:

Having a separate process that pushes the metrics into CloudWatch.
Have your Producers, Consumers or Stream Processors push the metrics you need into CloudWatch.

The process you follow will broadly be broken down into:

Publish Custom Metrics.
Create an AutoScaling Launch Configuration.
Create an AutoScaling Group and Policies based upon your Custom Metric.

Depending on your cost model I would also consider how you can use spot instances to keep the costs down, i.e. if you have non-critical but high compute workloads use Spot Instances as part of your autoscaling configuration.


Docker-compose + Kubernetes: backports.ssl_match_hostname.CertificateError I have a complete docker stack (source) which uses docker-compose.

It works fine by its own, but now I'm trying to deploy it in Kubernetes' Minikube and when after eval $(minikube docker-env) I try to run docker-compose up, it throws the following error:

  backports.ssl_match_hostname.CertificateError: hostname '192.168.99.100' doesn't match 'localhost'

What am I missing?

This is the complete trace:

  Traceback (most recent call last):

  File "/usr/bin/docker-compose", line 9, in
      load_entry_point('docker-compose==1.8.0', 'console_scripts', 'docker-compose')()

  File "/usr/lib/python2.7/dist-packages/compose/cli/main.py", line 61, in main
      command()

  File "/usr/lib/python2.7/dist-packages/compose/cli/main.py", line 113, in perform_command
      handler(command, command_options)

  File "/usr/lib/python2.7/dist-packages/compose/cli/main.py", line 835, in up
      remove_orphans=remove_orphans)

  File "/usr/lib/python2.7/dist-packages/compose/project.py", line 372, in up
      warn_for_swarm_mode(self.client)

  File "/usr/lib/python2.7/dist-packages/compose/project.py", line 539, in warn_for_swarm_mode
      info = client.info()

  File "/usr/lib/python2.7/dist-packages/docker/api/daemon.py", line 33, in info
      return self._result(self._get(self._url("/info")), True)

  File "/usr/lib/python2.7/dist-packages/docker/utils/decorators.py", line 47, in inner
      return f(self, *args, **kwargs)

  File "/usr/lib/python2.7/dist-packages/docker/client.py", line 140, in _get
      return self.get(url, **self._set_request_timeout(kwargs))

  File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 480, in get
      return self.request('GET', url, **kwargs)

  File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 468, in request
      resp = self.send(prep, **send_kwargs)

  File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 576, in send
      r = adapter.send(request, **kwargs)

  File "/usr/lib/python2.7/dist-packages/requests/adapters.py", line 376, in send
      timeout=timeout

  File "/usr/lib/python2.7/dist-packages/urllib3/connectionpool.py", line 560, in urlopen
      body=body, headers=headers)

  File "/usr/lib/python2.7/dist-packages/urllib3/connectionpool.py", line 346, in _make_request
      self._validate_conn(conn)

  File "/usr/lib/python2.7/dist-packages/urllib3/connectionpool.py", line 787, in _validate_conn
      conn.connect()

  File "/usr/lib/python2.7/dist-packages/urllib3/connection.py", line 277, in connect
      match_hostname(cert, asserted_hostname)

  File "/usr/lib/python2.7/dist-packages/backports/ssl_match_hostname/init.py", line 99, in match_hostname
      % (hostname, dnsnames[0]))

  backports.ssl_match_hostname.CertificateError: hostname '192.168.99.100' doesn't match 'localhost'

It seems you have missed DNS. First, don't reference the device by IP, instead use a FQDN. If you need, you can set an entry for 192.168.99.100 in /etc/hosts for now. Secondly, type "hostname" on the node and verify it does not return "localhost" (it likely will). Then type "hostnamectl set-hostname newhostname.fullyqualified.com" and then regenerate and re-sign the certificate.


GitLab Runner is using its Docker host's hostname as Git server FQDN/hostname When using a GitLab Runner with a "docker" executor, the following build error occurs:

Running with gitlab-runner 10.7.0 (7c273476)
  on main 80500676
Using Docker executor with image centos:latest ...
Pulling docker image centos:latest ...
Using docker image sha256:e934aafc22064b7322c0250f1e32e5ce93b2d19b356f4537f5864bd102e8531f for centos:latest ...
Running on runner-80500676-project-1-concurrent-0 via hostname-of-gitlab-runner-and-gitlab-server...
Cloning repository...
Cloning into '/builds/some-group-name/some-repository-name'...
fatal: unable to access 'http://gitlab-ci-token:xxxxxxxxxxxxxxxxxxxx@hostname-of-gitlab-runner-and-gitlab-server/some-repository-name/some-repository-name.git/':
Could not resolve host: hostname-of-gitlab-runner-and-gitlab-server
ERROR: Job failed: exit code 1

It looks like the GitLab Runner is using the hostname of its Docker host as the FQDN or hostname of the Git server to pull the code from. This doesn't make sense, as it's not in any of the configuration files:

This is the GitLab configuration:

$ sudo cat /etc/gitlab-runner/config.toml
concurrent = 1
check_interval = 0

[[runners]]
  name = "main"
  url = "http://127.0.0.1:1337/"
  token = "80500676b5518df3f3f3b1f772e20b"
  executor = "docker"
  [runners.docker]
    tls_verify = false
    image = "ubuntu:latest"
    privileged = false
    disable_cache = false
    volumes = ["/cache"]
    shm_size = 0
    #extra_hosts = ["hostname-of-gitlab-runner-and-gitlab-server:actual-fqdn-of-gitlab-server-where-git-repo-is"]
  [runners.cache]

This is the relevant configuration for external_url in the GitLab server configuration:

$ sudo egrep '\bexternal_url\b' /etc/gitlab/gitlab.rb
##! For more details on configuring external_url see:
external_url = 'https://server:actual-fqdn-of-gitlab-server-where-git-repo-is'
# gitlab_pages['artifacts_server_url'] = nil # Defaults to external_url + '/api/v4'

This is the relevant configuration for host: in the GitLab server's generated configuration:

$ sudo egrep -C 2 -n '^    host: ' /var/opt/gitlab/gitlab-rails/etc/gitlab.yml | head -n 5
11-  gitlab:
12-    ## Web server settings (note: host is the FQDN, do not include http://)
13:    host: hostname-of-gitlab-runner-and-gitlab-server
14-    port: 80
15-    https: false

When I change hostname-of-gitlab-runner-and-gitlab-server to actual-fqdn-of-gitlab-server-where-git-repo-is, and run sudo gitlab-ctl reconfigure, it gets reset.

If I edit gitlab.host in /var/opt/gitlab/gitlab-rails/etc/gitlab.yml directory, and run sudo gitlab-ctl restart without running sudo gitlab-ctl reconfigure, it will stay intact.

Some unanswered questions are:

How can I permanently set the gitlab.host configuration value that
gets generated in /var/opt/gitlab/gitlab-rails/etc/gitlab.yml when
sudo gitlab-ctl reconfigure is run?
Why does gitlab.host in
gitlab.yml keep getting set to the hostname of the GitLab server?

Neil, it's because the = sign in your /etc/gitlab/gitlab.rb file.

When you'll look on the full comment before the external_url line, you'll see a link to the documentation:

## GitLab URL
##! URL on which GitLab will be reachable.
##! For more details on configuring external_url see:
##! https://docs.gitlab.com/omnibus/settings/configuration.html#configuring-the-external-url-for-gitlab
external_url 'http://gitlab.example.com'

In the default gitlab.rb file (just after installing the Omnibus GitLab package) and also in the linked documentation you can see, that the external_url value is assigned without the = sign. It's done like that because the external_url is a method, not a variable. So here you're not assigning the value, but calling the method with this value. Ruby allows to call a method without brackets around the arguments, and this is a commonly used pattern when designing a DSLs based on Ruby.

When you try to set the value with:

external_url = 'https://server:actual-fqdn-of-gitlab-server-where-git-repo-is'

then the external_url method is not receiving any argument and in that case Omnibus GitLab falls back to a default value which is based on host's hostname.

In most cases the hostname of the server where GitLab is installed will be the same as the hostname that the server is accessible with. In some cases - including yours - the external_url needs to be set explicitly.

So the solution seems to be:

Change external_url = 'URL_HERE' to external_url 'URL_HERE'
Execute sudo gitlab-ctl reconfigure

This should set the proper values in gitlab.yml file and resolve your problems.


How to connect to host during a Docker build? I have a Dockerfile which I want to run some tests inside of as part of a multistage build. The tests attempt to make a connection to localhost, however it appears that they're not able to connect to the specific port (which is ready on the host machine) inside the RUN command of the Dockerfile. Is such a thing possible, and if so how do I achieve it? Obviously external connections can be made at build time (to download dependencies and such) and loopbacks at run time

FROM python:3.7-alpine as tester
COPY . .
RUN python -m unittest discover

And for reference, the Python code checking the connection

def setUp():
    timeout = 120.0
    port = 15903
    host = 'localhost'
    start_time = time.perf_counter()
    while True:
        try:
            with socket.create_connection((host, port), timeout=timeout):
                break
        except OSError as ex:
            time.sleep(1.0)
            if time.perf_counter() - start_time >= timeout:
                raise TimeoutError('Waited too long for the port {} on host {} to start accepting connections.'.format(port, host)) from ex

From your container's perspective localhost is the container itself.  If you want to communicate with the host machine, you will need the IP address of the host itself.  Depending on your Operating System and Docker configuration, this IP address varies.

edit host.docker.internal is recommended from 18.03 onward.

Here is an example of communicating with the Docker host from a container on Mac:

# Dockerfile

FROM python:3.7-alpine

RUN ping host.docker.internal -c 4

sending build context to Docker daemon  562.7kB
Step 1/2 : FROM python:3.7-alpine
 ---> a93594ce93e7
Step 2/2 : RUN ping host.docker.internal -c 4
 ---> Running in 54ce465f7386
PING host.docker.internal (192.168.65.2): 56 data bytes
64 bytes from 192.168.65.2: seq=0 ttl=37 time=0.735 ms
64 bytes from 192.168.65.2: seq=1 ttl=37 time=1.898 ms
64 bytes from 192.168.65.2: seq=2 ttl=37 time=0.720 ms
64 bytes from 192.168.65.2: seq=3 ttl=37 time=0.643 ms

--- host.docker.internal ping statistics ---
4 packets transmitted, 4 packets received, 0% packet loss
round-trip min/avg/max = 0.643/0.999/1.898 ms
Removing intermediate container 54ce465f7386
 ---> b2f32c26a135
Successfully built b2f32c26a135


Puppet directory sync does not delete absent files Repeating my question from SO.SE here ...

I'm trying to have my configs directory (on Windows server 2016 node) to be synced from puppet master (puppet --version = 5.5.3) modules folder: I have my desired structure repeated under module/files. This:

file { "c:\\":
        ensure => directory,
        recurse => remote,
        source => "puppet:///modules/configs"
        }

works for files that exist in source dir, but does not delete files that are deleted from source (from docs I understood that it should) also I tried:

file { "c:\\":
        ensure => directory,
        recurse => true,
        purge => true,
        source => "puppet:///modules/configs"
        }

but then I get an error:

  Error: /Stage[main]/Main/Node[nodename]/File[c:]: Failed to generate additional resources using 'eval_generate': CreateF
  ile(c:/Documents and Settings, 20000000000, 1, , 3, 210000000, 0):  Access is denied.

I was not able to find even descent source telling possible cause for the error; Access denied makes no sense as puppet agent runs with system privileges and should be able to access every path, also script should have nothing to do with c:/Documents and Settings path to my understanding.

Any help to solve this?

According to https://tickets.puppetlabs.com/browse/PUP-4759 the issue seems to be solved.


How to automate control of synchronization between Maven artefact version and Git tag? Given the following situation:

in a POM file in the repo root (Atlassian BitBucket) I can define that this POM has e.g. the version 0.0.1-SNAPSHOT
then I can tag the codebase whatever I want like v.0.1.

This would be then inconsistent versioning. Is there a better/established way than some custom scripting to control sync of this?

then I can tag the codebase whatever I want like v.0.1

One could use semantic versioning, i.e. X.Y.Z. instead of v.0.1


How does one build a build pipeline with versioned build artifacts? I'm not sure quite how to phrase this question. I'd like to learn either by answer here or sharing a reference to read from: How do I build a pipeline with versioned build artifact?
I've inherited an established Jenkins build infrastructure that I recognize needs to change. This is especially evident because we want to move into a modern build approach with branches (dev, main, etc.).
This is basically how our builds work
lib-A    is created first
lib-B    is created next
lib-C    is the last library created
prod-DA  uses lib-A
prod-EC  uses lib-C
prod-FAB uses both lib-A and lib-B

Our current build infrastructure is what I'd describe as a dev infrastructure. In that it always uses latest.
When lib's A, B and C are created they are dropped into a shared folder to be used by everyone else.
When prod's DA, EC and FAB need to link with the library they just pull from the shared folder.
This is what I inherited. It's adequate for a single branch, riding-latest build system.
Is there a resource, pattern, book, website or toolset to help me learn how to change this to a versioned artifact scenario? Or how to convert to a mechanism that will scale and support branching?
I don't really even know what to call this, I'm really wanting to learn what you have to share.
Thank you for your patience.

I know this question was asked a long time ago but I still want to answer it.
How products are build, versioned and delivered depend highly on the combination of the used tools, the infrastructure and the processes that have been established in your company.
This also means there is no general way to it. However there are of course best practices like continuous integration/continuous delivery, nightly builds, common branching patterns etc.
In your case I would start with common branching patterns for the used version control system (svn, git, etc.) and think about how to support versioned artifacts .
Next step is to get the build management tool (Maven, Gradle, Make, Cmake, etc.) build the specified version.
If these two parts are working you can think about how to automate it with jenkins.
One last thing: Your question was very general. I guess if you ask more specific questions like "How do I combine maven and git for building versioned artifacts" you get more answers.


Terraform Issue - Can't find Data Source in Sub Module? I'm fairly new to Terraform and have been following this example on creating an EKS cluster - https://github.com/terraform-providers/terraform-provider-aws/tree/master/examples/eks-getting-started

Only difference is that I've broken the structure up into modules:

-main.tf
-vpc
   vpc.tf
   vars.tf
-eks
   eks-cluster.tf
   eks-worker-nodes.tf
   vars.tf
   outputs.tf

In 'main.tf' I've got the following

provider "aws" {
  region     = "us-east-1"
}

# Using these data sources allows the configuration to be
# generic for any region.
data "aws_region" "current" {}

data "aws_availability_zones" "available" {}

When I run a 'terraform plan' I get the following error which comes from eks-worker-nodes.tf :

 * module.eks.local.demo-node-userdata: local.demo-node-userdata:
 Resource 'data.aws_region.current' not found for variable
'data.aws_region.current.name'

Is there any reason why it wouldn't be able to access the data source? I'm fairly new to Terraform but I've done some googling and couldn't find a clear answer :/

You need to pass that info to your module or also declare it in your module again. For example, for the region, pass it as a variable to the module:

module "foo" {
  ...
  current_region = "${data.aws_region.current.name}"
  ...
}

Or add it inline in your module, just like you did in your main.tf


Is there a better model for my CI/CD cycle? I am having some troubles finding a better model for my CI/CD cycle.
This is mainly gitflow issue.
My current model : We have 2 bitbucket branches : master and develop.
Both are linked via webhooks to jenkins to be deployed to prod. and dev. servers.
QA team for testing features and releases on dev. server.
Team of developers for both frontend and for backend. With lots of troubles.
The problem : Team manager wants to be able to have better control over accepted/rejected features. Say : developer1 committed a feature to develop branch, developer2 added a hotfix to develop, we want QA team to easily test them both separately, and then testing both together with the least hassle possible. And afterwards easily select what commit stays and moved to master branch.
As I said, this is mainly gitflow issue.
Any suggestions?

Key issue with this approach is that applying commits selectively is extremely problematic. Basically, how do you know whether commits are dependent or independent?
As you correctly mentioned, this is one of the issues with Gitflow itself, so the solution is shifting methodologies. Preferred way today is Trunk-Based Development (TBD) - https://trunkbaseddevelopment.com/ - refer to "Accelerate" book for evidence.
Under TBD you can choose a style of work where you have short-lived feature branches. Once a feature is complete, pull request is created and QA can test this feature selectively under such short-lived branch and then approve the merge to main.
This way you know that your branches are independent (unlike commits in Gitflow) and approve features selectively. Note, that you can achieve something similar in Gitflow with multiple different branches, but this gets extremely messy there.


Using Jenkins declarative when conditions in scripted pipelines Jenkins declarative pipelines offer very convenient when directive with conditions like changeset.

Is is possible the this changeset and other when conditions in a scripted pipeline?

It would seem not - in the scripted pipeline, groovy needs to be used - see the docs

node {
    stage('Example') {
        if (env.BRANCH_NAME == 'master') {
            echo 'I only execute on the master branch'
        } else {
            echo 'I execute elsewhere'
        }
    }
}

(taken straight from the docs)

This would be the same as when { branch 'master' }


Automating creation of new VM and pushing code to server Automating creating of new VM and pushing code to server

Hi all,

I am writing a integration test for a client-server application. To simulate
the production server setup, and to update the code, I set up VMs on my
computer. The application runs on a Windows server, so I have to use systems
compatible with Windows.

STEP 1: Set up VM properly:

Download the OVA file from MS
create a new VM from the OVA file
reset the mac id
set a static dns ip address and a VM ip address
open up the port 22
install bitvise ssh server
transfer public keys to the VM
install python 3.5.2
update pip
install virtualenv systemwide

STEP 2: Deploy code to VM:

create a requirements file using pip using: pip freeze -r requirements
zip the entire repo using powershell using:
powershell.exe -nologo -noprofile -command "& { Add-Type -A 'System.IO.Compression.FileSystem'; [IO.Compression.ZipFile]::ExtractToDirectory('atu.zip', 'atu'); }"
using pscp I transfer the files to the VM using:
pscp ./atu.zip ipython_user@10.15.33.31:/C:/Users/ipython_user/Desktop
unzip the files again using powershell using:
powershell.exe -nologo -noprofile -command "& { Add-Type -A 'System.IO.Compression.FileSystem'; [IO.Compression.ZipFile]::ExtractToDirectory('atu.zip', 'atu'); }"
delete the virtualenv folder called env in the folder using plink
create a fresh env folder using using virtualenv: virtualenv env
using the requirements.txt document, I install the requirements again by
direcatly using the pip installed in the virtualenv above using:
/path/to/env/Scripts/pip.exe install -r /path/to/requirements.txt
run the file with the python distribution installed in the virtualenv
using:
/path/to/python file_name.py

Because I was trying to execute relatively complex commands, which in turn had
multiple quotes (double and single), spaces etc, I had to spend quite a bit of
time just debbuging how to run these commands over the network.  I have
automated most of the deployment part(STEP 2) using a combination of
subprocess.call, plink, and pscp. However, I haven't yet automated
setting up the VM (STEP 1).

I am guessing, the above set of actions are not just something I am doing, and
probably others have done this as well.

Currently, I am using low level tools to acheive this. While this works for
setting up Windows VM, it wont probably work for Linux based VMs or if I change
python versions use a different language etc etc.

I have also searched a bit and came across this book:
https://www.safaribooksonline.com/library/view/test-driven-development-with/9781449365141/ch09.html

where the author goes into how to deploy to a linux server. However, it seems
that he is essentially doing what I am doing by directly running commands on
the linux server, just through the fabric wrapper. Because ssh doesn't work
as well on Windows, I am guessing, I will have to get back to the nitty-grittys
of deployment with fabric as well.

Question:

Is there a better way to setup the VM, and deploy code to the VMs, while
abstracting away the difference between different operating systems, python
versions, tools etc?
I have also heard of a git based workflow. Is that better than directly pushing the code to the server? Why so?

Instead of using custom scripts which it seems that you are using, I would consider using open-source tools, similar to what Tensibai has suggested in his comments.

I would further split your two Tasks a bit more differently, such that Step 1 would be to just "Set up the VM", and Step 2 would be to "Provision the VM with required programs and configuration".

My 2-cents would be to use tools such as Packer and Ansible.

You would use (for example), the VirtualBox Packer builder, to stand up your VM, which would handle points 1-4 of your Step 1, and then use the Ansible-local provisioner, or something to this effect, to run a playbook, which would handle the rest of Step 1, and all of Step 2.

The above suggestion would make your life a lot easier, and would be able to abstract different OS's as well as you'd just write a different Packer template and Ansible task. With installing different Python versions / Tools, you'll just need to update the Ansible role / have several different roles with the specific tools.

There are of course, other tools such as Vagrant, Puppet, and Chef, which can handle what you'd like to achieve, so it's up to you find your preferred tool.

Mini breakdown of Ansible modules you can use:

Open up port: win_firewal_rule_module
Installing bitvise and python: win_chocolatey
Setting up pip and virtual envs: pip
Transferring public keys / requirements.txt etc: win_copy

Etc. The full list of Ansible Windows Modules can be found here.


Cannot understand jenkins deployment error I am new to jenkins, and i am trying to make a deployment into a remote serve, but i am getting the following error:
    Current dependency:com.vmd positionWatchLoader latest.integration
:: loading settings :: file = /tmp/tmp.uhKXp22892/ivysettings.xml
:: resolving dependencies :: com.org#appli-caller;working
    confs: [default]
    found com.org#appli;300000 in local
    [300000] com.org#appli;latest.integration
:: resolution report :: resolve 82ms :: artifacts dl 0ms
    ---------------------------------------------------------------------
    |                  |            modules            ||   artifacts   |
    |       conf       | number| search|dwnlded|evicted|| number|dwnlded|
    ---------------------------------------------------------------------
    |      default     |   1   |   1   |   0   |   0   ||   0   |   0   |
    ---------------------------------------------------------------------

:: problems summary ::
:::: WARNINGS
        ::::::::::::::::::::::::::::::::::::::::::::::

        ::          UNRESOLVED DEPENDENCIES         ::

        ::::::::::::::::::::::::::::::::::::::::::::::

        :: com.org#appli;300000: configuration not found in com.org#appli;300000: 'runtime'. It was required from com.org#appli-caller;working default

        ::::::::::::::::::::::::::::::::::::::::::::::

:: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS

And i am not able to interpret the error message, it says:  configuration not found in com.vmd#positionWatchLoader;300000 which configuration ??
It was required from com.vmd#positionWatchLoader-caller;working default

I have no dependency called: com.vmd#positionWatchLoader-caller !!!!!
I do not know if it can help: i am using Ant to build and Ivy to manage dependencies.
Any help will be much appreciated

I resolved the problem publishing the ivy.xml file to nexus too, adding publishivy="true" to my publishing task in my build.xml file.


Ansible configure and compile in directory So I have have an Ansible Task that extracts a programs sourcecode zip into a certain directory and is then supposed to compile and install it.
The extraction of the zip works fine, but then I need to execute the commands:

./configure
make
sudo make install

And these commands need to be execute from within the extracted directory. I tried several things to execute it, I couldnt get it to work.

  - name: doit
    command: chdir=/tmp/proftpd/ ./configure && make && make install
    become: yes

I tried this but it fails:

  TASK [doit] ********************************************************************
fatal: [centos]: FAILED! => {"changed": true, "cmd": ["./configure", "&&", "make", "&&", "make", "install"], "delta": "0:00:00.152824", "end": "2019-11-21 14:20:48.656070", "msg": "non-zero return code", "rc": 1, "start": "2019-11-21 14:20:48.503246", "stderr": "configure: WARNING: you should use --build, --host, --target\nconfigure: WARNING: invalid host type: &&\nconfigure: WARNING: you should use --build, --host, --target\nconfigure: WARNING: you should use --build, --host, --target\nconfigure: WARNING: invalid host type: &&\nconfigure: WARNING: you should use --build, --host, --target\nconfigure: WARNING: you should use --build, --host, --target\nInvalid configuration `&&': machine `&&' not recognized\nconfigure: error: /bin/sh .//config.sub && failed", "stderr_lines": ["configure: WARNING: you should use --build, --host, --target", "configure: WARNING: invalid host type: &&", "configure: WARNING: you should use --build, --host, --target", "configure: WARNING: you should use --build, --host, --target", "configure: WARNING: invalid host type: &&", "configure: WARNING: you should use --build, --host, --target", "configure: WARNING: you should use --build, --host, --target", "Invalid configuration `&&': machine `&&' not recognized", "configure: error: /bin/sh .//config.sub && failed"], "stdout": "checking build system type... ", "stdout_lines": ["checking build system type... "]}

How can I achieve this properly with ansible?

Ansible command module requires only one command and cannot be changed. You should use the shell module at the very least. Note however that this will break the idempotency of the playbook.

You should probably split the task up into three:

configure task using the shell module. Use creates: to tell the task to be skipped if the configure task has already created the expected output file
make task, using the Ansible Make module
similarly for the make install task.

This should yield the same actual result, but would be more likely to be idempotent.


What are Deis Workflow Alternatives? We're looking to build an internal Heroku-like PaaS that supports clustering and a DFS. We found Deis Workflow that seems to do exactly what we're looking for, but the project has since been discontinued.

Are there any alternatives that are similar to Deis Workflow that achieve the same thing?

According to the README of the deis/workflow project on GitHub, it is no longer maintained since March 1, 2018.

However, the README indicates that there is a fork, called Hephy Workflow. At the time of writing, this is the first time I heard about Deis and Hephy and I have not worked with these solutions.

Alternatives to these products are OpenShift, Google App, AWS Elastic Beanstalk, Azure App Service, Cloud Foundry, Apollo according to StackShare.

You mentioned that you are looking to build something internal. This would probably mean that native-PAAS like AWS Elastic Beanstalk, Azure App Service and Google App are not an option as these are only available on their platform.

One of the options could be OpenShift. This product has an enterprise and open-source version that can be installed on your on-premise or cloud servers. You could also decide to create a bunch of VMs on Google Cloud and install OpenShift on these servers.


How to make one branch public while the others remain private on GitLab? I am currently working on a LaTeX template. I chose gitlab to host the template.

Now I want to give some colleagues access to my template, but only to the master branch. I use the other branches to write my personal documents based on the template.

Is it possible to set the master branch to public while the other branches remain private?

I'm not an expert in GitLab but I would assume it to be similar to other solutions like it (bitbucket, github).

A simple way to achieve this would be to clone the repo in another private repo instead of doing a separate branch.  In effect this would work similar to having your local repo and the gitlab repo.  Merging a branch and merging with another repo is not much different with git


How to structure DevOps related code and configs in a code repository? We've been growing as a company, our products are expanding and our DevOps related activities and efforts are growing as well - we've switched from Bamboo to a more flexible and configurable Jenkins, using deployment pipelines and other plugins; switched to Ansible and starting to use Docker here and there internally.

All of these things require some level of coding or configuration - Ansible scripts and configs, Jenkins groovy scripts, Dockerfiles and YAML configs.

For now, we've created a separate "ops" repository with high-level directories for jenkins, ansible, docker and other (which is a terrible name, but for now all "other" DevOps automation things are there).

Our approach does not feel right and might not scale, but what are the best practices and recommendations to keep DevOps-related code in a code repository or repositories?

I can answer bout docker, one of the best practice to use docker is to keep the docker file and the compose files in the same repository of the project, so wherever you clone the project you can build the docker image, and it is good to keep multiple versions of docker compose files for example (prod, staging, dev) so you can build the image and run the container with specific option for each env for example for dev machine you can use specific network and run more dependencies container or whatever.


How to manage Hashicorp Vault patching in AWS? I'm setting up a Hashicorp Vault open source deployment in AWS, but I'm trying to figure out the best approach for keeping the underlying EC2 instances patched. I'm using AWS Systems Manager for patching, but won't that cause the patched instances to restart, and then the Vault instances will need to be unsealed every time?

One workaround for this would be using the AWS KMS driven unseal approach. Anyone have any experience with this?

The canon solution is to use KMS auto-unseal: https://learn.hashicorp.com/vault/operations/ops-autounseal-aws-kms


How to automate initial server configuration? I’m trying to add new nodes (Linux servers) to my cluster but I have to first configure my nodes properly, i.e. setting up the firewall, closing ports, disabling password login, setting up git,  etc…
This is all very time consuming and can go wrong easily due to human mistakes. How do I automate this process ideally? What tools are used for this? I read about Terraform, Ansible,  but I feel kinda lost with all those different technologies. My desired result would be whenever I buy a new VPS, I run a script (or connect it to a master node) and then configuration happens based on a template or so
Thanks in advance

What you are trying to accomplish is referred to as infrastructure as code.
Which tools you select will depend on where you are operating, and what your business needs are.  If you are operating in AWS exclusively, you can skip 3rd party solutions and focus on their CloudFormation service.  Terraform (infrastructure) and Ansible (configuration) are popular cloud agnostic tools if you are operating On-Prem or across multiple cloud providers.


Is there any way to get New Relic or PagerDuty to not alert during nightly maintenance? Initial Problem

We have some badly behaved overnight maintenance jobs that cause alerts for AppDex and error rates for our apps.  These are metrics that would be problematic if they happened during the day when customers are using the service, but at night there are absolutely no customers so there's no motivation by devs to make the nightly jobs less error-prone.  So now I want to make it so it doesn't alert every night.  Between New Relic and PagerDuty I would think I could figure that out, but so far I've come up with nothing practical.

New Relic

I haven't spent tons of time with New Relic so I was learning as I tried adjusting things, but based on this thread on New Relic's site it seems clear that time-boxing alerts is not a feature they current support or will in the forseeable future.  I've dealt with using history as a predictor for future results using a few technologies and it is a mess that would not be a reasonable replacement for the simple alerts that are in place currently.  Going down that path would create more noisy alerts without detecting any more problems than we already find.

[ It doesn't seem very DevOps-friendly to me to ignore feature requests for simple features that would solve lots of people's problems and only offer complicated quagmires as an alternative.  I'm sure they're all jazzed about 24-7 cloud stuff, but ignoring the market of people who have prehistoric needs is sad.  My audience is literally captive and they are going to stay that way until we stop putting people in prison.  This use case will not go away regardless of how much cloud pixie dust you sprinkle on it. ]

PagerDuty

I've used PagerDuty merrily for years and I was sure there would be some way to handle this after the alert left New Relic, but:

Schedules apply to people and come after an alert has been through a Service.  If you could add a Service to a Schedule we could hack this, but no, that's not there.
Event Rule-based Suppression doesn't include time-of-day conditions.
Maintenance Windows might work, but I was really hoping this wouldn't turn into another cron job I have to keep an eye on.

The Question

Is there any way to get New Relic or PagerDuty to not alert during nightly maintenance?

We ended up creating a new PagerDuty service with a schedule attached to it.  That schedule only had anybody assigned for half of the day.  This ended up duplicating some info from the service we already had, but it was the least painful way we could find to deal with it.


Does the "cattle not pets" distinction apply as equally to machine instances as to containers? There is a great discussion of the Cattle vs Pets distinction from Randy Bias here. Martin Fowler talks about a SnowFlakeServer.

In a series of talks, Adrian Cockcroft talks about how they moved toward Cattle Servers for solving a scalability problem at Netflix.

The challenge with this distinction is always managing persistent state. Does it make sense to treat your database servers as Cattle? It does if you (a) manage the state outside of your cattle model (external volumes for your docker containers), or (b) use a distributed database like Cassandra that allows for individual nodes to fail, but still maintain state in the cluster.

I get that you can get very close to the 'disposability with persistent state' of Docker containers mounting a shared volume, with launching AMIs to machine instances mounting a shared drive. You can get closer this this idea of scheduled cluster management by having an autoscaling group that recreates machines that you've blown away.

To me - the machine instances lack the granularity of a docker container. They gravitate more towards the 'pets' end of the spectrum.

My question is: Does the "cattle not pets" distinction apply as equally to machine instances as to containers?

While technologically, containers and virtual machines are very different, there is no apparent difference from the perspective of your software. It seems like the argument in your question is that data is special and will always be a unique snowflake, so your question basically boils down to what to do about it in terms of DevOps, CI and Automation.

This is the perpetual challenge of the DevOps model. Ultimately, what you are asking is, either A) your data will be in a database or B) a datastore , so how do you manage that?

The answer is that yes, your database servers can and should also be treated as cattle - and I recently detailed several techniques and technologies for doing so in house. If you don't treat your data storage (databases, storage filers) as cattle you will find you have a single points of failure within your data(base) infrastructure and a lack of scalability and redundancy. These techniques I discuss in my linked answer basically allow you to distribute and cluster your relational database in a manner similar to Cassandra.

Managing your data(bases) and making data(bases) redundant is probably the most difficult challenge facing DevOps. The easiest way to solve the problem: outsource the headache to a cloud provider.

So in short, yes the "cattle not pets" distinction apply as equally to machine instances as to containers.


Why does Jenkins safeRestart not wait for job to finish? I have a simple pipeline job with a single stage:

timestamps {
    node ("jenkins-slave-small") {
            stage('Stage 1') {
                sh "echo 'stage2 echo'; sleep 500s"
            }
    }
}

I am testing whether or not /safeRestart waits for my pipeline to finish executing, but it seems like it's not.

Logs:

[Pipeline] { (Stage 1)
[Pipeline] sh
+ echo stage2 echo
stage2 echo
+ sleep 500s
Resuming build at Thu Sep 26 18:14:18 UTC 2019 after Jenkins restart
Waiting to resume part of test-folder » test-pipeline-no-parallels #6: Finished waiting
Ready to run at Thu Sep 26 18:14:55 UTC 2019

Am I misunderstanding how /safeRestart is supposed to behave or is that the intended behaviour?

Jenkins pipeline jobs are designed to be serializable to disk, to increase durability (their words not mine). Therefore they are considered safe to be suspended. The only things you can do is either wrap your stages in functions with @NonCPS decorator or using properties make the jobs non-resumable after restart, but this will cause them to not be resumed after restart.

Also, restarting jenkins is not really a thing that happens all the time, so it might not be worth the effort to protect against.


How to update all Bitbucket Repository Issues at once? Background: There are multiple Bitbucket Repository Issues (BRI). If an issue needs to be edited then an edit button could be clicked and several items, e.g. title and priority could be changed. This works well when some issues need to be changed, but when more than 250 issues need to be changed this approach is tedious.

Aim: to update all BRI at once.

Method: The current method is to update the issues manually, but when multiple issues need to be updated then it is time consuming. Another approach could be using the API.

Consulting the API: curl -X POST -u "<TOKEN>" https://api.bitbucket.org/2.0/repositories/<username>/<repository-name>/issues? did not return any issues.

In order to update all the issue the following was tried:

curl -H "Authorization: Bearer <access-token>" https://api.bitbucket.org/2.0/repositories/<username>/<repository>/issues -d "priority=trivial" -X POST

Instead of updating all BRI, the following was returned:

{"type": "error", "error": {"fields": {"title": "required key not provided"}, "message": "Bad request"}}

Discussion: As mentioned in the method section, perhaps the API could be used to close all BRI at once, but according to the API documentation there seems to be no update option.

Why create this Q&A on DevOps?: The reason is that the ultimate goal is to close an issue in bitbucket by Jenkins when a pull request has been merged. If updating issues using the API works then that will hopefully work as well.

Conclusion/question: It is possible to update all BRI at once?

At the moment it seems not be possible to update all issues at once using the API and therefore this issue has been created.


How to create credentials for Docker Hub? I was asked to use Docker Hub as part of a build/deploy pipeline.
I am supposed to provide some authentication credentials to a third-party tool, so that this tool can push container images to private repos.

Now, I don't see any place where I can generate secrets/API key that are safe to give to that third-party. Am I missing something? In Amazon for example I can create roles with IAM.

Note: The platform I tried to use is banzaicloud.com . Trying to setup a basic Nodejs app following their instructions. As far as I'm concerned, the problem of authentication should not be on the third-party tools side. The same way I can generate API keys for any kind of service and use them in whatever client I want. In the platform I refer to, it seems only the "full docker hub account" credentials can be used instead of some kind of dedicated third party mechanism (like API keys or IAM, etc).

You can generate an access token for docker hub by going here:

https://hub.docker.com/settings/security

And creating a new access token.

It's essentially the same process for organisational access tokens or for personal access tokens IIRC.

Once you have generated the access token you can literally just use that token instead of the password for most docker hub operations (ie. docker login --username $user --password $token)


Automatically write new bash history to change-log file Every time I ssh into a server I might issue some bash commands and then sign off (hup signal?). Is there a way to write the shell commands that I issued on the server to some changelog file?

I want to remove basic commands like ls/cd, and duplicate commands, as well.  It's basically as if every command I issue in bash is like this:

eval "$(my_command | tee "$HOME/change.log")"

but I don't know how to set this up, anybody know a good way to do this?

It would be really nice to eliminate commands like ls and cd, but also show the pwd of every command that was issued:

command  (/home/ubuntu)
command1 (/root/.ssh)
command2 (/home/ubuntu/teros)
command1 (/usr/local)
command1 (/usr/local/lib)
command3 (/etc)

Try using GNU Screen or Tmux. These are called "screen multiplexers" but you'd get a number of benefits

The commands you run will continue running even if you disconnect, or you have network related connection drops. Both software allow "re-attaching" to the running session
Command outputs will persist in memory. You can start a long running command, log off, then connect back some time later and see the output of the command.
Save the whole shell session to a file. You can have the log of all the commands you issued, as well as the output you got, in plain text format.


How to use a plugin parameter? In Ansible, I want to use a plugin parameter.  For example, the profile_tasks plugin supports a parameter to "Adjust the sorting output of summary tasks".

I enable this plugin by whitelisting it in the ansible.cfg file by:

[defaults]
callback_whitelist = profile_tasks

But how do I use the parameter supported by the plugin?

Note: I just want to use it but I don't know how to. The document says "parameter" and normally I think the way to use a parameter is to "pass a parameter to a function/command line" so that's why I asked "how to pass a parameter". What I want to know is "How to use a plugin parameter".

I've not tested it, but as the doc says, you can configure the plugin via two ways : ini entries or env.

Add below your snippet the following and it should work :

[defaults]
callback_whitelist = profile_tasks

[callback_profile_tasks]
sort_order = descending


Accessing TPM from inside a Docker Container I want to implement Vault as part of the DevOps build pipeline to store all the secrets like passwords and certificates. I understand that Vault keeps the data encrypted but I have been asked to look for a way to store the Vault's data in TPM as "it would be more secure and temper-protected". First off, is this argument valid? Vault already stores the encrypted data so is this additional layer of encryption really needed?
I read the Hashicorp's Vault Storage Documentation and it does not directly support any storage over TPM but if I could mount TPM on the container then I could probably use filesystem storage to store data inside it.
So, is it possible to mount the host system's Trusted Platform Module (TPM) onto a docker container?

I think that the statment "to store the Vault's data in TPM" is not valid.  The TPM is generally not for storage.  There are some slots for storing some values and likely some persitent keys, but it is only a handful. I think TPM 2.0 maybe has more storage than 1.2, which I'm more familiar with, but it is not a storage device.  It really depends on what and how much of it you need to store.

So while there might be a solution using "on-chip" TPM memory, it is scarce and really slow. The more common pattern is the use the TPM to wrap the keys or data, such that it can be stored in an encrypted blob on actual storage, like disk or network, and only safely unwrapped on THAT MACHINE.

The TPM is not really a "crypto co-processor" nor is it a storage device. It's slow and has minimal resources, for cost.

Having clarified that, it probably is valid to use the TPM, in the proper way, to help securely store the application data, it just depends on how much work it is for you.

Also, if the data doesn't need to be specifically tied to that specific machine, or that specific container, or the installation is on a VM, and the VM might spin up on different hardware, then using the TPM doesn't really make sense because it is a root-of-trust for a specific platform, a specific board or machine.

So to be semantically picky:

Storing the Vault data in the TPM: not valid
Storing the Vault data with the TPM: valid (potentially)

Disclaimer:  I don't know anything about Vault or the data you want to store.


Cloudformation YAML not well-formed I have a cloud formation template that I want to import some values from another template. Currently the other template exports a string by joining together an array (better way of doing this?).

In my template I want to import into I am getting the below error when trying to update my stack in the AWS console.

Template format error: YAML not well-formed. (line 71, column 28)

Here is lines 70 and 71 from my YAML file

SubnetId:
        Fn::Select [ "0", [ Fn::Split: [ ",", Fn::ImportValue [ !Sub "${ExportPrefix}-SubNets"] ] ] ]

Have also tried this..

Fn::Select [ "0", [ {Fn::Split: [ ",", Fn::ImportValue [ {Fn::Sub "${ExportPrefix}-SubNets"}] ]} ] ]

I have checked and all spaces should be spaces and all tabs be tabs.

The split function returns a list of strings. I believe encapsulating the function in a list is what is giving you the error. Have you tried removing the brackets around the split function? Something like:

!Select [ "0",  !Split [ "," , !ImportValue [ !Sub "${ExportPrefix}-SubNets" ] ] ]


Bitbucket pipelines: Deployment destination dependent on commit author I'd like to create a deployment pipeline triggered on PR creation that would deploy to e.g. 'staging1' environment if it's my commit and 'staging2' if it's a commit of other team member (the idea is for each team member to have his/her own environment assigned, but deploy using a shared parametrized pipeline). Is it possible to achieve this and if so, then how?

the idea is for each team member to have his/her own environment assigned, but deploy using a shared parametrized pipeline

You could use definitions for the steps for each environment.

definitions:
  steps:
    - step: deploy to staging1
        script:
          - echo "Staging 1"
    - step: deploy to staging2
        script:
          - echo "Staging 2"

Then just express each colleague branch and restrict his commits to just this branch.
For each of the branches you could call your needed definitions.

branches:    #these will run on every push of the branch
    feature/john.b:
      - step:
          # call needed definition step/s
    feature/pavel.m:
          # call needed definition step/s

This gives u flexibility to express multiple definitions steps for one person if needed.


How do i notify GitHub pull request for each stage of a parallel Jenkins pipeline? We have a parallel Jenkins pipeline script which deploy and run three kind of tests in parallel. We have to notify the status of each stage to GitHub pull request. Can anyone help me to get this?

def jobs = [
    [
        "name" : "LegacyUnitTests",
        "runFolder" : "recipes",
        "run"   : { -> sh "docker-compose -f docker-compose/docker-compose-maven.yml -f docker-compose/docker-compose.yml run maven-unit-test" },
        "report": { -> junit testResults: "docker-compose/volmnt/reports/*.xml" },
    ],
    [
        "name" : "AutomationTests",
        "runFolder" : "automation",
        "run"   : { -> sh "../run_automation_tests.sh" },
        "report": { ->
            junit testResults: "reports/junit/*.xml"
            allure([
                includeProperties: false,
                jdk: '',
                properties: [],
                reportBuildPolicy: 'ALWAYS',
                results: [[path: '/reports/allure']]
            ])
        }
    ]
]

def parallelStagesMap = jobs.collectEntries {
    ["${it.name}" : generateStage(it)]
}

def generateStage(job) {
    return {
        stage(job.name) {
            dir(job.runFolder) {
                try {
                    job.run.call()
                    setSuccess(this, job.name)
                }
                catch (err) {
                    setFailed(this, job.name)
                }
            }
       }
   }
}

pipeline {
    agent {
        node {
            label 'pr-slave-test'
            customWorkspace "/var/lib/jenkins/workspace/${JOB_NAME}_${BUILD_NUMBER}"
        }
    }
    post {
        always {
            //Reports
            script {
                jobs.each {
                    j -> j.report.call()
                }
            }//Cleanup
            cleanWs()
        }
    }
    stages {
        stage('Pre-Build') {
            steps {
                 script {
                    //set status as pending
                    jobs.each {
                        j -> setPending(this, j.name)
                    }
                    //checkout all repo
                 }
            }
        }
        stage('Build') {
            steps {
                dir () {
                    sh "echo \"Building\""
                }
            }
        }
        stage('CI') {
            steps {
                script {
                    parallel parallelStagesMap
                }
            }
        }
    }
}

I want to notify GitHub pr for each stage automation and legacy unit test status as pass fail as a GitHub check status

I don't think I can help you with this, but I think I can suggest you where to look for a relevant information.

You can report any status check to your PR using Status API (https://developer.github.com/v3/repos/statuses/)
You can configure which Status checks are required from PR (https://help.github.com/en/github/administering-a-repository/about-required-status-checks)
Since pipeline is actually groovy code, calling Status API should not be a challenge. (Example with CURL - https://www.openmakesoftware.com/restful-json-api-calls-jenkins-pipeline/, example using native groovy - https://www.deployhub.com/native-groovy-api/)