Branching model for nightly deploys I'm a lead developer for a small B2B SaaS company, using Django and AWS. We currently run biweekly deploys after hours with a mostly-automated custom Fabric-powered tool.

We're currently exploring options for moving to fully-automated nightly deploys, but want to move to a different branching model to support this.

Right now, topics are branched from dev and merged to dev, a long-running branch which is merged wholesale with --no-ff into master during the biweekly prod deployment process. This necessitates a feature freeze on release days, to ensure that broken code isn't merged right before a release, and isn't really a great solution.

My current plan is to move to a more branch-oriented workflow, similar to Gitworkflow: https://github.com/rocketraman/gitworkflow. This would allow us to merge feature/bug branches individually to dev, then only merge to master the branches (corresponding to Jira issues) that have passed manual QA.

Is this a good approach?

In my mind, there are a few potential open questions, like how to handle issues/branches which have failed QA, and should be removed from the dev branch/environment. I suppose that instead of reverting commits, we could then "refresh" the dev environment by overwriting it with master, and re-merging all branches currently undergoing QA...

You're bringing up a big pain point - and there are 2 camps here, people who do GitFlow and people who prefer other approaches (with TBD being most popular).

Personally, TBD with release branches is something I would always prefer, but you can find lots of other people who would tell you GitFlow is better - so you'd have to decide for yourself.

In your case TBD would work as following - you keep committing to your master, however before release time you cut a release branch out of master. Once you do that, you don't have to code freeze master and your developers can keep working there.

Some materials for research:
On Gitflow - https://nvie.com/posts/a-successful-git-branching-model/

On TBD - https://trunkbaseddevelopment.com/ (especially check "Scaled Trunk-Based Development" section)

Critics of Gitflow (note that myself I'm biased towards TBD) -

https://georgestocker.com/2020/03/04/please-stop-recommending-git-flow/
https://medium.com/@mattia.battiston/why-i-love-trunk-based-development-641fcf0b94a0


Docker Images via Container vs Dockerfile Pardon the dumb questions, but I've been learning about Docker and container technologies and I was learning about using Dockerfiles to specify build instructions for an image, and how each instruction creates a layer that will impact the overall size of the image generated.

I also learned that you can also attach to an image, as a base, and use it as any other server/vm, and commit the container as an image. This is the method I've been using in my few weeks of experimenting with Docker.

Is there any benefit to using one method vs the other? Is reusability enhanced either way? I was also able to push the image I created via container to the registry, is this best practice, or should I be keeping a repository of Dockerfiles?

If one uses a Dockerfile then a colleague could also understand what happened (documentation as code). If one runs a container, enters it, runs commits then it would be hard to understand what packages were installed. Especially after a couple of months.


How to get EC2 instance tags when pulling CloudWatch data? For monitoring purposes, we're pulling AWS/EC2 instances statistics data from CloudWatch and save it to InfluxDB.

We're using Telegraf to do this.

However, when pulling the data from Cloudwatch, it doesn't contain the instance tags or env_variables, only the standard instance fields (instance_id, for example)

Is there a way to get this data when pulling the stats? Some other agent? A Lambda function?

CloudWatch metrics contain "dimensions" that loosely match InfluxDB's concept of "tags". The list of dimensions available for EC2 metrics is available in the AWS documentation. Instances tags are not available and I'm not sure what you mean by env_variables but since all that is supported is

AutoScalingGroupName
ImageId
InstanceId
InstanceType

the answer is still going to be this is not possible.


Giving a meaningful name to Dockerfile's VOLUME In Dockerfile we can use VOLUME /some/folder to let docker always create a volume for the specified folder when calling docker run, without having to explicitly use the -v parameter. The volume created will have a random (unreadable) alphanumerical name.

Question: is it possible to specify a readable name for the volume on container creation?

You do not specify a volume source when you define a volume in an image. Doing so would be a potential security exploit where an image could mount a private volume with a common name and send the contents of that volume over the network. To create your volumes with known names, and have that defined in a file, you would do that at runtime with a docker-compose.yml file when the container is created. E.g.

version: '3'
volumes:
  - app_data
services:
  app:
    image: your_image:1.0
    volumes:
      - app_data:/data


What do you need for a monorepo for a large codebase? From certain size of the codebase, would you still have Git or are there more specialized solutions?

(Also to checkout just a part of the codebase)

The answer is: a bit of both. To satisfy the constraints of "use git" and "manage a vast codebase" Microsoft developed a new filesystem (previously they were using a variant of Perforce called SourceDepot). It's open source but I have no personal experience of using it.

Why would you want a monorepo? The most obvious reason is that you can modify an API and all the callers of that API in an atomic commit. Also there are advantages to being able to do a git log search across the entire codebase...


How to safeguard Ansible deployment to mitigate accidents? Recently the Amazon S3 had a major outage in the us-east-1 region. It looks like it was likely caused by a spelling error when running a maintenance playbook in Ansible or a similar tool. You can put a shell script wrapper around ansible-playbook to look like:

#!/bin/bash
/usr/bin/ansible-playbook "$@" --list-hosts --list-tasks
read -p "Are you sure? (y/n) " answer
test "$answer" = "y" || exit 0
exec /usr/bin/ansible-playbook "$@"

But what are some other ways you use to improve the safety and reduce a chance of error causing a major outage for your company.

We're using jobs in jenkins to trigger deployments. It ensures that no matter who does the deployment, the ansible command that is run will be the same. A nice bonus is the build logs record when deployments were triggered, who triggered them and what exactly happened during the deployment.

It's certainly not foolproof, but it's been a nice improvement over running ansible playbooks by hand.

For larger/riskier changes this should ideally be combined with some form of change management so changes are made only after another person/team reviews the change and the approach to the change to help identify and resolve potential issues early.

Additionally it never hurts to have a teammate who understands the change you're making be present and watching while you make big changes so they can watch for and help prevent mistakes in the execution of the change.


Simple CI/CD Containers in AWS I am using AWS Code Pipeline, Code Build to create a new Docker container and push it to ECR.

My application is a simple straight-forward single-container based. What would be less friction approach to pull down current running Container and re-launch a new Container from the ECS registry (output of Code Build thru Code Pipeline).

I tried CloudFormation with EC2 user data, custom scripts on one side and CloudFormation with ECS with task definition on the other side (not successful yet). I strongly feel there must be more obvious and simpler approach.

I would keep the ECS container instances (I'm talking about the Docker hosts - I don't like AWS terminology here) and the deployment as two separate things.

Get your ECS stack up and running. You can manage it through CloudFormation and Auto-scaling groups, that's fine. Just think of your cluster as a platform where you will deploy to, not something you need to redeploy.

Then, for CD, the easiest method by far is to update the service definition to use a new task definition and let ECS rolling update the containers for you.

Every time it start a task, ECS will run docker pull image:tag even if it has the image locally to make sure it has the latest version of that image:tag. So the image tag you use really don't matter (there is no need to change the tag on every build).

That means that you can build myimage:latest over and over in order to deploy it easily.

What you need is a task definition where the image = myimage:latest. Create a service with that task definition and every time that ECS start a task (an instance of your service) it will be the most recent "myimage:latest" you have built.

From there, you are missing only one piece in the puzzle, from CodeDeploy, you can call something, perhaps a lambda function, to create a new revision of your task definition and update your service and ECS will automatically create new tasks for that revision and remove the old tasks.

An example:

Let's assume you have created a service called MyService. That you have configured that service to run 2 tasks for the task definition MyTaskDefinition:1 (revision 1). In that task definition, you have one container definition which image is set to "myimage:latest".

Yesterday you have build myimage:latest which had the ID (SHA) 365d8f7bf565.
Your container instance ABC is running a task named MyTaskDefinition-1-containerName-someLongId. when you inspect that container, it's running the image "sha256:365d8f7bf565.........."
Your other container instance DEF is running another task. It has a similar name (only the ID differ), but it's running the same image.
You push a change to your repo.
CodePipeline picks up that change, build and publish the image to ECR.
That new Docker image is also myimage:latest, but its ID (SHA) is f7ec5e54ac96
Now you need to add a step to your pipeline to use Lambda functions and the AWS NodeJS SDK to do some calls to your cluster:

Create a new task definition (which will be exactly the same as before). That will be MyTaskDefinition:2
Update your MyService to use MyTaskDefinition:2 (instead of 1)

ECS will create new tasks. The container names will MyTaskDefinition-2-containerName-someLongId. When you inspect those containers, you'll see that they will be running "sha256:f7ec5e54ac96.......". Perhaps you'll have 2 tasks on the container instance ABC, perhaps they will be sprayed out (that depends on your service's configuration)
After some time ECS will remove the old task MyTaskDefinition-1-containerName-someLongId from ABC and DEF.

Note: you don't actually need to create a new task definition. If you want you could instead retrieve the service's tasks list and manually stop them one by one. You should wait for ECS to restart a task before stoping a new one (that is: stop the first container, wait for ECS to replace it, stop the second container). When ECS restart the container, it will grab the most recent myimage:latest built, as explained before. I just think creating a new task definition is easier and less error prone (no logic required to wait and check, ECS will handle the rolling update for you if you have a new task definition).


How do I create multiple versions of a lambda function in terraform I have it creating one version. It has latest and version 2 but can’t getting it to create more than one version of the same lambda?

resource "aws_lambda_function" "lambda" {
  function_name = "test-lambda2"
  publish = true
  runtime = "python2.7"
  source_code_hash =  "${data.archive_file.lambda_zip.output_base64sha256}"                                             handler = "index.handler"
  filename = "lambda_function.zip"
  role = "${aws_iam_role.iam_for_lambda_tf.arn}"
}

This is shown in the Hashicorp tutorial - "a new version of the lambda function. The suggestion there is to point to a different zipfile in S3.

I suppose in your case, the filename either needs to change, or the file contents need to change.

The suggestion from the tutorial is to use a variable for the filename, associated with the version of the application.


What is alert fatigue? Recently, we hear the term "alert fatigue" a lot.  I have a few questions regarding this:

where did this term come from?
was it invented by a company like PagerDuty?
what is alert fatigue actually?

The term Alert Fatigue, also known as Alarm Fatigue has been around for decades. It affects many professions, including medical, technical, and construction industries.  According to the Alert Fatigue by Other Names reference, the earliest literature on the topic came during the Israeli Arab conflict.  The practice/understanding of the issue is ancient.

No, PagerDuty did not invent the term.

Alert Fatigue:

  occurs when one is exposed to a large number of frequent alarms (alerts) and consequently becomes desensitized to them.

Aesop's Fable about the boy who cried wolf is a classic example of Alert Fatigue.  In Technology and Medical fields it occurs when alarms/alerts occur so frequently (with or without reason) that we become desensitized to them.  The outcome of this desensitization:

  dramatically reduces our reaction to similar subsequent threats.

In other words we are become so used to the alert that we either ignore it or take our time to acknowledge it.

References

Alert fatigue by other names
Alarm_fatigue
Cry Wolf - When Experience Becomes Fateful


How to test VM provisioning script without provisioning Currently I'm on a state that testing costs me money and a lot of time...

Background: I'm deploying VMs at softlayer and using a post-deployment script(bash) that will install every software that I need after the VM is ready.
The problem is, I can only test this script by deploying one VM, and it's currently taking around 4h for the script to finish... So every change that I make I need to create a new VM(costs money) and wait for around 4 hours to see if the script is broken or not... This is becoming chaotic and I won't be able to move forward if I stay this way.

I need a new way to approach this kind of situation and be able to test the provisioning script more quickly and without needing to deploy a new VM every time.

Do you guys know some tool to help me on this scenario ?

I can see some options:

Use Vagrant to create your VMs; it separates the process of creating the VM (including the base OS) and the actual provisioning. It also has some options to run certain provisioning steps at certain circumstances only.
Use Ansible, Puppet or something like that to switch to a provisioning mode where you do not do the same stuff everytime, but only what is needed. This means that you can start the job, and at the first failed part, stop. Fix that part, then continue.
Use Docker. This is slightly different than the Vagrant/Ansible approach in that it creates containers (which you don't actually need, as far as I can tell). It has the benefit, beyond the Ansible approach, that it gives you a very fine grained step-by-step development process. I.e., if one step fails, you still have all the images leading up to that, so during development, with a bit of discipline, you become very, very fast indeed.

All of these tools do much more than what you need, but all of them give you a way to do your job incrementally. Vagrant, Ansible and Docker are pretty easy to learn, as far as I'm concerned (as long as you are in Dev/Test mode, the "interesting" parts start when you go to production). Ansible is very minimalistic and needs nothing except a ssh connection. Vagrant and Docker might not be feasible in your infrastructure, you will quickly see.


How to only include directory contents in Azure Publish Artifact without full filepath? I've been using Azure Pipelines and Releases to build and publish artifacts.
In SolutionX, there are projects ProjectY and ProjectYTests.
I build everything in the solution (to run the tests) and then I want to publish just ProjectY's built contents, in an artifact called drop.
I deploy it to a folder called FolderZ.
When I download the published artifact, the filepath is like: FolderZ/drop/ProjectY/bin/Release/*
I'd much rather it was just FolderZ/*
How can I get rid of (at least) the bin/Release pipeline bit?

In the copy files task, there is an optional parameter flattenFiles (bool). This does the job.


All volumes are deleted when EC2 intance is deleted via terraform I am using a terraform template to create AWS resources.
My count is 2, everything gets created as per plan, and, I am attaching a 2nd EBS volume of 1 GB to both machines, which is also happening well.
However, the only issue is when I try to delete one EC2 instance using the command below, both of the 2nd EBS volumes of 1 GB are getting destroyed. I checked they are connected on separate instances.
$ terraform destroy -target=aws_instance.jumpserver[1]
aws_vpc.main_vpc: Refreshing state... (ID: vpc-06b59734024ad6adc)
aws_key_pair.ProdKeypair: Refreshing state... (ID: ProdKeypair)
aws_security_group.sg_internet_facing: Refreshing state... (ID: sg-05a2739733f4f8a32)
aws_subnet.public_subnet: Refreshing state... (ID: subnet-0a8c6ea2718a44224)
aws_instance.jumpserver[1]: Refreshing state... (ID: i-05646d53baa34a988)

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  - destroy

Terraform will perform the following actions:

  - aws_ebs_volume.vol_generic_data[0]

  - aws_ebs_volume.vol_generic_data[1]

  - aws_instance.jumpserver[1]

  - aws_volume_attachment.generic_data_vol_att[0]

  - aws_volume_attachment.generic_data_vol_att[1]

This is the main.tf:
    # Define webserver inside the public subnets
    resource "aws_instance" "jumpserver" {
      count                       = "${var.num_of_instances}"
      ami                         = "${var.ami}"
      instance_type               = "t2.micro"
      key_name                    = "${aws_key_pair.ProdKeypair.id}"
      subnet_id                   = "${aws_subnet.public_subnet.id}"
      vpc_security_group_ids      = ["${aws_security_group.sg_internet_facing.id}"]
      associate_public_ip_address = true
      source_dest_check           = false
      #   user_data = "${file("install.sh")}"

      root_block_device = {
        volume_type           = "gp2"
        volume_size           =  "8"
        delete_on_termination = "${var.delete_on_termincation}"
      }

      tags {
        Name = "${format("jump-%01d",count.index+1)}"
      }

      provisioner "remote-exec" {
        inline = ["sudo apt-get  -y install python"]

        connection {
          type        = "ssh"
          user        = "ubuntu"
          private_key = "${file(var.private_key_path)}"
        }
      }
    }

    resource "aws_ebs_volume" "vol_generic_data" {
      size              = "1"
      count             = "${var.num_of_instances}"
      type              = "gp2"
      availability_zone = "${element(aws_instance.jumpserver.*.availability_zone, count.index)}"

     tags = {
        Name = "${format("jump-%01d",count.index+1)}"
      }
    }

    resource "aws_volume_attachment" "generic_data_vol_att" {
      device_name = "/dev/xvdf"
      volume_id   = "${element(aws_ebs_volume.vol_generic_data.*.id, count.index)}"
      instance_id = "${element(aws_instance.jumpserver.*.id, count.index)}"
      count       = "${var.num_of_instances}"
    }

    # Define webserver inside the private subnet
    resource "aws_instance" "backendserver" {
      ami                         = "${var.ami}"
      instance_type               = "t2.micro"
      key_name                    = "${aws_key_pair.ProdKeypair.id}"
      subnet_id                   = "${aws_subnet.private_subnet.id}"
      vpc_security_group_ids      = ["${aws_security_group.sg_backend.id}"]
      associate_public_ip_address = false
      source_dest_check           = false
      user_data                   = "${file("install.sh")}"

      tags {
        Name = "backendserver"
      }
    }

I could see some spelling mistakes in the below mentioned line, rest all are fine i think.

delete_on_termination = "${var.delete_on_termincation}"

Better you could avoid this line, This might be deleting the EBS when u terminating the ec2 instance. Skip this option and check.


Trigger downstream pipeline job for the branch via from upstream freestyle job I have an upstream freestyle job and a downstream pipeline job.

The freestyle job is triggered via a gitHub webhook to run whenever code is pushed to gitHub branch.

The freestyle job should capture the branch name and pass to the pipeline job and should trigger the build for that particular branch.

How to pass the branch name to the pipeline job dynamically?

The "Add Parameters" can help you. You should set up your downstream pipeline with "GIT_BRANCH" parameter to build with that branch, and transfer it from this upstream job.


How to performance test websites using Gatling that are encrypted using Certbot? Aim

to performance test a UI

Method

Run Gatling recorder
Once the gatling recorder proxy has been started, configure this proxy in the webbrowser that will be used for the test
Login to the UI
Navigate to the website
Save the recording

Results

When a http site is recorded, then it is recorded, but when an https site then gatling records nothing

Discussion

Gatling can record websites, but when these are encrypted then one could define the following:

HTTPS mode:

self-signed certificate
provided keystore

keystore file
keystore pasword
keystore type

certificate authority

ca certificate
ca private key

How to record websites that are encrypted using Certbot?

This topic has been created in the Google user group.

If it is possible to switch https to http temporarly then gatling can record the clicks. When the recording was done, https was activated again and the test could be run against https as well. In summary, recording works if the site is http and running it works against https. Note that the recorder file contains http and this need to be changed to https.


How can I become a DevOps Engineer? Please suggest me a right path to this DevOps career. From blogs Im not getting a well defined path.
What all courses I have to take.

DevOps is not a role and in words of @Tensibai here:

  DevOps is roughly an organization pattern aiming at breaking the silo between developers and sysadmins.

So, you could already be on the DevOps moto and if you don't think you are, you can probably just try to implement the DevOps pattern on your organization or move to another organization with that pattern already implemented.

You may want to become an Infrastructure Engineer, a SysAdmin Engineer, an Automation Engineer, a Site Reliability Engineer, a Performance Engineer... but surely you can not become a DevOps Engineer.

You can get a good sense of your path by reading some questions and answers on this site. Here are a few ones:

What is DevOps?

What is the difference between DevOps engineer Vs DevOps consultant Vs DevOps architect?

What is the difference between SRE and DevOps?

What could be a valid definition of DevOps to introduce it to a novice?

What is the difference between the traditional Development and Operations
Model and Site Reliability Engineering?

What is the difference between Sysadmin and DevOps Engineer?


How to stop VPN connection started from Jenkins? I've just discovered this site and I would like to ask my first question.

I have a pipeline project on Jenkins that basically do this:

SVN update
Build
Deploy

As the deploy target machine is on a different network, I need to start an OpenVPN connection. This works fine using bat 'start openvpn config.ovpn'.

The problem is that I need to drop the VPN connection once the process has finished and there is no stop command for OpenVPN.

My first approach was to save the process ID that started OpenVPN and kill it later. Something like this:

stage('Connect VPN') {
        bat '''wmic process get parentprocessid,name | grep WMIC | tr -d \'WMIC.exe \' > wmic_pid
            start openvpn myconfig.ovpn'''
    }

stage('Deploy') {
    // Deployment process
}

stage('Disconnect VPN') {
    env.WMID_PID=readFile('wmic_pid').trim()
    bat 'taskkill -pid %WMID_PID% -t'
}

However, for some reason the process ID saved using wmic does not match with the one used by OpenVPN. This procedure works fine when its executed manually on a cmd.

Any ideas?

PS: I'm on a Windows Server 2012.

Your question is sitting unanswered here because you're way "out in the weeds" - running pretty idiosyncratic batch scripts that seem to leverage cygwin on Windows, in short, you're in the "wtf" category here.  That's ok!  In this work, all of us are in this place for a lot of our day.

I suggest you consider switching these up for powershell tasks, you can use:

node {
    powershell 'get-process "Openvpn" | stop-process -force'
}

to help with this.  This will simplify your build server and remove external dependencies on cygwin- which if you've ever grappled with at any scale, is a real monster.  It's tough to install automatically, tough to keep up to date, a vuln factory, and poorly maintained.

Perhaps even bigger picture though, using a VPN tunnel in a deployment process is slightly unorthodox.  You might consider pushing an artifact to a repository that both Jenkins and your target server have access to, then polling from the target server for the presence of a new build, then installing it.  Or simply decouple your build and deploy phases with a manual step- one where you log into your production environment to kick off a deployment after a build/test succeeds.


Jenkins API call (or Groovy builtin) that returns list of all currently running builds of jobs There's plenty of questions asking how to return a list of all jobs on your Jenkins instance, or even to get a list of which jobs are currently running, but I have a more specific use-case.

I need to see if a single job is already running. As in, return one value if there is 1 builds of the job running, and a different value if 2 or more builds are running.

The use case here is that we've automated our image creation to build after successful unit tests. It's set to build a maximum of one image every seven hours, by checking the build time of the last image and cancelling the build if it's been less than 7hrs. This works perfectly for most cases. However, these images take ~20min to build. If it's been over 7 hours and two unit tests finish within less than 20min of each other, the second build doesn't see the first creating, and continues building. We're getting two functionally identical images made once every 7hrs, only minutes apart. I'd like to check the status of the controller job to know if it's already running, so that the second build can know to cancel.

I've tried curl-ing the following API, but it turns out it doesn't register individual builds, just that the job is running in general, which is useless in this case as it self-triggers since it checking means it's already running; there's no difference between a single build and multiple builds.

http://JENKINS_URL/api/xml?tree=jobs[name,url,color]&xpath=/hudson/job[ends-with(color/text(),%22_anime%22)]&wrapper=jobs

TL;DR: Is there a different call I can make that will return some list/file/content that indicates that multiple builds of the same job are currently running?

Turns out you can hit the following URL to get an XML output of all currently running builds, including their build numbers.

http://jenkinsURL/computer/api/xml?tree=computer[executors[currentExecutable[url]],oneOffExecutors[currentExecutable[url]]]&xpath=//url&wrapper=builds

I greped it for a count and used that for my calculations.


Conflicting Kubernetes CPU usage & Docker Container Metrics We recently switched over our production environment to Kubernetes. I'd like to enforce CPU limits on the containers. I'm getting conflicting CPU metrics that do not fit together. Here's my setup:

DataDog agents running as a Daemonset
Existing applications running without CPU Limits
Containers in question are multi-threaded Ruby applications
Two metrics: kubernetes.cpu.usage.{avg,max} and docker.cpu.usage
c4.xlarge cluster nodes (4 vCPUs or 4000m in Kubernetes terms)

kubernetes.cpu.usage.max reports ~600m for the containers in question. docker.cpu.usage reports ~60%. It follows that a 1000m CPU limit would be more than enough capacity under normal operation.

I set the limit to 1000m. Then docker.container.throttles goes up significantly while kubernetes.cpu.usage.max and docker.cpu.usage stay the same. The system all fells to it's knees during this time. This does not make sense to me.

I researched Docker stats. It seems that docker stats (and the underlying API) normalize load according to CPU cores. So in my case, docker.cpu.usage of 60% comes (4000m * 0.60) to 2400m in Kubernetes terms. However this does not correlate with any Kubernetes numbers. I did another experiment to test my hypothesis that Kubernetes numbers are incorrect. I set the limit to 2600m (for some extra headroom). This did not result in any throttles. However Kubernetes observed CPU usage did not change. This leaves me confused.

So my questions are:

Does this feel like a bug in Kubernetes (or something in the stack?)
Is my understanding correct?

My follow up question relates to how to properly determine CPU for Ruby applications. One container uses Puma. This is a multi-threaded web server with a configurable amount of threads. HTTP requests are handled by one of the threads. The second application is a thrift server using the threaded server. Each incoming TCP connection is handled by it's own thread. The thread exits when the connection closes. Ruby as GIL (Global Interpreter Lock) so only one thread can execute Ruby code at a time. That does allow for multiple threads executing IO and things like that.

I think the best approach is limit the number of threads running in each application and approximating Kubernetes CPU limits based on the number of threads. The processes are not forking so total CPU usage is harder to predict.

The question here is: how to properly predict CPU usage and limits for these applications?

Multiple things here:

Youre on AWS ec2, hence whatever you are doing on your instance for measuring CPU is calculating CPU on hypervisor level and not instance level. To very this run any load test and check iostat -ct 1 and CPU usage in cloudwatch. CPU usage in cloudwatch is always 10-20% greater than what iostat will report and that is because iostat will give CPU usage on hypervisor level.
As docker to see how kubernetes and docker metrics compare i suggest to run the containers with --cpuset=1 or any number to allow all containers to use only a single vCPU.
Also in AWS 1 CPU= 2vcpu. It is hyperthreaded to 2. You can maybe take this into consideration while calculating.
Finally the best metric to use to see CPU usage for a particular application is using htop and correlating with your cloudwatch metrics.
I have also observed sometimes docker daemon pinning itself to one of the virtual CPUs and hence when you are reducing it to 1000m, maybe the whole setup is getting slow because the reduction is happening on any one of the vpcus. You can use mpstat to get into details of this.
Lastly on the host level you can pin docker to a single CPU and observe more.

Hope this brings you somewhat closer. Update me if you have already found a solution.


Docker bridge network Configuration for containers I have an Angular application deployed in the container . If I log in to the container I can curl to the port 4200.

I have mapped to port 9001 to connect from my host .

But I can't curl from my host machine. What could be the issue?  I think I should configure with docker bridge how to do that??

My Docker file:

FROM alpine:latest

#installing required pakages
RUN apk update
RUN apk fix
RUN apk add nodejs nodejs-npm apache2 curl yarn
RUN npm install -g @angular/cli
RUN npm install -g typescript
RUN mkdir -p /var/opt/angular

WORKDIR /opt

COPY . .

WORKDIR /opt/frontend

RUN npm install
RUN ng build
COPY start.sh /opt/frontend/start.sh
RUN chmod +x start.sh

EXPOSE 4200
CMD ["sh", "start.sh"]

Start-up script(start.sh):

#!/bin/bash

ng serve --host 0.0.0.0

While running the docker file I have specified.  -p 9001:4200

Since you are using the tag "Docker-Machine" I'm assuming that you are using Docker Toolbox to manage Docker.  The problem is most likely an issue with Virtual Box configuration than your Docker configuration.

When you try pinging your container on port 9001, use the ip 0.0.0.0 instead of 127.0.0.1 or localhost.  Another way to test would be to get the IP address of the docker-machine, docker-machine ip, and use this IP to ping your container (e.g. use 192.168.99.100:9001.

If you are trying to access the container from another computer you may have to look into a solution like port forwarding as described here.


DevOps practice on AWS instances I am new to DevOps, started learning the different tools.
I would like to practice all of the tools like Jenkins, Chef, GIT, Puppet, Ansible, Docker, Kubernetes, etc .

My question is If I would like to practice all of the tools on AWS Free tier Instances
How many of them are needed? and
which tools I can combine them and install on which instances ?

Please help me in this regard.

You might not be able to get this done on the free tier. Puppet for example isn't going to want to start because of the RAM limitations. The AWS free tier uses thje t2.micro instance which only has 1GB of RAM. Your operating system alone probably needs 512 to run at idle. This leaves you a mere 512 MB or ram for all of the things that you listed.

While the issue with puppet not starting can be overcome by editing /etc/sysconfig/puppetserver and setting something like:

JAVA_ARGS="-Xms512m -Xmx512m"

(this issue arises because puppet expects 2 GB or RAM out of the box)

You are bound to find that running all of these on a single server is debilitating and painful with only 1 GB of RAM. You should probably consider using something like virtualbox on your PC or laptop so that you can get some more ram than that offered by the AWS free tier. I'd say you want at least 4 or maybe 8 GB of RAM. While this probably can be done, I don't recommend it. You may have to stop one service to run another which then makes it difficult to have one talk to the other (eg, ansible talk to Jenkins while puppet is running, for example.)


How to change Jenkins' port to 80? I installed Jenkins on a Ubuntu 18.04 ec2 instance by following its official guide.

I want to change Jenkins from port 8080 (default) to port 80.

According to its official document:

  If your /etc/init.d/jenkins file fails to start Jenkins, edit the /etc/default/jenkins to replace the line ----HTTP_PORT=8080---- with ----HTTP_PORT=8081---- Here, "8081" was chosen but you can put another port available.

I managed to changed it to port 8081 but when I used port 80, it simply doesn't work:

curl -i localhost:80 returns connection refused

sudo systemctl status jenkins gives

jenkins.service - LSB: Start Jenkins at boot time
   Loaded: loaded (/etc/init.d/jenkins; generated)
   Active: active (exited) since Thu 2019-02-21 08:14:55 UTC; 4min 12s ago
     Docs: man:systemd-sysv-generator(8)
  Process: 6143 ExecStop=/etc/init.d/jenkins stop (code=exited, status=0/SUCCESS)
  Process: 6212 ExecStart=/etc/init.d/jenkins start (code=exited, status=0/SUCCESS)

Feb 21 08:14:53 ip-172-31-11-10 systemd[1]: Starting LSB: Start Jenkins at boot time...
Feb 21 08:14:53 ip-172-31-11-10 jenkins[6212]: Correct java version found
Feb 21 08:14:53 ip-172-31-11-10 jenkins[6212]:  * Starting Jenkins Automation Server jenkins
Feb 21 08:14:53 ip-172-31-11-10 su[6257]: Successful su for jenkins by root
Feb 21 08:14:53 ip-172-31-11-10 su[6257]: + ??? root:jenkins
Feb 21 08:14:53 ip-172-31-11-10 su[6257]: pam_unix(su:session): session opened for user jenkins by (uid=0)
Feb 21 08:14:54 ip-172-31-11-10 su[6257]: pam_unix(su:session): session closed for user jenkins
Feb 21 08:14:55 ip-172-31-11-10 jenkins[6212]:    ...done.
Feb 21 08:14:55 ip-172-31-11-10 systemd[1]: Started LSB: Start Jenkins at boot time.

I am sure nothing runs on port 80.

What am I doing wrong?

Assuming you are okay with moving your Jenkins(and settings) to a docker environment, you can do the following. This would give you the advantage to change ports(and manage jenkins) with greater ease in the future without ever having to touch IP tables or load balancers.

Host your Jenkins on a docker container and map the port 80 on your docker host daemon to 8080 on the container.

You can find the latest jenkins docker image here - https://hub.docker.com/_/jenkins

Install docker daemon and download the image using
docker pull jenkins

You can use docker run command to start jenkins along with the -p argument to supply port values.

For instance, you are hosting your docker daemon on 10.10.169.69, then you use this to start Jenkins
docker run -t container_tag -p 80:8080 image_name

And then to browse jenkins, use http://10.10.169.69:80/


What's the benefit of using Nginx to serve a Flask API on AWS? I want to deploy a Flask API on AWS using Docker. A common way seems to be using a combination of Nginx + app server (e.g Gunicorn) + Flask app.

The reason to use app server seems obvious to me. However, since I plan to use AWS ALB I'm not sure if there's a need for Nginx anymore? From my understanding Nginx is used for connection handling and serving static resources. In my case I don't need any of these: the connection is handled already by ALB (AFAIK) and I won't have to serve static resources.

Are there any other advantages of using Nginx for a low latency API service?

Unless you are using the AWS API Gateway, you'll still need some way of "serving" your Flask API.  You will also need to provision an EC2 instance and run something to serve your API.  Instead of running a windows EC2 instance with IIS you can save about 50% operating cost and use Nginx on a linux EC2 instance.  Additionally, Nginx can be used as a reverse proxy, and supports caching which you may find beneficial.

However, because you are using an Application Load Balancer and are looking into Gunicorn to serve the API you probably don't need Nginx, and it would just require more overhead for deployments if you are not going to setup any of the features in Nginx.


Bamboo Group Deploy Projects I have a numerous deploy projects in Bamboo that are used for deploying DevOps/release tools. I would like have all of these projects grouped together for easy reference. Is there a way to group or organize related projects in Bamboo? Ideally, this would be similar to how the build plans are grouped.

There is an improvement issue open for, I think, exactly what you're looking for: Bamboo should be able to group projects:

  Description

  I'm mulling over redoing our project heirarchy and grouping things a
  little more logically, and realize that being able to group projects
  in Bamboo would be brilliant. It's already stellar that each project
  can contain multiple build plans. Having project groups (a la Jira)
  would make my OCD squeal. Any chance of this?


Are microservices "cloud-ish medium-lived services"? According to https://events.static.linuxfound.org/sites/events/files/slides/cc15_mcguire.pdf p21-23, Kevin McGuire  of New Relics classifies the containers in three categories:

VM-like with a lifetime > 1000h
"Cloud-ish" with a lifetime between 24 and 1000 hours
Lightweight compute engine with a lifetime < 24h (and even <1h for 46% of them)

I have hard-time to clearly define what are the "cloud-ish" containers:

Are those typically "microservices" or is this something different?
Would you put applications that scale on demand as part of those "cloud-ish" services, or would they fall into the "lightweight compute engine" category instead?

Are those typically "microservices" or is this something different?

"Cloud-ish" services could include microservices. I would include services that are replicated and/or load-balanced, such that you can sustain downtime on a single machine, VM, or running binary without service disruption.

Their differentiation of VM-like services looks like services which have a lot of local state (in memory or on disk) where frequent restarting would incur lots of downtime (from either the time it takes to boot/populate a new VM or because the service is singly-homed.)

  Would you put applications that scale on demand as part of those
  "cloud-ish" services, or would they fall into the "lightweight
  compute engine" category instead?

They could be in either category. Scaling on demand is a commonly-used "feature" of Cloud computing. Think of "Lightweight compute engine" tasks as things like batch jobs, manually run jobs, one-offs etc.


Gradual rollout management for client applications I want to rollout software to clients (1m+) devices in a gradual fashion due to high risk. Ideally i would be able to target users based on analytical data i posses and decide if they should receive the upgrade.
Currently we are using our homegrown server calls but they don’t hook to any analytics.
Ive done quite a bit of research and haven’t found venders providing this type of functionality.
Are there platforms (sort of like the “app store”) which have ways to gradually rollout updates to client machines?

It sounds like you are talking about Canary Analysis ? If so, Spinnaker has deployment based on  canary analysis built on, using metrics provided by the application telemetry. To quote the Spinnaker docs:

Your application might send performance metrics which are published and available by default. You can also install a monitoring agent to collect more comprehensive metrics, and you can instrument your code to generate further metrics for that agent. In any case, you need to have access to a set of metrics, using some telemetry provider, which Kayenta can then use to make the canary judgment.
Support is built in for Stackdriver, Datadog, Prometheus, Signalfx, and New Relic.

So, you use Spinnaker as a continuous deployment service and instrument the application with one of the supported mechanisms. Canary analysis can then be done to manage the roll-out of new version.


PowerShell command to automate Azure Linux VM I am looking for some PowerShell command which can check if any update/patch need to be applied to existing Azure Linux VM and then apply the patch if required.
 I will then use this PowerShell command to automate the process.

There was an extension but that appears to be outdated now.

Apologies for not directly answering your question, but by the sounds of your question, you are trying to build an update management system to mitigate the threats of vulnerable operating system components and tools having unpatched vulnerabilities which expose you to risk.

This functionality is actually already available for both Windows and Linux via Azure Automation in the form of the unimaginatively named Update Management solution.

Essentially you will need to do the following:

Configure Azure Monitor and Log Analytics to provide information about vulnerable packages on your machines,
Ensure that any network security groups and firewalls are configured to allow access to/from the various components of the system,
Enable Change Tracking and Inventory,
Enable Update Management,
Onboard your machines.

I would also highly recommend employing Azure Security Center as this provides a clearer overview of vulnerabilities across your estate, not just as applied to specific virtual machines.


AWS: Using CloudFront as CDN for non-AWS hosted content Is there a way (say by providing a Route53 endpoint as origin) to distribute non-AWS hosted content using CloudFront?

(assuming the Route53 is used as dns for a site hosted outside of amazon)

CloudFront doesn't care where your origin server is hosted or even who you use for DNS.  As long as the Origin Domain Name you provide is resolvable from the Internet (and has valid SSL configuration, if enabled), CloudFront will fetch the content from that origin the same as it will fetch it from anywhere inside AWS.


Increase Kubernetes Nginx Ingress limits for large images in Docker registry When pushing images to a self hosted Docker registry installed using their official helm chart, this works for some images. But for one image, I'm not able to push two layers. Docker retry it several times, without success.

3a6fab8d5d23: Retrying in 2 seconds

The logs of the Kubernetes pod gave me the following error:

unknown error reading request payload: unexpected EOF

Both layers were relatively large (252.5 MB/ 393.8 MB).

What I already tried on the Ingress

I had the Kubernetes Nginx Ingress behind my registy. Using the direct NodePort without Ingress, it works perfectly. So it seems that some limits of the Ingress break the upload. The following annotations were applied:

ingress:
  enabled: true
  hosts:
    - my-docker-registry.domain
  tls:
    - hosts:
      - my-docker-registry.domain
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/proxy-body-size: "0"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "900"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "900"

This was taken from the Nginx Ingress documentation, where a Docker Registry behind Nginx acts as example. Are there any other limits I need to increase so that my registry will properly work behind the Ingress?

Background

I want to install the Component Pack for IBM Connections. IBM has a very special way to deliver this Kubernetes based application: You get a ~4GB Zip file. It contains all images and helm charts as well as a shell script, which tags and pushes all those images to a custom registry. The cnx-ingress image is relatively large:

:~/Downloads/IC-ComponentPack-6.0.0.8# l microservices_connections/hybridcloud/images/cnx-ingress.tar
-rw------- 1 daniel daniel 739M Mär 29 13:18 microservices_connections/hybridcloud/images/cnx-ingress.tar

I learned that it's not required to set the ingress class to nginx as I did in kubernetes.io/ingress.class: nginx. All annotations prefixed with nginx.ingress.kubernetes.io apply to the default Kubernetes ingress. Setting the timeouts is also not required. The only important annotation is

nginx.ingress.kubernetes.io/proxy-body-size: "0"

because the default value is way too small for Docker image layers. Without this annotation, we get 413 Request Entity Too Large errors.

The main problem in my case was somewhere else: We had a HAProxy loadbalancer in front of our Kubernetes workers. Being new in those HA technology, it's configuration was pretty default. It contains the following directives:

defaults
    # ...
    retries 3
    timeout connect     1000
    timeout client      1000
    timeout server      1000

According to HAProxys time formats, this got interpreted as ms. In our testing environment, I increased both client/server timeouts to 600s. This fixed all our pushing problems.


How do I load a Jenkins Shared Library in a Jenkins Job DSL seed? I have a seed job using the plugin Jenkins Job DSL. I also have a shared library.

I have tried using the @Library annotation to load the script and the library method. It cannot find the annotation and using library yields the following error:

No signature of method: simple_pipeline.library() is applicable for argument types: (java.lang.String) values: [platform-engineering-library@master]

How do I load a Jenkins Shared Library in a Jenkins Job DSL seed?

It's possible to use jobDSL from Pipeline. (Im using multibranch pipeline as it allows to configure pipelineTriggers)
You can configure your seed job to be a pipeline like this:
def gitCredentialsId      = 'github-jenkins'
def jobsRepoName          = 'https://github.com/my-jobs-repo.git'
def sharedLibraryRepoName = 'https://github.com/shared-library-repo.git'

properties([
   pipelineTriggers([githubPush()])
])

pipeline {
    agent any
    stages{
        stage('Seed Job') {
            agent any
            steps {

                checkout([
                    $class: 'GitSCM',
                    branches: [[name: '*/main']],
                    doGenerateSubmoduleConfigurations: false,
                    extensions: [[$class: 'RelativeTargetDirectory', relativeTargetDir: 'shared-library']],
                    submoduleCfg: [],
                    userRemoteConfigs: [[credentialsId:  gitCredentialsId, url: sharedLibraryRepoName ]]
                    ])

                git url: jobsRepoName, changelog: false, credentialsId: gitCredentialsId, poll: false, branch: 'main'
                jobDsl targets: 'jobs/**/*_job.groovy', additionalClasspath: 'shared-library/src'
            }
        }
    }
}

This way we can configure our seed job to checkout shared libraries repo into workspace subdirectory shared-library and specify additionalClasspath for jobDSL groovy scripts
So in your groovy scripts you can simply use import without @Library annotation


trigger jenkins jobs only when a specific branch is updated I require a setup in jenkins, which enables me to trigger a jenkins job based on a push event to a specific branch in github. tried so many methods neither worked.

Install the git plugin, add credentials for the account you want it to use, then set the job to use SCM Polling. It will check on a preset interval for changes to that repo and trigger when changes are detected. If you need instant triggering upon push, you're looking for webhooks, which is significantly more difficult to set up. I find that polling every 2 minutes or so is good enough in most cases. Here's a decent example: https://www.baeldung.com/jenkins-job-schedule


Why is one not allowed to use a Boolean in a docker-compose.yml? Defining a boolean in a docker-compose.yml file:

environment:
  SOME_VAR: true

and running docker up results in:

contains true, which is an invalid type, it should be a string, number, or a null

Attempts to solve the issue

If true is changed to True the issue persists.
Using 'true' is not accepted by the code itself (a play framework app is started using the ./target/universal/stage/bin/APPNAME -Dplay.evolutions.db.default.autoApply=, i.e. either -Dplay.evolutions.db.default.autoApply=true or -Dplay.evolutions.db.default.autoApply=false parameter):

VAR has type STRING rather than BOOLEAN
Using yes or no as a variable results in:

contains true, which is an invalid type, it should be a string, number, or a null
Using yes and using a script that transforms yes to true works

Discussion

According the docs Any boolean values; true, false, yes no, need to be enclosed in quotes to ensure they are not converted to True or False by the YML parser:

  Environment

  Add environment variables. You can use either an array or a
  dictionary. Any boolean values; true, false, yes no, need to be
  enclosed in quotes to ensure they are not converted to True or False
  by the YML parser.

  Environment variables with only a key are resolved to their values on
  the machine Compose is running on, which can be helpful for secret or
  host-specific values.

environment:
  RACK_ENV: development
  SHOW: 'true'
  SESSION_SECRET:

environment:
  - RACK_ENV=development
  - SHOW=true
  - SESSION_SECRET

Question

Why is it not allowed?

This come from a design choice of YAML language about booleans

Every unquoted value matching this "regex":

 y|Y|yes|Yes|YES|n|N|no|No|NO
|true|True|TRUE|false|False|FALSE
|on|On|ON|off|Off|OFF

Will be converted to True  or False.

This start causing a problem when your code will test an environment value to be yes or no for example taking this script (other examples in the PR discution):

if [ "$SOME_VAR" == "yes" ];
then
  echo "Variable SOME_VAR is activated"
else
  echo "Variable SOME_VAR is NOT activated"
fi

And setting in your compose file

environment:
  SOME_VAR: yes

Will result in SOME_VAR being True when the script run, hence taking the wrong case as it is not equal to yes.

So the choice has been made to disallow boolean to prevent unwanted behaviors hard to debug when you're not aware of the YAML rule.

I see two way to get over the problem:

Using an env_file instead, they are not parsed IIRC and should prevent the conversion.
As you already said, use a wrapper script around your launcher to define the value instead before launching the app, something along the line of this should do:

AUTOAPPLY=false
if [ "$SOME_VAR" == "true" ]
then
    AUTOAPPLY=true
fi

./target/universal/stage/bin/APPNAME -Dplay.evolutions.db.default.autoApply=$AUTOAPPLY


Route53 DNS records stopped resolving Last week, my TXT and other DNS records on Route53 stopped resolving:

https://dnschecker.org/#TXT/thuijls.net

The entries are set like:

What else could be missing here? As far as I'm aware, nothing has changed in the last week yet most of the entries are not being picked up. The domain was automatically extended earlier september if that makes a difference.

When I lookup your domain thuijls.net on MXToolBox I get the following messageL

Dns record not found

They explain this to mean:

  More Information About Dns Record Published
  We did not find a SPF (TXT) record for the domain. Either you are using the deprecated type "SPF" record (in which case you will see a warning message for this alongside this message) or no record exists.

  For a SPF record to be valid, it must at least contain v=spf1

A TXT lookup for thuijls.net yeilds:

Type   Domain Name     TTL     Record
TXT    thuijls.net     5 min   v=spf1 include:spf.messagingengine.com ?all

The problem is the ?all - you probably copy-pasted this from somewhere and the (non-standard) hyphen became a question mark.

Edit your TX record to read v=spf1 include:spf.messagingengine.com -all and all will be good.

Then you can add a 2nd TXT entry for your  "protonmail-verification-da....." setting.


What is a 'Feature Flag Toggle' and when to use them (or not)? There are some questions about feature flag toggles, such as:

How to persuade developers to start using feature flag toggles?
How to use feature flag toggles?

Questions:

What is actually a "feature flag toggle" (in the context of DevOps)?
Why are they used?

Without repeating the content of https://martinfowler.com/articles/feature-toggles.html, since it is an amazing in-depth explanation on what feature flag toggles are. I will just focus on the DevOps aspects.

According to the 2014 State of DevOps Report prepared by PuppetLabs, there are four major metrics to measure IT performance:

Lead time for changes
Release frequency
Time to restore service
Change fail rate

These also contribute to organisational performance overall. So it means that if your IT is doing great on these metrics, your bottom line gets more $$$.

Continuous Delivery is enabled by these metrics, and has been described in depth in the book Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation by Jez Humble.

In context of Continuous Delivery, there is an important distinction that differentiates it from Continuous Deployment. And that is the decision when to do a release of features (to customers).

Keeping changes smaller in size, and deploying (copying code) half-baked features to production systems with a feature flag toggled off allows to shorten lead time for changes.

When features are finally finished, doing a release is a decision left to the business. Maybe a release of a new feature needs to be aligned with some marketing, or a release in another part of the business like a feature in the mobile app.

Features can be released using A/B experiements to only a part of the customer base, or to specific people, or even direct to general availability (GA). Although releasing to GA is often done only after there is enough certainty that the feature works as expected. One might argue that this in effect affects release frequency to be higher.

This decoupling of release and deploy is almost impossible to achieve without feature flag toggles.

Naturally when no deployment is required to toggle a feature off, then the time to restore service is lowered substantially.

And by using feature flags that release features to a small slice of the customer base, the change fail rate metric can be improved significantly as well.

So a simple mechanism called feature flag toggles enables much better IT performance, and in turn improves organisational performance overall.

Great example of how this is done in real companies can be found at Flickr (on of the earliest public posts on the subject), and at Etsy. But many others have adopted the practice and talked about it in length, for example the famous engineering culture at Spotify videos.

Etsy are showing off their internal tool to manage feature flags, called Catapult, in multiple presentations found around the web. And Intuit release an open-source tool called Wasabi that helps manage feature flags.


How to determine whether the Jenkins build was started by a user or the scheduler? I have a Jenkins job which does some unit-testing for some code. It can be started in two possible ways:

By the scheduler - it polls the Perforce server, and starts the job when it detects a code change
By a user, on demand - it downloads a shelved/stashed code change, and checks whether it breaks the tests

The second way is called "Build Review":

If the job discovers problems, I want to send emails conditionally, to avoid spam:

If invoked by the scheduler, send mails to everyone who did check-ins since last successful build (called "developers" in Jenkins jargon)
If invoked by a specific user, send a mail only to that user

I am using the Any Build Step plugin to add two conditional instances of Editable email Notification into my job settings. But what are the conditions? How can I tell it to send mail only if invoked in a specific way, (1) or (2) above?

I'm not sure if there is a simple way to check if the "scheduler" kicked off the job, but you can check if a user executed the job.
In your Jenkins job, check for the presence of the BUILD_USER_ID environment variable. If it is present, retrieve the value and send your email to the specific user(s) (scenario 1). If it is not present, a non-user has executed the job, and you can email the users listed on the last successful check-in (scenario 2).
If you are looking for methods outside of the Jenkins job, you can also use the Jenkins API (http://jenkinsurl/job/[JobName]/[BuildNumber]/api/json?pretty=true) and parse the BUILD_USER_ID from the output.


Sending HTTP requests to private EC2 DNS from other EC2 host works, but not from container on EC2 host I want to make a container hosted on one EC2 host communicate with another container hosted on another EC2 host.
Currently, if I send an HTTP request from host A to host B using host B's private DNS and the exposed container port, I receive a response correctly.
However, if I try the same thing within the container on host B, I get a "Host is unreachable" error.
To me, this would indicate it is not a host networking issue, which is configured in AWS, so I won't enter into detail on it.
What puzzles me is that I have done the exact same thing in our pre-production environment and it works perfectly.
What could be the issue and where do you think I should look to troubleshoot this?

There are a couple of things that you can rule out, thanks to the testing that you have already done.  You know your container port has an active network socket, that is listening for and accepting connections.  You also know that your EC2 instance has working networking.
I can think of three obvious problems:

Your host is using local DNS resolution to identify your hostname as 127.0.0.1 and your container port is not listening on the loopback adapter. you can test this on the server by pinging your DNS name and see which IP address it pings.  You can also use telnet to connect to the port on 127.0.0.1 and see if that port accepts connections.  To fix this, you may choose to make listening on the local loopback work, but you could also fix this by using a DNS entry other than the hosts AWS generated name.

You are resolving the hostname to its private IP address, which then is filtered by the AWS Security Group. On your security group you have enabled incoming connections to this port, but you have specified the source either as a cidr that doesn't include its own IP address, or as security group that is not attached to this EC2 instance.  This is easy to fix in the security group editor.  To prove that this is working successfully and allowing the connection, you can set up an AWS Flow log which will allow you to see all TCP connections on the Network interface.

The Container hosting which provides the port mapping also has a firewall element that restricts incoming connections to only some IP addresses, and may also provide some diagnostics to assist in understanding whether it receives the connection, and whether it decides to route or reject the connection to the container.

If it is less obvious, i think you may be in the world of using IP Tables to generate log messages in the syslog as your connection routes though the IP stack.  This will at least allow you to confirm that at a network level you are generating a TCP connection and receiving the connection within the OS.


How do I achieve Continuous Integration and Continuous Delivery (Ci-Cd) via JMeter? I have prepared the JMeter test script for my project and it tests (both functionally and performance wise) various Rest Apis. I have even coupled it with Jenkins on my local machine and also implemented performance plugin.

Now I want to achieve CI/CD with my JMeter script wherein whenever any developer commits a code, the script runs automatically and if tests pass only then things proceed. In essence, Once the code is built, all tests should run to confirm that it behaves as the developers expect it to behave.

How do I achieve this? Step by Step detailed instructions please as I am a QA folk and devops is a new field for me!

Since you're talking about making delivery/deployment decisions based on the results provided by your script(s) you need to communicate with the team(s) responsible with maintaining the official CI/CD pipeline implementation for your organisation.

If they're using Jenkins and a particular pipeline implementation (multiple technological approaches are possible) compatible with the way you hooked up your script into Jenkins locally they may be able to directly use your approach.

Even if that's not directly possible, there probably still are ways of including your tests in the CI/CD pipeline, only with a more or less different implementation.

Note that there may be related technical and/or logistic questions/requirements that might need to be addressed depending on your organisation particularities, for example: ownership, development/maintenance process and policies, SLAs for the tests, script(s) and/or the resources for their execution.

Even if you don't get your tests integrated at first, you can still run them for every repository commit using your private Jenkins setup and make the results available for everyone to see. In time, if those results prove useful to others they may support your initiative and help with a reconsideration of the initial decision.


How to change Xmx settings for sonar runner? This Question has been answered on Stackoverflow too.

I have a Sonarqube instance running as a docker container. Since I updated it to version 7.1 the analysis of my greatest project fails with GC limit exceeded. If I restart the server, it might succeed once. After a while of researching this issue, I am tempted to believe, I need to increase the Xmx value for the background task.

Where and how can I configure this parameter?

docker-compose.yml

version: "2"
services:

  postgres:
    image: postgres:9-alpine
    container_name: postgres
    restart: always
    volumes:
      - /opt/docker-postgres/etc:/etc/postgresql
      - /opt/docker-postgres/log:/var/log/postgresql
      - /opt/docker-postgres/data:/var/lib/postgresql/data
    environment:
      POSTGRES_DB: sonar
      POSTGRES_USER: <SONAR_USER>
      POSTGRES_PASSWORD: <SONAR_PASSWORD>

  sonar:
    image: sonarqube:alpine
    container_name: sonar
    restart: always
    ports:
      - "9000:9000"
      - "9092:9092"
    environment:
      SONARQUBE_JDBC_USERNAME: <SONAR_USER>
      SONARQUBE_JDBC_PASSWORD: <SONAR_PASSWORD>
      SONARQUBE_JDBC_URL: jdbc:postgresql://postgres/sonar
    volumes:
      - /opt/docker-sonar/conf:/opt/sonarqube/conf
      - /opt/docker-sonar/data:/opt/sonarqube/data
      - /opt/docker-sonar/extensions:/opt/sonarqube/extensions
      - /opt/docker-sonar/bundled-plugins:/opt/sonarqube/lib/bundled-plugins

You can simply put a sonar.properties file under /opt/docker-sonar/conf/. This file will be available inside the container under /opt/sonarqube/conf/, because the folder gets mounted as volume.
A full example for a sonar.properties file can be found on github. However all you need to enter is:
sonar.ce.javaOpts=-Xmx<XMX_VALUE -Xms<XMS_VALUE> -XX:+HeapDumpOnOutOfMemoryError


Custom metric for a Windows process on a Windows ec2 instance I have a process (exe) on a Windows ec2 that is using up lots of memory. I would like to create a custom metric that will collect data and send to CloudWatch. I would like to also create an alarm when a certain threshold has been reached or surpassed.

Memory consumption is a metric that has to be reported from inside the guest (EC2 instance).  You can use a CloudWatch agent to collect metrics from the guest directly. The agent also supports custom metrics now.

Alternatively, install a third party monitoring system. Which also means that you'll have to get an agent running on the EC2 instance. There are many of those, choose whatever fits your environment.


Is there an elegant way to use a Dockerfile argument to tag an image? I use and argument (ARG) in my Dockerfile to download a particular version of an application. I would like to tag my image with that version number. I end up writing something like that:

sh$ PHPBB_VERSION=3.2.4 &&
    sudo docker build --build-arg PHPBB_VERSION=${PHPBB_VERSION} \
                      --tag phpbb:${PHPBB_VERSION} - < dockerfiles/bootstrap

But I don't find that very elegant since you have to type the word PHPBB_VERSION no less than four times in this command, with all the possible errors introduced by a typo.

So, is there a more elegant solution to tag an image with the value of a Dockerfile argument?

FWIW, here is the relevant part of my Dockerfile:

sh$ cat dockerfiles/bootstrap
FROM php:7.2-cli

ARG PHPBB_VERSION
RUN echo "You are building phpBB version ${PHPBB_VERSION:?undefined}"
...
RUN wget -nv -O- \
      https://github.com/phpbb/phpbb/archive/release-${PHPBB_VERSION}.tar.gz | \
    tar xzf - --strip-components=1

You can use a Makefile for this:

PHPBB_VERSION := 3.2.4

build:
    docker build \
        --build-arg PHPBB_VERSION=${PHPBB_VERSION} \
        --tag phpbb:${PHPBB_VERSION} - < Dockerfile

Dockerfile:

FROM php:7.2-cli

ARG PHPBB_VERSION

RUN wget -nv -O- https://github.com/phpbb/phpbb/archive/release-${PHPBB_VERSION}.tar.gz | \
    tar xzf - --strip-components=1

Then, in a shell: make build.


What are best practices to override complete sections using docker-compose? The docker-compose.override contains:

volumes:
  - ./a:/path/to/a

The docker-compose:

volumes:
  - a:/path/to/a

When docker-compose up is run the override overwrite the ./a overwrites the a.

When a new volume is added to docker-compose, the override will try to mount is as well.

docker-compose.override

volumes:
  - ./a:/path/to/a

The docker-compose:

volumes:
  - a:/path/to/a
  - b:/path/to/b

How to prevent that the override will use volumes that are defined in the docker-compose file?

When docker-compose.override contains

volumes:
  - a:/path/to/a
  - .b:/path/to/b

the b is still mounted...

According to this documentation it does not seem to be possible to override complete sections in docker-compose by using different docker-compose files.

  If both files are present on the same directory level, Compose
  combines the two files into a single configuration.

  The configuration in the docker-compose.override.yml file is applied
  over and in addition to the values in the docker-compose.yml file.


Jenkins pipeline fails on script approval, although it was approved I have a pipeline which fails on script approval:

Scripts not permitted to use method org.jenkinsci.plugins.workflow.support.actions.EnvironmentAction getEnvironment. Administrators can decide whether to approve or reject this signature.
[Pipeline] End of Pipeline
[BFA] Scanning build for known causes...
[BFA] No failure causes found
[BFA] Done. 0s
[JobCompletedListener] Build result is: FAILURE
org.jenkinsci.plugins.scriptsecurity.sandbox.RejectedAccessException: Scripts not permitted to use method org.jenkinsci.plugins.workflow.support.actions.EnvironmentAction getEnvironment

I went to in scriptApproval screen, approved the below:

signature: method org.jenkinsci.plugins.workflow.support.steps.build.RunWrapper getRawBuild
signature: method java.net.URL getProtocol
signature: method java.lang.Throwable printStackTrace
signature: method org.jenkinsci.plugins.workflow.support.actions.EnvironmentAction getEnvironment

but on the next run I'm getting the exact same error, and approval request in scriptApproval screen, comes up again. any idea what is the root cause of this behavior? is there a workaround for it?

Jenkins is running as a container, and the Security plugin version is 1.60. BTW, the same job works perfect on other Jenkins which is VM

The issue was with the wrong UID and GID of the user. after I've changed it from jenkins to 1000 1000 recursively, issue solved.


Ansible sporadic errors with Windows machines I'm running into some on and off issues when using windows hosts in my Ansible playbooks. I'm running Ansible 2.3 with pywinrm 0.2.2 installed. I'm using basic authentication with the local Administrator user.

Sometimes I receive this issue when I run a task:

 [WARNING]: FATAL ERROR DURING FILE TRANSFER: Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/ansible/plugins/connection/winrm.py", line 267, in _winrm_exec
  self._winrm_send_input(self.protocol, self.shell_id, command_id, data, eof=is_last)
File "/usr/local/lib/python2.7/dist-packages/ansible/plugins/connection/winrm.py", line 248, in _winrm_send_input
  protocol.send_message(xmltodict.unparse(rq))
File "/usr/local/lib/python2.7/dist-packages/winrm/protocol.py", line 207, in send_message
   return self.transport.send_message(message)
File "/usr/local/lib/python2.7/dist-packages/winrm/transport.py", line 191, in send_message
   raise WinRMTransportError('http', error_message) WinRMTransportError: (u'http', u'Bad HTTP response returned from server. Code 500')

Other times, when I try to run a win_shell/win_command/raw module and with_items on a group of windows hosts it seems to fail on temporary files created by Ansible.

The task I'm trying to run is:

- name: Check services up
  win_command: 'sc queryex {{ item }} | Findstr RUNNING'
  with_items: '{{ component_services }}'
  register: command_result
  ignore_errors: yes

And the error I may get is:

changed: [172.16.104.169] => (item=Dnscache)
failed: [172.16.104.176] (item=Dnscache) => {"failed": true, "item": "Dnscache",
  "module_stderr": "Exception calling \"Run\" with \"1\" argument(s): \"Exception calling \"Invoke\" with \r\n\"0\"
     argument(s): \"The running command stopped because
           the preference variable \r\n\"ErrorActionPreference\"
           or common parameter is set to
   Stop: (0) : cannot open \r\nC:\\Users\\ADMINI~1\\AppData\\Local\\Temp\\RESB3FF.tmp
  for writing\r\n(1) :
     using System;\r\n\"\"\r\nAt line:45 char:1\r\n+
     $output = $entrypoint.Run($payload)\r\n+
      ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\r\n+
  CategoryInfo          : NotSpecified: (:) [], ParentContainsErrorRecordE \r\nxception\r\n+
  FullyQualifiedErrorId : ScriptMethodRuntimeException\r\n",
  "module_stdout": "", "msg": "MODULE FAILURE", "rc": 1}
     changed: [172.16.104.141] => (item=Dnscache)
     changed: [172.16.104.168] => (item=Dnscache)
     changed: [172.16.104.145] => (item=Dnscache)

Both issues are absolutely random and may even not appear at all over a sequence of different runs.

Any assistance?

You should probably create an Ansible issue for this as it's most likely a bug in Ansible.

The first error makes me think about WinRM pipelining:

Ansible 2.3.0 introduced an always-on WinRM pipelining feature (similar to SSH pipelining), and that may be behind this.
SSH pipelining can cause issues in Ansible for Linux, and it can be useful to turn it off, but this isn't yet possible for WinRM pipelining.

This related issue includes some Git commits that will re-enable 'non-pipelined' mode in a future release (should now be released in 2.4, possibly with a backport as part of 2.3.2 - see this comment)

Try upgrading to Ansible 2.4.1+ (which generally works well) to get the fix.  Or try downgrading to Ansible 2.2.3 to see if this helps - this will disable WinRM pipelining and may avoid other regression bugs in this area.

If you installed Ansible using pip, you can do  pip install ansible==2.4.1
to upgrade (or ansible==2.2.3 to downgrade), then if that doesn't help, do the same with 2.3.1 to upgrade again.
you should also upgrade to latest pywinrm as mentioned in the issues above


How to make Octopus Deploy trigger a new release when a new release made on GitHub? I have a GitHub project that I'm using to trigger new releases on Octopus. I can see triggers for machine changes, or timed things, but not specifically when a new release is created on GitHub. This seems odd, certainly Azure offers it.
Does GitHub need to be configured to provide a feed, or is it possible for Octopus to check for new releases on GitHub and then use that to trigger a new release?

Octopus can be configured as either the target of a push from GitHub or you can set up a feed in Github, and pull from that feed.
In your case, it sounds like you want a push mechanism, such as GitHub Actions to do the heavy lifting.


How do I replicate a Jenkins setup via automation? I have a Jenkins setup running in production. I want to automate the Jenkins setup (installation) along with all the jobs that are set up in Jenkins.

One crude way I can think of is to copy the whole jobs directory to the new Jenkins setup.

How do I deal with this problem?

Jenkins itself can be configured in many ways. The same applies for Jenkins jobs.

Probably the most easy way is to configure Jenkins manually and replicate its setup by backing up and restoring. Unfortunately, this means dealing with copying XML files.

But fortunately there are other ways.

init.groovy.d directory

When Jenkins starts, it checks init.groovy.d directory and executes Groovy scripts there. To enforce locale I'm using a snippet.

// Copied from /var/lib/jenkins/init.groovy.d/set-locale.groovy
def pluginWrapper = jenkins.model.Jenkins.instance.getPluginManager().getPlugin('locale')
def plugin = pluginWrapper.getPlugin()

plugin.setSystemLocale('en_US')

Another example of setting up Jenkins by Groovy scripts is located on https://gist.github.com/hayderimran7/50cb1244cc1e856873a4 - I'm using that snippet to setup local Jenkins users on test machines.

Generating jobs

Jenkins Job builder

One approach of generating jobs is Jenkins Job Builder, which creates XML definitions jobs from YAML configurations. I don't like this approach personally as I consider it to be a bit hackish - looks like "generation of XML files without knowing what it means". But some big players are using this approach - JJB originates from Openstack.

Job DSL plugin

Another approach is Job DSL plugin (plugin page). There is special job type called seed job which creates other jobs from Groovy DSL. The seed job can pull the others job description in from Git, so you can easily version them, reuse them, etc.

I'm using personally this approach, as it was available when I started with Jenkins. I've even found some Jenkins tool to create DSL sources from existing jobs, but I'm not able to find it again.

To provide a seed job, you can still use the init.groovy.d approach.

Pipelines

The currently recommended approach is probably related to Jenkins Pipelines. You can store your building pipelines in the source VCS repository next to your code.

I haven't explored this very much, but I'm planning to do.


How do I automate the process of pulling the latest code and restarting a service on git push? I have an application (Logstash) running on a Linux VM and the settings for it are on a GitLab repo. What I want is for the VM to pull the latest version of the settings file and restart the service when I push an update to the repo.

What is the easiest way of achieving this?

There's no "easiest" way per se, there's the better way for you.
I'd recommend using a configuration management tool like Chef or Puppet (for a pull model) for which there's already resources to configure and handle logstash restarts.
That said logstash is able to watch its config file and reload it itself, quoting elastic.co documentation:

Starting with Logstash 2.3, you can set Logstash to detect and reload configuration changes automatically.
To enable automatic config reloading, start Logstash with the --config.reload.automatic (or -r) command-line option specified. For example:
bin/logstash –f apache.config --config.reload.automatic

So you may just set a crontab to git pull your repository periodically and have logstash reload itself.


How to get hostname from node-exporter in Docker Swarm? I have setup my monitoring with the cadvisor, node-exporter, prometheus, grafana stack in my clustered environment using docker swarm.

What is the easiest way to get my actual hostnames picked up by node-exporter, so I can configure my dashboards to filter by node names in grafana?

For my kubernetes environments this works ootb, because all metrics are extended by the kubernetes_io_hostname, that contains the actual hostname.

The only solution to this I found, so far, was to introduce my own metric, that creates the combination between node ids and node names (like done by https://github.com/bvis/docker-node-exporter). But I really like to use the vanilla image (for update reasons, for example) and also mounting some file with the configuration on all machines seems to be too much effort to do this.

Is there an easy way to get the hostname of the node into my node-exporter metrics? Maybe by just mounting the hosts /etc/hostname file to a specific place in the node-exporter container, or something like that?

You just have to use the hostname on your docker compose file like this:

hostname: whatever-{{.Node.Hostname}}

Note that you have to prepend something to the hostname to avoid hostnames duplicates.


Deploying a SaaS solution which is custom for every customer So we have 5 repositories (Individual web services ex. CMS/E-commerce and internal APIs) which are currently deployed as Docker containers.

What we plan to do is to use these services and make a standalone SaaS for customers. This is straightforward however, each customer over here will have some kind of Business customizations (Business Logic etc) so this means based on our current deployment technique each customer will need their own 5 repositories.

Just to explain things more clearly:

Each customer will have their own aws instance which will be having the docker containers of our products.

However, with this approach, we will have to create N * 5 repositories and custom deployment scripts (Ansible+Terraform) for each client.

One solution that I had was to create individual branches for each project within the same 5 repositories however, this is not a good scalable method.

Open to any idea, to be honest since this is something new for me.

I'd investigate what would it take to bundle all these customizations into something like a per-customer configurable theme that can be served by a single, more generic application code.

It probably won't be easy and it may take a long time to reach such capability, but it can be done gradually: start with the N*5 repos and, as you inch forward toward this ideal solution you'll see the deltas in the application code from the repos starting to shrink - the differences will gradually migrate towards these custom themes/artifacts.

When there are no more app code differences left then you can replace the N*5 repos with just 5 app code repos for now common app code and N*5 theme configurations/artifacts repos. Much easier to maintain.


Was AWS Secrets Manager removed from AWS CLI commands? If I enter:

 aws secretsmanager get-secret-value --secret-id  my secret id

It says invalid choice.

  aws: error: argument command: Invalid choice, valid choices are:

and doesn't list secretsmanager as one of the choices or any variation on secretsmanager?

aws secretsmanager get-secret-value --secret-id  my secret id

was used in a video from 2018

Similarly, this command doesn't work

and AWS help and AWS man don't contain secrets manager?

AWS secrets are still accessible via the CLI.

The link you provided goes to an old version of the AWS CLI.  Check out the latest version and commands here: https://awscli.amazonaws.com/v2/documentation/api/latest/reference/secretsmanager/index.html


How to get list of env variables from the root folder to jenkins job? Using configure option, when we check the environment variables, we
  get only the variables which are available with in the folder or job.
  But, I want to get complete list of env variable from the root folder
  to the folder/job, which I want to see.

I recommend that you use the printenv command for list system Variable:

printenv

and also An easy way to obtain the Jenkins environment variables list from your local installation is to append env-vars.html to the server's URL. the URL would be

http://{jenkins_url}/env-vars.html


Can I found a small company on DevOps? Suppose I am en entrepreneur with little seed money for my Great Idea (TM) and I start a company initially planning to implement my Great Idea (TM) on my own and hire additional help on a basis dictated by the rate of company growth.

Frequently, answers on this site center around the idea that DevOps is not a role, but a new way of doing things with a new philosophy of operation that allows a team of roles to actually do DevOps. This prevents burnout and people from feeling like a lone wolf

As a fledgling and budding company, then, should I even bother to pursue DevOps as a philosophy, or is this only appropriate for a more mature operation capable of having a development team of sufficient size?

DevOps has three parts: Tooling/automation, Organizational, and Cultural.

You're basing your organizational structure and process' on your business' maturity.  Which is smart.  My experience in a startup and with a fortune 1000 company has afforded me the experience to denote that starting with DevOps is better for company growth.  It's efficient and fits nicely with agile.

Tooling

Being your maturity is really young, you should buy your DevOps process as saas products to reduce time to market.  Instead of Jenkins use circleCI or similar product.  Instead of VM's and docker, use Heroku.  Try and incorporate the toolset but do it in a manner that works for the size of your business.  If deploying code manually takes 4 hours a week and automating it in 1 hour saves that.  Do it.

Organizational

Also from an organizational standpoint, you will naturally start with all developers as the responsible party for operations/features even product development.  Take a look at http://web.devopstopologies.com/  for organizational structures.  Our startup started with type 2, and as we mature will move toward type 7.

Cultural

DevOps is simply teamwork.  Work together and everything gets easier.   It usually is a problem because leadership doesn't keep this in mind so as you start out, ensure that everyone is apart of the business.  This is also reflected in agile.

So in short, yes pursue DevOps but do so at the level that meets your business maturity.


What's missing from this simple recipe that creates a user? I've spun many boxes in the cloud before, but was growing tired of going through the same dance; and as I'm trying to get familiar with Chef, I figured, that's a good use case to do a little learning.

Ah, that feeling when you think you know how it'll work - but it doesn't. Since I'm new to Chef, I suspect I'm not doing something right in this recipe, which is like my 2nd ever. NOTE: I understand there probably are 'cookbooks' out there that do something like that, but at this point I'm trying to build some ver basic knowledge with the simplest approach.

user 'foox' do
  comment 'test me'
  uid '6711'
  home '/home/foox'
  shell '/bin/bash'
  password '<password_shadow_here>'
end

directory "/home/foox" do
  owner "foox"
  group "foox"
  mode 00755
  action :create
end

directory "/home/foox/.ssh" do
  owner "foox"
  group "foox"
  mode 00600
  action :create
end

file '/home/foox/.ssh/authorized_keys' do
  owner "foox"
  group "foox"
  mode "00600"
  content IO.read('/root/.ssh/authorized_keys')
  action :create
end

Now, this does create the user with a password and home folder with .ssh, and copies the public key from root's own file. But this system ends up failing to let me log in -- Permission denied (publickey) -- when using the same private key as what I have as my root's. But, well, root user wasn't created by my recipe. And if I create my own user manually, it's working fine.

I guess there's something missing in my recipe, then?

I'll put in an answer although I'm not 100% sure. I think that you got the permissions wrong on ~/.ssh and it should be 0700 instead of 0600, otherwise the user cannot access it.

Another tip which is unrelated to your problem, the user resource has a property manage_home (https://docs.chef.io/resource_user.html#properties) which will create the home directory for you and will you save a redundant resource.


What Puppet modules do include best practices regarding CI and rspec-puppet like Geerlingguy's Ansible roles? It is possible to test Puppet code using rspec-puppet, but modules that are supported by the community like NTP do not seem to run test regularly like is done for all Geerlingguy's Ansible roles, e.g. Apache.

The aim is to find a Puppet module on Github that shows the best practices regarding testing using rspec-puppet on build using a CI like Travis. When such a module is found one could read and understand the .travis.yml and apply it to custom modules.

Go to https://forge.puppet.com and look for modules with a high "Quality Score"

For example https://forge.puppet.com/puppetlabs/stdlib/scores.

Then you can take a look at the Project URL and view it's .travis.yml and the tests in spec/


Choice for scheduled task in AWS I have deployed an AWS Lambda with Python 3.6 and CloudFormation executing a scheduled task which writes a JSON file once a minute to an S3 bucket. Now I wonder if this is a good way of doing it, or if an automation tool such as Ansible would be better suited for the task?

The background is that I am developing a PoC for data analysis.

CloudFormation is service that allows you to create your infrastructure from code to automate your deployments.

You can set up a rule to run an AWS Lambda function on a schedule by using CloudWatch event.
https://docs.aws.amazon.com/AmazonCloudWatch/latest/events/RunLambdaSchedule.html


Teamcity web-interface does not work I have Teamcity on Windows machine.
When installed it, I selected the 8080 port.
I have another application on the 80 and 443 ports.
It worked fine, but now I see an error:

But if I open localhost:8080 on a virtual machine - all works well.

What happened? Maybe you know :(
How I can change port or change working to "without SSL"?
How I can move all my projects to another Teamcity?

Thanks!

Thaks for all answers.
Teamcity began to work correctly when I installed all windows updates to my Windows Server 2012 R2.
Maybe it depended with last Microsoft security update.


What are some good Kubernetes training courses? I'm looking for Kubernetes training online or in person. What are some good training courses to take?

To understand kubernetes internal, it's good to have good understanding of containers, docker, etc.

Here, is the list of useful docker and kubernetes resources:

1) Play with Docker Classroom https://training.play-with-docker.com/

2) Docker Interactive Tutorials https://www.katacoda.com/

3) Play with Docker (PWD) https://labs.play-with-docker.com/
    Play with Kubernetes(PWK) https://labs.play-with-k8s.com/

4) Introduction to Kubernetes course on edX also his company, CloudYuga is a certified Kubernetes service provider and training partner

5) Blog by Docker Captain, https://collabnix.com/
    Also, search kubenetes blogs on Medium which are very useful.

6) Kubernetes The Hard Way github repo by Kelsey hightower updated with latest kubernetes 1.12.0 version.

Hope this helps.


Strange behaviour with docker-py Consider the following setup.

Precondition: have a healthy image i.e. service starts up.

E.g.

docker run -d IMAGE

Then, with the Python plugin docker-py, consider the following code which should be equivalent, only with added port mapping.

ingreedy = client.containers.create("openfoodbroker/ingreedy.service",
                                     name="ingreedy_service",
                                     detach=True,
                                     ports={'5000': 5000})
print(ingreedy.id)

This code prints out the container ID and I can remove the container using docker rm -f, but:

port 5000 is not open;
the container does not show up on docker ps

Any ideas?

docker.containers.create() is documented to:

  Create a container without starting it. Similar to docker create.

Either call docker.containers.run() instead, or given the container object returned from create(), call ingreedy.start().

(docker ps by default only shows running containers; docker ps -a should show the container in a "created" state.)


Best practise/standard practise for semantic versioning in development I have a project that looks like this:

SOME_MONOREPO, managed by lerna

- package-a
- package-b - depends on package-a, package-c
- package-c - is currently private, so not published as npm package.
When package-b builds, it accesses package-c directly via symlink.
As we move forward, we'll transition this to published packages like package-a.

ANOTHER_PROJECT

- package-foo- depends on SOME_MONOREPO.package-a

Now, say I'm doing some work on package-a, this could be a major, minor or patch change.

There are a few possible development flows:

The simplest: the work just occurs on package-a, I write tests for it, and then when ready version bump and push to master.
I'm doing work on package-a, but also updating package-b to adapt to the breaking changes/use the new functionality. In this scenario version bump isn't strictly necessary to do the code, because lerna's symlinks will still let package-b access the package-a code.
I'm also doing work on package-foo. In this case I need to be publishing to npm in order to access the code.

So, I'm thinking that nightly/unstable/prerelease builds are warranted here. That is - I build some functionality on package-a, build an unstable artefact, and then package-foo can use it, see if it's fit for purpose.

At some point though - a decision needs to be made whether this is a major, minor or patch update.

The question is - when should this happen? Does the developer, when making changes, decide on their first PR to develop each time what the version bump is? (And it not being a release PR, the CI automatically publishes it with the 'unstable' appendix, when it is release the unstable appendix is removed). What about if another developer is simultaneously working on the same package?

Is there a way to not think about version bumps until formal release?

Version bumps should happen every single build.  Waiting until a formal release and releasing everything as 1.0.0 is an option, but you should still know what build you are using in your alpha/beta.

Ultimately, the developer will have to determine if a major release increment is needed.
You should only do a major release bump if you introduced a breaking change.

For your regular builds, a common practice is to let your CI tool increment the "patch release" number or the X.X.Patch portion of the version.  This way your versioning is tied to your builds which will help the development and testing process.  If you choose to only do a nightly build, you can have your CI tool increment this number then so that you and other developers know what version of the package they are using.

NPM has published a good guide on how to handle semantic versioning.


post jenkins build console output to slack I know how to notify build status using the slack notification plugin. I can seeproject - #buildnumber Success after 1 min 18 sec (Open) in my slack channel. But I want the console output to be sent to slack as well (upon failure) instead of a link. I am using free style project since I need some scripts to build my project. The purpose is that I want engineers to know the result of their build without granting them Jenkins access.
edit:
if console log can be uploaded as a text, would be also good. Other forms of 'output' would be welcome as well.
new edit:
I found the answer by myself.

I figured it out by myself, there is no plugin needed.

build.sh &> /dev/stdout | tee jenkins.txt
and then curl -F file=@jenkins.txt -F channels=yourchannel -H "Authorization: Bearer your token" https://slack.com/api/files.upload

There is no '#' needed for slack channel and channel should not be private.

And if you use buils.sh | tee jenkins.log then you can't get the log from stderr.


Docker image delivery lead time SLA What is the best practice either reference value in terms of an SLA spec to expect the infrastructure to:

deliver a Docker image say for 100MB and 1GB size as reference?
unpack the above?
instantiate a container of the above?

Sure you can say "how important is that" but somewhere it gets irrational. SLA offers rational approach.

Service Level Agreement

https://en.wikipedia.org/wiki/Service-level_agreement

https://www.atlassian.com/it-unplugged/best-practices-and-trends/stop-hating-on-slas

https://www.cio.com/article/2438284/outsourcing/outsourcing-sla-definitions-and-solutions.html

What will happen if the docker registry is down? Are all departments blocked? What is the impact?

If all production images reside in a docker registry and someone wants to release a newer version to production, but the registry is down, is it still possible to release the product?

SLA is about defining processes, checking the impact of each of them and the more important the higher the SLA.

One could this tool to choose a certain SLA for a certain procedure:

https://uptime.is/

A SLA of 99.999 https://uptime.is/99.999 means a yearly downtime of 5m 15.6s, while https://uptime.is/99.9 means 8h 45m 57.0s.

One could set a SLA of 99.999 for the docker processes, but is a downtime of 5m 15.6s really required or is 8h 45m 57.0s in a year acceptable? If one hosts an ecommerce website and the webshop is down a couple of hours this will cause a lot of harm for the company as customers could not buy products and will probably buy products somewhere else.

For the docker processes that are used by developers and for releases to production one could argue that the first one could be 99.9 and the latter 99.999.


What is ansible-runner about? I've read the README file from the github repository as well as the documentation site of ansible-runner (from the official ansible project) but I don't understand what is ansible-runner about and in which case can it be useful ? Is it a project bring additional features or it is created to run ansible in a different way ?

This project is to support automation of running ansible playbooks. There is a GUI based project called Ansible Tower or AWX to do that, but ansible-runner seems to be the underlying layer of the automation in essence replacing ansible-playbook command.

There are some really serious issues for automation when trying to use ansible-playbook as part of some scripts. The output of ansible-playbook is the most awful thing under the sun when it comes to parsing. So you have to start with writing your own output plugin and then write a parser for that, then you need to figure out how to manage the environment variables and passwords and many other things. So ansible-runner is the answer to this. It would preserve the artifacts from running a task or playbook and make encapsulation of ansible in further automation possible.

You can run it from shell, you can run it as python module and there is even a reference container for it and obviously it is used by AWX. Further the project's event emitter will hopefully be much better pluggable to monitoring systems like ELK and to Message Bus type systems.


How to redirect tomcat logs TO console? So I have a problem that is the reverse of what everyone else on the internet wants.. I want to take tomcat logs that are being written out to /usr/local/tomcat/logs and have them be output to console instead. (Everyone else is asking the reverse.. how to get things away from console and to file). Reason why is because I'm on a system that captures output to console and directs it to an analytics program, it has to go to this program, and I have no control over how this is designed (ie. I can't change the program or capture process to use files as well as console).

To clarify, I am doing this on a Linux system, specifically Alpine.

There is Tomcat configuration file for the logging: TOMCAT_HOME/conf/logging.properties.
Edit the file and remove all the handlers except the console handler:

.handlers = java.util.logging.ConsoleHandler

This will direct all logging to the console, except for access logs (localhost_access_log.XXXX.txt). It is always written to file by the AccessLogValve, configured in 'server.xml'. To redirect that one you need to configure own implementation of the AbstractAccessLogValve... which requires coding.