Terraform apply exit code on error I'm using Terraform as part of a Jenkins Pipeline. When terraform apply encounters an error, it does not return a non-zero exit code. When running into a problem on Jenkins, it does not stop the pipeline or mark the build as FAILURE.

If terraform encounters an error, I need the job to stop and not progress.

Looking at the docs for apply, it doesn't look like I can provide a flag to return helpful exit codes. The docs for plan has a -detailed-exitcode flag, but doesn't look like that'll help me.

Turns out this was not a real issue. My apologies. At the end of that shell script, it did an echo which is what was returning the non-zero status code. After I took that out, it worked as expected.


Terraform: Route53, LB and CDN interdependencies My team uses Terraform to deploy the necessary infrastructure.
Currently, our stack is composed of: AutoScalingGroup fronted by a Load-Balancer, and a Route53 DNS record that points to this LB;

We would like to extend the stack to include a Cloudfront distribution that points to the LB, and make the Route53 DNS record point to the CDN.

I would like to be able to:

spin up Cloudfront and change the DNS record to point to the CDN, without causing any downtime;
tear down the CDN, change the DNS to point to the load-balancer, without causing any downtime;

The current configuration is as follows:

resource "aws_lb" "alb" {
  name                       = "${var.alb-name}"
  ...
}

resource "aws_cloudfront_distribution" "cdn" {
  count = "${var.use_cloudfront != "false" ? 1 : 0}"
  ...
}

resource "aws_route53_record" "www" {
  count = "${var.domain-name != "" ? 1 : 0}"

  depends_on = ["aws_lb.alb", "aws_cloudfront_distribution.cdn"]

  zone_id = "${data.aws_route53_zone.primary.zone_id}"
  name    = "${var.domain-name}"
  type    = "A"

  alias = {
    // we use concat because 'count' makes the response of the resource a list.
    // link to similar issue: https://stackoverflow.com/questions/45654774/terraform-conditional-resource
    name                   = "${var.use_cloudfront == "true" ? element(concat(aws_cloudfront_distribution.cdn.*.domain_name, list("")), 0) : aws_lb.alb.dns_name}"
    zone_id                = "${var.use_cloudfront == "true" ? element(concat(aws_cloudfront_distribution.cdn.*.hosted_zone_id, list("")), 0) : aws_lb.alb.zone_id}"
    evaluate_target_health = true
  }
}

The current behavior is as follows:

the spin-up of the CDN, followed by re-routing the DNS record to point to the CDN, does not lead to any downtime;
the tear-down of the CDN, followed by re-routing the DNS record to the load-balancer, does cause downtime;

I am aware that the depends_on = ["aws_lb.alb", "aws_cloudfront_distribution.cdn"] on the Route53 record might not be the best choice to create the desired behavior. However, I am not sure how to reach the desired behavior using terraform.

I've also added the Terraform output to a Gist file: https://gist.github.com/rafaelmarques7/03ec6e576faff81ef36d7fc878e85230

I am wondering if anyone has any suggestions to reach the desired behavior. Let me know if the provided information is insufficient. Thank you for your help, it is appreciated!

Modelling these permutations as a single module with lots of parameters is possible as you've seen, but it's often not an ideal way to model things because this sort of careful rollout strategy tends to require control over individual steps, rather than rolling everything all at once.

The Terraform team recommends using module composition instead. That means to split the problem into multiple smaller modules that the calling module can then combine to achieve the desired result.

In your specific case, it looks like there are three different building blocks: a load balancer, an optional Cloudfront distribution, and a DNS record. I'm going to show some examples of module composition using Terraform 0.12 features. If you're still using Terraform 0.11 then you can still follow a similar pattern, but Terraform 0.11's expression handling is more limited so the details would look different.

When breaking down this problem, my first step would be to identify what information needs to flow between these components. In this case:

The load balancer has some arguments of its own, but it doesn't depend on the other components at all.
The optional Cloudfront distribution needs to know the hostname of the load balancer.
The Route53 record needs a record name and zone id to populate an alias block; they could be for either the load balancer or the Cloudfront distribution depending on which of the two architectures the calling module is following.

In the full architecture where Cloudfront is used, these modules might be composed like this:

module "load_balancer" {
  source = "./modules/load-balancer"

  # ... load-balancer specific arguments
}

module "cloudfront" {
  source = "./modules/cloudfront"

  target = module.load_balancer.dns_name
}

module "dns" {
  source = "./modules/route53-record"

  alias = module.cloudfront.route53_alias
}

In the load-balancer-only architecture, we remove the cloudfront module and connect the DNS directly to the load balancer:

module "load_balancer" {
  source = "./modules/load-balancer"

  # ... load-balancer specific arguments
}

module "dns" {
  source = "./modules/route53-record"

  alias = module.load_balancer.route53_alias
}

Terraform 0.12 allows us to define that alias variable on the dns module with a specific object type, to easily pass the alias settings between modules as a single object value:

# (in the route53-record module)
variable "alias" {
  type = object({
    name    = string
    zone_id = string
  })
}

resource "aws_route53_record" "example" {
  # (other arguments as in your example)

  alias {
    name                   = var.alias.name
    zone_id                = var.alias.zone_id
    evaluate_target_health = true
  }
}

The idea here is that both of the other modules follow this conventional structure so that the route53-record module can be compatible with either of them without caring which one is used. This would then correspond with outputs in each of the other modules following the same structure:

# in the load-balancer module
output "route53_alias" {
  value = {
    name    = aws_alb.example.dns_name
    zone_id = aws_alb.example.zone_id
  }
}

# in the cloudfront module
output "route53_alias" {
  value = {
    name    = aws_cloudfront_distribution.example.domain_name
    zone_id = aws_cloudfront_distribution.example.hosted_zone_id
  }
}

With this different design, we can address your original use-case by giving control to the author of the calling module to drive the switch between these two models. In order to avoid downtime when switching from Cloudfront back to the load balancer, we need to repoint the DNS first and then destroy the Cloudfront distribution only when it is no longer used. So we can do that in two steps, first by modifying the first example above to point the DNS record at the load balancer while retaining the Cloudfront distribution:

module "load_balancer" {
  source = "./modules/load-balancer"

  # ... load-balancer specific arguments
}

module "cloudfront" {
  source = "./modules/cloudfront"

  target = module.load_balancer.dns_name
}

module "dns" {
  source = "./modules/route53-record"

  alias = module.load_balanccer.route53_alias
}

Then, once the Cloudfront distribution has become idle, remove that module altogether at your leisure and apply again to clean it up.

Composition is the primary way we model relationships in Terraform, so it's most natural to use composition when working with modules too, rather than building big modules with lots of conditionals to handle every permutation. Composition allows flexibility in how to make changes to the infrastructure in future so that the maintainer of the top-level module can be the one to decide what tradeoffs to make, such as making smaller changes to avoid downtime.

I used modules above because the original question was about modules, but I'd note also that after this decomposition we're left with one module per resource whose name is the resource type, which is one of the "smell tests" in When to write a module?.

If decomposing these into separate modules ends up just creating modules that are thin wrappers around single resource blocks then it's usually better to just eliminate the module altogether and just use the resource types directly.

However, if one or more of the modules still raises the level of abstraction (e.g. by providing hard-coded good values for most of the resource arguments and only exposing a small subset of them for customization) then that module in particular may still be warranted.


What is "black box" testing? What is "black box" testing and how it is different from ordinary testing? For example black box tests prior to deploying to production.

Note: "black box" testing is a common requirement when applying for devops roles (not necessary a tester thing).

I will make my answer, based on the knowledge for security testing, but IMHO this can be generalized.

Black box testing - when the tester know nothing about the system, components, liaisons, connections, etc. This can be helpful more like UI/UX testing, functional testing. Example: you do not work for Microsoft and also you do not have the source code and need to test MS Office
Gray box testing - when the tester have some knowledge about how the system work, some information about internals.
White box testing - when the tester have or can get all the information about internals, connections, principles of work of the tested product. Example: testing Linux kernel.

For reference:

Black box testing (Wikipedia)

White box testing (Wikipedia)

Gray box testing (Wikipedia)


Processing message queues quickly without wasting money on AWS Lambda What are some strategies and patterns for processing message queues quickly?

I have a Lambda function that is invoked on a schedule every x minutes to process a number of messages in the queue. I could reduce the interval, but a lot of the time there won’t be any messages to process and as Lambda is priced per request this would be cost inefficient. So I’m looking for a few ideas on other ways to achieve this

An SQS queue can now be wired directly to a Lambda function as an event source, so you don't have to start the Lambda function on a schedule.

https://aws.amazon.com/blogs/aws/aws-lambda-adds-amazon-simple-queue-service-to-supported-event-sources/


Who is responsible for running and maintaining shared services in DevOps As a member of a small agile development team developing an application heavily relying on Kafka I wonder what implications the DevOps paradigm of You build it, you run it has regarding running and maintaining a shared service like Kafka used by other teams as well.

Does the You build it, you run it not only revolve around our application?

Does DevOps really mean you are using another service so you have to manage that service as well?

I think we need to view this in the larger context of Flow and Systems Thinking which is one of the core values of DevOps.

Specifically, DevOps Values require an understanding that each individual is part of a larger system and work toward increasing the flow of value through that system to the customer.
DevOps also focuses on helping the organization set a shared vision and work incrementally and collaboratively on achieving that vision.

In such a scenario, the focus needs to shift from "technology-specific silos" to "cross functional teams". And each DevOps practitioner needs to achieve a T-Shaped Skillset with core as well as shared responsibilities.

So, to answer your question:

  Does DevOps really mean you are using another service so you have to manage that service as well?

I think it depends.

If your being able to manage the service leads to increased flow of work through the pipeline to the customer then yes.

If however, it leads to you becoming a bottleneck or constraint to flow (perhaps due to you becoming overloaded with additional responsibilities), then no.

If you are able to share this responsibility with others than manage the service (and have a Systems and Flow mindset), and collaborate with them so that none of you get too overloaded with work to become a constraint to flow, then yes.

References:

DevOps Values
YBI/YRI
What is DevOps
Cross-Functional Teams and DevOps


Using Google IAM for GKE service web access I am hosting an application on GKE and would like to be able to let users from my organization access this application from the web. I would like them to be able to log-in using their Google Account IAM credentials.

Is there a way to configure a service exposing the clusters web endpoint such that to access this service the user simply needs to login with their google account?

For example, when testing a service I can easily do a web-preview in the cloud-shell and then access the web application in my browser.

Is there a way to configure this such that any users authorized in my organization can access the web interface of my application?

Okay, I managed to make it work perfectly. But it took a few steps. I am including the manifest here that is required to setup the IAP using an ingress. It requires a few things which I listed in the manifest below. Hopefully this can help others since I could not find a single source that had all of this put together. Essentially all you need to do is run kubectl apply -f secure-ingress.yaml to make everything work (as long as you have all the depenedencies) and then you just need to configure your IAP as you like it.

secure-ingress.yaml

# Configure IAP security using ingress automatically
# requirements: kubernetes version at least 1.10.5-gke.3
# requirements: service must respond with 200 at / endpoint (the healthcheck)
# dependencies: need certificate secret my-secret-cert
# dependencies: need oath-client secret my-secret-oath (with my.domain.com configured)
# dependencies: need external IP address my-external-ip
# dependencies: need domain my.domain.com to point to my-external-ip IP
# dependencies: need an app (deployment/statefulset) my-app
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: my-secure-ingress
  namespace: default
  annotations:
    kubernetes.io/ingress.class: "gce"
    kubernetes.io/ingress.allow-http: "false"
    kubernetes.io/ingress.global-static-ip-name: my-external-ip
spec:
  tls:
  - secretName: my-secret-cert
  backend:
    serviceName: my-service-be-web
    servicePort: 1234
---
kind: Service
apiVersion: v1
metadata:
  name: my-service-be-web
  namespace: default
  annotations:
    beta.cloud.google.com/backend-config:
      '{"default": "my-service-be-conf"}'
spec:
  type: NodePort
  selector:
    app: my-app
  ports:
    - protocol: TCP
      port: 1234
      targetPort: 1234
      name: my-port-web
---
apiVersion: cloud.google.com/v1beta1
kind: BackendConfig
metadata:
  name: my-service-be-conf
  namespace: default
spec:
  iap:
    enabled: true
    oauthclientCredentials:
      secretName: my-secret-oath


CORS in Docker Engine REST Api I have activated my Docker Engine V1.41 REST API in docker with this artcile: https://blog.usejournal.com/how-to-enable-docker-remote-api-on-docker-host-7b73bd3278c6
After that in my browser it works fine like http://localhost:port/container/list will show me all lists of my containers in json.
But now, when I try to reach the API with axios in react I get an CORS error:
So I think I have to setup cors for docker?
I found a bunch of articles about how to enable CORS for a docker image / container but nothing about enabling it for the docker agent / host itself.
Know someone who can answer?

CORS is applied at the application layer, not the infrastructure (docker) layer. So, this has nothing to do with docker. It also has nothing to do with the axios client. Rather, your server needs to allow requests from your client by returning the correct headers via the OPTIONS method.
If you're using nodejs and express, you can add something like this to your service:
app.use((req, res, next) => {
  res.header("Access-Control-Allow-Origin", "*");
  res.header("Access-Control-Allow-Headers", "Origin, X-Requested-With, Content-Type, Accept, Authorization");
  res.header("Access-Control-Allow-Methods", "GET");

  if (req.method === "OPTIONS") return res.status(200).json({});

  next();
});

Or, there are plenty of CORS libraries that can help you out.


How to properly achieve dynamic parallel action with a declarative pipeline? Currently, I'm going to need an implementation that must find all files within a directory and start a parallel task for every file found.

Is it possible to achieve this using declarative pipelines?

pipeline {
    agent any
    stages {
        stage("test") {
            steps {
                dir ("file_path") {
                    // find all files with complete path
                    parallel (
                        // execute parallel tasks for each file found.
                        // this must be dynamic
                        }
                    }
                }
            }
        }
    }
}

Managed to solve it with the following code:

pipeline {
    agent { label "master"}
    stages {
        stage('1') {
            steps {
                script {
                    def tests = [:]
                    for (f in findFiles(glob: '**/html/*.html')) {
                        tests["${f}"] = {
                            node {
                                stage("${f}") {
                                    echo '${f}'
                                }
                            }
                        }
                    }
                    parallel tests
                }
            }
        }
    }
}


Minikube - Is there a way to run Windows Containers? How do I run Windows containers on Minikube?

Is it possible?

It is not officially supported (as of Minikube 0.25.0, Kubernetes 1.9, January 2018). But there is Beta support for Windows server containers in Kubernetes.

These articles contain more information:

http://blog.kubernetes.io/2018/01/kubernetes-v19-beta-windows-support.html
https://kubernetes.io/docs/getting-started-guides/windows


run ansible command as another user using their ssh key i am trying to figure out how to run an ansible ping command as another user, using their ssh key.

User setup i have essentially:

master_user (me)
ansible (sudo user, pwdless sign in)

I logged in as ansible user and did:

ssh-keygen (generate key)
ssh-copy-id XXXX (copy the ssh key to corresponding server)

Tested it all and it works fine (each server has ansible user already fyi).
I updated the hosts file to have all my hosts as well.

Now when i remote into master_user, i want to run a ansible command but as ansible user something like this:

ansible all --become-user=ansible -m ping

This fails because it is trying to ssh into the servers using the master_users ssh key.

Error message: Failed to connect to the host via ssh

When i run -vvvv flag i can see it trying to user master_user ssh by looks of it

Offending key for IP in /home/master_user/.ssh/

How can configure it so that it uses ansible user always and use ansible users ssh key?

become_user will tell which user to become to run a particular ansible module on the remote server.

If you want to run the ansible command on your control machine as an other user, you need to use sudo or su on this machine e.g.

sudo -s -u ansible ansible all -m ping

The -s option will load the user's shell an make sure homedir and all other envs are correctly set prior to running the command.

An other possible solution would be to use the ssh key you created for the ansible user from your account

ansible all --private-key=/home/ansible/.ssh/id_rsa -m ping

This will potentially work only if your current user has read access to the given key.


npm not found but installed from the shell script file in Dockerfile I have a following Dockerfile:

FROM ubuntu:18.04

RUN mkdir app && cd app
WORKDIR app/

RUN apt-get update && apt-get install -y \
    software-properties-common \
    curl \
    sudo \
    && curl -O https://hyperledger.github.io/composer/latest/prereqs-ubuntu.sh \
    && chmod u+x prereqs-ubuntu.sh \
    && ./prereqs-ubuntu.sh \
    && npm install -g \ # this line failed
    && yo \
       composer-cli@0.20 \
       composer-rest-server@0.20 \
       generator-hyperledger-composer@0.20 \
       composer-playground@0.20

    CMD /bin/bash

The prereqs-ubuntu.sh is from https://github.com/hyperledger/composer/blob/master/packages/composer-website/jekylldocs/prereqs-ubuntu.sh.

However, it says /bin/sh: 1: npm: not found on the line npm install -g when I try to build the image. But npm is installed when running prereqs-ubuntu.sh already and can be ran successfully at the end of the file (npm --version).

Also, when I remove the npm-install part from the Dockerfile (shown below) and do docker run --name test -it myimage to see if npm is installed, but it is installed.

FROM ubuntu:18.04

RUN mkdir app && cd app
WORKDIR app/

RUN apt-get update && apt-get install -y \
    software-properties-common \
    curl \
    sudo \
    && curl -O https://hyperledger.github.io/composer/latest/prereqs-ubuntu.sh \
    && chmod u+x prereqs-ubuntu.sh \
    && ./prereqs-ubuntu.sh \

    CMD /bin/bash

The issue is that prereqs-ubuntu.sh uses bash to install the npm. While the RUN directive uses sh to run the commands.

Where is the npm installed

root@2cd4a6af90f4:/app# type npm
npm is /root/.nvm/versions/node/v8.16.0/bin/npm

The PATH for the RUN directive

# echo $PATH
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin


How to upload existing local project to a remote existing and empty github repository I'm using wsl/ubuntu20 with a laravel project on it. now when I am done, I want to upload and sync my project to an existing github repo.
I don't mind if anything will be deleted on the remote, but i must use the existing repo'.
What are my options?
I prefer to do something like:

init git repo in the folder(what is the proper syntax?)
'connect' the local and the remote repo's.
add / commit / push
work...

thanks

When you create a new repository in Github, all the steps required are shown:

You just need to modify this accordingly for your use case:

you already have files, so no creating files
you may want to create the correct .gitignore for your project type, usually you exclude temporary files, such as caches, logs as well as files containing credentials
For this to work as per SSH example, you already need your primary public key uploaded to the same github account. If not, you can do this with a Github token, or look at the https instructions
if the existing repository already has commits in it, are you supposed to wipe them out? In this case use -f on git push.


How to navigate to an index.html with nginx? This builds and runs without apparent error:
root@mordor:/home/nicholas/docker/nginx#
root@mordor:/home/nicholas/docker/nginx# docker exec -ti some-nginx bash
root@2c912aa5ce5c:/#
root@2c912aa5ce5c:/# ls /usr/share/nginx/html/
50x.html  index.html
root@2c912aa5ce5c:/#
root@2c912aa5ce5c:/# cat /usr/share/nginx/html/50x.html
<!DOCTYPE html>

<title>Error</title>

An error occurred.

Sorry, the page you are looking for is currently unavailable.

Please try again later.

If you are the system administrator of this resource then you should check
the error log for details.

Faithfully yours, nginx.

root@2c912aa5ce5c:/#
root@2c912aa5ce5c:/# cat /usr/share/nginx/html/index.html

hello world

how do I put this file in a diff dir?

root@2c912aa5ce5c:/#
root@2c912aa5ce5c:/# exit
exit
root@mordor:/home/nicholas/docker/nginx#
root@mordor:/home/nicholas/docker/nginx# docker container ls
CONTAINER ID        IMAGE                COMMAND                  CREATED             STATUS              PORTS                  NAMES
2c912aa5ce5c        some-content-nginx   "/docker-entrypoint.…"   5 minutes ago       Up 5 minutes        0.0.0.0:8080->80/tcp   some-nginx
root@mordor:/home/nicholas/docker/nginx#
root@mordor:/home/nicholas/docker/nginx# ls
dockerfile  index.html
root@mordor:/home/nicholas/docker/nginx#
root@mordor:/home/nicholas/docker/nginx# cat dockerfile
FROM nginx
COPY index.html /usr/share/nginx/html
root@mordor:/home/nicholas/docker/nginx#
root@mordor:/home/nicholas/docker/nginx# cat index.html

hello world

how do I put this file in a diff dir?

root@mordor:/home/nicholas/docker/nginx#

Yet, there's no output when browsing to localhost:
nicholas@mordor:~$
nicholas@mordor:~$ lynx localhost --dump

Looking up localhost
Making HTTP connection to localhost
Alert!: Unable to connect to remote host.

lynx: Can't access startfile http://localhost/
nicholas@mordor:~$
nicholas@mordor:~$ lynx localhost/index.html --dump

Looking up localhost
Making HTTP connection to localhost
Alert!: Unable to connect to remote host.

lynx: Can't access startfile http://localhost/index.html
nicholas@mordor:~$
nicholas@mordor:~$ lynx localhost/50x.html --dump

Looking up localhost
Making HTTP connection to localhost
Alert!: Unable to connect to remote host.

lynx: Can't access startfile http://localhost/50x.html
nicholas@mordor:~$

Neither am I getting a welcome file from nginx as I'd expect.

Did you only try to reach localhost? Or even 0.0.0.0 or 127.0.0.1?
Depending on the operating system, docker config and the internal networking setup, some problem can occurs to reach the docker network.
You can inspect the container to get the network IP, maybe you can go there https://stackoverflow.com/a/20686101/2904965


SSH tunneling on steroids I have a remote redis machine with no ssh and no open ports except to the same lan + a remote machine (on the same lan) with ssh + my pc outside the lan. What I'm trying to do (mostly to see if it's possible) is redis-cli on my pc -> ssh tunnel to the ssh machine -> something -> redis so it would work.
Is netcat + mkfifo the right way?
Thanks :)

Assuming the following:

YOUR_PC, has Redis client, SSH client, SSH access to SSH_SERVER
SSH_SERVER, has SSH server, redis access to REDIS_SERVER
REDIS_SERVER, has Redis server

Set up the tunnel from YOUR_PC, port 1234, to REDIS_SERVER:REDIS_PORT, via SSH_SERVER

ssh SSH_SERVER -L 1234:REDIS_SERVER:REDIS_PORT

On another terminal, on YOUR_PC, run the redis client (based on https://stackoverflow.com/a/40678950):

redis-cli -h localhost -p 1234


Is it bad to use sudo from Jenkins? I use Publish Over SSH plugin to deploy my apps from Jenkins to different environments. Some deployment jobs do environment preps and things like stop and restart the app server system service. Some of those commands require sudo.

I am just curious if it can be a bad security practice to require sudo within remote publish and execution Jenkins jobs. Should we change the security policy on the target host to allow the required functions to be performed without sudo?

Whether you allow remote sudo or remote access to something that does SUID root you have a pretty similar attack surface.  I would keep sudo in the chain because it lets you limit the commands easily and has logging that will be vital if you need to audit things later.  sudo also has a much longer history in production.  Doing something else will have less history and higher changes of unpleasant surprises.

There are other things you can do to make this more secure though:

tighten up ssh
only allow the restart commands for a few specific users including one for jenkins
only allow logins for those specific users from all internal IP's or just jenkins and jump box IP's
store logs on remote boxes so they can't be mucked with


How to "inject" AWS Lambda code into a CloudFormation? I'm exercising an idea of not using S3 bucket to refer to the code of a lambda function.

The usual example of a lambda CloudFormation code might look like:

    MyLambda:
        Type: AWS::Lambda::Function
        Properties:
        FunctionName: hello
        Handler: index.handler
        Runtime: nodejs8.10
        Role: !GetAtt LambdaExecutionRole.Arn
        Code:
            ZipFile: |
            <can we "inject" the code and/or binary here?>

But what I would like to do instead is:

inject that code from the file system, after potentially running my (unit) tests against that code;
maybe even just inject a compiled binary instead of actual code;

It is possible to do similar using CLI (create a zip manually, refer to it). I understand that CloudFormation changes would be running on the AWS infrastructure and not locally or a build server, so I'm open to other non-CloudFormation options.

So my question is - what is the best way to deploy the AWS functions sticking to "infrastructure as code", preferably sticking with CloudFormation. Or is S3 + CloudFormation really the best option?

You've got a couple of options:

1. Embed the source code to the CloudFormation template

I do just that in my ec2-start-stop demo. You'll see file ec2-start-stop.template.yml with these lines:

    StartStopLambda:
      Type: AWS::Lambda::Function
      Properties:
        [...]
        Runtime: python3.6
        Code:
          ZipFile:
            Fn::Join:
            [...]
            -
              - "%%{ec2-start-stop.lambda.py}%%"

Then I've got the actual Python script referred in the %%{ec2-start-stop.lambda.py}%% and a simple import-files.py script that somewhat intelligently embeds the python file into the yaml file and creates a valid standalone CloudFormation template with the lambda code in it.

Note that you're limited to 4096 bytes including new lines for your lambda code.

2. Use aws cloudformation package / deploy

If you've got a more complex Lambda that needs external libraries or doesn't fit in the 4kB limit you can use aws cloudformation package and aws cloudformation deploy to facilitate the deployment to CI/CD.

In this case your Lambda resource in the template points to a local directory, e.g. lambda/. The package command then zips up the contents of the directory, uploads to the provided S3 Bucket with a unique name and generates an updated CFN template with the link to the uploaded file. The aws cloudformation deploy command then takes the template, creates a change set and applies the changes, thus updating the Lambda.

I've used both methods in CI/CD pipelines and both work well because the source files are kept in separate files until deployment.

Hope that helps :)


Add mapping ip and host into /etc/hosts with docker I am using docker to scale my service.
I know there's a command called --add--host for adding host into /etc/host

However, I have a bunch of hosts need to be added. It's just too long to launch a docker in this way

docker run -it \
  --add-host=testing1.example.com:10.0.0.1 \
  --add-host=testing2.example.com:10.0.0.2 \
  --add-host=testing3.example.com:10.0.0.3 \
  ...

Therefore, I was thinking to create a function to read a mapping file and solve the problem.

The file format is like the following

host-list

14.64.65.3 A
11.64.65.18 B
12.64.65.23 C
13.10.200.63 D
13.10.5.77 E
14.10.5.72 F
15.10.5.81 G
...

I am new on shell script and have no idea to start with.

You could try to use docker compose and the docker compose "extra_hosts" tag, as described on the docker compose specification at https://docs.docker.com/compose/compose-file/.

  extra_hosts

  Add hostname mappings. Use the same values as the docker client --add-host parameter.

  extra_hosts:

  "somehost:162.242.195.82"
  "otherhost:50.31.209.229"

  An entry with the ip address and hostname is created in /etc/hosts
  for this service, e.g:

  162.242.195.82  somehost

  50.31.209.229   otherhostinside containers

So, this will mean, that you need to use docker compose in addition to docker.


Is there a limitation in AWS to open an IPSec VPN connection? Consider the following situation (for test purposes):

there is a public IPSec VPN endpoint (StrongSwan);
a machine with either Ubuntu (Strong Swan client) or Windows OS (native client) running in the cloud should be the VPN guest.

Result:

Windows Server 2019 machine establishes VPN connection successfully but can't send any packet
Ubuntu 18 machine establishes connection to some degree but does not create PPP0 interface.
Ubuntu 16 machine makes a handshake but then there is a message about protocol mismatch (maybe in this specific case there is an incompatibility between both endpoints).

Are there any AWS limitations to IPSec VPN?

For me it looks to be tunnel misconfigured -  "there is a message about protocol mismatch"

Regarding the limitations quick google took me here: [strongSwan] strongswan limits


Continuous Deployment with jenkins I am a beginner in DevOps.
I created a continuous integration cycle using jenkins(docker image) and bitbucket.
I used a simple spring boot hello-world project and after CI i got an executable Jar.
I created a dockerfile and docker-compose for the deployment process.
I tried this on my local server and it worked successfully(hello world successfully displayed on browser).
Now i want to begin the continuous deployment process on remote server how can i do that ?

Your next step will be to identify a solution for container orchestration.  For a quick dev test solution, you could host your own docker registry, push your built image to the registry, and add a deployment step that stops the existing container and spins up a new one from your registry.
However, in a production setting there are many other aspects to consider such as downtime, versioning, security, etc.  There are several products and tools that are designed to handle containers such as AWS Elastic Container Solution (ECS), Kubernetes.


How to sync two directories on same host using Ansible? I am using the below playbook for mounting partitions on another EBS volumes while provisioning instance on AWS

    ---
    - name: Mounting necessary partitions
      hosts: localhost
      become: yes
      become_user: root

      tasks:

        - name: Ansible formating partitions.
          command: "{{ item }}"
          loop:
            - mkswap -f {{ ebs_swap }}
            - mkfs.xfs -f {{ ebs_home }}
            - mkfs.xfs -f {{ ebs_var }}
            - mkfs.xfs -f {{ ebs_tmp }}
            - mkfs.xfs -f {{ ebs_app }}
            - mv -f /home /home_old
            - mv -f /var /var_old

        - name: Dispaly uuid & store in variable
          command: blkid -s UUID -o value {{ ebs_swap }}
          register: uuid_swap

        - name: Dispaly uuid & store in variable
          command: blkid -s UUID -o value {{ ebs_home }}
          register: uuid_home

        - name: Dispaly uuid & store in variable
          command: blkid -s UUID -o value {{ ebs_var}}
          register: uuid_var

        - name: Dispaly uuid & store in variable
          command: blkid -s UUID -o value {{ ebs_tmp }}
          register: uuid_tmp

        - name: Dispaly uuid & store in variable
          command: blkid -s UUID -o value {{ ebs_app }}
          register: uuid_app

        - name: Creates directory
          file:
            path: "{{ item }}"
            state: directory
          with_items:
            - /home
            - /var
            - /tmp
            - /app

        - name: Add the below lines
          blockinfile:
            path: /etc/fstab
            state: present
            block: |
              UUID={{ uuid_swap.stdout }}   swap      swap defaults                         0   0
              UUID={{ uuid_home.stdout }}   /home     xfs  defaults,nodev                   0   0
              UUID={{ uuid_var.stdout }}    /var      xfs  defaults,noexec,nosuid,nodev     0   0
              UUID={{ uuid_tmp.stdout }}    /tmp      xfs  defaults,noexec,nosuid,nodev     0   0
              UUID={{ uuid_app.stdout }}    /app      xfs  defaults                         0   0

        - name: Mounting disks permanently
          command: "{{ item }}"
          loop:
            - swapon -a
            - mount -a

        - name: Copy multiple Directories
          copy:
            src: "{{ item.src }}"
            dest: "{{ item.dest }}"
            mode: "{{ item.mode }}"
          with_items:
             - { src: '/home_old/',dest: '/home/',mode: preserve}
             - { src: '/var_old/',dest: '/var/',mode: preserve}

I have tried multiple ways to copy the data back to newly mounted EBS volumes at task name
Copying /home data on new EBS.

I have tried using copy synchronization & rsync modules. I know using cp is not recommended option but I can not see any other option related to syncing the data between to directories with preserving permissions.

But when I do it manually it works perfectly.

I am mounting these partitions on another EBS is just coz of CIS hardening steps includes it.

I have followed a couple of blogs already. such as
1. one 2. two

Can anyone help in solving this? This is very urgent.

  ERROR: failed: [localhost] (item={u'dest': u'/var/', u'src': u'/var_old/'}) => {"ansible_loop_var": "item", "changed": false, "item": {"dest": "/var/", "src": "/var_old/"}, "msg": "could not find src=/run/acpid.socket, Could not find or access '/run/acpid.socket' on the Ansible Controller.\nIf you are using a module and expect the file to exist on the remote, see the remote_src option"}

The copy module copies a file from the local or remote machine to a location on the remote machine.

In order to copy the file from the remote machine itself, set remote_src to yes.

- name: Copy multiple Directories
          copy:
            src: "{{ item.src }}"
            dest: "{{ item.dest }}"
            mode: "{{ item.mode }}"
            remote_src: yes
          with_items:
             - { src: '/home_old/',dest: '/home/',mode: preserve}
             - { src: '/var_old/',dest: '/var/',mode: preserve}


Gitflow with pull request or pull request My team is currently using gitflow workflow into develop branch.

Here is an overview for what we are doing. We are a sql server data warehouse team  using Azure repos and Azure devops to build and deploy our code to SSIS, SSAS, and 30+ databases in our environment. I typically deploy to a integration server and then to a prod server. I have an auto build for our SQL server solution that triggers on checkin to the develop branch. I run manually a build that generate the deployment scripts. All the SSAS and SSIS builds and deployments are triggered on the checkin to the branch.

I am looking at trying to migrate to a workflow that would be a pull request so I don't have a broken build that holds everyone eases deployments.  If someone breaks the build in the develop branch then I cannot deploy by the current method.

Prior to moving to the azure devops I was using a release workflow to create the deployment artifacts manually and I would deploy everything manually. This was a completely manual process and time consuming process that was very tedious.

I would like to figure out a path to get to using pull request. Does this mean I would do away with gitflow process? My team is currently using VSCode for creating features and this part works well. I am looking for advice for moving forward with pull request.

You can set up Branch Policies on your develop branch which requires a build to pass successfully before the Pull Request can be completed.

I have set this up for my team with great success; we very rarely will have a broken master branch. The PR does not run the main build, but instead runs a separate, slimmed down build that checks that the code compiles and the unit tests pass. The "Gated" build runs on Pull Request creation, the "Full" build is triggered on change in master branch (PR completion).


How does the creation of an AMI on AWS work and why is the creation time unpredictable? Creation date: September 23, 2019 at 9:22:06 PM UTC+2

21:48: still waiting for completion

21:53: completed

Sometimes it takes 15 minutes and sometimes longer. How does a creation of an AMI work and why is the creation time unpredictable?

When creating an AMI, you are making a copy of your EBS volumes. This utilizes block storage and will only need to process the diffs of your EBS snapshots. If you changed a lot of data in your EBS or don't have a previous snapshot, it will take longer to create your AMI.

Here is a quote from the documentation at https://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/Creating_EBSbacked_WinAMI.html

  Depending on the size of the volumes, it can take several minutes for
  the AMI-creation process to complete (sometimes up to 24 hours). You
  may find it more efficient to create snapshots of your volumes prior
  to creating your AMI. This way, only small, incremental snapshots need
  to be created when the AMI is created, and the process completes more
  quickly (the total time for snapshot creation remains the same).


Expose kubernetes deployment on master's IP I have simple deployment which has 3 replicas. I want this deployment to be load balanced between this 3 replicas, but available under IP of my master and on some specified port.

This cluster was created by using kubeadm, I have one master and 20 worker nodes.

I tried to expose the service as LoadBalancer, but this only works when you have cloud load balancer configured. I don't have such thing and EXTERNAL-IP is always <pending>.
I tried to expose the service as NodePort, but this only works on nodes where pods were created. So for example after rollout, when pods will be created on different nodes, it will stop working. Also I guess this is not load balanced properly.
I tried to create ingress to make it work, but unsuccessfully.

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: registry-ingress
  namespace: registry
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  rules:
  - http:
    paths:
      - path: /registry
        backend:
          serviceName: registry-service
          servicePort: 5000

What is the proper approach here?

If the above is possible then second question: is it possible to expose it on 2 masters if I will want to create 2nd one?

I was wrong about the NodePort approach. It was firewall blocking the connection to my master on ports 30000-32767.

After unblocking this ports everything was working fine with NodePort service.


branch specific jenkins pipeline trigger I am trying to trigger a jenkins pipeline job only when a pull request has been approved & merged to a specific github branch in our CI/CD implementation.
currently whenever i merge a PR to release, jenkins job that is created to get triggered when PR is merged to master is also getting triggered, while I only want release jenkins job to be triggered.
strangely echo env.GIT_BRANCH cmd shows branch name as MASTER in master job, parallelly release job shows branch as RELEASE, when i only merged PR to release.
Things tried:

Branches to build feature in Jenkins
when condition in Jenkinsfile (its acting weird, both staging and production jobs are getting triggered when one branch is targetted)

Environment:
Jenkins ver. 2.176.3, Docker,Generic webhook trigger for Jenkins
I'm convinced that when condition will work, but for me it isn't, following is my partial when condition in Jenkinsfile:
stage("Step 1: Build Docker"){}
stage("Step 2: Push to DockerHub"){}
stage("Step 3: Production Deployment"){
            when {
                expression { return params.current_status == "closed" && params.merged == true && env.GIT_BRANCH == "origin/master" }
            }
            steps{
                -----
                sh 'kubectl apply -f **.yaml'
            }
}

In my jenkinsfile I'm writing logic to deploy my docker images to kubernetes cluster, so all that i  am looking for is to accurately trigger the job based on the branch.
Any suggestions on how to get it to work, what am i missing here?

Do you have 2 different jenkins jobs configured for master and release branches? Since github webhook event is sent to Jenkins irrespective of branch, both your jenkins jobs may be getting triggered by this github webhook event. And since both jobs are configured with separate branches, they may be checking out those branches during jenkins job execution which is why you may be seeing respective branches being printed in both jenkins jobs.
Checkout jenkins multibranch pipeline if you haven't already so you don't have to maintain multiple jenkins jobs for branches.
Now coming back to your Jenkinsfile code, you can try following:

Replace env.GIT_BRANCH with env.BRANCH_NAME

Use allOf inside when condition instead of &&  -
stage("Step 3: Production Deployment"){
    when {
        allOf {
            expression { env.BRANCH_NAME == "origin/master" }
            expression { params.merged == true }
            expression { params.current_status == "closed" }
        }
    }
    steps{
        -----
        sh 'kubectl apply -f **.yaml'
    }
}

Note that github webhook will trigger the jenkins job for every branch push or PR merge event (irrespective of branch), so your jenkins job would have to handle multi-branch scenarios like explained in this official doc example.


Is it required to move a domain to Google DNS in order to use certbot in kubernetes on Google Cloud platform? Follow-up to What are trusted providers that offer wildcard certificates and associated costs to be used in an Orchestration Platform like Kubernetes?

With https://github.com/jetstack/cert-manager it would be possible to automatically renew certificates.

At the moment my domain is registered somewhere else than google. Is it required to move the domain to Google DNS in order to use certbot in kubernetes on Google Cloud platform?

At the moment when I add a new subdomain, e.g. abc.somedomain.com I will add a CNAME to my DNS provider. At the moment I have two subdomains and I would like to switch to kubernetes. At the moment the IPs refer to some Virtual Machine instances that reside on another provider.

Current situation

What I understand of Certbot is that it is able to create certificates that are valid for three months. I certbot for almost a year right now and I am happy about it. When I create a new subdomain in my DNS provider I create a CNAME and run certbot and a new certificate is created. This works fine.

Preferred situation

Now I would like to phase out my VM and replace it with kubernetes. I was able to create a kubernetes cluster and expose the microservice to the internet using an ingress. One the IP was exposed, it is possible to access it.

Problems

The problem is that I now would like to get a secure Ingress, i.e. access the microservice via https rather than http, but I am struggling to understand how to configure it in k8s.

Attempt to solve the issue

A Google Cloud Platform project has been created
A kubernetes cluster has been created
...
...
...

@euank indicated that it is not required to create a DNS record on GCP in order to make it work.

https://github.com/jetstack/cert-manager/issues/399


Kubernets docker registry behind nginx-ingress I try to setup docker-registry in Kubernetes cluster behind the nginx-ingress controller.
The issue is, when I try to push the image into private registry, it tells me:

  Get https://registry.local/v2/: x509: certificate is valid for ingress.local, not registry.local

When I curl it, I get response from ingress' backend - 404.

Here's ingress' manifest:

kind: Ingress
apiVersion: extensions/v1beta1
metadata:
  name: docker-ingress
  annotations:
    kubernetes.io/ingress.class: "nginx"
spec:
  tls:
  - hosts: [ 'registry.local' ]
  - secretName: registry
  rules:
  - host: registry.local
    http:
      paths:
      - backend:
          serviceName: docker-registry
          servicePort: 5000
        path: /

And here's docker-registry manifest:

kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  labels:
    app: docker-registry
  name: docker-registry
spec:
  replicas: 1
  selector:
    matchLabels:
      app: docker-registry
  template:
    metadata:
      labels:
        app: docker-registry
    spec:
      containers:
      - command:
        - /bin/registry
        - serve
        - /etc/docker/registry/config.yml
        env:
        - name: REGISTRY_HTTP_ADDR
          value: 0.0.0.0:5000
        - name: REGISTRY_STORAGE_FILESYSTEM_ROOTDIRECTORY
          value: /var/lib/registry
        - name: REGISTRY_HTTP_TLS_CERTIFICATE
          value: /certs/ca.crt
        - name: REGISTRY_HTTP_TLS_KEY
          value: /certs/domain.key
        image: registry:2.6.2
        imagePullPolicy: IfNotPresent
        name: docker-registry
        ports:
        - containerPort: 5000
          name: http
          protocol: TCP
        volumeMounts:
        - mountPath: /var/lib/registry
          name: image-store
        - mountPath: /certs
          name: certs
      volumes:
      - name: image-store
        emptyDir: {}
      - name: certs
        configMap:
          name: certs
---
kind: Service
apiVersion: v1
metadata:
  labels:
    app: docker-registry
  name: docker-registry
spec:
  ports:
  - name: http
    port: 5000
    protocol: TCP
    targetPort: 5000
  selector:
    app: docker-registry
  type: ClusterIP

Separetly, nginx-ingress and docker-registry, work fine, but don't together.

Nothing valuabe from the ingress controller logs can be found.

First of all 404 error is resource error in kubernetes. It means your container in the pod is not created by your first script. Even if it is created and you can see it then your labeling is wrong in the second script.

I would suggest you check matchlabels of both YAML scripts. So in your case I can see that matchlabel is docker-registry in the spec section of the first script, but I don't see any docker-registry in the second script. I believe that in your metadata part of ingress script should have "name = docker-registry" instead of "name=docker-ingress".


Amazon ECS Basic Definitions and Terms I'm a beginner starting to learn about Amazon EC2 and I want to get clear on the following terms, if anyone can supply a basic definition and perhaps an example to tie them together that would be phenomenal:

Task,
Service,
Container,
Instance,
Cluster (of containers),
State (of a cluster, most odds),
ECS.

A partial answer will also be valuable.

AWS provide different services and categories them on base of computation, storage, databases, management, Analytics, Messaging, Developer Tools etc.

ECS comes under the computing services.

Amazon EC2 Container Service (ECS) – Container Management for the AWS Cloud

  Run your applications packaged as Docker containers.EC2 Container Service (or ECS for short). This service will make it easy for you to run any number of Docker containers across a managed cluster of Amazon Elastic Compute Cloud (EC2) instances using powerful APIs and other tools. You do not have to install cluster management software, purchase and maintain the cluster hardware, or match your hardware inventory to your software needs when you use ECS.

Here I will explain the steps to set up the docker based application in ECS and terminology.

Cluster: A cluster is a pool of EC2 instances in a particular AWS Region, all managed by ECS. One cluster can contain multiple instance types and sizes, and can reside within one or more Availability Zones.

This is the first step to creating ECS.before anything you need cluster it can be default or you can create new one.

EC2 Instance type: In cluster you need to choose the Amazon EC2 instance type for your container instances. The instance type that you select determines the resources available for your tasks to run on.

Amazon Elastic Compute Cloud (Amazon EC2) is a web service that provides secure, resizable compute capacity in the cloud.it's like your PC you can install mongo, MySQL (for MySQL RDS is best), PHP, node or whatever you want in it.While Creating EC2 you have to define which OS you want to use in ec2.You can access EC2 using pem file  that will be provided by AWS during creating ec2.https://aws.amazon.com/ec2/

Container: – A container is a packaged (or “Dockerized,” as the cool kids like to say) application component. Each EC2 instance in a cluster can serve as a host to one or more containers.
For every task definition, you need to specify container which tells about CPU, memory and other resources.
ECR Registries Amazon EC2 Container Registry (Amazon ECR) is a managed AWS Docker registry service that is secure, scalable, and reliable. Amazon ECR supports private Docker repositories with resource-based permissions using AWS IAM so that specific users or Amazon EC2 instances can access repositories and images. Developers can use the Docker CLI to push, pull, and manage images.

Once the above environment is ready you can place task or start services.

Task definition: A task definition is like a blueprint for your application. In this step, you will specify a task definition so Amazon ECS knows which Docker image to use for containers, how many containers to use in the task, and the resource allocation for each container.
service services manage the task that you have created.once you have created a task definition, you will configure the Amazon ECS service. A service launches and maintains copies of the task definition in your cluster. For example, by running an application as a service, Amazon ECS will auto-recover any stopped tasks and maintain the number of copies you specify.

NOTE: Elastic load Balancing can be setup before any thing done. when you want to create services you can configure with load balancer.load balancer conttain target group and container which need to be balanced.https://aws.amazon.com/elasticloadbalancing/

Elastic Load Balancing Elastic Load Balancing automatically distributes incoming application traffic across multiple Amazon EC2 instances. It enables you to achieve fault tolerance in your applications, seamlessly providing the required amount of load balancing capacity needed to route application traffic


WinRM / WSMan - getting SSL handshake error / TCP RST when connecting with IP address, correct connection when connecting with hostname While using Ansible to manage some AWS based Windows machines, when connecting using the IP as destination, on some machines I'm getting an SSL Error

  credssp: HTTPSConnectionPool(host='18.217.233.26', port=5986): Max
  retries exceeded with url: /wsman (Caused by SSLError(SSLError(\"bad
  handshake: SysCallError(-1, 'Unexpected EOF')\",),))", "unreachable":
  true}

In wireshark it looks like TCP RST is sent by server

When connecting using hostname or FQDN the connection succeeds.

I don't see any relevant log in the event logs on WSMan side.

Anybody have any experience with setting up winrm for Ansible and can assist?

I faced the same error .. You probably want to run ConfigureRemotingForAnsible.ps1 script with -ForceNewSSLCert option on your remote machines.

.\ConfigureRemotingForAnsible.ps1 -ForceNewSSLCert


What do you tell someone new to going oncall or PagerDutyl? [[ Based on tags and the wording this sounds like a PagerDuty-specific question, but it could apply to any tech or monitoring stack in my mind.... PagerDuty is just so dominant in SF-based jobs I've seen that is like tech Kleenex.... ]]

I've been searching in my spare time for a few months for an answer to: what do you tell a new recruit to pager duty (or PagerDuty or pagerduty)???  And I don't mainly mere "install the app and prey" training, but those are key parts of it.  I'm more looking for generic, general, globalizable advice like:

ask for help as early as possible
don't go sleep deprived "too" many nights in a row ~3
know who is likely to answer with low latency and who will notice work again at Monday 9am DOH!
it is easier to find answer on StackExchange than random forums at 3am free plug
triage: how big of a fire is it?  how many fire trucks do you call in?  Etc.
when to merge incidents
when to snooze incidents... but that is so scary!!  "You don't get fired for snoozing here?"  hahaha.
good notification settings
good phone/volume settings for getting woken up, even if you sleep like a log.
scheduling and calendar layers
rules of thumb and recommended tools
know who to call for what subsystem (varies based on locality and org) but using a wiki or Confluence is often part of that

Maybe I'm searching wrong.  I'm open to meta-suggestions for better ways to find this needle in the haystack.  Searching for "starting pagerduty" makes sense to me, but it gets results of how to setup these things programmatically or technically, not how to instruct the humans in dealing with sorts of things.  PagerDuty does try to address this, but it doesn't include the breadth of responsibilities that the job entails to me.

I also admit that I'm asking for everything and the kitchen sink and the city water supply for a bonus round.  Getting little pieces of this as links and pointing folks at it would be a huge improvement over what I've found so far.  I doubt anybody has done all of this in one doc or would ever want to do so.

I've actually already turned this question into an internal class.  I'm happy to get permission from management to share my fun version of this after I pull out the proprietary bits.  I'm still holding out for it having been done at some point before and I'm just too lazy to go that far into the search results.  :)

There are a number of things that I think it's important to know before going on-call:

How to triage an issue.  There should be a defined list of priorities with examples so that you can identify how serious a problem is.
What the priority levels mean.  Can you ack this and then go back to sleep?
Who to contact post-triage. The on-caller's primary responsibility is triage, not fixing.  Fixing the problem is good, too, but once you've triaged the issue, then it's up to you to rope in other folks from other teams and manage communications between them.

People jumping straight into fixing problems before triaging is the most common mistake I've seen (because, after all, we're used to fixing things and think of that as our job).  But communications are often vital, and doing things in the wrong order can mean customers are left in the dark because you got caught up in trying to fix a problem before letting your support people know there is a problem.


Reuse Alertmanger Slack Integration into Jiralert template What kind of changes need to be made to a slack template in order to reuse into Jiralert one? Can I utilize the whole body as it is, can somebody post an example of a working resuse, even for the default example in vendor site.

In the end solved it myself, I had to just define jira.summary and description respectively and copy/paste the contents from the slack integration in the given fields.

{{ define "jira.summary" }}]
    {{ .Alerts.Firing | len }} {{ .CommonLabels.alertname }} for {{ .CommonLabels.job }}
    {{- if gt (len .CommonLabels) (len .GroupLabels) -}}
    {{" "}}(
    {{- with .CommonLabels.Remove .GroupLabels.Names }}
        {{- range $index, $label := .SortedPairs -}}
        {{ if $index }}, {{ end }}
        {{- $label.Name }}="{{ $label.Value -}}"
        {{- end }}
    {{- end -}}
    )
    {{- end }}
{{- end }}
{{ define "jira.description" }}
    {{ with index .Alerts 0 -}}
    :chart_with_upwards_trend: *<{{ .GeneratorURL }}|Graph>*
    {{- if .Annotations.runbook }}   :notebook: *<{{ .Annotations.runbook }}|Runbook>*{{ end }}
    {{ end }}
    *Kubernetes Cluster:* `default`
    *Prometheus Alert Details*:
        {{ range .Alerts -}}
        *Prometheus Alert:* {{ .Annotations.title }}{{ if .Labels.severity }} - `{{ .Labels.severity }}`{{ end }}
        *Prometheus Alert Description:* {{ .Annotations.message }}
        *Prometheus Alert Details:*
            {{ range .Labels.SortedPairs }}  *{{ .Name }}:* `{{ .Value }}`
            {{ end }}
    {{ end }}
{{- end }}


error while loading shared libraries: libtinfo.so.6 : two behaviors for same docker image I am developing an Erlang-OTP application, for that I use a docker image.

docker-compose.yml:

FROM centos
LABEL Name=myimage Version=2.0.0

RUN set -e \
   && yum -y install \
      autoconf \
      gcc \
      gcc-c++ \
      git \
      glibc-devel \
      make \
      ncurses-devel \
      tar \
      curl

ENV SSL_VERSION="1.0.2p"

RUN set -e \
   && mkdir -p /tmp/openssl \
   && cd /tmp \
   && curl -fSL -o openssl.sha256   http://www.openssl.org/source/openssl-${SSL_VERSION}.tar.gz.sha256 \
   && curl -fSL -o openssl.tar.gz http://www.openssl.org/source/openssl-${SSL_VERSION}.tar.gz \
   && SHA=$(printf "%s  openssl.tar.gz" `cat openssl.sha256` | sha256sum -c -); if [ "openssl.tar.gz: OK" != "${SHA}" ]; then exit 3; fi \
   && tar -zxf openssl.tar.gz -C /tmp/openssl --strip-components=1 \
   && cd /tmp/openssl \
   && ./Configure \
         --prefix=/usr/local/ssl \
         --openssldir=/usr/local/ssl \
      linux-x86_64 \
      shared \
   && make -j4 \
   && make install \
   && rm -Rf /tmp/openssl*

ENV OTP_VERSION="20.3.6"
ENV WITH_NATIVE=true

RUN set -e \
   && mkdir -p /tmp/otp_src \
   && cd /tmp \
   && curl --tlsv1.2 -fSL -o otp_src.tar.gz https://github.com/erlang/otp/archive/OTP-20.3.6.tar.gz \
   && tar -zxf otp_src.tar.gz -C /tmp/otp_src --strip-components=1 \
   && cd /tmp/otp_src \
   && ./otp_build autoconf \
   && ./configure \
         --prefix=/usr/local/otp \
         --enable-threads \
         --enable-smp-support \
         --enable-kernel-poll \
         --disable-dynamic-ssl-lib \
         --with-ssl=/usr/local/ssl \
         `test "${WITH_NATIVE}" = "true" && echo "--enable-hipe --enable-native-libs"` \
   && make -j4 \
   && make install \
   && ln -s /usr/local/otp_${OTP_VERSION} /usr/local/otp \
   && rm -Rf /tmp/otp_src*

ENV PATH $PATH:/usr/local/otp/bin

RUN set -e \
   && yum -y install \
      which

ENV REBAR3_VERSION="3.6.1"

RUN set -xe \
    && REBAR3_DOWNLOAD_URL="https://github.com/erlang/rebar3/archive/${REBAR3_VERSION}.tar.gz" \
    && REBAR3_DOWNLOAD_SHA256="40b3c85440f3235c7b149578d0211bdf57d1c66390f888bb771704f8abc71033" \
    && mkdir -p /usr/src/rebar3-src \
    && curl --tlsv1.2 -fSL -o rebar3-src.tar.gz "$REBAR3_DOWNLOAD_URL" \
    && echo "$REBAR3_DOWNLOAD_SHA256 rebar3-src.tar.gz" | sha256sum -c - \
    && tar -xzf rebar3-src.tar.gz -C /usr/src/rebar3-src --strip-components=1 \
    && rm rebar3-src.tar.gz \
    && cd /usr/src/rebar3-src \
    && HOME=$PWD ./bootstrap \
    && install -v ./rebar3 /usr/local/bin/ \
    && rm -rf /usr/src/rebar3-src

RUN mkdir /myimage
WORKDIR /myimage

Commands to run docker-compose

sudo docker-compose build
sudo docker-compose run --rm myimage bash

When I try to run production tar created by Ubuntu-18.04, I get:

  error while loading shared libraries: libtinfo.so.6: cannot open shared object file: No such file or directory.

I dont get any error while compiling. When I create this tar from Debian-9, it runs without error.

Debian machine has locally installed erlang-otp, but Ubuntu doesnt have.

There is Docker's known issue #37531

To fix dependencies, you might try:

Either upgrade Docker to 17.02

curl -sSL https://get.docker.com/ | sudo bash

or install lib inside of container.
Since your container based on Centos (FROM centos), so try yum install

rpm -ql libXtst-devel-1.2.2-2.1.el6.x86_64


Vagrant box take long time to access shared /vagrant folder I have a vagrant box running on windows 10 hyper-v

config.vm.box = "kmm/ubuntu-xenial64" #A standard Ubuntu 16.04 LTS (x64).
config.vm.provider "hyperv"

Just after provisioning it, it seems to be running o.k.
but then if I exit and ssh back to it then trivial actions such as pressing tab after typing cd /vag would take tons of time (e.g 30 sec or more)
and after that a cd /vagrant worked o.k.
then ls -la worked o.k.
and another ls -la took something like 3 minutes or so.

During that time hyper-v is showing the virtual machine at 0% CPU.

Any suggestions on where I should look to try and figure out why is this happening?

It seems ther reason for this was that the SMB connections would disconnect.
using

net config server /autodisconnect:-1

in a "run as administrator" command window fixed it.

as explained here:
https://www.vagrantup.com/docs/synced-folders/smb.html#preventing-idle-disconnects


How to monitor a micro-service oriented solution? I run a micro-service oriented system, running under a private network, composed mainly of:

databases (clusters)
internal API's
external/public API's
workers (services)
monitoring servers

All are running Debian (7/8) and mainly using nginx and supervisor.

I'm currently monitoring logs using ELK stack. But I do want to have a monitoring/alerting system to include the following:

internal/public API's (calls)
databases status (only basic, to know when the cluster does not have issues)
ram/cpu for the servers

What do you recommend to use? I'm more open to open-source solution, but also can pay for services that are mature (not interested in startup solutions)

Prometheus is a popular tool for monitoring microservices deployed to the cloud in combination with Grafana. For a better understanding of your inter-service issues, a distributed tracing tool is needed as well.


The ansible_local var is not being populated when using the setup module to gather custom facts on windows The ansible_local var is not being populated when using the setup module to gather custom facts on windows.
Playbook:
---
- name: Testing and Debug PB
  hosts: all
  tasks:

    - name: Gather Facts
      setup:
        fact_path: 'c:\custom_facts'

    - debug:
        msg: "{{ hostvars[inventory_hostname] }}"

    - debug:
        msg: "{{ ansible_local }}"

Contents of "C:\custom_facts\foo.fact" on remote host:
[general]
asdf=1
bar=2

Playbook Output:
ansible-playbook -i inv/dev 301_pb_testing.yml -vv
ansible-playbook 2.9.12
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/home/alex/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python3.6/dist-packages/ansible
  executable location = /usr/local/bin/ansible-playbook
  python version = 3.6.9 (default, Jul 17 2020, 12:50:27) [GCC 8.4.0]
Using /etc/ansible/ansible.cfg as config file

PLAYBOOK: 301_pb_testing.yml ************************************************************************************************************************************************1 plays in 301_pb_testing.yml

PLAY [Testing and Debug PB] *************************************************************************************************************************************************
TASK [Gathering Facts] ******************************************************************************************************************************************************task path: /home/alex/git.sigma/SDG-Tech-DevOps/ansible/301_pb_testing.yml:2
ok: [demi-win00]
META: ran handlers

TASK [facts : Gather Facts] *************************************************************************************************************************************************task path: /home/alex/git.sigma/SDG-Tech-DevOps/ansible/roles/facts/tasks/main.yml:14
ok: [demi-win00]

TASK [debug] ****************************************************************************************************************************************************************task path: /home/alex/git.sigma/SDG-Tech-DevOps/ansible/301_pb_testing.yml:15

---------REMOVED SOME HOSTVAR OUTPUT FOR BREVITY AND PRIVACY--------

ok: [demi-win00] =>
  msg: |-
    ansible_architecture: 64-bit
    ansible_bios_date: 03/31/2014
    ansible_bios_version: rel-1.13.0-0-gf21b5a4aeb02-prebuilt.qemu.org
    ansible_check_mode: false
    ansible_connection: winrm
    ansible_date_time:
  ......
    ansible_facts:
   ......
    ansible_kernel: 10.0.17763.0
    ansible_lastboot: '2020-08-29 11:07:46Z'
    ansible_machine_id: S-1-5-21-49790847-4020553596-361474921
   ......

---------REMOVED SOME HOSTVAR OUTPUT FOR BREVITY AND PRIVACY--------

TASK [debug] ****************************************************************************************************************************************************************task path: /home/alex/git.sigma/SDG-Tech-DevOps/ansible/301_pb_testing.yml:18
fatal: [demi-win00]: FAILED! =>
  msg: |-
    The task includes an option with an undefined variable. The error was: 'ansible_local' is undefined

    The error appears to be in '/home/alex/git.sigma/SDG-Tech-DevOps/ansible/301_pb_testing.yml': line 18, column 7, but may
    be elsewhere in the file depending on the exact syntax problem.

    The offending line appears to be:

        - debug:
          ^ here

ansible 2.9.12
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/home/alex/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python3.6/dist-packages/ansible
  executable location = /usr/local/bin/ansible
  python version = 3.6.9 (default, Jul 17 2020, 12:50:27) [GCC 8.4.0]

The ansible Controller is running debian, The remote is Windows Server 2019
I would expect ansible_local to be populated once setup is run over a valid custom fact, but It doesn't. What am I doing wrong?

According to documentation:

NOTE - For windows clients, the results will be added to a variable
named after the local file (without extension suffix), rather than ansible_local.

According to your example it should be in ansible_foo, but take into consideration that the files in the path must be PowerShell scripts (with extension .ps1) so maybe foo.fact wasn't actually executed.
Hope it helps.


Jenkins vs Azure DevOps Pipelines I've been doing some research on the Azure DevOps solution that Microsoft recently unveiled.

Some background, I've been using Jenkins for years, and recently started exploring the containerisation rabbit hole (Docker, Kubernetes, etc),

I'm unsure about some of the aspects of Azure DevOps Pipelines, and it'd be awesome to get some feedback and clarity.

TL;DR

Is Azure DevOps now competing with Jenkins, where they previously supported the Jenkins community via open source plugins for Jenkins?
Is it possible for me to host Azure DevOps Pipelines completely on premise, not just agents?
Is it possible to use a self hosted git solution (like GitLab on-premise) with Azure DevOps Pipelines?

Longer version

Microsoft has developed various open-source plugins for Jenkins, such as the Azure App Service Jenkins plugin. There are many examples. To me it seems like the Azure DevOps (specifically Pipelines) solution is now competing directly with Jenkins, or am I missing something?
To me it seems that there is no way to host the Azure Pipelines solution on premise, or in a cloud provider of my choice. Yes, I can host my own agents where the bulk of the work is performed, but the actual execution of the pipeline logic happens on Microsoft's servers. And then I have to pay for parallel jobs. I'm comparing this to an on-premise (or cloud provider of choice) hosted Jenkins instance, where no payment is required, and no 3rd party is even aware that our pipelines are running. Am I missing something?
From what I can tell, Azure Pipelines has no support for self hosted repositories, I either put my code on GitHub or on Azure repos, nothing else. Thus, I can't even use this Pipelines solution if I have a privately hosted GitLab instance?

Thanks for any input.

I would love to get some clarity too maybe from a more authoritative source than myself (I'm sure I'm guilty of FUD too).

I've been looking to deploy to Azure using Kubernetes.  I am similarly confused but I think that is because there isn't one clear path.  I can't speak for Microsoft but their attitude to cloud seems to be they are happy to become "all things to all men".  If you are using Azure, Microsoft don't seem to have a strong opinion on how you get there.

Microsoft are actively trying to evolve; e.g.
Linux now dominates Azure

Clearly Azure DevOps deployment pipeline does exactly what Jenkins can do - so in that sense it is a competitor.  There is also that slightly off-putting upfront Azure DevOps pricing for parallel pipelines.

On the one hand, Azure DevOps appears to be a re-branding of Visual Studio Team Services.  So it will be familiar to people already within the traditional Microsoft development ecosystem.  At the same time Microsoft are making great efforts to embrace technologies that are not traditionally Microsoft (Go, Java, Node.js, Python, Ruby, PHP etc).

It is early days - but it is not a surprise that Azure DevOps pipelines should prefer the Microsoft Git properties of Azure Repos and GitHub.  In the same way that BitBucket pipelines prefers BitBucket and GitLab pipelines prefer GitLab.

Meanwhile;

The Azure Marketplace offers a Jenkins deployment that comes with Azure deployment plugins out of the box.

Many Jenkins plugins have been developed by the Azure DevOps team.

in addition;

Microsoft have put their backing behind Kubernetes related projects Helm and Draft.  Helm is a package manager for Kubernetes.  You can even use Helm to deploy a Jenkins installation into Kubernetes.

--

It would be interesting to see someone weigh up the benefits, costs and effort of running your own Jenkins in Azure verses using Azure DevOps pipelines.

... and then there is Jenkins-X.

So it would seem we have many options and decisions to make.


What Golang function could be used to enforce a minimum code coverage and return an Exit1 if false? Follow-up to this Q&A.

How to ensure that a certain minimal code coverage is reached in Golang? E.g. if the CI runs, the build should fail if the code coverage is less than X percent.

In Java, one could use Jacoco and define a minimum code coverage. Other language like Python support minimum code coverage as well.

One option is to simply create a bash script that parsers the output of go test -cover, but I would like to know if there is a off the shelf solution.

There aren't any off the shelf solution currently that allows you to specify a variable number of code coverage percentage.

There is the tool Courtney which enforces 100% code coverage. This tool also allows you to comment on blocks of code which do not need to be covered.


Docker compose to ECS, two services in one task I'm using this method to deploy my docker-compose project to AWS ECS.
A simplified version of my docker-compose.yml looks like this:
version: '3'

services:
  web:
    image: .dkr.ecr.eu-west-3.amazonaws.com/django:${IMAGE_TAG}
    build:
      context: .
      dockerfile: ./Dockerfile

  nginx:
    image: .dkr.ecr.eu-west-3.amazonaws.com/nginx:${IMAGE_TAG}
    build:
      context: .
      dockerfile: ./nginx.Dockerfile
    ports:
      - 80:80
    depends_on:
      - web

So, run docker compose up with my AWS ECS context and it works beautifully, cloud-formation creates all my resources and so on.
But, I can't figure out how to get these two services (web and nginx) to run in one task. Since they are in separate tasks at the moment it will create two instances of FARGATE or EC2 which is a bit overkill. I'd like two run the two containers on one instance but be able to scale up to more duplicate instances.
As stated in the link attached, I can modify the cloud-formation template to my liking, but something are limited to what I can modify. I don't think I can move that container service to another task since the image tag etc are created dynamically by docker compose.
My ECS cluster looks like this atm.

What you are looking for is a sidecar pattern where you run a container of your main application and a number of additional containers (one or more) in support of it. This is a well established pattern with orchestrators such as ECS but it's not possible to declare it with a docker compose syntax today. There is an open request talking about this. Can I ask you to +1 it and add your commentary/request there? It would help Docker to sense demand for this feature. Thanks!


How to define global variables in groovy file, which I want to load and consume? I have a Jenkins parent groovy file which consumes the child groovy file

Parent grooy file:

...
def child = load 'child-groovy-file';
child.execute();

Child groovy file:

a = "first letter";
b = "second letter;
def execute
{
echo "a is "+a;
...
}

  But this is not working. Unable to access variables, which are defined
  in Child Groovy file, when trying to access from Parent groovy file.
  When I create a node block and trying to consume directly the child
  groovy file, It is working.

Is the Environment variables only the solution to access globally?

try this one

static void main(String[] args) {
      // Initializing 2 variables
      def x = 5;
      def y = 10;

      //Performing addition of 2 operands
      println(x+y);
 }

}

check this URL


What is the difference between DevOps and GitOps? I am hearing nowadays a lot about GitOps. What is the difference between DevOps and GitOps?

Is it just the SCM does the build and deploy when there is a commit instead of the CI/CD tool exclusively?

GitOps is a technique of using Git to manage infrastructure provisioning and software deployments.  This technique uses many of the features of Git such as Pull Requests to manage and trigger deployments, and allows "diffing" (looking at the difference between commits) to view the differences between deployments.

DevOps is more of a culture or philosophy.  It is not tied to the specific tool Git.  Instead, DevOps is the joining of Development (Dev) and System Operations (Ops).  Often continuous integration and continuous deployment are big focuses of companies that invest in a DevOps culture.

A company that has a DevOps culture may choose to use a technique like GitOps to manage their infrastructure.  They may choose to use other tools though.

Note: DevOps is not a process. It is a culture. GitOps is a specific System Operations (ops) process tied to a specific tool (Git). A company that uses GitOps may not have a DevOps culture. Conversely, a company that has a DevOps culture may not use GitOps.


How to serve a custom index.html using the stock nginx docker image? How do I serve static content from the default nginx docker image?  Given:
root@mordor:~#
root@mordor:~# docker image ls nginx
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
nginx               latest              c39a868aad02        8 days ago          133MB
root@mordor:~#

where and how can a hello world be configured?
In the context of:
https://askubuntu.com/q/1292182/847449

The "official" nginx image has a specific path where it looks for files. You can either copy a file into the image by creating a new image from it or mount your files into this path as a volume.
To copy files, have a folder with your index.html and a Dockerfile that looks like this:
FROM nginx
COPY index.html /usr/share/nginx/html

Then build a new image with docker build -t my-nginx .
To start a container with your own files as a volume, you can run the container like this:
docker run -p 8080:80 -v ${PWD}:/usr/share/nginx/html nginx

All the files in your current folder (including the index.html) will now be available via http://localhost:8080/index.html.
Most of this information is covered in the documentation for the image at https://hub.docker.com/_/nginx


In jenkins, how to re-run old builds of a job? Some old builds are failed in my jenkins job and I need to re-run them again. How to do that?

If the job is a jenkins pipeline, open the failed build and select the Replay option. Otherwise select the Rebuild option.

Sample Replay Option:

Sample Rebuild Option:


org.jenkinsci.lib.envinject.EnvInjectException: java.io.IOException: remote file operation failed I've the following error in Jenkins console when running job which uses Windows slave:

Started by user
  [EnvInject] - Loading node environment variables.
 ERROR: SEVERE ERROR occurs
 org.jenkinsci.lib.envinject.EnvInjectException: java.io.IOException: remote file operation failed: C:\Program Files\Java\jre1.8.0_91 at hudson.remoting.Channel@2f4af34f:Management VPC Slave: Windows (sir-rvhi7z8q): java.io.IOException: Unable to delete 'C:\Program Files\Java\jre1.8.0_91\bin\awt.dll'. Tried 3 times (of a maximum of 3) waiting 0.1 sec between attempts.
 at org.jenkinsci.plugins.envinject.util.RunHelper.getBuildVariables(RunHelper.java:137)
 at org.jenkinsci.plugins.envinject.EnvInjectListener.setUpEnvironmentWithoutJobPropertyObject(EnvInjectListener.java:236)
 at org.jenkinsci.plugins.envinject.EnvInjectListener.setUpEnvironment(EnvInjectListener.java:50)
 at hudson.model.AbstractBuild$AbstractBuildExecution.createLauncher(AbstractBuild.java:534)
 at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:454)
 at hudson.model.Run.execute(Run.java:1737)
...
 Caused by: java.io.IOException: remote file operation failed: C:\Program Files\Java\jre1.8.0_91 at hudson.remoting.Channel@2f4af34f:Management VPC Slave: Windows (sir-rvhi7z8q): java.io.IOException: Unable to delete 'C:\Program Files\Java\jre1.8.0_91\bin\awt.dll'. Tried 3 times (of a maximum of 3) waiting 0.1 sec between attempts.
 at hudson.FilePath.act(FilePath.java:994)
 at hudson.FilePath.act(FilePath.java:976)
...
 Caused by: java.io.IOException: Unable to delete 'C:\Program Files\Java\jre1.8.0_91\bin\awt.dll'. Tried 3 times (of a maximum of 3) waiting 0.1 sec between attempts.
 at hudson.Util.deleteFile(Util.java:250)
 at hudson.FilePath.deleteRecursive(FilePath.java:1211)
...
Caused by: java.nio.file.AccessDeniedException: C:\Program Files\Java\jre1.8.0_91\bin\awt.dll
 at sun.nio.fs.WindowsException.translateToIOException(Unknown Source)

which stops the job at the very start.

I've no idea why EnvInject plugin would want to remove some Java files (such as awt.dll).

Any idea how to solve this problem?

The EnvInject plugin is basically attempting to load the build environment which uses configuration of JDK tools in Jenkins which fails during reinstallation of Java JDK library on the Windows slave (as per given configuration).

This JDK configuration can be found in Global Tool Configuration (at /configureTools) for the given JDK profile which was selected in that job configuration. The installation process failed to reinstall Java JDK due to running Java process in the background (such as java -jar slave.jar) which was using these DLL files.
So it sounds like it's not possible to reinstall the same Java JDK files which are used to run the Jenkins slave process.

The solution was to disable Install automatically option in Global Tool Configuration, as Java JDK was already installed on that slave.

Related: JENKINS-46989 - EnvInjectException: remote file operation failed.


What tool or type of tool am I looking for? We have an operations team that looks after our running tenants. They respond to requests and incidents from users by following specific processes, they might for instance restart a container on an instance or undelete a resource a user deleted by accident.
Because these are repeatable actions they should be automated by rolling them into scripts. We want the jobs to be executed centrally so we can be able to track who ran which job and ideally have a central dashboard.
I need to investigate a platform to provide to the Operations team to get this job done.
Being from a dev background Jenkins would be my obvious choice but I feel like this must be a solved problem in the devops/ops world.
I've tried to research this but part of the problem is I don't know the name of the class of tool I'm looking for. When I look for automation platforms I find CI systems but they don't really fit with my problem.
Which tool or type of tool is appropriate to my needs?

Jenkins can work just fine for this purpose.  In my view, Jenkins is a general-purpose automation tool, not necessarily a CI/CD tool specifically.
I'm on an Ops team and we use both Jenkins and Rundeck.  We generally prefer Jenkins because we find it difficult to navigate the Rundeck UI and find things like job output, but on the other hand it's often easier to write jobs for Rundeck, because you don't need to know Groovy/Pipeline.  Here are some examples of what we use each for:

Install package updates on managed systems (yum/dnf packages) - Rundeck
Deploy & provision new VMs - Jenkins
Run Chef on managed systems - Rundeck
Clean up stale Computer objects in Active Directory - Rundeck
Clean up stale subscriptions in Red Hat Satellite - Rundeck
Create & configure new repositories in Bitbucket from templates - Jenkins
Build new vagrant images - Jenkins
Backup & upgrade some services that my team manages (AWX, Red Hat Satellite, etc.) - Jenkins or Rundeck depending on the service

I'm not claiming that either of these tools is the best choice for any of these particular examples, just that my team uses them for these purposes and they generally work well.  Historically, which tool we use for each job generally has come down to personal preference of the person who created the job.
In terms of the name of this type of tool, we just call them "automation tools" or "automation platforms".  I'm not sure if there's a more "industry-standard" term for this type of tool.


Why won't InSpec admit that port 5432 is listening inside a PostgreSQL container? Background

I'm trying to test a PostgreSQL container with InSpec 2.1.10, but I cannot get a simple port check to return true, even though Docker, netstat, and psql all report that the port is properly mapped. While it's possible this is a bug in InSpec, it seems more likely that the problem exists between the keyboard and the chair. What have I missed?

Repeatable Tests

My Spec

# port_test.rb
describe port(5432) do
  it { should be_listening }
end

My Testing Commands

container=$(docker run -d -P --rm postgres:alpine)
inspec exec port_test.rb -t docker://$container

My Output

Profile: tests from test.rb (tests from test.rb)
Version: (not specified)
Target:  docker://8caceaa7e2eefa4a2bf50a58ea1ddf9c927c08d3397e3d9ad5b60d3519971380

  Port 5432
     ×  should be listening
     expected `Port 5432.listening?` to return true, got false

Test Summary: 0 successful, 1 failure, 0 skipped

But It Should Work!

Everything but InSpec reports that things are working as expected.

$ docker port $container
5432/tcp -> 0.0.0.0:32770

# from inside container using netstat
$ docker exec $container netstat -lntp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:5432            0.0.0.0:*               LISTEN      -
tcp        0      0 :::5432                 :::*                    LISTEN      -

# on macOS Docker host; flags are the macOS equivalent to `-lntp`
$ netstat -anv | fgrep $(docker port $container | cut -d: -f2)
tcp6       0      0  ::1.32770              *.*                    LISTEN      131072 131072    709      0
tcp4       0      0  *.32770                *.*                    LISTEN      131072 131072    709      0

# connecting to database in container from host
$ psql -V \
       -h localhost \
       -p $(docker port $container | cut -d: -f2)
psql (PostgreSQL) 10.3

You did everything right, it is the correct InSpec test and use with Docker!

You encountered a bug in InSpec that is related to InSpec's netstat output parsing combination with Alpine's netstat output.

See InSpec's output for postgres container instead of postgres:alpine:

$ inspec exec docker_test.rb -t docker://1c8162517c22                                                               1d15h master[a5ff79d3]

Profile: tests from docker_test.rb (tests from docker_test.rb)
Version: (not specified)
Target:  docker://1c8162517c229ff8c87ab3e8c909f46b4370d7c63a65c0f973a5c10fdebda800

  Port 5432
     ✔  should be listening

Test Summary: 1 successful, 0 failures, 0 skipped

In your case, the problem is that Apline's netstat reports a different output then other Linux eg. Ubuntu, CentOS:

# netstat -tulpen
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       User       Inode      PID/Program name
tcp        0      0 0.0.0.0:5432            0.0.0.0:*               LISTEN      999        108791     -
tcp6       0      0 :::5432                 :::*                    LISTEN      999        108792     -

On Alpine, the inode output is missing:

netstat -tulpen
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:5432            0.0.0.0:*               LISTEN      -
tcp        0      0 :::5432                 :::*                    LISTEN      -

That is why InSpec is not able to parse the netstat output. It is clearly a bug in InSpec. As a mitigation, I recommend to install the iproute2 package for alpine containers via apk add iproute2. InSpec prefers ss over netstat. Please report this bug to InSpec team.


Can the Angular CLI be used to build and deploy on a remote server via ssh? I'd like to use the Angular CLI to build the app and deploy it on my remote server via SSH. Is this feature available?

You need to use continuous deployment tools.
Not available yet.


Auto-managed docker-compose volume not shared among containers I have a bit of hard time understanding the way docker-compose handles auto-managed volumes. Here is an example docker-compose.yml

version: '3.7'
services:
  one:
    image: alpine
    command: sh -c 'echo one >> /data/db.txt && cat /data/db.txt | wc -l'
    volumes:
      - /data

And here is what I get (docker-compose logs stripped out)

$ docker-compose up
1 # as expected

$ docker-compose start
$ docker-compose up
3 # as expected

$ docker-compose run one
1 # why ? I expected 4

So it seems the volume is not shared between containers of the same service (in this example, the service one). Actually a new volume seems to be created every time run is used

test $(docker volume ls | wc -l) -eq 0 # ensure clean env

repeat 10 docker-compose up
test $(docker volume ls | wc -l) -eq 1 # works

docker-compose down && docker volume prune -f # cleanup

repeat 10 docker-compose run one
test $(docker volume ls | wc -l) -eq 1 # fails, result is 10 instead

The docker-compose documentation does not really help, the only explanation seems to be

  Just specify a path and let the Engine create a volume

  volumes:
  - /var/lib/mysql

I do not encounter this issue when using named volumes (e.g. data:/data). So why is a new volume created for new containers of the same service ? And what is the point of such behavior to be the default ?

N.B. I know I should use explicitly defined volumes instead, but I want to understand the default behavior, and why it is this way

   docker and docker-compose version info https://pastebin.com/8TuFCg2i

Define the volume in the following way

volumes:
  - ${local_path}:/data

or using docker volumes

version: '3.7'

volumes:
  db-data:

services:
  one:
    image: alpine
    command: sh -c 'echo one >> /data/db.txt && cat /data/db.txt | wc -l'
    volumes:
      - db-data:/data


Is DevOps compatible with ITIL? In my career, I have been both a software developer and ITIL practitioner in an operations role. Thus DevOps was a natural progression for me.
However, I have always struggled with the highly specialised language that ITIL introduces and making that "Developer Friendly" enough to not be a complete turn off to developers.

ITIL is an internationally recognised IT Service Management framework that has been developed over 30 years as a set of practices that have a proven benefit to the operational stability and maturity of an organization.

Is DevOps truly compatible with ITIL, or in essence do we need to take the spirit of ITIL and "translate" it to language that is better understood by development teams:

Incident & Problem Management → Production Defects, Bugs or Issues
Change & Release Management → Continuous Delivery
Event Management → Logging, Telemetry, Instrumentation and Alerting

In my opinion, the DevOps culture come along with a methodology change toward Agile process management.
ITIL is heavily aimed at a clear formalism of the process and the results and thus more adapted to a Waterfall model.

This doesn't mean ITIL is incompatible with Devops, but usually this will be two separate process with different timelines.
I mean that the inclusion of a new product within ITIL referential will usually be delayed until the product/application has been released in production for a while, where early pitfalls and some documentation needed to integrate ITIL has been done and adapted after the product is "live".

One of the thing in ITIL is the Service Design, which is assumed to be defined before any development task, an agile process will/may review the design in each iteration, breaking the formalism needed in an ITIL process.

The main goal of ITIL is, as you said, to provide a framework to ensure nothing is omitted between the design/conception and maintenance phase (Build/Run).  In a devops culture, the whole team is responsible of all phases on long term, hence why the formalism is reducted.

That doesn't mean we have to forget ITIL, the core principles are absolutely good and, in my opinion, should be used as a checklist to build the initial backlog of a product.
It's just that following the ITIL principle with all its formalism goes against the time to market reduction goal of a quick iterative software development and sometimes is not even applicable as there's less transmission of information needed between teams, as the tasks are done by the same team.


How to avoid output being truncated when running AWS SSM command? I've got the following AWS CLI commands to run the PowerShell's Invoke-WebRequest command:

instanceId=i-0xyz # Change this.
cmdId=$(aws ssm send-command --instance-ids "$instanceId" --document-name "AWS-RunPowerShellScript" --query "Command.CommandId" --output text --parameters commands='"While ($i -le 10) {Invoke-WebRequest -UseBasicParsing -Uri http://example.com; $i += 1}"')
sleep 1
aws ssm list-command-invocations --command-id "$cmdId" --details --query "CommandInvocations[*].CommandPlugins[*].Output[]" --output text

However, the output is truncated:

...
StatusCode        : 200
StatusDescription : OK
Content           : <!doctype html>
                    <html>
                    <head>
                        <title>Example Domain</title>

                        <meta charset="utf-8" />
                        <meta http-equiv="Content-type" content="text/html;
                    charset=utf-8" />
                        <meta name="viewport" conten...
------------------Output truncated-------------------

How I can run aws ssm list-command-invocations command, so the output won't be truncated?

According to this ssm tutorial, SSM send-command output is truncated at 2500 characters.  To work around that, you must send your output to an S3 bucket using the --output-s3-bucket-name parameter.


How comes a GitLab CI/CD job cant' find its artifacts? In the Open Source ventilator project, we aim to provide those who focus on testing with a ready-to-go flash file for the Arduino controller board.

While the binaries seem to be there, for some reason the CI job can't pickup the artifacts.

What is happening:

a CI/CD pipeline creates two files which can be listed using find
in the next step, this pipeline reports it can't find these artifacts. It says, "no matching files/no files to upload".

What's the error and how to fix/troubleshoot? Why does this job end up as successful?

CI job definition:

compile:
  stage: build
  tags:
    - docker
  script:
    -  |

        git clone -b ci-gh https://github.com/mattd3v/Open-Source-Ventilator.git
        cd Open-Source-Ventilator/OpenSourceVentilator/
        arduino-cli compile --warnings all --fqbn arduino:avr:uno -o osv-controller-uno.bin OpenSourceVentilator.ino
        find .
  artifacts:
    paths:
    - ./osv-controller-uno.bin.hex
    - ./osv-controller-uno.bin.elf

Pipeline output:

.
./test
./test/RMVS_TestCaseData.xlsx
./OpenSourceVentilator.ino
./osv-controller-uno.bin.hex
./osv-controller-uno.bin.elf
./README.md

section_end:1586713988:build_script
[0Ksection_start:1586713988:after_script
[0;msection_end:1586713990:after_script
[0Ksection_start:1586713990:archive_cache
[0K[0K[36;1m Saving cache[0;m
[0;msection_end:1586713991:archive_cache
[0K section_start:1586713991:upload_artifacts_on_success
[0K [0K [36;1m

Uploading artifacts for successful job[0;m
[0;m[32;1m
Uploading artifacts...[0;m

[0;33m WARNING: ./osv-controller-uno.bin.hex: no matching files[0;m
[0;33m WARNING: ./osv-controller-uno.bin.elf: no matching files[0;m

[31;1mERROR: No files to upload                         [0;m
section_end:1586713993:upload_artifacts_on_success
[0K[32;1mJob succeeded
[0;m

As Casey Vega commented: provide the complete path.


Dealing with ambiguity of the terms publish vs. deploy DevOps terminology sometimes originates in products as introduced by their authors.

As a result, there are some standard actions described by ambigous terms. In mixed teams and large heterogeneous environments we need to agree on terms to undertstand what we actually do.

publish: term specific to Ant/Ivy meaning putting an artefact into binary repository. Also, some people would say "publish content updates to website".
deploy: while "in the Maven world" this word describes putting an artefact to binary repository afaik, more "opsy" people deploy their servers also without having Maven in place.

How to deal with this ambiguity (ideally, please refer to "success stories")? An explorative talk on terms meaning is not always welcome in delivery oriented stressful environments; maybe there are already better terms around?

For terms definitions, please refer to sources.

The terms are actually quite well defined and some authors are simply making a mess of things by using a term like 'publish' for deployment, in cases where publishing a website for example constitutes for their own process also a deployment, as there is no extra deployment step after publishing. Sometimes they use word publishing both for uploading and publishing together as all of their artifacts are public. That creates unnecessary confusion. Just to make it clear, here are the definitions:

Environment Definitions:

  build environment - Environment setup for build process.
  runtime environment - Environment setup for execution process (test/staging/production).

Artifact Lifecycle Stages:

  building - process of creating an artifact in build environment
  uploading - moving artifact from build environment to repository.
  publishing - making artifact in repository public.
  downloading - moving artifact from repository to runtime environment
  installing - applying artifact to runtime environment
  deploying - using artifact in runtime environment
  delivery - making artifact (effects) available to customer

Note: People often use any of those terms to mean one or more of the previous terms together.