What about installing an Antivirus on Windows K8s nodes? We're building an on-premise Kubernetes cluster with Windows nodes. As a company policy, all Windows Servers needs an antivirus (symantec endpoint security).

The antivirus would be installed on the nodes, and not as a pod.

What do you think about installing an antivirus on Kubernetes nodes? Considering the great chance of false-positives when an antivirus has to deal with Docker.

I don't have much experience with Symantec (I don't really like the product).
But searching on Google apparently has some steps to set up, as they don't natively support Kuberentes as an application.

Look if these two articles can help you.

https://support.symantec.com/us/en/article.tech246815.html

https://www.symantec.com/content/dam/symantec/docs/solution-briefs/securing-containers-think-your-containers-are-secure-en.pdf


What is the difference between Sysadmin and DevOps Engineer? When applying for a job, usually you can find two types of similar jobs: Sysadmin Engineer and DevOps Engineer.

Both of them deal with server configuration and ensure the reliable operation of computer systems. It can be hard to tell the difference between the two. What are the main differences between them?

Mainly DevOps is not a role (when used as such it's more a buzzword than a real role).

DevOps is roughly an organization pattern aiming at breaking the silo between developers and sysadmins.
The main goal is to build teams with devs and sysadmins (along with testers usually) responsible for a product (application) from its definition, architecture decisions up to the maintenance in run of this product.
Each member of the team will be part of the decision on the whole life-cycle of the product, a dev will do some sysadmin tasks in production, and a sysadmin will participate  in the design phase of the product to avoid caveats from the infrastructure perspective for example.

At ideal, a sysadmin would also be part of the development team for the product, in real world sysadmin code more on the configuration around the product and monitoring solutions, but being able to voice concerns to other members of the team avoid a lot of misunderstanding on the deployment process.


How to manage Docker images and their child images in a pipeline? I have various Jenkins agents images and I am trying to add automation to it.

I have a base java image called agent-base and those who inherit from that one e.g. agent-ansible, agent-terraform and so on. In some case, I even have a 3rd level.

My questions:

What would be the best way to update the child images when a parent is updated? Downstream pipelines perhaps?
Or should I keep all the Dockerfiles in a single repository and have a script that know the images hierarchy?
More importantly, how should I tag (version) my child images? Should it contain my parents version too? In the tag itself or as labels.

There is a good practice called multistage build, in which you have one Dockerfile containing various image definitions that depend from one another, this helps you keep all your images updated with the latest changes and copy artifacts from one to another; this also helps reducing the image size if done right.
A simple example is like the following:

FROM golang:1.9.2 as builder
ENV CGO_ENABLED=0 GOOS=linux
RUN go build -a \
             -installsuffix cgo \
             -o app .

FROM alpine:3.6
WORKDIR /root/
COPY --from=builder app .
CMD ["./app"]

In this example, in the first image you compile a go example application using the golang official image and then, using alpine to keep the size of the image small, you copy the executable to a new image and execute it.
This way you can keep your containers in one file and build all at the same time or you can build just a target stage:

docker build --target builder -t somehub/sometag:latest .

You can even use remote images as stages:

COPY --from=nginx:latest /etc/nginx/nginx.conf /nginx.conf

If you need anymore information regarding Docker multi-stage builds you can find it in the following link.


Fully enabled End to End CI/CD Solution I've been working in the field of DevOps for some time now and have learned about some tools like Jenkins, Docker, Kubernetes, Puppet, Artifactory, etc.. Although I know where each of these tools could be used and what purpose each of these serves, I really wanted to see how a fully implemented CI/CD looks like.

I understand that the solution would definitely change with the change in the requirements, technology stack, team size, etc. and that there is no God solution for everything but, I haven't seen a fully implemented DevOps lifecycle ever.

I'm not sure if asking for resources is prohibited here (Feel free to close the question if it is) but could you please redirect me to some article, blog, white paper or suggest me a book where I could learn and get a feel of what fully implemented CI/CD looks like.

could you please redirect me to some article, blog, white paper or suggest me a book where I could learn and get a feel of what fully implemented CI/CD looks like.

Sure, look how Google does it.

  I haven't seen a fully implemented DevOps lifecycle ever.

There seems to be a misunderstanding here. "DevOps" is a lot of things; but a "full" CI/CD (meaning each individual developer commit turning up in production fully automated shortly after the commit - continuous deployment (not only delivery)) is by far not the norm. In my experience with a few larger local and national customers, where production or business critical things are controlled by the software, and there usually are large amounts of critical data involved, it always stops before the production stage.

Full CI/CD is complicated. Yes, you can cobble the tooling together pretty quickly, but the prevention, detecting and backing out of unwanted commits is very hard. If you look at the above article about Google you will see that huge effort went into getting this to run (I seem to remember only their core DevOps team is about 100 persons, doing nothing than tooling; and then they invest a lot of work into peer reviews, knowledge transfer, etc. etc.). Whole ecosystems of components, procedures and mindsets were created specifically for this. The fact that we mere mortals get to play with things like Kubernetes is a result from that, their breadcrumbs if you will.

Even only the one fact that they are doing Trunk-based development would be too complicated (regarding processes, necessary quality requirements etc.) for many more normal, smaller teams.


Is this okay to use double proxy-server setup (on host and in container) on one machine? One nginx server on host machine with renewable certificate (via certbot).
And another nginx server inside docker container.
First one should redirect all traffic to the second one.

Currently I have one nginx service inside container and I connect certificates that stored on host to that server via volumes. But with this kind of setup I can't renew certificates automatically...

Sure, you put in as many proxies as you need. You don't want to over-complicate things, but it seems you have a good reason for your architecture.


Loop chmod with Ansible I'd like to remove the execute bit from some ubuntu motd (Message of the day) files with ansible to disable some ad-like Ubuntu news.
The module should only act if files exist and permissions are wrong.

If there are no files the tasks are running fine, but as soon as a file exists, I'm getting this:

Error Message:

failed: [ip] (item={'_ansible_parsed': True, u'stat': {u'isuid': False, u'uid': 0, u'exists': True, u'attr_flags': [... redacted the rest of the file attributes from stat ...]

Mayboe someone can help?

Here the files of my role:

defaults/main.yml

---
motd_path: '/etc/update-motd.d'
unwanted_motd_files:
  - 10-help-text
  - 50-motd-news

tasks/main.yml

---
- name: Register unwanted motd files
  stat:
    path: "{{motd_path}}/{{item}}"
  register: filecheck
  with_items: "{{ unwanted_motd_files | default([]) }}"

- name: Check if unwanted motd files are executable and remove the executable bit
  file:
    path: "{{motd_path}}/{{ item }}"
    state: touch
    mode: u-x,g-x,o-x
  with_items: "{{ filecheck.results }}"
  when: item.stat.exists == true

Ok, solved it :)

Here the task for everyone who intends to do the same.

Only the skip/changed console output could be replaced by a nicer message.

- name: Register unwanted motd files
  stat:
    path: "{{motd_path}}/{{item}}"
  register: filecheck
  with_items: "{{ unwanted_motd_files | default([]) }}"

- name: Check if unwanted motd files are executable and remove the executable bit
  file:
    path: "{{motd_path}}/{{item.item}}"
    state: touch
    mode: u-x,g-x,o-x
  with_items: "{{ filecheck.results }}"
  when: item.stat.exists == true and item.stat.executable == True


Deploying lambda conceiving sensitive variables using serverless I want to deploy AWS lambda written in NODEJS. And I have sensitive variables that can't be disclosed in AWS lambda variables section. Is there a way to deploy lambda with an encrypted variable which no one can see and can't be decrypted from the machine which was used to deploy the lambda. Only some other user who has the key should be able to decrypt the variable.

As far as I'm aware. There is no way to encrypt the environment variable value in the variables section of the lambda function. And to be fair, even if there was, it wouldn't provide any benefit in terms of keeping the variable secret. If the variable could be hidden/encrypted in the variables sections to be decrypted later, a user could simply execute the lambda function and print the results of the decrypted variable to stdout.

One approach could be to use aws secrets manager to store and retrieve your secrets. You can then assign your lambda function a role to allow access to the secret store, and use AWS SDK to retrieve your value.


Is there any Jenkins plugin for pylint results? I have Pipeline which run Jenkinsfile. one of the stage is pylint:
python -m pylint --rcfile=.pylintrc --exit-zero ${MODULE} > ${WORKSPACE}/testreport/pylint.log

I know there was used to be violations plugin which is now deprecated. is there any alternative?

Jenkins accepts JUnit-formatted test results with the JUnit plugin, so you could try pylint-junit to output JUnit-formatted results (I have not tried this myself).
Also it looks like a very similar question was asked and answered over on StackOverflow.


Is "running several Linux distributions on the same host" a good justification for container adoption? I need to justify container adoption.

I have several solid arguments, but one is making me dubious. Is it correct to say software can "run better" or "worst" depending on the Linux distribution? Here is what I heard:

  some software run better on RedHat-based distributions, other better on Debian-based distributions. With containers, you can run each software on its preferred environment regardless of the operating system installed on the host.

Is it true? And if it is, do you have an example of such software?

That is not a good argument and it shows a basic misunderstanding of what containers are and how they should be used. Containers are NOT virtual machines.

A container runs a (short lived) process. Think executable, like ls, or top, or a shell or Python script. When the process ends, the container is destroyed. The process is isolated from other processes running on the same machine. Basically the kernel reserves some space (memory/CPU) for that process, runs the executable inside that isolated space, then releases the reservation. The reserved space is the container.

A virtual machine on the other hand is an emulator. It pretends there is an entire computer and it needs an operating system to, well, operate it. It boots a kernel, starts an init system and then a whole bunch of processes to deal with every aspect of the emulated hardware.

A container is supposed to be small and fast. And it's not generally supposed to run for days, or even hours. Compared to a virtual machine, which gives the user more freedom, but also takes more resources. Packing an entire distribution like Debian or RedHat inside a container would defeat the purpose.

For example, you're not supposed to run a web server (say, Nginx + PHP) inside a container. When a page is requested, you run the PHP script inside a container to generate the page, hand the output over to Nginx running outside, then the container is destroyed. If you want Nginx+PHP in the same space, you're better off with a virtual machine. There may be reasons to set it up as a container, but technically speaking Nginx workers already do the work of containers, so you'd just be duplicating that.


How to apply using Terraform to launch multiple EC2 Resource with different configs (VPC not maintained by TF) We are planning to use Terraform for provisioning EC2 instance which will host Oracle DB. While we are able to achieve this, there are some questions for which I am searching for suggestions / feedback's.

Below is the directory structure we are planning:

|aws
  |Environments
    |STAGING
        |Modules
           |Compute
           |Networking

    |PRODUCTION
        |Modules
           |Compute
           |Networking

Questions:

We are able to create EC2 instances using existing VPC. As VPC is not something that will be owned by my team. Is it enough to keep VPC ID, SUBNET ID, etc as variables in EC2 resource or is there a best practice to keep them in modules for existing VPC ? Overall, our tf codes are not going to manage the VPC's, just use what is being giving to us, but want to make it dynamic so that changes are simple, yet DRY
We are planning for Continuous Development, meaning, developers / requester will have the ability to request for a new ec2 instance with certain version of database. Right now, we are installing db with bootstrap scripts. Though this works as expected. Increasing count for more than 1 instance works, but what if the configuration slightly changes.

  For example:

  1) If I need more  capacity on root volume for the new instance,    changing the existing resource shows a plan that will
  destroy    existing resource and recreate. How can we achieve this
  ? Is    duplicating the aws_instance resource, the only way when
  the    configuration changes for new EC2 instance?

  2) If new instances launch as t2.large(Instance type passed as variable) while old ones run on t2.micro, will it recreate old
  instances too if I just increase the count

Also, is there a way to automate this process i.e. append something like a template for new EC2 instance to existing .tf file.

What do we have now

Main.tf

provider "aws" {
  access_key = "${var.access_key}"
  secret_key = "${var.secret_key}"
  region     = "${var.region}"
}

resource "aws_instance" "oracle_ec2" {
  ami           = "${lookup(var.ami, var.region)}"
  instance_type = "${var.instance_type}"
  key_name      = "${var.key_name}"
  subnet_id     = "${var.subnet_id}"

  vpc_security_group_ids = ["${aws_security_group.oracle_sg_new.id}"]

  root_block_device {
    volume_type           = "gp2"
    volume_size           = 10
    delete_on_termination = true
  }

  ebs_block_device {
    device_name           = "/dev/xvdb"
    volume_type           = "gp2"
    volume_size           = 200
    delete_on_termination = true
  }

  count = "${var.instance_count}"

  tags {
    Name = "Ora"
  }

}

You can use a remote backend as a data source. That is working well for us thus far but this project is not very mature and we will likely refactor a time or ten.

data "terraform_remote_state" "network" {
  backend = "s3"
  config {
    // some variables related to config would likely go here
  }
}

then for the teams not managing the network to consume that data you could create an instance like (special attention to the subnet_id here):

resource "aws_instance" "oracle_ec2" {
  ami           = "${lookup(var.ami, var.region)}"
  instance_type = "${var.instance_type}"
  key_name      = "${var.key_name}"
  subnet_id     = "${data.terraform_remote_state.network.subnet_id}"
}

The network team should be able to manage the networking infrastructure and you should be able to consume those values based on current state.

Terraform Remote State Data Source

  2) If new instances launch as t2.large(Instance type passed as variable) while old ones run on t2.micro

If you change the instance type from t2.micro to t2.large then yes the t2.micro instances will be destroyed and replaced with the t2.large. The name of the game is immutability outside of meta data changes like altering tags. Look into lifecycle hooks to alleviate some of this pain

lifecycle {
  create_before_destroy = true
}

  will it recreate old instances too if I just increase the count

No, current instances would not be destroyed. Terraform will check the state to see how many instances you currently have. If more instances are necessary, terraform will know how many instances to create and create the instances necessary

  Also, is there a way to automate this process i.e. append something like a template for new EC2 instance to existing .tf file.

Not sure what exactly you mean by this one. What I will say is that having the data coupled to the instance itself makes instance immutability and using terraform more difficult. A (untested and unproven) thought and potential solution in my head is to:

//create an ebs volume
resource "aws_ebs_volume" "oracle-data" {
  ...
}

//attach that volume to the instance
resource "aws_volume_attachment" "oracle-data-attachment" {
  ...
}

with some bash involved for an init script that creates and mounts the new volume to be used in conjunction with the aws_volume_attachment resource. That could potentially create a new instance preserving the EBS volume then reattaching it to the new instance.


Run command from Jenkins to Linux Server I have a HAProxy server setup in Azure Linux Machine. How can I run a HAProxy start/stop command from Jenkins to multiple Linux server?

If you have ssh access from the Jenkins machine to the HAProxy machine, you could run an Ansible task:

ansible -i ha-inventory.ini ha-servers -m service -a name=haproxy state=restarted

Of course you could also have Jenkins do a simple ssh :

for server in ${server_list} ; do ssh server 'sudo service haproxy restart' ; done

For both of these, you would need

ssh keys set up between Jenkins and the HAProxy servers
sudo rights for restarting the service (at least) for the connecting user (probably jenkins)


Confluent Kafka on K8s - offset is stuck, lag increases, what can be done? We are using Confluent Kafka cluster on production K8s.
It looks like our consumers are stuck in a loop as offset is static and lag keeps going up.
When I restart zookeper/brokers (persistent volume) the offset moves by a small amount and then it is stuck again. I have checked my consumers and only 1 of them has assigned something, that is processed and all assignments finish after. What happens on consumer side is that sometimes we have an error, we tried ignoring the message but Kafka is still affected. What can be done outside of the default config. We are not sure why message are stopped being consumed or it looks like there is some loop state leading to attempting to process same message over and over agian.

Things we have checked were what happens with the background io queue and the buffer. It turned out our producers were ok as lag was increasing, but as offset was static on check of consumer handlers it turned out we don't send acknowledgement to Kafka on read, leading to Kafka sending the same message after timeout on each attempt to consume. On java code side we have just made sure to send ack.


Failed to start Container after reboot I am hosting an own registry.
After rebooting the Server my registry container is unable to start.

I used this command to start the registry

docker run -d -p 5000:5000 --name registry -v /var/lib/registry/:/var/lib/registry -v /root/certs:/certs -v /root/auth/:/auth -eEGISTRY_HTTP_TLS_CERTIFICATE=/certs/domain.crt -e REGISTRY_HTTP_ADDR=0.0.0.0:5000 -e REGISTRY_HTTP_TLS_KEY=/certs/domain.key -e "REGISTRY_AUTH=htpasswd" -e "REGISTRY_AUTH=htpasswd" -e REGISTRY_AUTH_HTPASSWD_PATH=/auth/htpasswd -e "REGISTRY_AUTH_HTPASSWD_REALM=Registry Realm" --restart always registry:2.7.1

After a reboot i get this message when i try "docker start registry":

  Error response from daemon: OCI runtime create failed: container with id exists: dfb0bef21bdfc8a89b59498befd37f83513e75527c0beb552e0400df2a2b7c7d: unknown
  Error: failed to start containers: registry

Starting a new container works fine.
How can a fix, it and sould the container not start by itself because of "--restart alway"

docker --version

Docker version 18.09.6, build 481bc77

Thx for your help.

Update

Intesting news I have written an init script to to the job,
but the problem is exactly the same.  The container exists but isn't started.
If I try to start it, I get the error message from above.

On the boot screen in get this information.

So the docker daemon seam to be not ready.
Do you have any suggestions why?

Try to check is registry really stopped using docker ps -a if container runned stop container then start.

Part related to the --restart always option, restart always will proceed only if container works at least 10 seconds and docker itself can monitor this service.

See more info at this link restart policy detail


How to introduce(join) new worker node to an existing k8s cluster with selective scheduling only Given a working Kubernetes cluster consisting of a master and some workers.

We need to add a node to run a very specific pod and be part of the cluster for networking reasons. Then being largely ignored for later pod creation by the master.

Adding selectors to every deployment to avoid this node is out of question.

Thanks!

Possibly you are looking for this  feature which is closed with no action since there are ways to work around this problem.

  kubeadm already allows to set taints for the joining node using the configuration file at join time (check my other answer for details on how to do that)

One Other way to achieve this is by running join as usual but taking few extra steps as below after join

  1) After successful join , Force Drain the new node to make sure it removes unwanted pod which might have scaled to it post join operation kubectl drain <node name> --force

  2) Drain command will auto cordon off the node. (Note : also consider special handling to remove any daemonset which might have scaled to this node after join)

  3) Now Taint the new node as needed and then uncordon it

  4) Add nodeselector plus tolerance to pods you want to be scheduled on this new node.

Logs of above steps for reference are listed below.

On below example i have joined a new node-02 and assume that some pod were scaled in an existing deployment so new node got used immediately on join, so before adding we need drain action and then taint the node so going forward no scheduling can be done unless pod has tolerance to the taint added

Nodes Before Drain or Taint

$ kubectl get all -o wide
NAME                         READY   STATUS    RESTARTS   AGE   IP               NODE                NOMINATED NODE   READINESS GATES
pod/nginx-86c57db685-8cdkx   1/1     Running   0          27s   192.168.58.120   k8s-node02-calico   <none>           <none>
pod/nginx-86c57db685-xnh6q   1/1     Running   0          65s   192.168.182.78   k8s-node01-calico   <none>           <none>

NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE   SELECTOR
service/kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP   2d    <none>

NAME                    READY   UP-TO-DATE   AVAILABLE   AGE   CONTAINERS   IMAGES   SELECTOR
deployment.apps/nginx   2/2     2            2           65s   nginx        nginx    app=nginx

NAME                               DESIRED   CURRENT   READY   AGE   CONTAINERS   IMAGES   SELECTOR
replicaset.apps/nginx-86c57db685   2         2         2       65s   nginx        nginx    app=nginx,pod-template-hash=86c57db685

Drain The node

$ kubectl drain k8s-node02-calico
node/k8s-node02-calico cordoned
evicting pod "nginx-86c57db685-8cdkx"
pod/nginx-86c57db685-8cdkx evicted
node/k8s-node02-calico evicted

$ kubectl get nodes
NAME                STATUS                     ROLES    AGE   VERSION
k8s-master-calico   Ready                      master   2d    v1.17.2
k8s-node01-calico   Ready                      <none>   2d    v1.17.2
k8s-node02-calico   Ready,SchedulingDisabled   <none>   2d    v1.17.2

Taint the new node

$ kubectl taint node k8s-node02-calico newnode=nobodyallowed:NoSchedule
node/k8s-node02-calico tainted

Now we are good to uncordon the node, no more pods should be able to use this noe unless they add toleration to the new taint.

$ kubectl uncordon k8s-node02-calico
node/k8s-node02-calico uncordoned

$ kubectl get nodes
NAME                STATUS   ROLES    AGE   VERSION
k8s-master-calico   Ready    master   2d    v1.17.2
k8s-node01-calico   Ready    <none>   2d    v1.17.2
k8s-node02-calico   Ready    <none>   2d    v1.17.2

We test by scaling the deployment and nothign should be added on node02

$ kubectl scale deployment --replicas=3 nginx
deployment.apps/nginx scaled

$ kubectl get all -o wide
NAME                         READY   STATUS    RESTARTS   AGE     IP               NODE                NOMINATED NODE   READINESS GATES
pod/nginx-86c57db685-vm8ql   1/1     Running   0          3m30s   192.168.182.75   k8s-node01-calico   <none>           <none>
pod/nginx-86c57db685-xnh6q   1/1     Running   0          10m     192.168.182.78   k8s-node01-calico   <none>           <none>
pod/nginx-86c57db685-zh2sj   1/1     Running   0          8m19s   192.168.182.76   k8s-node01-calico   <none>           <none>

NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE   SELECTOR
service/kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP   2d    <none>

NAME                    READY   UP-TO-DATE   AVAILABLE   AGE   CONTAINERS   IMAGES   SELECTOR
deployment.apps/nginx   3/3     3            3           10m   nginx        nginx    app=nginx

NAME                               DESIRED   CURRENT   READY   AGE   CONTAINERS   IMAGES   SELECTOR
replicaset.apps/nginx-86c57db685   3         3         3       10m   nginx        nginx    app=nginx,pod-template-hash=86c57db685


In what all cases would one use an external load balancer over the Ingress routing mesh in a Docker cluster? Also, would the routing mesh act as an application load balancer?

Like for example, if I am having a Docker cluster in AWS, how does the ELB added to it, work? And how would that be different from what the internal load balancer is doing?

Or, did I get the concept of the routing mesh completely wrong?

You missed a point that there's still a need in balancing traffic before nodes in Docker cluster, and ELB is great for it (managed cluster, healthchecks, you don't pay for number of ELB instances per VPC, you can spawn spare nodes and register them in ELB automagically with Cloudwatch), great discussion on ServerFault. Basically, external balancer will split traffic between all nodes, while internal balancers will route it to nodes where service containers are running (which is where routing mesh is used).


Enable/disable resource in terraform using count I added a piece of code to support enable/disable to the feature - I used the count. Now I have the following problem:
ERROR:
on modules/eventhub/main.tf line 110, in resource "azurerm_eventhub" "events":

 110:   namespace_name      = azurerm_eventhub_namespace.eventhub.*.name
    |----------------
    | azurerm_eventhub_namespace.eventhub is tuple with 1 element

Inappropriate value for attribute "namespace_name": string required.

# Create EventHub namespace
resource "azurerm_eventhub_namespace" "eventhub" {
  count               = var.enable_eh ? 1 : 0
  name                  = "${var.name}-ns"
  location              = var.location
  resource_group_name   = var.resource_group
  sku                   = var.sku
  capacity              = var.capacity
  dedicated_cluster_id = azurerm_eventhub_cluster.eventhub[count.index].id #for eventHub Cluster
  auto_inflate_enabled     = var.auto_inflate != null ? var.auto_inflate.enabled : null
  maximum_throughput_units = var.auto_inflate != null ? var.auto_inflate.maximum_throughput_units : null

    dynamic "network_rulesets" {
    for_each = var.network_rules != null ? ["true"] : []
    content {
      default_action = "Deny"
      dynamic "ip_rule" {
        for_each = var.network_rules.ip_rules
        iterator = iprule
        content {
          ip_mask = iprule.value
        }
      }

      dynamic "virtual_network_rule" {
        for_each = var.network_rules.subnet_ids
        iterator = subnet
        content {
          subnet_id = subnet.value
        }
      }
    }
  }

  tags = var.tags
}

resource "azurerm_eventhub_namespace_authorization_rule" "events" {
  for_each            = local.authorization_rules
  name                = each.key
  namespace_name      = azurerm_eventhub_namespace.eventhub.*.name
  resource_group_name = var.resource_group

  listen = each.value.listen
  send   = each.value.send
  manage = each.value.manage

}

resource "azurerm_eventhub" "events" {
  for_each            = local.hubs
  name                = each.key
  **namespace_name      = azurerm_eventhub_namespace.eventhub.*.name  <------ problematic line**

  resource_group_name = var.resource_group
  partition_count     = each.value.partitions
  message_retention   = each.value.message_retention

}

I don't know what the issue is.

I believe you should be able to reference the namespace name like so:
namespace_name      = azurerm_eventhub_namespace.eventhub[0].name
However, if the namespace_name field is a required parameter on the resources you are passing it to, you might run into other issues while you are trying to set enabled to false, as the resources depend on the output of the resource you are setting to count[0]. If you want them all to be conditional on the creation of the namespace, i would recommend continuing down your approach with the for_each over the local values.
If you need to the other resources to not be created unless the namespace is created you can create a condition on your local value like this (assuming map):
locals{
    hub = var.enable_eh ? {foo = "bar", baz = "qux"} : {}
}

The for_each will omit the resources that are empty.


Docker Node.js Error: Cannot find module When I start a dockerized Node.js testapp with

sudo docker-compose up

I get the following error:

Starting testapp_web_1 ... done
Attaching to testapp_web_1
web_1  |
web_1  | > testapp@0.0.1 start /usr/app
web_1  | > node index.js
web_1  |
web_1  | internal/modules/cjs/loader.js:613
web_1  |     throw err;
web_1  |     ^
web_1  |
web_1  | Error: Cannot find module 'mongodb'
web_1  | Require stack:
web_1  | - /usr/app/index.js
web_1  |     at Function.Module._resolveFilename (internal/modules/cjs/loader.js:610:15)
web_1  |     at Function.Module._load (internal/modules/cjs/loader.js:526:27)
web_1  |     at Module.require (internal/modules/cjs/loader.js:666:19)
web_1  |     at require (internal/modules/cjs/helpers.js:16:16)
web_1  |     at Object.<anonymous> (/usr/app/index.js:4:21)
web_1  |     at Module._compile (internal/modules/cjs/loader.js:759:30)
web_1  |     at Object.Module._extensions..js (internal/modules/cjs/loader.js:770:10)
web_1  |     at Module.load (internal/modules/cjs/loader.js:628:32)
web_1  |     at Function.Module._load (internal/modules/cjs/loader.js:555:12)
web_1  |     at Function.Module.runMain (internal/modules/cjs/loader.js:826:10)
web_1  | npm ERR! code ELIFECYCLE
web_1  | npm ERR! errno 1
web_1  | npm ERR! testapp@0.0.1 start: `node index.js`
web_1  | npm ERR! Exit status 1
web_1  | npm ERR!
web_1  | npm ERR! Failed at the testapp@0.0.1 start script.
web_1  | npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
web_1  |
web_1  | npm ERR! A complete log of this run can be found in:
web_1  | npm ERR!     /root/.npm/_logs/2019-05-04T15_34_04_615Z-debug.log
testapp_web_1 exited with code 1

The project structur is as follows

- testapp
--- docker-compose.yml
--- dockerfile
--- src
----- index.js
----- package.json

index.js

'use strict';

const MongoClient = require('mongodb').MongoClient;
const express = require('express');

// Constants
const PORT = 8080;

// App
const app = express();
app.get('/', (req, res) => {
  res.send('Hello world\n');
});

app.listen(PORT);
console.log(`Running on Port:${PORT}`);

package.json

{
  "name": "testapp",
  "version": "0.0.1",
  "description": "Testapp",
  "author": "hi there",
  "main": "index.js",
  "scripts": {
    "start": "node index.js"
  },
  "dependencies": {
    "express": "^4.16.4",
    "mongodb": "^3.2.3"
  }
}

dockerfile

FROM node:12-alpine

WORKDIR /usr/app

COPY package*.json ./

RUN npm install

COPY . .

EXPOSE 8080

docker-compose.yml

version: '3'
services:
  web:
    build: .
    command: npm start
    volumes:
      - ./src:/usr/app/
      - /usr/app/node_modules
    ports:
      - 80:8080

I have alread read this stackoverflow thread but I couldn't solve this issue. I'm getting this error no matter what module I'm trying to use: mongodb, spdy, etc. Locally without docker the testapp is working. But I don't know what I'm doing wrong with docker. Can anyone help me?

volumes:
      - /usr/app/node_modules

This volume could be overwriting what you have installed during the RUN npm install stage of the Dockerfile. As far as I can see you're not doing any installation when starting the container, so the contents of the mount are empty. If you remove that mount, the node_modules path will be populated with what was installed during the docker build (as defined by the Dockerfile) phase.


Kubernetes nginx ingress controller - one pr. environment? I have a kubernetes cluster in Azure. In this i have three namespaces:

development
staging
production

I have installed nginx ingress controller into its own namespace, ingress-nginx. I am kind of new to the ingress part, and I cant seem to find anywhere any advice if I should have a controller pr namespace, or share one controller with all.

Should I have one that is shared or install one pr. namespace? What is the pattern here?

I would suggest at least one for production and one for npe.  The nginx-ingress helm chart appears to only support setting a scope to all namespaces or just one namespace (a frustrating limitation not in other ingress controller options):

https://github.com/helm/charts/tree/master/stable/nginx-ingress

controller.scope.namespace


What are best practices for using Ansible Vault on public CI's and Source Control Repositories like BitBucket? Introduction

On a private CI's and Source Control Repositories like Gitlab-ce it is possible to copy an ~/.vault_pass.txt to the server and let it use by the CI to decrypt files using Ansible.

Problem

On public CI's and Source Control Repositories like Bitbucket it is not possible to copy a ~/.vault_pass.txt to the CI server it self.

Discussion

In Bitbucket it is possible to define encrypted variables, but when this file is checked the only VAULT related variables are:

ANSIBLE_ASK_VAULT_PASS
ANSIBLE_VAULT_PASSWORD_FILE

These variables are not an option to solve the issue as when the ANSIBLE_ASK_VAULT_PASS is set ansible-vault still prompts:

user@host $
Vault password:

When the same password is entered it can open the encrypted file, but the aim is to open the file without needing a file or entering a password in a prompt.

Another attempt to solve the issue was running export ANSIBLE_ASK_VAULT_PASS=<ansible-vault-password>, but the interactive mode persists.

Another option is export ANSIBLE_VAULT_PASSWORD_FILE=~/.vault_pass.txt, but then this file needs to be pushed to the repository, but source Control repositories should not contain passwords.

--vault-password-file can instead point to an executable script that writes to stdout. This little-known feature should solve your problem.

First, write a simple, executable script that prints a system environment variable, and check that into your source control. Then, use Bitbucket's encrypted variable feature to set that environment variable to your ansible-vault secret. Finally, execute like so:

ansible-playbook site.yml --vault-password-file ./mypass.sh.

References:

http://docs.ansible.com/ansible/playbooks_vault.html#running-a-playbook-with-vault
https://groups.google.com/forum/#!topic/ansible-devel/1vFc3y6Ogto


Migrating to AWS with Devops What are they key stages of migrating to AWS with Devops ?
Can anyone list a use case or workflow?

Continuous Deployment is certainly a no brainer for mission critical applications but you may try Azure DevOps from MS which integrates with AWS beautifully and has provisions for environment approvals which can help move to that state in a user friendly and elegant manner. IMO, AWS code deploy is still not at par with Azure DevOps. And perhaps you may want to incrementally move there and refine your used case.
e.g. Instead of weekend deploy, you can make the process decoupled and efficient that you can do it on a Friday night, then setup blue/green in Beanstalk or EKS et to reach that state. Also for the 6000 hosts, you definitely want to try and see which can live in Kubernetes for ephemeral instances and review the infrastructure footprint. Don't forget cloud is expensive, a lift and shift of that many VMs would have exponential costs in cloud.
I work in healthcare so unsure about retailers, but thought I will suggest that tool (Azure DevOps) incase your client isn't familiar with DevOps for easier transition.
Also to be more comprehensive, consider monitoring, security etc also as a part of your effort in your white paper.
One of the most comprehensive DevOps roadmap (evolving) is this, you may consider all the aspects from this chart for your whitepaper -
https://roadmap.sh/devops


Is there no Oracle JDK for docker? REPOSITORY          TAG       IMAGE ID       CREATED             SIZE
an-image            1         X              26 seconds ago      279 MB

when the docker image will be run, the following message is shown:

No java installations was detected.
Please go to http://www.java.com/getjava/ and download

When Oracle JDK is deployed the docker image size is more than doubled!

REPOSITORY          TAG       IMAGE ID       CREATED             SIZE
an-image            2         X              26 seconds ago      666 MB

Discussion

When "Is there no Oracle JDK for docker" is googled only some links to docker image that contain docker are returned. When Oracle JDK is deployed in the docker image the size is more than doubled. I want to keep the docker images as small as possible, but the Oracle JDK seems to be larger than the image itself!

Question

Is there no Oracle JDK for docker?

No. Because you cannot distribute Oracle JDK or JRE, the license in effect doesn't allow distribution. When distributed by a third party (embedded with your app) all the liability for it not working is on that party. This is why you will not find Oracle JDK/JRE on any of the public Docker registries, or in any Linux package repositories for that matter.

You can create your own image, and install Oracle JDK or JRE using the webupd8team/java package for Ubuntu/Debian.

And if you are using Alpine Linux this blog post explains which dependencies are required, and links to StackOverflow for installation instructions.

Update: Oracle has an official Docker image on the Docker Store now. https://blogs.oracle.com/developers/official-docker-image-for-oracle-java-and-the-openjdk-roadmap-for-containers

Link to Docker Store Oracle Java Image - https://store.docker.com/images/oracle-serverjre-8


What is an artifactory? There are quite some questions and answers that mention "artifactory".

I wouldn't be surprised if it is somehow related to artifacts.

My questions:

What is actually an "artifactory" (in the context of DevOps)?
Why are artifactories used?

Artifactory is a product by JFrog that serves as a binary repository manager.  That said very often one will use a 'artifactory' as a synonym of the more general binary repository, much like many people use Frigidaire or fridge to denote the refrigerator regardless if it is a Frigidaire brand or not.

The binary repository is a natural extension to the source code repository, in that it will store the outcome of your build process, often denoted as artifacts.  Most of the times one would not use the binary repository directly but through a package manager that comes with the chosen technology.

In most cases these will store individual application components that can later be assembled into a full product - thus allowing a build to be broken in smaller chunks, making more efficient use of resources, reducing build times, better tracking of binary debug databases etc.

Here are some of the most popular package managers that can be managed using a binary repository:

Java: jar, ear, war etc has Maven and the official MavenCentral.  There are many other package managers that will use the maven binary repository format as well (ivy, gradle etc).
.Net: nuget for .NET components (DLL and EXE) but can also be used as a distribution mechanism under windows thorugh systems like Chocolatey.  Newer versions of Powershell can also leverage this to distribute powershell modules though the powershell gallery of which one could build a local distribution with a binary repository and a repository in nuget format.  Also check OneGet if Windows distribution management is of interest to you.
In JavaScript: we have npm which is one of the most popular, will require nodejs.
In python: there is pip and the official package index pypi, which one can also create a local instance through binary repository that will support the format.

This list is far from complete, just gives an idea of what's out there.

The binary repository can allow to host all of these under one roof, making their management much simpler for teams.  Note that you do not need a very large team to start reaping benefits from binary package management.  The initial investment is not very large and the benefits are felt immediately.  Especially now that more and more platforms, frameworks and languages are integrating this dependency management directly in them.
Their biggest advantage I have found however was to create an environment that your programmers will find natural and comfortable making it essential.  It helps you as a devops creating a solid tool-chain and it helps them making the overall experience fit naturally in their stack of choice.

As I said earlier there are many products out there that can serve as binary package managers, some more generic than others in their target usage, varying also widely in their accessibility and prices.

My personal opinion is that binary repositories are as vital a part of a well designed devops setup as the source code repository or continuous integration.


Is there any way I can rotate AWS KMS keys every time interval < 1 year? According AWS KMS, keys can be rotated once every year. However, there is no way I can define a custom time period for that.

However, I can keep changing the same in the bucket's policy manually.

So, is there anyway I can automate that?

The AWS documentation is clear that automated key rotation is only supported by Amazon on a yearly basis. However, you can use the CLI to manually create new keys, and then use an automation technique such as cron to automate the manual process.

In this case you can likely have cron run a bash script to read from a list of keys and credentials and then use the AWS kms cli to "manually" change your keys.


How does Cacti compare to Nagios? So far I have been using the Nagios monitoring system.  However, I'm not familiar with Cacti. How does Cacti compare to Nagios?

While Romeo has laid out the basic facts I'd like to add a bit of experience to the answer:

Basics Cacti is focused on graphs.  It doesn't do any of the up/down monitoring that is provided by something like Nagios.  Classically Nagios does not provide any useful graphs, but more on that later.
Config Scaling Cacti tends to be configured through the GUI manually.  Eventually you will hopefully scale enough to find this frustrating.  I've switched to Munin in places where there's lots of configuration management and it is easy to turn a server inventory into a config file with templates.  Nagios tends to be configured with config files so hopefully you script the translation of your inventory in that format too.  Then you can scale to thousands of nodes without redesigning your monitoring system.
Distinctions blur. The classic distinction between Cacti/Nagios of graphing vs up-down status has been blurred in a number of Nagios derivatives.  Icinga and OMD, for instance, integrate graphing into a Nagios-style monitoring architecture.
Plugin Availability Doing something in the Nagios world gives you access to tons of well tested plugins to monitor almost anything under the sun.  Cacti also has plugins, but not nearly as many.

So obviously my Cacti experience hasn't been that great and I'd recommend you look elsewhere for your long-term graphing needs.


SaltStack: Use Temporary Scripts And Not Cause State Changes To Reported I am doing some operations with Kafka via salt, and using scripts to perform them.

My current SLS file looks roughly like:

/tmp/script.sh:
  file.managed:
    - source: salt://blah/blah

Run script:
  cmd.run:
    - name: /tmp/script.sh

delete /tmp/script.sh:
  file.absent
    - name: /tmp/script.sh

script.sh is idempotent itself. But I always get a changed=2 status report at the end of applying the state, because of it putting the script down and then deleting it.

Is there a more idiomatic way in Salt to have temporary scripts that don't live on the target box? Or do people usually just leave them in /tmp so that they don't see a reported change when applying a state.

You might want to consider adding an unless or onlyif requisite in the first directive - especially if script.sh is particularly complicated or expensive. With unless or onlyif, you will use the reults of several BASH commands to determine if you need to push down and execute your script. If your system is out of plumb only then is the state enforced. If everything checks out OK, then the script will never be pushed and executed preventing changed=2 in the final status report. You will want to use a require requisite for the second and third directives to prevent them from running if the script is not pushed by file.managed otherwise you will get to errors instead of two changes.


How to buy more Azure DevOps pipelines build minutes? As mentioned on the Azure DevOps official pricing page, the plan provides 1 Free Microsoft-hosted CD/CI parallel job with up to 1800 minutes per month and 1 Free Self-hosted CD/CI parallel job with unlimited minutes per month.

When we exceed the limit of Microsoft-hosted minutes, what I do is configure a VM as the build agent. How can I buy more build minutes after exceeding this 1800 minutes, rather than launching VMs for agents by myself?

You can set up billing in your Organization settings in DevOps

https://dev.azure.com/~your organization name~/_settings/billing


Is local / staging / production still industry standard? I've inherited a project that uses local / staging / production.
It is a simple project that lets users create a login and download customized PDF documents. We keep the document templates and the per-user-customisations in S3.
But I've heard that githubflow allows direct deploy-to-production--upon--merge-PR-to-master.
The project comprises a React front-end and a Django backend.
The backend is hosted on AWS and administered through Kubernetes.
There is also a database and S3 bin on AWS.
The infrastructure was never completed, so I'm in the process of automating deploy-to-{staging/production}, and wondering if I'm being a dinosaur.
My thought is that if deploy-to-production automatically backs up the database and S3, this could give a faster/more-efficient workflow.

When you say "Industry Standard" that's very much going to depend on what industry.
You can drop code into production however you want (legally restricted industries obviously being the exemption to this), but the question you should probably be examining is if you should be doing that.
I've recently worked on products which range from having incredibly test coverage, huge amounts of infrastructure set up to perform every imaginable regression test against any integration point that exists, and even though it was not the done thing with that particular product, after passing all of the automated testing required for a branch to be mergable, I probably would have been pretty comfortable lodging straight to production. On the other end of the spectrum I've also worked on projects which arguably were developed in production given that the production instances were so dissimilar to what was actually checked in to source control or running in any test environment and, of course, no test cases to be seen.
The standard approach to making production deployments as safe as possible is to try and ensure that the thing you are planning to deploy is as rigorously tested as possible, and to do that most of the testing will need to be automated, otherwise you'll just end up spending your life testing.
Additionally, most modern systems use standard patterns to make a production roll back an extemely frictionless task, if not to automate it entirely. If you are deploying canaries of your software, or you are running a blue/green deployment and can have built in mechanisms to roll back if certain success conditions aren't met, then it further reduces the risk of a production change breaking something.
Essentially, I would question anyone telling you how to manage and derisk your production releases unless they are just as intimate with the software as you are.


Add an Actuator Button to Grafana Dashboard Hi everyone and Thank You for reading. I have  built a dashboard with Grafana. The sensor readings comes to me from a MQTT broker and are stored in Inflixdb. Now I want to add a button on the Dashboard in order to toggle an actuator (a water pump). Can you help me and suggest me how to do? I do not see any widget in Grafana to add an actuator.
Thank you for your help.

I think to have found a suitable grafana plugin that can activate/deacttivate an actuator by posting a value with a call api rest. The name of this plugin that can be easily donwloaded and istalled in the grafana is cloudspout button panel plugin. and can be found at grafana.com/grafana/plugins/cloudspout-button-panel


How does dependency management for Node.js differ from Maven? Strange enough node.js unlikely maven installs the packages the software needs inside the project directory.

Is there a way to introduce a central repository which also caches different versions of components and would allow to pull them on build like we know it from other ecosystems?

Strange enough node.js unlikely maven installs the packages the software needs inside the project directory.

I don't see fundamental differences between npm on node.js and Maven. In both cases, the dependencies finally end up in the project directory (npm: /node_modules, Maven: /target). Otherwise, you cannot continue building the software. (When you run the build system to create the final artifact, it needs the dependencies.)

In addition, both have a global cache to avoid downloading already fetched dependencies: In Maven, the default cache directory is ~/.m2 (on Linux and MacOS). For npm, it is ~/.npm.

  Is there a way to introduce a central repository which also caches different versions of components and would allow to pull them on build like we know it from other ecosystems?

If the system cache is not enough and you do not want to depend completely on the public repository (npm: registry.npmjs.org, Maven: Maven Central), you can run your own mirror. For instance, if you want to be able to rebuild the software even if the public repository is down.

In my previous job, we used Nexus as a caching proxy and local registry (for hosting code that is only accessible within your organization). It supports both npm (see documentation) and Maven (see documentation). (There are other tools, but the point is that both ecosystems can be handled by one tool.)

Of course, there are differences between Maven and npm. For instance, Maven is also a full build system, whereas the scope of npm is primarily dependency management. Maven also defines conventions to structure a Java project ("The Maven Way"). But when looking only at the dependency aspect, package managers for Node.js like npm are conceptually not so different from Maven.


consul throwing an error Trying to run consul on my system.

Have followed link to download consul:

consul and consul-template setup

consul agent -server -bootstrap-expect 1 \
    -data-dir /tmp/consul -node=agent-one \
    -bind=127.0.0.1:8500 \
    -client=0.0.0.0 \
    -config-dir /etc/consul.d \
    -ui-dir /opt/consul/

But getting an error:

WARNING: BootstrapExpect Mode is specified as 1; this is the same as Bootstrap mode.
==> WARNING: Bootstrap mode enabled! Do not enable unless necessary
==> Starting Consul agent...
==> Error starting agent: Failed to start Consul server: Failed to start lan serf: Failed to create memberlist: Failed to parse advertise address!

It looks like that the official Consul documentation is more up to date than the link that was posted in the question.

If one starts with the Getting Started guide, then one could start a consul agent in no time and the documentation also explains how to proceed.

user@localhost /tmp $ ./consul agent -dev
==> Starting Consul agent...
==> Consul agent running!
           Version: 'v1.0.6'
           Node ID: '777c5475-bc78-6e38-75a6-42038eef7cb8'
         Node name: 'localhost.localdomain'
        Datacenter: 'dc1' (Segment: '<all>')
            Server: true (Bootstrap: false)
       Client Addr: [127.0.0.1] (HTTP: 8500, HTTPS: -1, DNS: 8600)
      Cluster Addr: 127.0.0.1 (LAN: 8301, WAN: 8302)
           Encrypt: Gossip: false, TLS-Outgoing: false, TLS-Incoming: false

==> Log data will now stream in as it occurs:

    2018/04/09 21:33:04 [DEBUG] Using random ID "777c5475-bc78-6e38-75a6-42038eef7cb8" as node ID


Are there cheaper alternatives to cloud watch logs for creating dash boards? Iv set up a grafana instance.

I want to create dashboards to get metrics on lambda functions but useing cloud watch logs would be expencive.

Are there any cheaper alternatives to achive this that might be worth looking at?

Grafana was not really built to view logs from the get go.

They however introduced this feature in v6.0 via the Explore tab

This can be leveraged in 2 ways.

Via Elasticsearch data source (in v6.3+)

Check the Elasticsearch docs for more.

If you want, you can still integrate this with Cloudwatch. Check out the log subscirber-consumer solution & this

Via Loki & AWS Lambda

If you want to stick to Cloudwatch as your primary source of logs, you can create a Lambda function that ships them to Loki which in tern can be used in Grafana as a data source. See 1, 2 & 3


Can large scale services use a public cloud managed database service? Theoretical question:
Is it possible for a very large scale service to use a major public cloud managed database service?
For example, a service with a scale of Netflix and a service like Amazon Aurora.
If not, which solutions are available for very larges scale services?

Yes, it is certainly possible, but it entirely depends on the architectural design of the entire system. Separating application services into manageable components (i.e. microservices) is one approach to this. While Netflix as a whole is one application in terms of the user experience, it is comprised of many applications/services that provide a small fraction of the functionality needed to run Netflix as a whole.
In terms of a database, the same principal can be applied. Netflix doesn't manage one database for the entire application, each component utilizes it's own scalable database service.


Which SCM solutions exist for managing mainframe software? Imagine a company using mainframes to run (part of) their (often mission critical) business applications, and using z/OS (also known as OS/390, or MVS).

What are the typical softwares they use to facilitate Software Change and Configuration management, for software that is deployed/used on those mainframes?

From my own experience, these are some the typical softwares packages:

CA Endevor.
SERENA ChangeMan ZMF.
Compuware ISPW.
IBM SCLM.

All of these packages can manage, more or less out-of-the-box, anything that's stored in regular "PDS" components (a typical file structure used in z/OS).

When it comes to a company evaluating which one best fits for them, it often boils down to these criteria:

IBM SCLM is perceived as free of any license/maintenance fees (actually it's included in the z/OS license, which itself is not free). So if no dedicated budget is available, then this is often the software package that gets selected (better then no package at all). If there is a budget, then this one is often the one that doesn't make it to the shortlist.
CA Endevor has the highest install base. Its key strength, IMO, is the way that you can trace down for each executable how it has been compiled/linked using which version of which building blocks (copybooks, etc).
SERENA ChangeMan ZMF's install base is quite a bit below the one from CA Endevor. Some of its key strengths are:

the notion of "packaging" related software changes, which is at the core of it.
its capabilites to deploy softwares towards physically remote sites.

Compuware ISPW is like the "new kid in town" (as compared to the CA Endevor or SERENA ChangeMan ZMF alternative). It is commonly perceived as the solution where "any custom SCM requirements can be implemented with it, with a relatively low effort to do so".

Looking at it from an architecture standpoint, then SERENA ChangeMan ZMF and Compuware ISPW appear to have the most open architecture, which is what you'll need if you want to tune it to make it manage software components written in a 4GL language that is (what some call) more exotic, as compared to 3GL languages such as COBOL, PL/I, etc. I.e. because the software components are stored in file systems that are not stored in standard PDSs. Some examples of those languages are:

Natural (stored in ADABAS DBMS).
ADS Online (stored in IDMS DBMS).
QMF (stored in DB2 DBMS).
CA Gen (stored in "encyclopedias").
CSP (stored in VSAM file format).

Attention: having an "open architecture" is great to have it fit your custom requirements (the sky is the limit). However when it comes to upgrading to new releases, it comes with a price to get those custom requirements upgraded also.

Note: rather accidentally, during a CA Endevor training for SERENA ChangeMan ZMF experts, we discovered that CA Endevor and SERENA ChangeMan ZMF appear to have the same roots (from somewhere in the late 1980s ...). For those who are a bit familiar with both of them: go check what the functionality is of these utility programs, with similarly named names ... (you'll be shocked ...):

PGM=CONWRITE versus PGM=CMNWRITE.
PGM=CONPRINT versus PGM=SERPRINT.


What is the common term for Apache virtual hosts or Nginx block systems? A webserver environment as with Apache or Nginx includes a type of file (or a file system) that I can call a webserver substrate; In Apache it is called a Virtual host and in Nginx I can call it a "Block system", mainly comprised of a server{} block, among other blocks.

This "webserver substratre" connects an app's directory with the webserver, and also enables or disables it (by means of webserver requests and responses associated with that app directory). Thus, each web application has one such substrate.

What is the common term for such file, or file system, in the Linux System Administration (LSA) field?

All my searches for a term brought me to conclude that the general term either for one Apache virtual host file or a constellation of Nginx block files (each containing at least one block) is just that --- a virtual host.

Per project; either the virtual host (entity) is a single file or a combination of files, and that's true if the webserver software is different than these two.


Continuous Deployment best practices for managing application configuration My CD pipeline:

Get Docker image (created and pushed during CI stage).
Copy docker-compose .yml template to target server.
Run the app.

Let's go straight to the 3rd point. I can run the app using docker-compose up e.g., but I have no idea how to handle configuration. Target server is a Ubuntu Server created by Chief on AWS, I have an access to all AWS services like Parameter Store etc.

So here are my ideas about managing configurations:

save secrets to configuration file before running the app and attach this file to specific location in docker-compose template (I don't like this solution, I can handle this by some bash script that inject secrets to file).
set environment variables before running the app (I talked with some DevOps and basically they like this solution).
get secrets from AWS Parameter Store (unfortunately I can't have an access by role management, only by key which I need to set on server by env or in config file).

How does managing configuration look like in real case scenarios? I'm worried about case of server crash or something similar. 1st solution will survive this, but in case of 2nd and 3rd ones the environment variables will be lost after crash/reboot. What is the best solution here?

[Copying answer over from duplicate post]

If you start a container with an environment variable, that variable is part of the container definition. As a pro, when the server is rebooted, the same variable will be there when the container restarts. But as a con, anyone with access to the docker API can view that secret in clear text, and some apps log their environment when debugging.

With swarm mode (Kubernetes has their own similar solution), you can define your secret on the manager and it will be given to nodes running the container on a need to know basis. The secret is accessible inside the container as a file, but that file is only stored in memory on the worker node, and you can use an entrypoint script in your container to load that into the environment if that's how the application expects it. The swarm mode secrets and configs have the advantage of being portable as the container is rescheduled on other nodes in the event of a worker crash. Migrating from a docker-compose install to a single node swarm cluster and deploying a stack is a very simple task.

There are also external solutions for storing configs and secrets in key/value stores like CoreOS's etcd and Hashicorp's Vault. Both Swarm Mode and Kubernetes use a key/value store for their own settings, giving you an idea of how the community sees it as a best practice. The downside is you often want to configure the application to read and write configuration from one of these key/value stores directly. If you use something like swarm mode secrets and configs, these will be stored in docker's internal key/value database, so you get a similar solution with little effort on your part.


Tfs octopus migration to k8s and docker My current client uses .net and .net core,  tfs and deploys via octopus. They want to use helm, k8s and docker.  With all of the environments, env variables are already in octopus.

I am thinking to restore the nuget packages in the dockerfile, build the image, push to our local registry. Add our docker registry as a feed, add helm as a feed then creating a lifecycle for dev and prod. Then modify the octopus jobs to point to the local registry to pull the images.

Anyone see any issues here? Should I strip out the env variables from octopus and just get rid of octopus?

Helm is a Kubernetes tool so this answer assumes you are deploying to Kubernetes or OpenShift not Docker Swarm.

If you are using Helm well then I don't think it is particularly important what build pipeline tech you are using to invoke it. I think what is more important is how you manage all your configuration within Git and how you structure the Helm deployments.

There are two different paradigms you can follow:

Build the app container image with the correct environment variables baked within it for each environment.
Build only one app container image that picks up environment variables from within the environment it has been deployed.

If you follow 12factor.net best practices then you go for Option 2. IMHO it feels far cleaner and less error-prone to move the exact same image between kubernetes namespaces that represent different environments. We store environment variables inside of Kubernetes as secrets or configmaps. We then get to two more questions:

How do you load secrets and configmaps as environment variables into your .Net Core runtime images when the container starts up?
How to you manage the secrets and configmaps in each environment?

With respect to the first question, Kubernetes has this covered. Any deployment can name a bunch of different secrets that should be loaded as environment variables. Ditto for configmaps. You just need each environment to be a different namespace so that the same logical secret can have the same name in each environment. Your database credentials can then be a secret called main-database-credentials that holds different credentials in different environments.

To deal with the second question we use Helm to manage both secrets and configmaps. We have one generic Helm chart that deploys all our different secrets in all our different environment:

helm upgrade --install -f $NAME.yaml $NAME secret-1.1.tar.gz

So to upgrade our main-database-credentials secret we would change the settings in main-database-credentials.yaml then tell the chart to recreate the main-database-credentials deployment using that command. The chart simply uses the Helm templating language to build yaml that defines the secret using the name set in the yaml file. Likewise, we have a generic chart that updates configmaps that is invoked with a very simlar command as above. Helm uses a standard Golang templating library so it is very straight-forward to create those generic charts.

At first, it may seem a bit heavyweight to use Helm to update something like a database password secret. Yet helm gives you a deployment history and lets you diff the history of deployments. It also lets you easily roll-back to a previous deployment. That is really powerful as an audit trail of config changes that is really helpful when managing a complex microservices deployment. Since we also use Helm to deploy all our apps it makes perfect sense to use the same power tool to manage the deployment of configuration into our environments.

As we can use a generic helm chart to deploy many secrets into many environments why not use a generic helm chart to deploy all our microservices? We do:

helm upgrade --install -f $NAME.yaml $NAME deployer-1.3.tar.gz

This means that NAME=my-microservice and NAME=my-admin-webapp or anything else can be deployed using one generic deployer chart. The path and tag of the application container image to deploy is in the yaml file. The container path is different for different apps and the container tag can be different between environments.

That might sound strange as normally Helm packages a particular app so you would expect a one-to-one relationship between charts and apps. Yet if all your microservices are containers following 12factor.net best practices shouldn't they all "look the same" from the perspective of deploying them; it is just the container and the environment variables you deploy into the one generic chart. I see it that the app is the pure container with none of the configurations to run it and the Kubernetes configuration to run it is "application infrastructure". So I have a Helm chart for each type of application infrastructure. I don't want each bit of application infrastructure to be hand-rolled I want to cookie cutter create it using Helm.

Now we have covered all the Helm mechanics we are left with the question "how do we manage all our configuration within git?". We use different repo for different purposes:

Each microservice has its own repo. We use CircleCI to build the code into an image that is configured via environment variables. We don't store any Helm or Kubernetes config in this repo. We do have a properties file that names the latest version of the deployer chart this code is compatible with.
We have a GitHub repo that holds the source of the generic Helm charts that are used to deploy all our apps and secrets. These don't actually change very often now they are fairly stable. CircleCI builds and releases the charts as tar.gz onto the GitHub releases page for this repo.
We have a git repo per environment. This has a folder per deployed microservice within that environment. Each contains the yaml file to pass to the helm deploy encrypted with git secret. It has a folder per deployed secret within that environment. Each contains a yaml file that is git secret encrypted. Ditto for configmaps.

Why a repo per environment? A branch per environment sounds more normal. Only a database credential for the live environment never moves between environments. As we use a separate repo for live we can give it separate permissions. We might let all developers go wild in the dev-environment repo we have a bit more discipline with the live-environment repo.

We then have CircleCI do automated deployments of configuration. This works as Helm data detects changes and doesn't deploy a change to any Kubrernetes object if the latest yaml it generates is already deployed. To change a password in live you change it in the git secret encrypted yaml in the live repo and apply a release tag. CircleCI pattern matches on the tag name to see that its a release tag and it runs the deploy step. That uses a container with the helm client to run a bash script that loops over all app folders, secret folders and configmap folders running Helm for the right chart and correct yaml. Helm will only apply updates where there are changes within git.


What is the difference between using brackets (`[` `]`) and just specifying the command in the CMD option in Dockerfiles? Why does CMD ["mysqld"] result in:

db_1   | 2017-05-14T16:34:11.829163Z 0 [Note] mysqld (mysqld 5.7.18) starting as process 1 ...
db_1   | 2017-05-14T16:34:11.833159Z 0 [Note] InnoDB: PUNCH HOLE support available
db_1   | 2017-05-14T16:34:11.833190Z 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
db_1   | 2017-05-14T16:34:11.833199Z 0 [Note] InnoDB: Uses event mutexes
db_1   | 2017-05-14T16:34:11.833211Z 0 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
db_1   | 2017-05-14T16:34:11.833222Z 0 [Note] InnoDB: Compressed tables use zlib 1.2.3
db_1   | 2017-05-14T16:34:11.833232Z 0 [Note] InnoDB: Using Linux native AIO
db_1   | 2017-05-14T16:34:11.833583Z 0 [Note] InnoDB: Number of pools: 1
db_1   | 2017-05-14T16:34:11.833729Z 0 [Note] InnoDB: Using CPU crc32 instructions
db_1   | 2017-05-14T16:34:11.835500Z 0 [Note] InnoDB: Initializing buffer pool, total size = 128M, instances = 1, chunk size = 128M
db_1   | 2017-05-14T16:34:11.844395Z 0 [Note] InnoDB: Completed initialization of buffer pool
db_1   | 2017-05-14T16:34:11.846532Z 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().

and CMD mysqld in:

db_1   | 2017-05-14T16:35:43.121085Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
db_1   | 2017-05-14T16:35:43.123152Z 0 [Note] mysqld (mysqld 5.7.18) starting as process 8 ...
db_1   | 2017-05-14T16:35:43.126844Z 0 [ERROR] Fatal error: Please read "Security" section of the manual to find out how to run mysqld as root!
db_1   |
db_1   | 2017-05-14T16:35:43.126882Z 0 [ERROR] Aborting
db_1   |
db_1   | 2017-05-14T16:35:43.126905Z 0 [Note] Binlog end
db_1   | 2017-05-14T16:35:43.127142Z 0 [Note] mysqld: Shutdown complete
db_1   |
strength_db_1 exited with code 1

CMD with brackets is contained within an array. Without brackets, the command is executed as /bin/sh -c mysqld.  Best practice afaik is to always use brackets.


Persistent Storage with Docker in Production - Which Solution and Why? I've recently started working for a company that wants to break their monolithic SaaS application up into containerized microservices. I'm having a hard time grasping a fundamental part of persistent storage, though. Why are there so many different competing platforms? Portworx, Rexray, StorageOS, Flocker, Inifint, etc.

My Questions

Why wouldn't someone simply spin up an NFS server and use a hierarchical folder structure there as their storage backend? What gains do you get when using one of these tools?
How dangerous is it to use something like this with Docker? What are the common causes for catastrophic data loss in a docker-based environment?
What persistent storage solution would you recommend and why? My company operates a SaaS platform. The data payloads are small in size (5kb-100kb). Data processing is small-medium in resource consumption. Overall volume is medium, but continues to grow. We're hoping to completely move our monolithic application to the cloud as separate containerized microservices. Including our data warehouse.
Somewhat unrelated, but it ties in. What are the strengths of using Kubernetes as an orchestrator as opposed to Rancher/Cattle? Isn't Kubernetes over-engineered for a small-medium sized platform? Are there any strengths to using Kubernetes in Rancher aside from the one-click installation?

Thank you for the insight. Sorry for the naivety. I welcome all documentation and supplemental reading material.

EDIT: For context we are using Azure as our underlying Cloud platform.

I can answer the 2nd point:

Docker is most suited in a micro service based architecture when the application runs inside the containers but the storage or any other live sessions are maintained in shared RAM or the database.

Basically you just should not store anything inside the docker container. There are many reasons to it:

Consider upgrade: Someone from your team has created a newer image of the application and you need the container running with the latest image. The current docker and popular way of doing this is to bring the existing container down and spin a new container with the same run time parameters as the older container but with the newer image. This is one of the biggest reasons why containers should always be stateless and not contain any data. You can have all your data mounted to some place and sessions kept inside a db or something like memchached etc.
One of the big use case of docker is to build clusters. If you start keeping data inside your containers then its an overhead to maintain that data in sync between the application containers.
The docker community in general does not recommend to keep any data in container and hence nobody has tried taking this risk in production and nobody wants to be the first storyteller of how they messed up production :)


Should local Ansible playbooks be loaded each day/week with cron? I ssh into a remote 5$ debian:stable IaaS machine with Putty. The machine has only debian:stable and Ansible installed on it and I run an Ansible playbook locally on it with my current working user (that has sudo rights).

I aim to establish a LAMP server environment on this machine.

playbook.yml:

 - name: Establish a Debian-LAMP with some extras
  hosts: 127.0.0.1
  connection: local
  become: no
  tasks:

  - name: update the apt package index i.e. apt update
    apt: update_cache=yes

  - name: Install ufw
      apt: package=ufw state=present ## Install only if package-index is already present in the apt package-cache.

  # etcetra

$ ansible-playbook playbook.yml

That single Ansible playbook is an orchestration and deployment of my entire server environment (Installation and setup of ufw, unattended-upgrades, SSHGuard, Apache, MySQL, PHP, ssmtp and further tools like curl, wget, zip, unzip tree and so forth).

When I will learn enough Ansible I hope to add directives that will also transduce all my virtual-host files of Apache from say Apache 2.4 to 3.4 to 4.4 to 5.4 ecetra so I won't have to manually change all virtual host files in case I need to.

Given the playbook is filled with apt upgrade package1 package2 package3 etcetera I assume I should run this playbook (or at least parts of it) each time anew, if Ansible won't do it itself.

Should I run this Ansible playbook daily/wwekly with cron or Ansible will recursively run it itself by some internal scheduling mechanism of its own?

I would discourage you from using Ansible to run Ansible (though this can be done) as you will run the risk of breaking idempotent rule of Ansible. Instead, choose a singular starting point for running your playbooks; either command line, cron or some other program.

With that said, I would use cron to run an Ansible playbook, however I would also setup your ansible.cfg file for logging and check the log periodically if there is breakage in your playbook.

Ansible by default doesn't do logging so I would set this up in ansible.cfg, just in case:

[defaults]
log_path = ./ansible.log

Also, make sure to check your email regularly to make sure your cron/Ansible job ran without any problem.


How can I access the console of a booting EC2 instance in AWS? I'm new to AWS and am looking for a way to access my EC2's console so I can triage issues such:

boot to single user mode
view console output as system boots
gain console access to debug GRUB issues

Background

When using cloud services such as AWS or Azure your ability to access the console of booting instances is typically restricted to a read-only type of interaction.

AWS does provide tools, so you just need to adjust your approach, if you're expecting to drop to single user mode or watch a booting system on a console.

For example, in the EC2 services area of AWS you can select a EC2 instance and then under the actions menu select either:

Get System Log
Get Instance Screenshot

Example

Step 1: Screenshot of actions menu

Step 2: Screenshot of "System Log"

Step 3: Screenshot of "instance's console"


How does Vagrant differ from Ansible and Docker? I understand that Ansible is a Configuration Management (CM) tool for orchestrating, deploying (and continuously upgrading) software and conf via a special YAML format created for that software, and that Ansible actions can be done on the local machine via a locally executed Ansible playbook, or remotely for 1 or more machines by listing IP addresses for an Ansible playbook in an Ansible storage file and executing both the storage and associated playbook (without needing to install Ansible on the one or more remote machines).

I also understand that Operating-System-Level-Virtualization ("Containerization") software like Docker (also utilizing YAML in a unique format) allows us to "copy away" containers of "frozen; just-bake" data, from one machine to 1 or more other machines to have a pre-processed data to work with and to change minorally per specific nneds to save time of repeatitive creation of that data in the container time and again.

Sometimes when I read about either Ansible or Docker I come across mentions of Vagrant which is another software.

I understand Vagrant to be a what I can call "All-Level-Virtualization" (operating-system and possibly also some stack above it), yet I'm not sure if that's correct.

How does Vagrant is different than these two in principle (and also the term "Vagrant box" makes me to wonder if Vagrant is an actual "competitor" for Docker, is it?)

Vagrant can complement Ansible and Docker. It is used to spin virtual machines inside which you install Docker and test your Ansible playbooks. You describe desired VM configuration with Vagrantfile and then spawn them with vagrant up command. Then Vagrant software downloads one of shared boxes (VM templates, like bare ubuntu or centos, or with preinstalled software like laravel), and provisions it (executes setup instructions). There are several provisioners, like simple shell, or Ansible.

One of possible scenarios: you want to test mysql:5.7.24 image inside Docker 18.06.1-ce on Ubuntu 18.04. So you:

Spin up ubuntu/bionic64 box with Vagrant
Provision it with Ansible playbook that installs Docker 18.06.1-ce
Start Docker container based on mysql:5.7.24 image


Reconciling Continuous Deployment with strict change management policies I'm curious how others architect their DevOps practices in a strict change management environment, such as with a Change advisory board (CAB) approval process.

I understand that automation can improve your audit process by guaranteeing a more rigorous, provable, and repeatable process.  But it feels like Continuous Deployment is more or less impossible in such a situation.  Since it can take a week or more to get changes approved, you lose the ability to deploy fast and often.  What steps do you take to work within these processes short of just submitting change requests and waiting for approval?

If you must adhere to the change process, you'll be limited according to the limitations of the change process, full stop. If changes must be approved prior to deployment, you cannot do continuous deployment. If approval takes a long time, you cannot deploy quickly. There's no workaround whereby you can both follow the process and not be impacted by it. That's the cost of following the change process, and it's a cost worth bringing to the attention of the stakeholders in that process.

All is not lost... you can maximize the automation around the process, in order to minimize errors; all of the CD steps except for the linkage between generating a stable artifact and deploying that artifact to production. That linkage would be replaced by either some kind of user intervention (button, CLI command, etc.), or linked to the record of approval (e.g. when a change request ticket is moved to the "approved" state, fire off the associated deployment). You just have to squeeze as much benefit as you can out of it, while following whatever mandatory process you've been saddled with. It won't make approvals go any faster, of course.


Enable TLSv1 TLSv1.1 for specific Ingress with Nginx Ingress I try to enable TLSv1 and TLSv1.1 for one specific ingress on a AKS cluster.
By default, only TLSv1.2 is enabled.
I don't want to apply this configuration on my all cluster, just on a specific webserver service.
Regarding the documentation, I need to add this annotation if I don't want to change nginx config by ConfigMap :
nginx.ingress.kubernetes.io/proxy-ssl-protocols: TLSv1 TLSv1.1 TLSv1.2
but this has no effect.
Is it really possible to do this by annotation ?
Did I do something wrong ?
Thank you !

The ConfigMap and Annotation configuration have different purposes.
As per nginx-configuration/configmap/#ssl-protocols:

Sets the SSL protocols to use. The default is: TLSv1.2 TLSv1.3
ngx_http_ssl_module.html#ssl_protocols
Context:  http, server

i.e., enable the specified protocols to your Nginx Controller.
client --ssl-protocols-> controller --> backend

On nginx-configuration/annotations/#backend-certificate-authentication, we can read:

Enables the specified protocols for requests to a proxied HTTPS server.
ngx_http_proxy_module.html#proxy_ssl_protocols
http, server, location

i.e., enables the specified protocols for requests to a proxied backend using HTTPS.
client --> controller --proxy-ssl-protocols-> backend

As you can see, to set the protocols served by your Nginx controller, you only have http, server contexts available (i.e., you can't set this to specific ingress objects).


How to start the latest version of k8s using minikube? When a k8s cluster is created using minikube, the output indicates that an outdated version of k8s is deployed:

user@host ~ $ minikube start
Starting local Kubernetes v1.6.4 cluster...
Starting VM...

https://rominirani.com/tutorial-getting-started-with-kubernetes-on-your-windows-laptop-with-minikube-3269b54a226

user@host ~ $ minikube start --kubernetes-version="v1.7.5"
Starting local Kubernetes v1.7.5 cluster...
Starting VM...


How to properly setup a multi-environment release pipeline in Azure YAML pipelines? We have just migrated from TFS 2018 to Azure DevOps Services and we're having a hard time trying to figure out the new Azure YAML pipelines.

We want to be able to:

Run a build/test pipeline when a PR is pushed to develop.
We've already accomplished this by creating a dedicated YAML pipeline that only builds and runs the tests (no publish artifact or anything). This pipeline is triggered by a "Build Validation" branch policy on the develop branch.
Run the full CI/CD pipeline that will build, test and deploy to DEV and QA environments when a commit is pushed to the develop branch.
We've already accomplished this by using a multi-stage pipeline triggered by any change in the branch. Although we're not entirely happy about the duplicated tasks here (this has exactly the same tasks to build and test the project as the above pipeline).
Deploy to our production-staging environment when a commit is pushed to the master branch.
We're stuck here. We're not sure how to do this without creating yet another new pipeline that will do exactly the same as the previous one, but targeting our staging environment. We want to "build once, deploy many". This was easy using TFS build and release pipelines (I guess this is what Azure DevOps calls "Classic Pipelines", right?)

How we can accomplish this workflow without needing to create several similar YAML pipelines, but also following the "build once, deploy many" pattern?

Assuming that deploy to DEV and QA steps look a lot like your deploy to production-staging environments, put the code that does that work into a yaml template and take advantage of parameters to allow you to pass in differing environment settings for DEV, QA, and Staging-Production.

Then in one pipeline create a stage for each environment.  The stage should have a job which references your template.  You should use conditions on each stage to ensure your code goes to the environment when you want it.  Here's a pseudocode example:

stages:
- stage: MyDevStage #No condition so this stage always runs
  jobs:
    - template: '../MyTemplate.Yaml
      parameters:
        ServerName: MyDevServer
- stage: MyProdStage
  condition: and(
    succeeded() # if my dev deploy doesn't go well don't deploy to PRD
    , eq(variables['Build.SourceBranch'], 'refs/heads/master') # confirm this is a commit to master
    )
  jobs:
    - template: '../MyTemplate.yaml'
      parameters:
        ServerName: MyProdServer

At the top of your pipeline yaml file you'll need to specify a trigger which may look something like this:

trigger:
  batch: true
  branches:
    include:
    - master


Blue/green deployment with micro services and DNS Currently, we have on our test environment an ECS Cluster with micro services deployed. All micro services are talking to each other through Route53 DNS CNAMEs like service1.company.com, service2.company.com, etc.

Here is a simplified schema of the infra:

The ALB listens on port 80 and everything works fine.

Now comes the troubles: I'm trying to build a blue/green deployment infra.

I basically know how to do it (HAProxies, 2x ALBs, 2x ECS Clusters). What I want to acheive can be seen https://www.youtube.com/watch?v=KJOORneBTHI

I'm stuck with the fact that we are using CNAMEs for our micro services communications.

So on the blue part:

CNAME service1.company.com -> ALBBlue

On the green part:

CNAME service1.company.com -> ALBGreen

But that's something I cannot do of course.

How can I do?

You should have 3 names when you're willing to do a blue/green deployment.

Two set of names, one for blue, one for green, which will works as is, and a production entrypoint which will point to either the blue name or green name.

The point of a blue/green deployment is to be able to test the full deploy before switching the clients entry point, so your services calls themselves don't have to use the entry point name itself.


Trigger jenkins pipeline job with Bitbucket hook I'm missing something obvious here.

Is there no way to trigger a Jenkins scripted pipeline job via Bitbucket hook?

There's a bug out about it but it's been around for years with apparently no fix. Is the only way to make this work really to make an upstream trigger job that kicks off the pipeline? That seems insanely clunky.

If you just need generic post-receive hooks, you can just do a regular web hook and hit the build API endpoint.  For instance, if the name of your Jenkins server is jenkins.example.com and the name of the Pipeline job to trigger is my_awesome_job, then you can fire off a POST request to https://jenkins.example.com/job/my_awesome_job/build.  Depending on your security model, this request may or may not need to be authenticated and authorized.  If you do need authentication and authorization, you can use HTTP Basic auth (unfortunately I haven't found a reasonable way to do keypair auth).

If you need to trigger builds on pull requests, then you'll need a plugin. I've been using the pull-request-notifier plugin for Bitbucket to trigger builds from Bitbucket and it's worked pretty well.  It works especially well in combination with the Bitbucket Branch Source plugin for Jenkins, which automatically creates Jenkins jobs for every branch and open PR of every single repository in a Bitbucket project.  The combination of these two plugins allows me to automatically trigger builds as soon as a PR is opened and prevent merging the PR until the build passes (this is a pretty common workflow so I figure it's worth mentioning even if you didn't explicitly ask about it).


A service by AWS that routes traffic based on simple logic Background
we currently use google distance matrix (GDM) apis a lot to help in core business functions. As the cost of GDM went up, we worked on switching on cheaper alternatives. We decided to use OpenStreetMap (OSM) as an alternative. Now we have OSM hosted on aws Elastic Service Container (ECS).
Problem
Sometimes (for reasons beyond the scope of this question) the OSM service fails or times out. We put logic in the application code that if it fails (or times out, with a timeout being 1s), then switch back to GDM as a backup. The problem is that if there is a sudden burst of requests, using the ECS architecture these requests will stack up, and so if we have 20,000 requests that's a total potential lag of 20,000 seconds, and the the system crashes.
One way of solving this problem is putting in the application code logic that keeps track of the number of OSM timeouts TOs per time T(persisting that number into redis or something) and once it reaches past a certain number, we will switch back to GDM, and after Z time elapsing, we want to try using OSM again (the idea is that the server would be operational again).
Question
Is it possible to do the above in an AWS Service that is specialized for such routing problems? I was thinking something like AWS Gateway but I couldn't find a place where routing logic can be put there.

You can use AWS APP MESH to route traffic between OSM and GDM. First, all traffic will be sent to OSM using App Mesh's VirtualRouter.
VirtualRouter is an object to route traffic between K8S services. One service will be pointed to the ECS hosted OSM. Another service will be pointed to GDM using External Service
You can use a Cronjob which will run in a pod and sense timeout-related metrics from ElasticCache. If the condition matched - the code host in the pod and running as a CronJob will modify VirtualRouter.


Pipeline flow get stuck at “Notifying stash” after finishing all stages All our CI projects are getting stuck in the Notify stash. eventually, it's passing, but we're trying to understand why it takes so much time. example:

17:03:01 Notifying Stash at "https://bitbucket"
17:15:44 Notified Stash for commit with id 925ceb567087f94c5b9d44d1330995860f11d1e5

what can cause such a delay? Bitbucket (Stash) Notifier Plugin version: 1.4.0

We removed the OCP plugin configuration from Jenkins


Azure Windows Container: How to read secrets from an azure vault I am coming from AWS / Linux and try this in Azure / Windows, now.

The Question:

I have a windows container in Azure and want it to get some secrets from an Azure Vault. How can I do that?

This is how I thought it would work (but didn't):

I had expected that I could just run

az keyvault secret show --name "my_secret" --vault-name "my_fault"

... at least after adding permissions to the container. Unfortunately, I cannot find out how to add permissions to the container to access the vault.

What I have tried so far:

I have created the vault and added an "application" under "Access Policies". I do not find any way how to make the container act as the "application".

I have read that items need to get an "identity assigned" 1, but this seems to be not supported with windows containers 2. I had tried it out before I found the documentation, and it did not work.

How can I access the azure key vault from an azure container?

Assuming this is transient container then my suggestion to work around the not yet implemented issues would be that whatever creates it does so by passing in secure environment variables for the values you want from the key vault; then your scripts can access the data that way.

I'm not sure what your security considerations are so you'll need to take that into account so people can't login to the container to view the environment variables.

As an example I have a function that uses the .Net Fluent API to create a container group and then sets the environment variables for the container using ".WithEnvironmentVariablesWithSecuredValue()" this means anyone whose looking at the container in the portal won't be able to see the values in plain text.

Then you assign a managed identity to the function and then give that identity get permissions to secrets in the key vault and then you can either access the key vault in code in the function or the easier option is the put them as app settings for your function and then just get them as environment variables in your function - https://docs.microsoft.com/en-us/azure/app-service/app-service-key-vault-references

Hope this helps or gives you an idea on a workaround?


CodeBuild - multiple input artifacts in AWS Pipeline I am building an AWS CodePipeline which depends on two source repositories. If the 'CodeBuild' step is specified with multiple Input artifacts, then you get the error: .. declares 2 input artifacts which is more than the maximum count. Documentation has a strict requirement for CodeBuild to have only a single input artifact.

Lambdas, on another hand, do not have this limitation. I've been looking for a way how a Lambda could "merge" two artifacts into a single one which I can pass to the CodeBuild then. Unfortunately, information on this subject is quite limited.

Does anyone have a working CodePipeline Lambda which would merge two Artifacts into one?

Note: Documentation about How to access artifacts.

We've had exactly the same problem - our build stage needs to use information from two separate locations.

To solve this in a generic manner, I've created two general purpose Lambdas - one that will merge artifacts, and one that will extract specific files from an artifact. In both cases a new artifact is created that can be passed on to a subsequent CodePipeline stage.

See https://github.com/tooltwist/codepipeline-artifact-munge

If it does not meet your exact requirement, it should be easy enough to tweak to do what you need.


How to hook up Jenkins to Endevor? I'm not DevOps, I only come from a .net background, but I've been asked how to hook up Jenkins to kick off jobs, listening to Endevor mainframe?

I only found this which talks about SCLM which as I understand is a competitor company.

Apart from your "this", there is also Compuware Source Code Download for Endevor, PDS, and ISPW Plugin. Some details about it (from the linked page):

  ... allows Jenkins users to download Endevor, PDS, or ISPW members from the mainframe to the PC. Source can then be accessed on the PC, for example, for SonarQube analysis and reporting.

So you may want to give "that" a try ...

PS: I'm assuming your question is about Endevor SCM, and not Endevor DB (that a completely different story, and a challenging one ...).


How to refresh user's content of the site after release? I have an Angular / Web Api 2 site that uses ui-router for SPA behaviour and gulp for file transformations.

There is an issue where when I release the site, the user has to refresh their browser to view new content (usually). I have cache busting implemented on my js / css files, but it seems that the HTML views themselves are a problem.  At the same time I can't help feel like I'm missing something blatantly obvious here.

How do I get my tech stack to refresh the user's content (eg. when adding a new field to an existing form) so that the user receives the most up to date version of the site after release?

Here is what works for us.

We have a "version" file deployed with the application itself. Then, application issues a "check version" request periodically that checks if the version that the application has initially loaded is different from the version specified in this version file. If it is different, we show a notification that there is a new version available and suggest to refresh the browser window to get it.

I think that's what Google Inbox, for example, does. They have this periodic checkJsVersion request and, if there is a newer version of the web app available, they show this information box on the bottom left with a "New version is available, hit Refresh to get it" button.

There is also a different approach when the information about a new version of the app is pushed from the server to the client.