Incorrect timezoneID inside docker container I have created a docker image from the base Centos7 image and while running the container from it, the time and timezone were not in sync with my host. To give the container the same timezone as that of the host. I mounted /etc/timezone and /etc/localtime from the host to the container using the '-v' argument while running the container. After running the container with this, this is what i get inside the container and host:

For docker container running timedatectl:

  [root@d7a24c723f6a /]# timedatectl Local time: Tue 2017-08-01 10:28:30
  IST Universal time: Tue 2017-08-01 04:58:30 UTC RTC time: n/a Time
  zone: UTC (IST, +0530) NTP enabled: n/a NTP synchronized: no RTC in
  local TZ: no

For host running timedatectl:

  [root@indiadev72 ~]# timedatectl Local time: Tue 2017-08-01 10:27:29
  IST Universal time: Tue 2017-08-01 04:57:29 UTC RTC time: Tue
  2017-08-01 04:57:29 Time zone: Asia/Kolkata (IST, +0530) NTP enabled:
  yes NTP synchronized: no RTC in local TZ: no DST active: n/a

I am not sure why my docker container is not able to take the timezone of the host.

EDIT: Also while running some of my test cases, i am able to see that the application is reading timezoneid as 'GMT+530' instead of Asia/Kolkata. While GMT+530 is indeed Asia/Kolkata but due the timezone ID mismatch inside the container my test cases are failing. Maybe its just the timezoneID issue and the container is picking up the correct time. Can someone please help here?

The case is dpkg-reconfigure tzdata simply creates /etc/localtime as a copy, hardlink or symlink (a symlink is preferred) to a file in /usr/share/zoneinfo. So it is possible to do this entirely from your Dockerfile. Consider:

ENV TZ=Asia/Kolkata
RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone

And as a bonus, TZ will be set correctly in the container as well.

This is also distribution-agnostic, so it works with pretty much anything Linux.

The original answer: https://serverfault.com/questions/683605/docker-container-time-timezone-will-not-reflect-changes


Blacklist network ranges Docker is allowed to use for its containers (swarm mode) Basically I need to blacklist an IP range (and maybe multiple ranges) so that Docker can use any other IPs for its containers BUT IPs in this range.

Assume I have my docker nodes in a swarm of n nodes. They are in IP range 10.1.1.0/24. There are multiple other VMs in this IP range like a VPN Server, a firewall, etc.

Can I restrict Docker from assigning IPs from this range to containers it starts?

Docker should avoid using a network if there's a route to that network (see ip r). You can whitelist networks that docker is allowed to use with configurations in several places:

You can adjust bip in /etc/docker/daemon.json which controls the docker bridge network named bridge. This needs to be set to a gateway ip with CIDR notation.
You can adjust default-address-pools in /etc/docker/daemon.json which is a list of CIDR blocks for all other bridge networks.
You can pass --default-addr-pool when initializing a swarm to a CIDR block for overlay networks.

Make sure the two pools are large enough to create multiple networks, e.g. a /16 subnet to allow creation of 256 /24 networks.

More details of this are available in my DockerCon presentation (press P for presenter notes).


Minecraft docker-container refuses connection So I have this Dockerfile:
FROM openjdk:8u212-jre-alpine

LABEL maintainer "cowboy_patrick"

RUN apk add --no-cache -U \
  openssl \
  imagemagick \
  lsof \
  su-exec \
  shadow \
  bash \
  curl iputils wget \
  git \
  jq \
  python \
  tzdata \
  rsync \
  nano \
  sudo \
  knock \
  ttf-dejavu

EXPOSE 25565 25565

COPY minecraft /root/minecraft
WORKDIR /root/minecraft

RUN chmod +x /root/minecraft/start_minecraft.sh

CMD /root/minecraft/start_minecraft.sh

This just copies an existing world into an alpine with java and executes some prewritten scripts, I had written for running it on a vps (doing automatic backups etc.)
When I execute it with

docker run --network host -p 25565:25565 --name mcs mcserver

I can see the following in the container-dashboard:

For good measures you can see my console-output here but it's pretty standard. Just a minecraft-server that boots up.
The only thing I don't understand is why the connection gets refused when I try to join localhost:25565:

Does anyone have an idea on how I can allow this connection? And in extent allow connections from outside my machine (assuming that port-formwarding and everything works).

I’m glad my comment helped. I read the article OP mentioned in his answer and reread his question and I think the issue isn’t using -p instead of —publish as the two work the same, but specifying --network host. In Linux’s version of Docker daemon this prevents port publishing from working and since both Docker for Windows and Docker Toolbox use some sort of virtualization to run that version of Docker for Linux containers, I can only assume it has the same effect. The difference being that host in question is not Windows, but whatever Linux is running on top, so the container is completely unreachable.


New Kubernetes Cluster: remote error: tls: bad certificate This is my first attempt at setting up a Kubernetes cluster in my test environment. In preperation, I created 3 instances running Fedora Atomic:

10.11.184.8: master/etcd
10.11.184.5: node01
10.11.184.6: node02

Then using contrib/ansible playbooks for Ansible, I deployed kubernetes to my instances. It completed with "0" failures for each host.

I then connect to my master and begin to check that status of it:

[root@kubemaster ~]# kubectl get cs
NAME                 STATUS    MESSAGE              ERROR
scheduler            Healthy   ok
controller-manager   Healthy   ok
etcd-0               Healthy   {"health": "true"}

However, I go to check my nodes and it returns nothing:

[root@kubemaster ~]# kubectl get nodes
No resources found.

I begin to check logs, and I find the below repeating over and over from all three of the IPs listed above:

http: TLS handshake error from 10.11.184.5:32788: remote error: tls: bad certificate

It appears that the ansible playbook did generate some certificates:

[root@kubemaster ~]# ll /etc/kubernetes/certs/
total 40
-r--r-----. 1 kube kube-cert 1220 Aug 15 19:11 ca.crt
-r--r-----. 1 kube kube-cert 4417 Aug 15 19:11 kubecfg.crt
-r--r-----. 1 kube kube-cert 1704 Aug 15 19:11 kubecfg.key
-rw-rw----. 1 root kube-cert 4417 Aug 15 19:11 kubelet.crt
-rw-rw----. 1 root kube-cert 1704 Aug 15 19:11 kubelet.key
-r--r-----. 1 kube kube-cert 4917 Aug 15 19:11 server.crt
-r--r-----. 1 kube kube-cert 1704 Aug 15 19:11 server.key

And the kube-apiserver binary is being passed these as a parameter:

/usr/bin/kube-apiserver --logtostderr=true --v=0 --etcd-servers=http://10.11.184.8:2379 --insecure-bind-address=127.0.0.1 --secure-port=443 --allow-privileged=true --service-cluster-ip-range=10.254.0.0/16 --admission control=NamespaceLifecycle,LimitRanger,ServiceAccount,ResourceQuota --tls-cert-file=/etc/kubernetes/certs/server.crt --tls-private-key-file=/etc/kubernetes/certs/server.key --client-ca-file=/etc/kubernetes/certs/ca.crt --token-auth-file=/etc/kubernetes/tokens/known_tokens.csv --service-account-key-file=/etc/kubernetes/certs/server.crt --bind-address=0.0.0.0 --apiserver-count=1

If I view the logs on one of the nodes, I see a lot of messages like so:

Failed to list *v1.Node: Get https://10.11.184.8:443/api/v1/nodes?fieldSelector=metadata.name%3D10.11.184.8:443/api/v1/nodes?fieldSelector=metadata.name%3D10.11.184.5&resourceVersion=0: x509: certificate is valid for 10.254.0.1, 192.168.0.11, 172.16.63.0, 172.16.63.1, not 10.11.184.8

Is it likely that my certificates were not generated correctly? I am looking at this documentation. Perhaps I should attempt to manually create the certificates using openssl or easyrsa?

From the documentation you linked, it states,

  The referenced file must contain one or more certificates authorities to use to validate client certificates presented to the API server.

This means your client's certificate is invalid because either:

The server doesn't trust the client's signing certificate authority
The client doesn't trust the server's signing certificate authority
The certificate's DN doesn't match the hostname

The most likely scenario is that 1. The server doesn't trust the client's signing certificate authority since the server doesn't verify DNS for the client certificate and the error indicates this is a remote error not on the client. Most likely this means the client's certificate was signed on the client itself, not on the server.

You will need to make sure to generate the certificate on the client, have the server sign it and then transfer the certificate back to the client. Otherwise, you need to add the server's CA certificate to the Client's keystore.. You can ensure you do not have a DNS/DN mismatch by setting hosts file entries.


How to setup artifact filters to differentiate for deployment We would like to deploy only the artefacts that are built from our dev branch to the dev environment.

and master branch to be deployed to staging(stage) and production(PRD) environments

As a tool, we are using VSTS

https://imgur.com/QmBsyM5

How should we coordinate this process if we can not implement artefact filters in VSTS?

I am considering to create a separate container registry for master builds

https://imgur.com/a/eSF4EHx

Wanted to learn the best practice before i implement this

Azure DevOps supports artifact filters for each environment. Below is screenshot of how to enable the feature.


How/where are programs installed in Docker containers? Apologies for being such a basic question but all the resources I've found seem to take this knowledge for granted.

I'm brand new to docker and containerization in general.

Today when opening a program for a project i.e. RStudio, Jupyter, Notebook, Spyder or an IPython Console I would run a bash script in anaconda3/bin, either from a custom menu item or Anaconda Navigator or just the terminal.

How does this work with Docker? If I'm using a Docker container for my project and I install Anaconda and the other packages I use, is there a folder within the container containing the bash scripts or does it link to bash scripts installed on my system or something else?

Hi and welcome to DevOps SE and Docker!

Now the concept of Docker is to create isolated virtual  environments, actually it's just a process or more but wrapped in its own environment.

So at first hand you could take any Docker image and the scripts inside would be there where the creators of the image placed them.

If you decide to create your own Docker images, and I believe you will sooner or later, then it is up to you how layout the system.

Additionally, you could link - proper term is here mount - your system's folders with those of the container but still the execution will take place inside the container.

For example to install the RStudio package in an Anaconda3 environment, I have tried the following.

$ docker run -it continuumio/anaconda3 bash
(base) root@7162c3acc9e3:/# conda install -y -c r rstudio
(base) root@7162c3acc9e3:/# which rstudio
(base) root@7162c3acc9e3:/# /opt/conda/bin/rstudio

As you can see, if installed as root user which is default, the path is /opt/conda/bin/rstudio. Most packages not using special system capabilities would considerably not "know" whether they are in a Docker environment or not, so in many cases you could consider a Docker container to be just a (fluid) environment you have the root access too.

Additional note on the terms:

image is a blueprint for a container defined by a Dockerfile. "Installing" something in an image means you add instructions to its Dockerfile. Own images can be derived from others by the statment FROM.
container is basically process isolation in its own OS environment. If you install something in the container during its uptime, this change will be gone with the container.


Jenkins pipeline lock resources I need to run commands on my pipeline during a certain stage, on devices which connected to my Jenkins as slaves.
But I want them to run only when another one finished.
On the other hand, I want them to run all previous stages before they "wait" for another device to finish.

What is the best way to implement this lock? Again, it is the same job, but different slaves.

I found exactly what I was looking for-

[Jenkins plugin] (https://wiki.jenkins.io/display/JENKINS/Lockable+Resources+Plugin)

Which let you create a lock between, slaves , different jobs and more..

(in contrast to the plugin description)


Docker: Let containers communicate to each other but don't expose to internet I was recently shocked to find out that docker writes its own iptables rules and ignores ufw bindings completely. I had blocked all ports using the ufw deny incoming command and then selectively allowed only the ports that I wanted to expose.
I have a very simple setup that utilizes a postgresql container and a simple nodejs application. My primary goal is to keep the postgresql container port off the internet but let the node backend accept requests through port 3333. How do I do that?
My current docker-compose.yml file looks like this:
version: "3"

services:
  redis:
    image: redis:latest
    ports:
      - 6379:6379
    command: ["redis-server", "/redis.conf"]
    network_mode: host
  database:
    image: "postgres"
    env_file:
      - ./database/database.env
    volumes:
      - database-data:/var/lib/postgresql/data
      - ./database/init-db.sql:/docker-entrypoint-initdb.d/init.sql
    ports:
      - 5432:5432
    restart: always
    network_mode: host

  backend:
    depends_on: [database]
    build: ./server
    volumes:
      - ./files:/var/lib/files
    ports:
      - 3333:3333
    restart: always
    network_mode: host

volumes:
  database-data:

If you're looking to access your database only from localhost, you can do something like that:
    ports:
      - "127.0.0.1:5432:5432"


Custom inventory file failing to parse properly I've got the following production inventory file (playbooks/production.yaml) for Ansible:

all:
  children:
    control:
      moriarty.server.com
      toby.server.com
    managed:
      sherlock.server.com

Based on what I'm reading online, it looks like it should be correct, and I can't find any syntax errors in the file using a YAML linter. However, when I run ansible-playbook on my local machine, I get the following error:

$ ansible-playbook -i playbooks/production.yaml --extra-vars target=moriarty.server.com playbooks/create-server.yaml
[WARNING]: log file at /var/log/ansible.log is not writeable and we cannot create it, aborting

[WARNING]: Skipping 'control' as this is not a valid group definition

[WARNING]:  * Failed to parse /Users/username/infra/playbooks/production.yaml with auto plugin: no root 'plugin' key found,
'/Users/username/infra/playbooks/production.yaml' is not a valid YAML inventory plugin config file

[WARNING]:  * Failed to parse /Users/username/infra/playbooks/production.yaml with yaml plugin: control is not a known host nor group

[WARNING]:  * Failed to parse /Users/username/infra/playbooks/production.yaml with ini plugin: Invalid host pattern 'all:' supplied, ending in ':' is not allowed, this
character is reserved to provide a port.

[WARNING]: Unable to parse /Users/username/infra/playbooks/production.yaml as an inventory source

[WARNING]: No inventory was parsed, only implicit localhost is available

[WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all'

[WARNING]: Could not match supplied host pattern, ignoring: moriarty.server.com

PLAY [moriarty.server.com] ************************************************************************************************************************************************
skipping: no hosts matched

PLAY RECAP **************************************************************************************************************************************************************************

I'm at a loss, since it looks to be valid YAML. Do I need to format it some other way when I have groups in this manner?

The keyword hosts is missing. Also colons : are needed after the hostnames. Fix the syntax, for example
all:
  children:
    control:
      hosts:
        moriarty.server.com:
        toby.server.com:
    managed:
      hosts:
        sherlock.server.com:

See How to build your inventory.


How to setup a permanent storage volume for EC2 instance? I created an EC2 instance of r5ad.xlarge and installed Jenkins on it. However, after a few jobs I configured and ran I saw a warning about no space left. Then realised Jenkins was installed on 8 GB drive and that it's a non-persistent drive.

1st try: After reading on the subject I changed the 2 volumes of the machine with modify-instance to be DeleteOnTerminate to false and copied Jenkins to the bigger disk and configured auto-mount, following this and this. Then I stopped and started the machine and everything got deleted from the disk.

2nd try: So I defined a new EBS volume (SSD gp2)(with this article) and attached it to the instance. Again with DeleteOnTerminate=false, formatted then setup auto mount and copied Jenkins directory. Stopped and Started the machine, again everything gone...

What am I doing wrong? How do you make the attached volume persistent?

That 150GB disk that r5ad.xlarge comes with is known as ephemeral storage - its content is wiped when you stop the instance. If you need persistent storage you can use r5a.xlarge that doesn't have the ephemeral storage disk and simply give it a bigger EBS disk.

Either you can increase the size of the root disk during instance creation or create another EBS and use that. EBS volumes are persistent, they don't go away when you stop the instance (they may be deleted if you terminate the instance though).

If you don't see the previous content of EBS volume after a reboot make sure you're properly mounting it during boot (hint: /etc/fstab).

Hope that helps :)


How to get exposed ports from inside of container? I want to know which port(s) are exposed when user run the image.

docker run -p 1000:1234 myimagename

I want to get 1234 value in container since I will use it for some configurations(IIS bindings etc.) inside of container.

Is there any env variable for this purpose?

TL;DR: You don't.

From inside of the container, you do not know which ports are exposed since exposing a port is only documentation visible in the image and container metadata. To detect which ports are published, you'd need to know the host IP, which can be difficult to lookup depending on your environment, and then you'd need to do something like a port scan of the host and tcpdump inside the container to detect the incoming request.

All of this is non-trivial because it's solving the problem in the wrong direction. You build your app inside the container to listen on a specific port, configure that default port and potentially let users override it with a variable or config file. Then you document this decision be exposing the port in the Dockerfile with the EXPOSE command. This is only documentation, it goes in the image metadata, users and programs can see it, but by default docker will not do anything with it. Finally, users of your image will publish the port on the host when and where appropriate, and map to the port of your application.

In other words, the right side of the published port (the container port) is largely controlled by the image creator, not the user running the container. If they publish to the wrong port, it is not be the applications responsibility inside the container to dynamically adjust what port it is listening on, and this would be difficult to impossible to do if you had multiple ports anyway.


How to make Tiller highly available? I'm using helm to deploy my services to a Kubernetes cluster.
I want to improve the service availability of my tiller pod. So I scaled the tiller deployment to 2 replicas.
Helm is working as before with no issues that might be related to this.

Does anyone have an idea why this might NOT be a good idea to increase reliability of helm and tiller?

EDIT:

Helm v3 is expected to drop the usage of tiller. Making this issue not relevant
If you want a tillerless experience in helm v2, you can use a helm-tiller plugin

Running two tillers does not make it HA. There is an open issue at the moment.


How to filter by entity name using sensuctl? Using sensuctl for sensu-go, I want to be able to search for entities by name, preferably using wildcards.

I've tried this:

.\sensuctl.exe entity list --field-selector "testfilter in entity.name"

This still shows all of the entities regardless of the name.

It seems that --field-selector may only be available in the licensed version and that may be why this isn't working.

Using some shell commands in cmder to parse the json and filter the results I was able to filter the results. Here is the command:

.\sensuctl.exe dump entity --all-namespaces --format wrapped-json | jq ".metadata.name" | grep -i <filtertext>


What is the most used tool for Cloud Infrastructure deployment Ansible vs Puppet vs Terraform? I have some time for research at work right now, and I was thinking on expanding my knowledge on my favorite subject, which is DevOps (love you all).

I have some advanced experience with Ansible, as it has always been my choice when preparing and provisioning hardware in my Cloud tenancies. I started working with Ansible because it was the first software that popped up based on a quick SE search, a couple of months ago. However, I'd like to know how Ansible compares with Terraform and/or Puppet, in the following aspects:

Which is the most commonly used tool between the three in the DevOps community, more specifically when deploying Cloud infrastructure to customers, and why is that?
Which of the three use the simplest syntax (to my knowledge, Terraform is best at this)? I’m looking for a human-readable syntax if possible which makes things easier to remember.
Which is most respectable in the DevOps community, with regard to community standards, and why? Do any of these tools have a feature that makes them stand out from a development POV?

Thank you as always for your time.

@jasper welcome!

This really might depend on the context. On long term it's worth to take a deeper look for a moment if you are not after quick shots and go beyond experiments for a sustainable (and large) infrastructure.

So we have got many clouds out there. Deployment automation means, they have some API.

So what does make a deployment automation tool delivering infrastructure as code capability?

(1) Unified, programmable, configurable abstraction layer, also called domain specific language (DSL).
(2) An object model of (multiple) cloud assets and available services.
(3) Corresponding logic to execute actions on the assets typically: create, read, update, delete (CRUD) and synonyms.

Now there are some challenges to address human readability and sustainability varying by programming language ecosystems and communities:

Cloud products being themselves computer infrastructure and network engineering (these are different disciplines!) abstractions are not following some official "cloud domain" ISO/IEEE naming standards as far as I know. That is, all the names are pushed by vendors and might be marketing biased. So either you invent your own terminology or decide which terms are most generic, or a mix of that. There is therefore a good reason that you find also cloud vendor specific deployment languages and tools for example like Cloud Formation by AWS.
Somebody needs to maintain related components to support new API versions for every cloud platform!

Some deployment tools delegate the cloud driver (that's another term for cloud platform specific code bases) to separate projects, which in their term delegate responsibility for API compatibility to one or another boilerplate (=often boring and tedious) API implementations.

Therefore my conclusion is that depending on your purposes the most straight-forward way to write infrastructure as a code is the programming language of your choice using a well-supported SDK which is though vendor-specific!

Still this is the point where you have got greatest flexibility because for a small standard use case you could write your own abstraction layer and your custom logic whatever you want.

Python examples:

boto3 for AWS
exoscale/cs for Apache CloudStack
python-openstackclient for Apache CloudStack

Recent Terraform example: its Apache CloudStack driver has backward compatibility problems because it's based on some less-visible CloudStack API SDK written in Golang. I am sure you can run into problems of this sort with any deployment tool; so if you commit to one, bring enough courage and patience to connect with and contribute to the corresponding community (same is for SDKs but they at least typically have less abstraction).

That is, before deciding for a certain cloud deployment automation tool, consider besides the things you see on the surface like syntactic sugar and community support also the following aspects:

If it's open source, is it written in a programming language where you/your team could create an emergency patch if things go really worse?
How well supported are those cloud drivers critical to your business: these typically can't be all of them, if so, you might run into issues like GitLab with Docker Machine (also a specific deployment tool after all), where a significant technical base for a competitive product feature suddenly breaks away because all the responsibility in this context had been factually outsourced to the open source community.


Issue integrating acunetix with jenkins The first of all, thank you for reading my question, I really appreciate it! I am trying to integrate jenkins and acunetix (a vulnerability scanner software), but it seems to be near impossible...My situation is as follows:

There are two machines, a debian 9 machine [DEB] and a win10 machine [WIN]. In [WIN] I installed Acunetix v11 and jenkins in [DEB], and I configured acunetix to be accessible from outside (following this tutorial: https://www.acunetix.com/blog/docs/use-acunetix-host-localhost/). After that, I followed another tutorial (https://www.acunetix.com/blog/docs/install-acunetix-root-certificate-another-computer/) to connect jenkins with acunetix. I managed to access to acunetix webpage from [DEB] via web (Not issues in network or configuration in that sense).

However, I could not connect with acunteix from jenkins despite the fact I followed every single step explained in acunetix web page. I got this error: "Please add the Acunetix scanner certificate to Java CA store" when I tested the connection from jenkins (obviously, acunetix plugin is installed in jenkins). I checked that the certificate obtained from acunetix installation (in [WIN]) "ca.cer" was correctly added in the keystore of [DEB]. Then, I made a custom keystore to be used by jenkins (as it is explained in this other tutorial: https://support.cloudbees.com/hc/en-us/articles/203821254-How-to-install-a-new-SSL-certificate-) and I got the same result...At that time, I thought that maybe something was wrong in [DEB], so I used another windows 10 machine [WIN2] to repeat the process...and I got the same result... The acunetix certificate was also included in the OS apart from java keystore in all cases.

My company contacted Acunetix support about this issue but no solution has been provided so far... Currently, we don't what to do right now... Have you dealt with this? Have you found a solution?

Thank you in advance, mates!

See you!

We've configured the plugin properly (we're able to invoke acunetix from jenkins although we're having issues getting the report copied into the Jenkins workspace). The issue he might have run into is that you might have more than one jre installed and you have to register the certificate on the keystore JRE that Jenkins is using (you can check which one is being used by jenkins by looking at the "<executable>" tag in the jenkins.xml config file)

In a default installation the jre being used by jenkins and its keystore are located inside the jenkins folder: .\jenkins\jre\bin\keytool and .\jenkins\jre\lib\security\cacerts

With this configuration, you can verify if the certificate is in the keystore by using the command:
keytool -list -keystore "jenkins\jre\lib\security\cacerts"


View logrotated log continuously Is there a command to view logrotated logs as a continuous stream with proper ordering?

I need to view all entries in syslog in historical order starting from the oldest entry available. So far I was inspecting logs one by one manually.

-rw-r-----   1 syslog    adm             140368 Jun 12 09:41 syslog
-rw-r-----   1 syslog    adm              45498 Jun 12 06:25 syslog.1
-rw-r-----   1 syslog    adm               1874 Jun 11 06:25 syslog.2.gz
-rw-r-----   1 syslog    adm               1758 Jun 10 06:25 syslog.3.gz
-rw-r-----   1 syslog    adm               1615 Jun  9 06:25 syslog.4.gz
-rw-r-----   1 syslog    adm              28494 Jun  8 06:25 syslog.5.gz
-rw-r-----   1 syslog    adm              92359 Jun  7 06:53 syslog.6.gz
-rw-r-----   1 syslog    adm              13849 Jun  6 09:21 syslog.7.gz

If all the files are there without any extension then yes you can execute the cat command with * (cat ) it will display the content of each and every files underneath this directory, but here in your output it has the zip files (.gz) too, so you need to execute 2 commands rather than single command...

cat syslog syslog.1
zcat *.gz

Hope this will helps.


How is it a good practice to version your helm charts for each deployment? I'm trying to set up a process to release a web app with helm in my company and I can't find a satisfying way.

Basically each for each push in a feature branch, I want to:

Build a docker image and push it to a registry
Upgrade the app through helm

Step 1 is easy, we keep the Dockerfile in the sources, which makes sense because it allows developers to test the docker image and the CI can also build the image and push it automatically.

Step 2: I'm not sure. I was told it could be interesting to keep track of the deployment history and therefore to update the helm chart version for each deployment which seems do-able but I'm not sure I see the point because helm history already provides the info I guess?

Also it forces the CI to actually commit and push the new chart version itself!

My idea was to keep the helm chart in a separate repository with all our charts and to just use keep track of the changes of the chart but nothing to do with any deployments so: the CI could basically build the docker image, push it, clone the helm repository and do the following:

helm upgrade $APP_NAME --set image.tag=$APP_VERSION ./path/to/chart

(with APP_VERSION being auto generated and matching the docker build tag)

I haven't seen this approach when reading about it, most people seems to keep their app version up to date in their helm charts and I'm wondering how is that supposed to work for continuous deployment?

Approach you are describing - where app version is substituted at helm upgrade command - is pretty popular as well. Maybe it's not well documented - I'm not sure about that.

Yet another approach that people use is doing helm template which would compile everything (including variable resolution) into fixed k8s definitions and then doing GitOps on that.

There is no universally accepted approach here among those 3 (1st being updating helm chart itself with new versions every time). So you should decide what works better for your use case.

One thing that is worth mentioning is versioning issue (disclaimer: I'm solving it with Reliza Hub, which I develop). Basically, you need some system which would tell you what version tag of each microservice to actually use on different environments. That problem may become quite complex, especially if you have many microservices and many environments.


How to solve `Error creating config for VM: Invalid JSON` in minishift? minishift start

was issued and openshift started:

Login to server ...
Creating initial project "myproject" ...
Server Information ...
OpenShift server started.

The server is accessible via web console at:
    https://192.168.42.37:8443/console

You are logged in as:
    User:     developer
    Password: <any value>

To login as administrator:
    oc login -u system:admin

-- Exporting of OpenShift images is occuring in background process with pid 32067.

At a certain moment, the laptop ran out of free disk space. Once this issue was solved, an attempt was done to restart the minishift, but all commands return:

Error creating config for VM: Invalid JSON

Attempts to solve the issue

It turned out that a minishift process was still running. Killing this process and subsequently run minishift status returned the same issue.
sudo systemctl restart libvirtd did not solve the issue
@tm1701 suggested minishift delete --clear-cache, but that does not solve the issue either

One option to solve this issue is a suggestion as provided by @AndreasGajdosik, i.e. deleting the .minishift directory.

Once rm -rf ~/.minishift was issued and minishift status was run, the issue has been solved:

user@host$ minishift status
Does Not Exist


How to get Docker Container stats using DataDog API? I have a running DataDog-Agent on a Docker server.

Is there a function in the DataDog API to get the stats of every single container, so I could print the stats to an external Webpage?

Or is there an alternative?

I'm not sure I understand what you're trying to do, but .. DataDog has a Docker Intergration, which when configured provides a status of every single container on a host. Then, by simply querying the DataDog metrics, you can see if  a particular container is up or not, or how many containers you have available per image, or what's the CPU consumption etc.


Trigger a cross project GitLab trigger I am trying to call a trigger on one GitLab project using a job on another.

Let’s call one A, and the other B. I want a job in A to trigger a job in B.

In A, this is what my .gitlab-ci.yml looks like:

deploy:staging:
  image: alpine
  variables:
    GIT_STRATEGY: none
  script:
    - apk add --no-cache curl
    - curl -X POST -F "token=$CI_JOB_TOKEN" -F "ref=$CI_COMMIT_REF_NAME" https://gitlab.com/api/v4/projects/1234/trigger/pipeline

In B, this is what my .gitlab-ci.yml looks like:

image: google/cloud-sdk

deploy:staging:
  only:
    - triggers
  script:
    - echo "$CI_ENVIRONMENT_SLUG"

However, when A runs, I get the following result from curl:

{"message":{"base":["Reference not found"]}}

When I hardcode the ref to master, I get this message:

{"message":{"base":["No stages / jobs for this pipeline."]}}

How can A be configured, so that the deploy:staging job in B is triggered on master?

Thinking ahead, I would like to add more deployment targets, so it should only trigger the deploy:staging job.

In your job, I noticed the below curl command, where two things need to be corrected:

token=$CI_JOB_TOKEN - this TOKEN should be generated in project B under Settings -> CI/CD -> Pipeline triggers (Add triggers)
"ref=$CI_COMMIT_REF_NAME" - this is pointing to a branch with name $CI_COMMIT_REF_NAME on project B and this will only execute if it find a project with that branch name, so if you are trying to trigger it on master branch or any specific branch, you should replace it with something like -F "ref=master" in your curl command

 - curl -X POST -F "token=$CI_JOB_TOKEN" -F "ref=$CI_COMMIT_REF_NAME" https://gitlab.com/api/v4/projects/1234/trigger/pipeline

I use the same trigger with few variables to execute one of my job in another project

- "curl -k -X POST -F token=$PROJECT_B_TOKEN -F ref=master -F variables[PROJECT_NAME]=$CI_PROJECT_NAME -F variables[TAG]=$CI_BUILD_REF_NAME -F variables[SOME_VARIABLE_IDENTIFIER]=DevOps -F variables[USER]=$GITLAB_USER_LOGIN https://gitlab.mycompany.com/api/v4/projects/1234/trigger/pipeline"

The person who created in token in project B, needs to be owner in project A to execute this trigger, else it fails with a 404 message.


Application version and Database Schema version mapping I am trying to integrate database schema versioning in an existing project where schema is not versioned at all.

Is it a good practice to create a tight mapping between application version and database schema version so much so that the application deployment will fail if the schema version is not the one expected by the app version? This is because the application has frequent schema changes.

What are the pitfall of using such a logic in application init code?

Honestly, you have several questions to unwrap and there is no perfect guidance.  Database versioning is difficult at best.  However, there is guidance that I have been successful with.  In all cases, starting where you are as "version 0" is your best bet.

For your first statement, use one of the following methodologies (or a combination that works best):

EntityFramework Code First Migrations - Treat the current state of your schema as well as any reference tables as version 0.
DbUp and the DbUp Azure Pipelines tasks - This is a very simple, but effective way to get a "poor-man's" database migration methodology, if you follow some simple rules/conventions.  There are also ways to integrate "running" DbUp migrations as part of the start-up of your application, too, but I've had success executing DbUp migrations via Azure Pipelines releases.
Red Gate Database Lifecycle Management (DLM) Automation Suite - This requires a bit more planning and comes with a cost, but Red Gate's tools integrate well and give you more of a complete DLM picture from integration of SSMS with Git version control, drift detection, Azure Pipelines tasks, and more.

As far as the question of should you couple versioning of the database with the application, you'll need to answer this question.  Is the database shared between more than one application or is it dedicated to that application.  If "dedicated" is your answer, then versioning of the database with the application is the right thing to do and should be done so within the same source code repository.  If not, it will be difficult to decide which application "owns" the versioning of the database or if it needs to be versioned entirely independent of your apps.  If it is versioned independent of your apps, then you better have strong unit testing in all of the apps.

Lastly, I will assume your "pitfalls of using such logic" refers to using your applications initialization/startup code to run an upgrade of the database to the intended version.  If I've assumed correctly, there is no right/wrong way.  What matters is that you understand where you need the automated process to fail to ensure your application does not incur a downtime because of a failed database migration.  If you upgrade a DB during initialization/startup of an application, then your app better have a fallback mechanism where it can function with the previous version of the DB until which time it's fixed.  This is where I'd recommend making failure of a DB upgrade part of the deployment process itself and not part of the application.  That way, if the DB upgrade fails (and you're doing it correctly), reverting to the previous version of the app upon failure ensures that downtime is minimized.

And, when I refer to doing DB upgrades "the right way," I refer to using methods that ensure the state of the DB schema is only changed atomically and transactionally such that failures in the middle of a deployment can be completely rolled back.


What are signs to hire a CIO? If for example the number of complex Java issues is over growing exponentially on the backlog this could be a sign to hire a Senior Java Engineer, but what are signs for hiring a CIO?

The sign to hire a CIO is that your CEO is not able to handle the relevant tasks on his own, plain and simple. The reasons for that may be manifold - maybe he has too much on his plate, as the company grows, or he does not have a sufficiently technical background, etc.


Why do I need a Nat Gateway for ECS containers using AWSVPC? So here is a scenario:
I have a route table that routes all traffic to the internet gateway, and I have a subnet associated with that route table, which makes it a public subnet. Now I create another route table and I route all its traffic to a Nat gateway that I created I created in the public subnet abovesubnet. Now I create another subnet in this private route table, and associate that with my ECS containers. So in effect, this is how the traffic is being routed:

But what is wrong with doing this:

Why doesn't associating it with the internet gateway directly work? The first one seems like the same thing but with extra steps. I would really like to know why exactly that is necessary, and what happens "under the hood" in both cases, and what causes the second way to not work.

While this is completely possible, it would mean that your instance would be accessible from the internet. Public subnet means that your instances are accessible directly  from the outside. Private subnet are not accessible from the outside as instances do not have a publicly  routable IP.
In short, security


aws - Connect two private subnets? I'm testing some cloudformation infrastructure templates, and I noticed that most of the examples include routing tables to connect private subnets to public subnets, but my question then is:

if two private subnets exist in different AZs (and are attached to individual routing tables) and the resources within them may need to access a resource on the other subnet (for example a private service on the subnet 1 accessing a database on the subnet 2)

Should I create a route to connect the two private subnets in each of their routing tables?
Would it be a better practice to attach both subnets to a single shared routing table (both CIDRs are contiguous)?

Are they in the same VPC? If yes it should just work as the VPC CIDR route will take care of all the subnets intercommunication.


Load Balancing of Jenkins Nodes without Load Balancing Plugin? we currently want to introduce a loadbalancing on the Jenkins server and our nodes. I have already created several nodes.

Currently it is also so that if a node is blocked accordingly, that the next free option is chosen. But only if a node is issued. Unfortunately, this is only part of what the whole task looks like.

Task to be solved:

If a node is under load, the next free option should be selected automatically.
If a node is issued, then the next inactive node should be taken.

Problem:

For security reasons, you do not want us to use the loadbalancing plugin. So there has to be a solution that we have through a script or some other solution

This is similar to:
https://stackoverflow.com/questions/55023570/how-to-run-a-job-on-first-available-node/55567958

Have you considered Least Load plugin?

https://plugins.jenkins.io/leastload

By default Jenkins tries to allocate a jobs to the last node is was executed on. This can result in nodes being left idle while other nodes are overloaded. This plugin overrides the default behavior and assigns jobs to nodes with the least load.

Don't imagine there's a security issue there.


Terraform AWS AZ error - manually specify AZ? I'm fairly new to terraform and when creating AWS infrastructure I use the following to fetch the 'available' availability zones for my region (us-east-1) where I'm trying to create an EKS cluster.

data "aws_availability_zones" "available" {}

However when I run a 'terraform apply', it generates the following error because us-east-1a is at capacity.

aws_eks_cluster.demo: error creating EKS Cluster (test-cluster):
UnsupportedAvailabilityZoneException: Cannot create cluster 'test-cluster' because us-east-1a, the targeted availability zone, does not
currently have sufficient capacity to support the cluster. Retry and
choose from these availability zones: us-east-1b, us-east-1c, us-east-1d

Is there any way I can get terraform to handle this automatically or specify that I only want to use 1b,1c,1d etc.

EDIT:

I've realised this is tied with where I create my subnets, which I do using this line:

 availability_zone = "${data.aws_availability_zones.available.names[count.index]}"

So I assume I somehow need to skip us-east-1a by doing something here?

Thanks in advance.

If anyone else has this issue, here's how I got around it.

Specify a variable with the availability zones you want to target

variable "zones" {
  default = ["us-east-1b", "us-east-1c", "us-east-1d"]
}

When creating your subnets, use this line

availability_zone = "${var.zones[count.index]}"


AWS launch template create-before-destory? In terraform, to reduce downtime when creating a new auto-scaling-group behind an existing load balancer, you can use a launch configuration and set create-before-destroy.

Launch templates are the recommended/new replacement for launch configs... but I can't seem to figure out how to achieve the same thing.

How can you mimmic this behavior in launch templates?

It seems you're misunderstanding what the create-before-destroy lifecycle parameter of the launch configuration does.
It's about replacing the launch configuration within an autoscaling group, not replacing the autoscaling group.

Launch templates have versions which makes this parameter useless as you'll create a new version of the launch template before updating the ASG to use it.

Terraform gives an example in the ASG documentation.

The underlying idea is to be able to roll back more easily than with launch-configurations.


Why does my Docker composer not work with volume? I'm trying to create a volume to be shared among the containers, and this volume binds to a location of my host, the problem is that every time I run the docker-compose up gives the  following error:

In file '. \ docker-compose.yml', service 'volumes' must be a mapping not an array.

and so far I didn't understand why.  Following is the compose file:

version : '3.4'

services:
    testando-volume-compartilhado-a:
      image: carloshenriquecarniatto/teste:latest

    volumes:
       - D:\App:/app
    ports:
        - "10001:80"
        - "44378:443"

It seems there is a spacing issue in your yaml

You should change the compose file to

version : '3.4'

services:
    testando-volume-compartilhado-a:
      image: carloshenriquecarniatto/teste:latest

    volumes:
        - D:\App:/app  # <- extra space here
    ports:
        - "10001:80"
        - "44378:443"


Pass information between Jenkins Job builds Please excuse my non-fluent terminology..
There is a Jenkins server, master and a few slaves, there is a pipeline job.. let's call it playground
I run the job once, I want to save some data somewhere.
Run the job again, I want to get the data from previous build.

Given that the Jenkins server is bounded to a VPN
Given that the Job may run on master or any of the slaves
Given that the Job deletes the workspace with every build

I couldn't find a way to save and load the data..
One plugin that actually do this is the git plugin, that saves the commit id and reads it in the next build for the changelog
I went through the git plugin code.. I am very confused and cannot figure out how or where it is done!
Can someone point me in the right direction.

There's a simple two step process...
In your originating job, tick step [ X ] Archive the artifacts and set the files to archive.

The files are copied to the master, stored in the jobs folder tree, under the builds directory for each build, along with the log. (Delete the build and it deletes the artifacts that came from it).
Install the Copy Artifact plugin.
Configure the job accordingly.

Despite the plugin's screenshot example, this is actually a bad practice for artifacts like jars. Those should be properly managed using an artifact repository like Nexus or JFrog.
See my response regarding keeping the the Jenkins space under control, managing the build and artifact retention soundly.


GCP - Can't create a tpu on europe-west1 I'my trying to create a tpu for my vm instance from this tutorial but get this error message:

ERROR: (gcloud.compute.tpus.create) PERMISSION_DENIED: Location europe-west1-b is not found or access is unauthorized.

I have enabled all of the access the vm can have but i still have this error.

Thanks for the help

According to "TPU types and zones" TPUs are only available in a very limited set of regions, namely:

Europe, Middle East and Africa

europe-west4-a (Eemshaven, Netherlands)

Americas

us-central1-a, b, c and f (Council Bluffs, Iowa, USA)

Asia-Pacific

asia-east1-c (Changhua County, Taiwan)

Note: not every SKU is available in each region, so double check the above page when planning your deployments.

In addition to specifying a valid zone, you will need to make sure the TPU API is enabled, and your account has the relevant IAM permissions: Cloud TPU\TPU Admin.

At the moment TPUs are predominantly designed for centralised data processing rather than close-to-edge processing. Hence the somewhat limited number of locations TPUs are available within.


How to start a VPN connection using Jenkins? I'm on a Windows Server and I want to have a pipeline project on Jenkins that basically does this:

SVN update
Build
Deploy

As the deploy target machine is on a different network, I want to know how to start a VPN connection before to do the deploy.

Note: My general goal is to automate deployment to a server in a VPN. Currently the VPN is Cisco AnyConnect.

solutions is: OpenConnect Plugin

See here: https://wiki.jenkins.io/display/JENKINS/OpenConnect+Plugin

OR also you can use below method:

On Linux (Debian based)

sudo apt-get install openconnect

Add the following lines to the bottom of /etc/sudoers (ubuntu configuration)

jenkins ALL=NOPASSWD:/usr/sbin/openconnect*
jenkins ALL=NOPASSWD:/bin/kill*

Then you will get an option in the Jenkins run task called:

Connect to Cisco AnyConnect VPN


Bugzilla 4.2 to 5.0.4 'Can't locate parent.pm in @INC' I've just updated Bugzilla from 4.2 to 5.0 on our Centos 6 vps, using the tarball method:
bash$ cd /var/www/html
bash$ wget https://ftp.mozilla.org/pub/mozilla.org/webtools/bugzilla-5.0.4.tar.gz
bash$ tar xzvf bugzilla-5.0.4.tar.gz
bash$ cd bugzilla-5.0.4
bash$ cp ../bugzilla/localconfig* .
bash$ cp -r ../bugzilla/data .
bash$ cd ..
bash$ mv bugzilla bugzilla.old
bash$ mv bugzilla-5.0.4 bugzilla

I'm getting the following error from ./checksetup.pl:

Can't locate parent.pm in @INC (@INC contains: . lib /usr/local/lib64/perl5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5) at Bugzilla/Constants.pm line 14.
BEGIN failed--compilation aborted at Bugzilla/Constants.pm line 14.
Compilation failed in require at ./checksetup.pl line 27.
BEGIN failed--compilation aborted at ./checksetup.pl line 27.

What did I do wrong and/or how to fix?

I needed to install a new perl module

yum install perl-parent

then I had to install make to install other dependencies:

cpan
o conf make '/usr/bin/make' #or path to your make
o conf commit


How to backup Amazon Cognito? Amazon Cognito is a user management and authentication service provided by Amazon through Amazon Web Services:

  Amazon Cognito lets you easily add user sign-up and sign-in to your mobile and web apps. With Amazon Cognito, you also have the options to authenticate users through social identity providers such as Facebook, Twitter, or Amazon, with SAML identity solutions, or by using your own identity system. In addition, Amazon Cognito enables you to save data locally on users devices, allowing your applications to work even when the devices are offline. You can then synchronise data across users devices so that their app experience remains consistent regardless of the device they use.

What you are effectively doing is storing your customer data in a closed system hosted by Amazon.  If you are a company, such as a social network, who's data represents a significant asset within your organisation how to you ensure that this data is protected from:

Amazon Cognito being shut down unexpectedly, i.e. at request from a law enforcement or government agency.
Your access keys being compromised and some or all records being corrupted.
The relationship between your organisation and Amazon breaking down to the point Amazon withdraws all services with immediate effect.

Amazon Cognito is a huge benefit to organisations and developers as it takes a particularly hard piece of software and makes it easy to implement; from a service operability perspective, we need to find ways to mitigate, remediate or accept risks to the companies assets at all times.

The Amazon Cognito streams feature can be used to backup data.

Currently, Amazon does not provide a solution to backup their Cognito user Pools. You can use the following NPM package called "cognito-backup":

Install:

npm install -g cognito-backup

Usage

cognito-backup backup-users <user-pool-id> <options>  Backup all users in a single user pool
cognito-backup backup-all-users <options>  Backup all users in all user pools for this account

Examples

cognito-backup backup-users eu-west-1_1_12345
cognito-backup backup-users eu-west-1_1_12345 --region eu-west-1 --file mypool.json
cognito-backup backup-all-users eu-west-1_1_12345 --region eu-west-1 --dir output

Source: https://www.npmjs.com/package/cognito-backup


Access vault secret from bash script I am looking for help on a direction on where to go from here. I have a Hasicorp Vault server running and accessible over DNS (local). I need to have a script get a secret from it then use that secret. I am not sure where to go from there. I know I want to use a bash script for this but past that I don't know what to do.

Until now I have been using curl to test it.

Suggestions?

The vault docs mention a -field parameter for the read subcommand.  So you should be able to put this into a shell script:
SECRET=$(vault read -field foo secret/mysecret)

Other vault docs use the vault kv get in the same way so you might try:
SECRET=$(vault kv get -field foo secret/mysecret)


Where should I put my application configuration? I've been reading lately a debate about "Where should properties that depend on the environment be stored?".

The classical way is to have multiple property files, one by environment, and based on an environment variable (DEV, PROD...), you choose where to read them when starting the application (like with Spring profiles).

On the other hand, if you are using a container to deploy your application, it's said that this kind of configuration should come from the environment itself (using environment variables that the application reads), so the image doesn't change between environments.

What are the pros and the cons of each approach? Is there a "best" approach for the container scenario?

Who said that properties files and environment variables where mutually exclusive?

There is a distinction to be made between "where do I store my app configuration?" And "where does my app source it's configuration? "

The most likely outcome is that everyone probably should just keep doing what they are doing with configuration files as a storage mechanism (think long term, persistent state for as long as the environment exists).

However, rather than dropping that configuration file into the application context and letting it run the application should be able to just expect those variables to already be available in the environment when it starts up.

This means you need to have two deployment work flows -

I deploy may application into an environment by going through X change control process and doing Y reviews with Z tool, whatever.
I deploy my environment configuration into an environment by going through A change control process and doing B reviews with C tool, same process, different outcome.

To use an example of managing environment variables as Key Value pairs in a tool like consul, if you are storing configuration files in git then tools like git2consul with handle getting that configuration into the environment when it's updated.

If you have an app that is expecting that there will be config available as a configuration file then you can avoid shipping multiple copies of the configuration file with the app by building a deploy process with something like consul-template which has the capability to turn your consul values back into a file.


How to know where a docker image came from? I have a Docker image on my system 12346789ABCD.

Is there a way to know where it is coming from?

Specifically, I would like to know if it was built locally or downloaded from a repository, and in that latter case, downloaded from which repository.

Typically, if it's from a registry, you see that in the name of the image (if it's not from the default registry, Docker Hub). Images from Quay look like quay.io/<namespace>/<image name>:<tag>.

Some images are from from anywhere else at all. For example images that are FROM: scratch and images created via docker import will not be associated with any registry. These are the famous none:none images (no repository and no tag). In this case, you only have the ID (SHA) and you need do be Sherlock Holmes (or Jessie Frazelle) to match it with something.

So, you can only see what is written in the manifest. Should you trust it? The manifest can be signed, using docker content trust even if the image contents themselves are not.  If you need more certainty on the provenance, you probably need notary  to ensure that trust is maintained throughout the delegation chain.

Hope that answers the question.


Escaping curly braces in Bamboo In Atlassian Bamboo, in script tasks you can use special Bamboo environment variables e.g. for current build number

${bamboo.buildNumber}

Now I have the impression if I have a code piece  like ${VAR} which I want to pass to other context, Bamboo's templating logic will kick in and render this part as an empty string.

Proof:

echo "my foo ${VAR}!"

Output:

01-Feb-2018 20:47:58    my foo !

Is there a way to escape curly braces in this context?

What I have tried:

googling it: nope
${{VAR}} - bad substution error
$\{VAR\} leads to exactly same output as it escaped which I do not want to have either.

UPDATE from comments - making sure there is no existing variable being just empty.

Input:

echo "my foo random ${RANDOM_VAR_ABCFOO}!"
echo "my foo random with prefix ${bamboo.RANDOM_VAR_ABCFOO}!"

Output:

Bamboo 5.13.1

02-Feb-2018 10:26:11    my foo random !
02-Feb-2018 10:26:11    /home/bambooagent/temp/FOO-JOB1-4-ScriptBuildTask-3130782940072218698.sh: 2: /home/bambooagent/temp/FOO-JOB1-4-ScriptBuildTask-3130782940072218698.sh: Bad substitution

Bamboo 6.3.0

02-Feb-2018 10:22:02    my foo random !
02-Feb-2018 10:22:02    /data/bamboo-agent/temp/FOO-JOB1-24-ScriptBuildTask-7913400670965638044.sh: line 2: my foo random with prefix ${bamboo.RANDOM_VAR_ABCFOO}!: bad substitution

I don't know the inner of bamboo, but I assume it works like a bash script.

What happens is that variables are replaced before execution, in bash that would be echo "echo $VAR" > test.sh and test.sh will only contain "echo" because VAR is replaced by it's value before the command is executed.
If you want test.sh to contain echo $VAR you have to tell bash to ignore the replacement on the first call by escaping the $ sign:

echo "echo \$var" > test.sh will give echo  $VAR in the file.

In the same note, the notation ${VAR} and $VAR are the same, using braces is a good practice when you do concatenation like in echo "Size is ${VAR}Kb" as without the braces like this $VARKb bash would try to find a variable named VARKb and return an empty value.

To address PrestonM different behavior, I assume there's either
 - the fact running under windows with the powershell interpreter doesn't behave the same
 - or just that bamboo has a special case for variables prefixed with bamboo. and as such escape them.

If someone want to test you may try:

echo "${VAR}"
echo "${bamboo.VAR}"
echo "\${VAR}"

and edit the results in this answer.


docker-compose invalid type, it should be a string I have the following environment variable configured in a docker-compose.yml file:

version: '3'
services:
  server:
    ports:
     - 13045:3000
    environment:
     - NODE_CONFIG: '{"DATABASE_URL":"http://db:5984"}'

When trying to run docker-compose up, I'm getting this error:

services.server.environment contains {"NODE_CONFIG": "{\"DATABASE_URL\":\"http://db:5984\"}"}, which is an invalid type, it should be a string

I need the environment variable to be set to a JSON string (see https://github.com/lorenwest/node-config/wiki/Environment-Variables#node_config)

Am I doing something wrong here? Can I get this to work somehow?

The Docker Compose file reference states, that environment variables are defined as VARIABLE=value array elements. For your case, the docker-compose.yml file would need to be changed to this:

version: '3'
services:
  server:
    ports:
     - 13045:3000
    environment:
     - NODE_CONFIG='{"DATABASE_URL":"http://db:5984"}'


How can I make artifacts available to the QA team with a user-friendly UI? Currently developers push to master, jenkins picks that up, builds it, and then I go to artifacts, download the APK files and upload them in a Microsoft teams group so that the QA team can download either the latest or any previous version I had uploaded before.
I want to change this process to make it more efficient. I want something with a UI that the QA team can go to and download whichever version of the app (APK files) they want.
I want it all to be automated, so as soon as I push the master, Jenkins will build and the files will eventually show in this portal, webpage or section of Jenkins and the QA team can download which ever version they want.
How can I achieve this? Is there a Jenkins plugin for this?

It seems to me that you're looking for an artifact store such as Nexus or Artifactory.
Both solutions have Jenkins plugins which will simplify the process of uploading your artifacts to them.
You can find an example as to how to upload artifacts from your Jenkins pipeline to Nexus in this link and an example for Artifactory in this link.
Both solutions are very common and relatively easy to manage and use and are known by the community for a long time so you can find many tutorials that explain how they can be used and benefit you.
Then, you just need to allow your QA team read access to the relevant artifacts repo and they can download whatever version of the artifacts they need.


How to install Certbot plugins? I installed Certbot by following this tutorial:

sudo add-apt-repository ppa:certbot/certbot
sudo apt-get update
sudo apt-get install python-certbot-nginx

And now I want to set up my cert, but I need the dns-digitalocean plugin:

# certbot certonly --dns-digitalocean
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Could not choose appropriate plugin: The requested dns-digitalocean plugin does not appear to be installed
The requested dns-digitalocean plugin does not appear to be installed

I tried installing it with pip:

pip install certbot-dns-digitalocean

But apparently the apt-get version isn't "seeing" it.

How do I install it properly?

Better method, thanks to the others responses for helping me get to this.

Determine what plugins are installed currently:

# certbot-auto plugins
Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
* apache
Description: Apache Web Server plugin - Beta
Interfaces: IAuthenticator, IInstaller, IPlugin
Entry point: apache = certbot_apache.entrypoint:ENTRYPOINT

* nginx
Description: Nginx Web Server plugin
Interfaces: IAuthenticator, IInstaller, IPlugin
Entry point: nginx = certbot_nginx.configurator:NginxConfigurator

* standalone
Description: Spin up a temporary webserver
Interfaces: IAuthenticator, IPlugin
Entry point: standalone = certbot.plugins.standalone:Authenticator

* webroot
Description: Place files in webroot directory
Interfaces: IAuthenticator, IPlugin
Entry point: webroot = certbot.plugins.webroot:Authenticator
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Determine where your certbot (in my case certbot-auto) is installed:

# find / -name certbot
/opt/eff.org/certbot
...

Get into the Virtual Env and install plugin

cd /opt/eff.org/certbot/venv
source bin/activate
pip install certbot-dns-google
deactivate

Verify certbot plugins again

# certbot-auto plugins
Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
* apache
Description: Apache Web Server plugin - Beta
Interfaces: IAuthenticator, IInstaller, IPlugin
Entry point: apache = certbot_apache.entrypoint:ENTRYPOINT

* dns-google
Description: Obtain certificates using a DNS TXT record (if you are using Google
Cloud DNS for DNS).
Interfaces: IAuthenticator, IPlugin
Entry point: dns-google = certbot_dns_google.dns_google:Authenticator

* nginx
Description: Nginx Web Server plugin
Interfaces: IAuthenticator, IInstaller, IPlugin
Entry point: nginx = certbot_nginx.configurator:NginxConfigurator

* standalone
Description: Spin up a temporary webserver
Interfaces: IAuthenticator, IPlugin
Entry point: standalone = certbot.plugins.standalone:Authenticator

* webroot
Description: Place files in webroot directory
Interfaces: IAuthenticator, IPlugin
Entry point: webroot = certbot.plugins.webroot:Authenticator
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -


docker-compose local variable declare and use In the programming language we can define and declare a variable and use it later in the program, can we do same in docker-compose file?

For example in Java:

int x = 100;
System.out.println("Number is: "+ x);
Similarly, in docker-compose file, can we declare a local variables and use it later.

docker-compose file

In the docker-compose file we want to declare tomcat_home as a variable and use it later as shown in example.

values:
  - tomcat_home: "/usr/local/tomcat"

# Creating Tomcat Server 1

Server1:
  image: tomcat
  hostname: TomcatServer1
  container_name: TomcatServer1
  volumes:
    - ${PWD}/tomcat-users.xml:${tomcat_home}/conf/tomcat-users.xml
    - ${PWD}/server.xml:${tomcat_home}/conf/server.xml

https://docs.docker.com/compose/environment-variables/#the-env-file

One could set a variable in an .env file:

TAG=1.2.3

and call it in a docker-compose file

docker-compose.yml

version: "3"
services:
  web:
    image: nginx:${TAG}
    ports:
    - 80:80

docker-compose up

returns:

Pulling web (nginx:1.2.3)...
ERROR: manifest for nginx:1.2.3 not found

No need to run docker-compose to check the config

According to the documentation one could run:

docker-compose config

to verify the docker-compose config

networks: {}
services:
  web:
    image: nginx:1.2.3
    ports:
    - 80:80/tcp
version: '3.0'
volumes: {}


Scaling replication controllers via API in Kubernetes Is there a way to scale replication controller via API in kubernetes.

By using kubeclt command, it may be done by running:

kubectl scale --replicas=1 --namespace=kube-system rc my_replication_controller

But how to achieve it using API?

Moving through docs, I found only "replace scale of the specified Scale" and "partially update scale of the specified Scale" among write operations. But I think this is not what I need.

After long playing with it and consulting on slack DevOps chat, I was able to figure it out. Here is a workable curl:

curl -sSk -H \"Authorization: Bearer ${KUBE_TOKEN}\" -H 'Content-Type: application/merge-patch+json' -X 'PATCH' $URL_NAME/api/v1/namespaces/kube-system/replicationcontrollers/kube-registry-v0 -d '{"spec": {"replicas":  1}}'

One of the important components here is "Content-Type: application/merge-patch+json" Without it specification, Kubernetes will not recognize data in API request correctly.


Connecting to a Windows machine with Ansible I'm trying to simply connect to a machine and run the win_ping module.

There are several tutorials online that outline a few key steps:

Create a directory ./windows
Create a file hosts in ./windows
Create a directory ./windows/group_vars
Create a file windows.yml in ./windows/group_vars
Insert several variables with your Windows credentials into windows.yml

The problem is that none of these actually specify where to put these relative to the '.'.  I've tried placing this structure in /etc/ansible/ and unfortunately it's looking like Ansible isn't picking this up at all.  I've also tried moving the structure around but no luck.

How do I get Ansible to reference the directory I've created?  I believe ansible.cfg would make sense but I can only see how to point it to a default inventory file. Interestingly enough - even adding a group called 'windows' in the base hosts file and calling ansible with the group there - it says no hosts found. I have also uncommented the inventory location in ansible.cfg to point to /etc/ansible/hosts and it still says no hosts found when calling ansible windows -m ping (or win_ping for that matter).

There are two places where Ansible looks for group_vars subdirectory:

the playbook directory - the one containing a playbook you run
the inventory directory - the one specified with -i option at run time, or a default one (usually /etc/ansible, or /usr/local/etc/ansible)

In most typical scenario you might place group_vars along with the playbook.yml.

The example you posted uses the second method, so to reference the files you need to add -i to the command:

ansible windows -m ping -i ./windows


How to split traffic in Google AppEngine for specific IP addresses I'm trying to find a way to split the Google AppEngine for some Ip address for testing purpose is there any way to do that?

Yes, it is possible to split the incoming traffic between several GAE app/service versions based on the IP address from which the requests originate. From Splitting traffic across multiple versions:

  When you have specified two or more versions for splitting, you must
  choose whether to split traffic by using either an IP address or HTTP
  cookie. It's easier to set up an IP address split, but a cookie split
  is more precise. For more information, see IP address splitting
  and Cookie splitting.

Read the respective linked section (same page) for details and limitations.

Configuring the traffic split is done from the Versions page.


How to get memory usage per process with sar, sysstat? Can I get memory usage per process with Linux? we monitor our servers with sysstat/sar. But besides seeing that memory went off the roof at some point, we can't pinpoint which process was getting bigger and bigger. is there a way with sar (or other tools) to get memory usage per process? and look at it, later on?

As Tensibai mentioned, you can extract this info from the /proc filesystem, but in most cases you need to determine the trending yourself. There are several places which could be of interest:

/proc/[pid]/statm

          Provides information about memory usage, measured in pages.
          The columns are:

              size       (1) total program size
                         (same as VmSize in /proc/[pid]/status)
              resident   (2) resident set size
                         (same as VmRSS in /proc/[pid]/status)
              shared     (3) number of resident shared pages (i.e., backed by a file)
                         (same as RssFile+RssShmem in /proc/[pid]/status)
              text       (4) text (code)
              lib        (5) library (unused since Linux 2.6; always 0)
              data       (6) data + stack
              dt         (7) dirty pages (unused since Linux 2.6; always 0)

cat /proc/31520/statm
1217567 835883 84912 29 0 955887 0

memory-related fields in /proc/[pid]/status (notably Vm* and Rss*), might be preferable if you also collect other info from this file

          * VmPeak: Peak virtual memory size.

          * VmSize: Virtual memory size.

          * VmLck: Locked memory size (see mlock(3)).

          * VmPin: Pinned memory size (since Linux 3.2).  These are
            pages that can't be moved because something needs to
            directly access physical memory.

          * VmHWM: Peak resident set size ("high water mark").

          * VmRSS: Resident set size.  Note that the value here is the
            sum of RssAnon, RssFile, and RssShmem.

          * RssAnon: Size of resident anonymous memory.  (since Linux
            4.5).

          * RssFile: Size of resident file mappings.  (since Linux 4.5).

          * RssShmem: Size of resident shared memory (includes System V
            shared memory, mappings from tmpfs(5), and shared anonymous
            mappings).  (since Linux 4.5).

          * VmData, VmStk, VmExe: Size of data, stack, and text
            segments.

          * VmLib: Shared library code size.

          * VmPTE: Page table entries size (since Linux 2.6.10).

          * VmPMD: Size of second-level page tables (since Linux 4.0).

          * VmSwap: Swapped-out virtual memory size by anonymous private
            pages; shmem swap usage is not included (since Linux
            2.6.34).

server:/> egrep '^(Vm|Rss)' /proc/31520/status
VmPeak:  6315376 kB
VmSize:  4870332 kB
VmLck:         0 kB
VmPin:         0 kB
VmHWM:   5009608 kB
VmRSS:   3344300 kB
VmData:  3822572 kB
VmStk:      1040 kB
VmExe:       116 kB
VmLib:    146736 kB
VmPTE:      8952 kB
VmSwap:        0 kB

Some processes can, through their behaviour and not through their actual memory footprint, contribute to the overall system memory starvation and eventual demise. So it might also be of interest to look at the OOM Killer related information, which already takes into account some trending information:

/proc/[pid]/oom_score

          This file displays the current score that the kernel gives to
          this process for the purpose of selecting a process for the
          OOM-killer.  A higher score means that the process is more
          likely to be selected by the OOM-killer.  The basis for this
          score is the amount of memory used by the process, with
          increases (+) or decreases (-) for factors including:

          * whether the process creates a lot of children using fork(2)
            (+);

          * whether the process has been running a long time, or has
            used a lot of CPU time (-);

          * whether the process has a low nice value (i.e., > 0) (+);

          * whether the process is privileged (-); and

          * whether the process is making direct hardware access (-).

          The oom_score also reflects the adjustment specified by the
          oom_score_adj or oom_adj setting for the process.

server:/> cat proc/31520/oom_score
103

/proc/[pid]/oom_score_adj (or its deprecated predecessor /proc/[pid]/oom_adj, if need be)

          This file can be used to adjust the badness heuristic used to
          select which process gets killed in out-of-memory conditions.

          The badness heuristic assigns a value to each candidate task
          ranging from 0 (never kill) to 1000 (always kill) to determine
          which process is targeted.  The units are roughly a proportion
          along that range of allowed memory the process may allocate
          from, based on an estimation of its current memory and swap
          use.  For example, if a task is using all allowed memory, its
          badness score will be 1000.  If it is using half of its
          allowed memory, its score will be 500.

          There is an additional factor included in the badness score:
          root processes are given 3% extra memory over other tasks.

          The amount of "allowed" memory depends on the context in which
          the OOM-killer was called.  If it is due to the memory
          assigned to the allocating task's cpuset being exhausted, the
          allowed memory represents the set of mems assigned to that
          cpuset (see cpuset(7)).  If it is due to a mempolicy's node(s)
          being exhausted, the allowed memory represents the set of
          mempolicy nodes.  If it is due to a memory limit (or swap
          limit) being reached, the allowed memory is that configured
          limit.  Finally, if it is due to the entire system being out
          of memory, the allowed memory represents all allocatable
          resources.

          The value of oom_score_adj is added to the badness score
          before it is used to determine which task to kill.  Acceptable
          values range from -1000 (OOM_SCORE_ADJ_MIN) to +1000
          (OOM_SCORE_ADJ_MAX).  This allows user space to control the
          preference for OOM-killing, ranging from always preferring a
          certain task or completely disabling it from OOM killing.  The
          lowest possible value, -1000, is equivalent to disabling OOM-
          killing entirely for that task, since it will always report a
          badness score of 0.

          Consequently, it is very simple for user space to define the
          amount of memory to consider for each task.  Setting an
          oom_score_adj value of +500, for example, is roughly
          equivalent to allowing the remainder of tasks sharing the same
          system, cpuset, mempolicy, or memory controller resources to
          use at least 50% more memory.  A value of -500, on the other
          hand, would be roughly equivalent to discounting 50% of the
          task's allowed memory from being considered as scoring against
          the task.

          For backward compatibility with previous kernels,
          /proc/[pid]/oom_adj can still be used to tune the badness
          score.  Its value is scaled linearly with oom_score_adj.

          Writing to /proc/[pid]/oom_score_adj or /proc/[pid]/oom_adj
          will change the other with its scaled value.

server:/> cat proc/31520/oom_score_adj
0


Re-using Python virtual environment vs Re-building it Currently, we build our application on a build server where we create a virtual environment using virtualenv command, install all the Python dependencies into it, then "patch" it using the following bash commands to make it relocatable:

# Adopt virtualenv to the target system
echo "* Patching virtual envirinment for target system"
for f in "${VENV_PATH}/bin"/* ; do
    if [ -n "$(file "${f}" | grep "text")" ] ; then
    echo "  + ${f}"
        sed -i "s;"${VENV_PATH}";/path/to/the/target/directory/venv;" "${f}"
    fi
done
rm -rf "${VENV_PATH}/lib64"
ln -s "/path/to/the/target/directory/venv/lib" "${VENV_PATH}/lib64"

Then, we pack it into an RPM package including some other parts of the application, deploy to multiple target servers and install it via yum.

The main motivation behind making the virtual environment relocatable is to "build once, deploy multiple times" - in other words, to avoid installing the same dependencies on all the target servers.

The problem is that, this patching script looks really fragile and, from what I understand, this is quite tied to the current Python version and also brings the requirement for the build server to be the same operating system as the target servers (or, at least, to have all the python related paths match).

Is my understanding correct? Would it be better to rebuild the virtual environment on every single target? (we can accelerate pip installation commands with, for example, pip-accel or caching) What are the general ways to tackle this problem?

What are the general ways to tackle this problem?

The ideal way to resolve your problem would be to dockerise your application, as it would remove any worries you have about compatibility between your build and running servers. Installing dependencies using pip in a Dockerfile is trivial, and as long as you're deploying the same container image you never need to worry about inconsistency.

Edit: as luck would have it I actually found myself wanting to have a venv that worked both inside and outside Docker, so I came up with the following:

python3.6 -m venv --copies venv
sed -i '43s/.*/VIRTUAL_ENV="$(cd "$(dirname "$(dirname "${BASH_SOURCE[0]}" )")" \&\& pwd)"/' venv/bin/activate
sed -i '1s/.*/#!\/usr\/bin\/env python/' venv/bin/pip*


Passing in a file with configuration One of the steps in our deployment process requires a configuration file that we don't want to commit to the repository and rather have it passed in by the Bamboo itself. Is there some sort of task/way to conveniently add it? It can be either a task that creates a file in work directory or assigns the file content to a variable.

What we want to avoid is to have the file on the disk and passing in the path to the file - I'd rather have a way of pasting the file directly into a task.

Currently, the deployment step calls a script that expects the content of said config file to be passed in (shortened for brevity):

deploy.ps1 --settings=<file_content>

edit:
The settings file is a lengthy json file, with several arrays on few levels. I've been considering just assigning an inlined content of that file to the environment variable, but I was wondering if there was a better way to pass it as well as allow maintainers edit the values, if necessary.

Splitting the config into several Bamboo environment variables is not really feasable, especially since one of our use cases requires the ability to run the deployment script without Bamboo, in case of server failure, for example.

It is convenient to use Environment Variables to provide runtime configuration for such scripts. Especially when it contains secrets that you don't want to hold in any persistent storage or in version control.

When you do actually have secrets, it is even better to use some kind of secret management system that will provide these secrets on request. One such tool that I highly recommend is HashiCorp's Vault.

Depending on how you are executing the jobs on the Bamboo Server, another option might be to store a local file on the server and reference that during builds. This is not really recommended since the server then becomes a special snowflake that only a few people know how to actually set up. It is much better to allow malfunctions with the server, or even the loss of all server data, to not affect your ability to deploy code. Thus having special files on the server itself is not so great.


In a modern IaaS microservice based system is Zookeeper still a useful tool? I've used zookeeper before as a configuration management service to keep track of application settings, feature toggles etc.  However, that was back in the day before microservices and AWS/Azure/GCP.  Since zookeeper tends to need multiple VMs to operate, is it still a good tool in the modern ecosystem?

Are there better ways to handle feature toggles and centralized configuration settings for multiple environments using a more modern (aka serverless style) system?

It's difficult to say whether other tools will be better if there is nothing really concrete to measure against. A good, partially biased answer comes from Hashicorp's Consul

This seems to describe why Consul would be inherently better than Zookeeper, from the point of view of ease of use:

  ZooKeeper et al. provide only a primitive K/V store and require that application developers build their own system to provide service discovery. Consul, by contrast, provides an opinionated framework for service discovery and eliminates the guess-work and development effort

... scalability:

  ZooKeeper provides ephemeral nodes which are K/V entries that are removed when a client disconnects. These are more sophisticated than a heartbeat system but still have inherent scalability issues and add client-side complexity. All clients must maintain active connections to the ZooKeeper servers and perform keep-alives.

... and simplicity :

  Consul provides first-class support for service discovery, health checking, K/V storage, and multiple datacenters. To support anything more than simple K/V storage, all these other systems require additional tools and libraries to be built on top. By using client nodes, Consul provides a simple API that only requires thin clients. Additionally, the API can be avoided entirely by using configuration files and the DNS interface to have a complete service discovery solution with no development at all.

It does not address the need for extra resources, but since the architecture is different, I would assume the need for resources is also less:

  Instead of only having server nodes, Consul clients run on every node in the cluster.

Hope this can provide some form of answer.


SSH tunneling to hosts deployed behind firewalls I'm working on a deployment that will require me to place Ubuntu servers that monitor industrial equipment at various sites and behind firewalls that are out of my control.  I need to maintain SSH access to the servers to perform maintenance and updates, but am struggling on the best way to securely accomplish this.

Port forwarding and creating firewall pinholes are not options, so I can't quite get my head around what to do.  Is there some system that I could use to maintain the ability to open an SSH tunnel to these machines from the outside world?  Is there a better way to accomplish this?

Thanks!

Assuming that the monitoring host is able to make outbound connections, you can solve this by having a bastion host elsewhere which you can connect through. Obviously do make sure to clear this with your customer.

To do this, set up a server somewhere under your control. This might be a VPS with a generous traffic limit, or it might be a physical server under your control, or something else. Give it a stable DNS name, possibly on a one-per-customer basis, which maps to the IP address of the bastion host. Mind the possibility that you may need to renumber that host in the future, possibly without being able to access the monitoring host.

Now, configure the monitoring host (the one that you want to be able to ultimately connect to) to connect out to the bastion host to set up a SSH tunnel. This will probably include creating a non-privileged user account on the monitoring host to run the tunnelling SSH process, and protecting that appropriately. (Remember: this account won't be used for logging in to; it will only run a local, non-interactive SSH client process, and own the tunnel.) Also set up a non-privileged account on the bastion host, with only sufficient privileges to log in with SSH (not to a shell) and set up a SSH tunnel to the monitoring host.

With that in place, you can set up something like autossh on the monitoring host to maintain a connection to the bastion host to keep the tunnel alive. Add some decent logging on the bastion host, since on the monitoring host, the connection will almost certainly look like it originates from localhost.

When you need to connect to the monitoring host, first connect to the bastion host setting up a SSH tunnel to the bastion host's local port that forwards to the monitoring host; and then connect to the local port that maps to that tunnel. This will tunnel the SSH traffic via the bastion host, allowing your system to talk to the ultimate target host without requiring the host that is ultimately being connected to to be able to accept incoming connections.


Is there an equivilent of a Kubernetes ingress for Docker Compose? I'm more of a software developer than a networking/ops person to give you some context.

I'm currently creating an application that is a standard containerised microservices architecture, where my frontend is some static assets in an nginx container, and my backend APIs are expressjs servers running on node containers.

When my frontend makes a call to a backend endpoint, it's going to look like:

const response = await fetch("/api1/v1/items");

Because by default the browser is going to make this request to the frontend container, we need  some way of redirecting that traffic to the correct backend container.

One standard way to do this, is that I can setup a reverse-proxy in  the frontend container's nginx.conf to direct all /api1/* traffic to another service.

ie. you would do something like this:

location /a {
   proxy_pass         http://api1;

Where you have defined api1 as a service in your docker-compose.yml.

I'm a bit uncomfortable with this, at the cluster configuration is mixed up with the implementation.

I am aware with Kubernetes ingresses, this routing can be done from outside of the container - where essentially the traffic is routed as it enters the cluster.

This is a much nicer solution in my view.

But the reason I'm using docker-compose is just for simplicity - it's more for running the cluster on your local machine and just getting the whole thing up and running.

Is there a similar configurable ingress concept for docker-compose?

Kubernetes ingress is implemented by changing the configuration of another container or the cloud load balancer to route traffic to the service port/ip. Docker by itself is fairly lightweight, only runs the containers you start, and does not have external dependencies on things like your cloud load balancer. The solutions to setup a reverse proxy inside a container are almost identical to the kubernetes nginx based ingress controllers, with the key difference being the configuration of that reverse proxy.

There are a variety of docker based reverse proxies that configure themselves, including docker-flow-proxy, nginx-proxy, and Docker-EE ships with interlock that configures HA proxy. Probably the easies solution I've seen to implement is traefik which can be configured to monitor the docker socket for changes and automatically reconfigure based on labels set on your containers.

One important difference from Kubernetes is that docker networks are not flat, you will need to explicitly place the proxy container on the same docker network as your application in order for them to be able to route traffic between each other.


How to deploy applications to k8s using bitbucket pipelines? The aim is to implement Continuous Deployment (CD) to a k8s cluster.

It works to deploy a Nginx image to the k8s.

https://github.com/kubernetes/kubernetes/blob/master/examples/simple-nginx.md

kubectl run my-nginx --image=nginx --replicas=2 --port=80

When the same command is run in a CD tool then it complains that it cannot find the cluster. Off course it is possible to copy the ~/.kube/config, but is that recommended?

When the Kubeconfig is stored in a variable and kubectl is run by bitbucketpipelines then the following error is thrown:

error: Error loading config file "/root/.kube/config": yaml: mapping values are not allowed in this context

Inspecting the files turns out that the whole file consists of one line instead of multiple.

You need to separate your daily authentication method of the CD authentication.

You should start understanding the different methods that K8S offers https://kubernetes.io/docs/admin/authentication/

On the other hand if you use some cloud solution like GCP you can benefit of the service account that the platform puts in all your instances, setting the right permissions to a service account allow you to to controll any cluster inside your project like in your local machine.


How can I share a (global) variable between salt states? I am using Salt for deployment and configuration management. In order to attach to a FreeIPA server, I need to randomly generate an OTP token and then use it to attach to IPA. How can I share a variable between SLS files?

For example,

init.sls:

{%- from "freeipa/map.jinja" import client, ipa_host with context %}
include:
  - freeipa.example

freeipa_client_install:
  cmd.run:
  - name: >
    ipa-client-install
    --server server.example.com
    --domain example.com
    --realm EXAMPLE.COM
    --hostname client.example.com
    -w {{ otp }}                                   <===== !!!
    --unattended

./freeipa/example.sls:

{% set otp = salt['random.get_str'](20) %}         <===== !!!

get_ticket:
  cmd.run:
    - name: kinit service-account@EXAMPLE.COM -kt /tmp/salt-service.keytab
ipa_host_add:
  cmd.run:
    - name: >
      curl -k
      -H referer:https://{{ client.get("server", {}) }}/ipa
      --negotiate -u :
      -H "Content-Type:application/json"
      -H "Accept:applicaton/json"
      -c /tmp/cookiejar -b /tmp/cookiejar
      -X POST
      -d '{
        "id": 0,
        "method": "host_add",
        "params": [
          [
            "client.example.com"
          ],
          {
            "all": false,
            "force": false,
            "no_members": false,
            "no_reverse": false,
            "random": false,
            "raw": true,
            "userpassword": "{{ otp }}",          <==== !!!
            "version": "2.156"
          }
        ]
      }' https://server.example.com/ipa/json

What I find is that everything looks good in example.sls, but I cannot reference this variable in init.sls. Do I need to be doing salt['random.get_str'](20) inside of init.sls instead? If so, how can I reference the otp variable in example.sls - or vice-versa - how can I reference the otp variable set in example.sls inside of init.sls?

When I encountered a related problem, I wound up having Jinja run execute Salt modules.

In my case, it was cmd.run to run mktmp (and, at the end of the template, rm for cleanup), with file.append and file.grep, though in your case, you might be able to use Salt's load_yaml extension to Jinja.

So, something like this (rough, untested):

{% set tmpfile = salt['cmd.run']('mktmp') -%}
{% salt['file.append']('my_key: ' ~ your_key) -%}
{% set my_key = load_yaml(tmpfile)['my_key'] -%}
{% salt['file.remove'](tmpfile) %}

This certainly isn't the best place to put data, but it demonstrated using calling into Salt modules from Jinja to store data. I don't know of any existing Salt modules suitable for storing ephemeral data that only lives as long as the templates compile. I'll check back later and see what other solutions might work.


Questions about Ansible connection to remote hosts I want to ask some questions about working of Ansible.

1 - Which shell Ansible uses when logging to remote host ?

https://askubuntu.com/questions/937354/track-logins-made-via-non-interactive-shells/937716?noredirect=1#comment1486867_937716

2 -  If I have 10 tasks in a playbook, then ansible makes connection 10 times to remote host OR uses a single connection to perform all these 10 tasks ?

3 - If I am running 10 tasks on 2 hosts,

It will do first task on host 1 and then on host 2
then second task on host 1 and then on host 2,
..
. so on,

.

So, here also the connection persists to both the hosts?

OR

does it firstly connect to host 1, runs task 1 and then disconnects AND then connects to host 2, runs task 1 and then disconnects ?

Can't answer 1 (I always thought it just uses the default /bin/sh unless specified otherwise?).

It will make the connection more than 10 times.  A single task will typically have an SCP or SFTP connection to copy the taskfile that'll get remotely executed, then another connection to trigger the script.  You can monitor this happening by running your playbook with -vvv, such as:

ansible-playbook deploy_app.yml -u maplebird -vvv

3rd-level verbosity shows all connections to client.
By default, Ansible will execute tasks concurrently on all hosts up to the max configured number of forks.  So, it'll run task 1 on both hosts, then it'll run task 2 on both hosts, etc.  Forks are defined in ansible.cfg, and default to 5.  Change this variable to a higher number or comment it out:

forks = 5

Optionally, you can also do a rolling batch when running playbooks by specifying the serial option in the playbook.  This will only concurrently execute tasks for however many hosts you've defined.

Say you're running a playbook against 5 hosts, and have the serial option set.  With serial = 1, it will run the full playbook 1 host at a time.  With serial = 2, it will run hosts 1 & 2, then hosts 2 & 3, then host 5.  Example:

name: deploy to all webservers
hosts: webservers
serial: 2
roles:
   - deploy_application

More reading:

Serial (rolling) playbook runs:

http://docs.ansible.com/ansible/latest/playbooks_delegation.html#rolling-update-batch-size

Forks:

http://docs.ansible.com/ansible/latest/intro_configuration.html#forks

Hope this helps.


Does RAM Come Into The Container vs VM Decision? I am currently running a Java trading system on a VM that needs ~32 GB RAM and runs (almost) 24/7.
There are a couple of engines linked to it that require 4GB + 8BG respectively.
I am considering moving from VMs to containers.
Most of the articles I read talk about running web apps and the benefit of a container starting up fast but they don't talk about heavyweight applications.
For this question I would like to know whether the RAM requirements of the processes running within them are a consideration when comparing VMs to Containers, or is there no difference between the two?

the benefit of a container starting up fast but they don't talk about heavyweight applications.

yes and no. The container only contains the app and runtime (e.g. JVM), but the app and runtime starts up with the same speed as when running on a VM. But a VM usually takes longer time to create and start since you also have to boot the operating system. So it depends on how you compare.
Running an app in a container does not do your app startup faster.

For this question I would like to know whether the RAM requirements of the processes running within them are a consideration when comparing VMs to Containers, or is there no difference between the two?

The app will use the same amount of resources when run as a container as on VM. But with containers it is easier to manage multiple instances - so you could consider that - this is usually done to achieve better availability and zero downtime deployments - however your app need to be stateless if you consider scaling out to multiple instances.


Jenkins Email notification for multiple builds I need suggestion for Jenkins project (multi-conf or pipeline) and plugins that will fit my work.

I have 10 "flavors" of the product, so I must build 10 times every time I commit to the repository (all in the same workspace).
Today I have 10 jobs (freestyle) and a "master" job that trigger the rest.
I tried to add Email notification (using Email Extension Plugin) but I want only one Email report for all the builds, not 10 Emails.

I understand that I should change to one multi-configuration project or one pipeline project that will handle all the builds, so it will be easier to trigger only one Email, but what is the best practice to get only one Email report on multiple builds?

The other option you could try is to add email notification as part of your build script. I use ANT build script and call the email notification target from my jenkins job instead of using the email plugin