What does implementing Hashicorp Vault mean in practice? I have read several Hashicorp Vault tutorials and one of the features that could be used is generating ssh keys for git clone repositories, but that implies that everybody in the company that clones repositories should install vault and an helper like https://github.com/hashicorp/vault-ssh-helper

In the hope of resurrecting this question, I will give my opinionated answer.

As I understand it, the use of Vault is part of the drive to give more autonomy to teams, by removing fear of leaking or changing secrets. By providing somewhere to store and manage secrets, ops can give dev the means to continue developing at their own pace, following the practices of continuous integration and working in small batches.

With a product like Vault in the pipeline, secrets can be consumed safely in the various phases - build, test, bake, deploy, etc. Implementing Vault in practice means less personal interaction with the service, but more machine interaction with the service, from trusted components in the pipeline.


Properly shutdown k3s on local machine I've just installed k3s on my notebook (Ubuntu). Works fine. Great project. However, i do not need it to run 24/7 so, how do I properly shutdown the k3s cluster. Oddly, I cannot find anything about this in the documentation. Thanks

There is a systemd shutdown task (/etc/systemd/system/k3s.service) which stops the server, but not the k8s resources (pods, networks, etc). For this there is a utility script k3s-killall.sh which should be at /usr/local/bin/k3s-killall.sh


AWS ECS vs EKS for hybrid cloud strategy What can EKS deployment do that ECS deployment can't do? is there any specific circumstances that you have to deploy your application on EKS but not ECS.

This is a very broad question (and BTW you have a typo in the title, it should say ECS not ECR).
In terms of general positioning this may be a good starting point.
Given in the title you mention "hybrid" deployments you may be interested in investigating and investigating ECS Anywhere and EKS Anywhere. They are both coming in 2021.


Would choosing the integration branch strategy be part of DevOps? In a mature DevOps organisation would choosing a product/product line integration branch strategy be part of DevOps?

What do I mean by integration branch strategy?

For example a product team has to integrate 20 features in a certain interval of time to meet some time to market release requirements.

One potential approach would be to use feature branches in a waterfall manner as they've deen doing for years - each feature is developed and tested in their own branch and only merged into the main branch (from which the release branch will be pulled) after metting the respective feature QA goals.

But the total integration time estimate - 20 feature branch merge windows multiplied by the average merge duration (from the team's branch merge historical data) is maybe too high.

Another approach is to pull say 4 intermediate integration branches, each of which being used by 5 of the features branches. This way all 20 features will be integrated into the 4 intermediate branches in a 5 branch merge windows interval. Which would be followed by a 4 other branch merge windows interval for integrating these 4 branches into the main branch, bringing the total integration estimate to 9 windows, much better than 20. Yes, some of the gain will probably be cancelled by higher degree of difficulty of last 4 merges.

Another approach would be to go straight for trunk-based development (true CI integration into the main branch) with practically no branch merges.

The process of selecting one such strategy is what I'm referring to as branch strategy.

Note: the actual SCM repository is irrelevant, it can be any combination of repos managed together as a single one, the term branch denotes just the common logical representation of whatever physical/actual repo branches exist in that combination.

Branching strategy is a very clearly a business decision in which the DevOps engineer should be the primary expert, explaining all the trade offs. There are some strong positions on this issue in the community, very clearly Jez Humble is on one side with his book of Continuous Delivery. You could see him talking about it here. Here would be a video by panel of experts on the topic. And here is my highly up-voted answer on StackOverflow specifically on topic of feature branches. So clearly this is a big topic for DevOps.

One thing that people often say (Jez Humble mostly) is that feature branches are evil. You will need a (n-1)(n-2)/2 merges [combination number n-1 over 2] for n independent feature branches. Which comes to some very high numbers at 10+ feature branches. And because merging is hard and no publicly available version control systems do it will, then the entire Software Industry is moving away from software releases and software on premises to Software as a Service, where the only way is forward and software is both developed and maintained for you by the vendor.

As far as I know, there is only one company that can do high quality, business critical software on premises, with maintaining 10+ years of past major releases, back-porting patches through multiple waves of code refactoring and capable of working regularly on dozens of major feature branches per release and thousands small features developed on their individual feature branches and being continually merged as full features and not individual commits. The code has also been released on 14 different architectures. That company is Oracle, the product is RDBMS and they have been practicing what is now called DevOps since at least 1990. Hundreds of code deployments a day back in 2001 would be pretty standard. The reason why they are able to do it is through rigorous coding standards, invariant test suite and a proprietary version control system I've spent 10+ years of my life developing. A system that can do 99% of all merges through automation.

Everyone else pretty much only patches the most recent major release version past a certain stage/size of a project or it takes extreme amounts of human resources to merge all the patches to all the possible release versions and extreme amounts of human or compute resources to test them all. For example in Linux kernel you would have a maintainer for X.odd development version and later stability branch maintainer for X.even version, but you would see not patches for X-1.Y versions. Sometimes Linux distributions would maintain kernel version in their LTS release, but only at high cost and with varying code quality among distributions. And that is a high visibility project with enormous resources poured into it.

The primary reasons for this discrepancy are differences in:

Adherence to Coding Standards and Code Styles
Complete coverage by invariant test suite, which can catch bugs for which no tests were written specifically.
Version Control System capable of high levels of Automation.


What are ways to mitigate the effects of Mythical Man Month? Brooks's law: Adding manpower to a late software project makes it later.

In his book No Silver Bullet — Essence and Accidents of Software Engineering Frederick Brooks defines the concept of Mythical Man Month:

Brooks's assumption is that complex programming projects cannot be perfectly partitioned into discrete tasks that can be worked on without communication between the workers and without establishing a set of complex interrelationships between tasks and the workers performing them.

Since 1982 we have certainly moved forward and gathered some more experience in mitigating this problem. What are some of the solutions that you have successfully applied at your work to add resources to a project without creating more problems.

What is the MMM

First I want to explain the context for Brook's Law. What was the assumption that made him create it back in 1975?

  A Man-month is a hypothetical unit of work representing the work done by one person in one month; Brooks' law says that it is impossible to measure useful work in man-months.

source: https://en.wikipedia.org/wiki/The_Mythical_Man-Month

Back in the day, complex programming projects would mean big monolith systems. And Brooks claims that these cannot be perfectly partitioned into discrete tasks that can be worked on without communication between developers and without establishing a set of complex interrelationships between tasks and the people performing them.

This is very much true in highly cohesive software monoliths. No matter how much decoupling is done, still the big monolith mandates time required for the new programmers to learn about the monolith. And increased communication overhead which will consume an ever increasing quantity of the time available.

But does it really have to be this way? Do we have to write monoliths and keep communication channels to n(n − 1) / 2 where n is the number of developers?

We know there are companies where thousands of developers are working on big projects ... and it does work. So there must be something that changed since 1975.

Possibility to mitigate the MMM

In 2015, PuppetLabs and IT Revolution published the results of the 2015 State of DevOps Report. In that report, they focussed on the distinction between high performing organizations vs. non-high performing.

The high-performing organizations show some unexpected properties. For example, they have the best project due-date performance in development. Best operational stability and reliability in operations. As well as the best security and compliance track record.

One of the surprising things highlighted in the report is the deployments per day metric. But not just deployments per day, they also measured deploy/day/developer and what is the effect of adding more developers in high-performing organizations vs. non-high performing.

This is the graph from that report -

While low-performing organizations align with Mythical Man Month assumptions. The high-performing organizations can scale the number of deploy/day/dev linearly with the number of developers.

An excellent presentation at DevOpsDays London 2016 by Gene Kim talks about these findings.

How to do it

First, how to become a high-performing organization? There are a couple of books that talk about this, not enough space in this answer so I'll just link to them.

Good to Great: Why Some Companies Make the Leap...And Others Don't by Jim Collins
The High-Velocity Edge: How Market Leaders Leverage Operational Excellence to Beat the Competition by Steven Spear
The New Economics for Industry, Government, Education by W. Edwards Deming

For software and IT organizations, one of the critical factors for becoming a high-performing organization is: focus on quality and speed.

For example Ward Cunningham explains Technical Debt as all the things we allowed to be left unfixed. This is accepted by management because it always comes with a promise that it is going to be fixed when there is time.

There is never enough time, and technical debt just becomes worse and worse.

What are these things that cause technical debt to grow?

legacy code
manual configuration of environments
manual testing
manual deployments

Legacy code As defined in Working Effectively with Legacy Code by Michael Feathers is any code that doesn't have automated testing.

Any time shortcuts are used to get the code to production; the operations is burdened with the maintenance of this debt forever. Then the process of deployment becomes longer and longer.

Gene tells a story in his presentation about a company who has six-week long deployments. Involving tens of thousands of extremely error prone tedious steps, tying up 400 people, and they would do this four times every year.

One of the tenets of DevOps is that reliability comes from doing smaller deployments more frequently.

Example

Jon Jenkins at Velocity 2011: Amazon are doing 15k deploys/day
Ken Exner at AWS Summit 2015: Amazon are doing 136k deploys/day

These two presentations show all the things that Amazon did to decrease the time it takes them to deploy code to production.

According to Gene, the only thing that is changed across time in these high-performing organizations is the number of developers. So from the Amazon example, you could say that in four years they increased their deployments ten times just by adding more people.

This means that under certain conditions, with the right architecture, the right technical practices, the right cultural norms, developer productivity can scale as the number of developers is increased. And DevOps is definitely in the middle of all this.


GitLab runner not able to fetch AWS ECR images in the current AWS account GitLab and GitLab runners are running on the same host (I know how bad it is). There is an IAM Role attached to the EC2 on which GitLab is running.

This IAM Role gives the permission to perform some actions on multi-account ECR's.

Runners use docker as executor and assume role perfectly to push,pull images.
But, if images need to be pulled/pushed to the account on which GitLab is running, it doesn't work. I first need to pull images on the GitLab host so they are accessible within the runners.

I did a small test: I manually started a container directly on the GitLab host and ran aws ecr get-login... and I was able to login on the account.

I've installed amazon-ecr-credential-helper and configured like stated in the doc.

I don't understand why runners cannot pull images from the account...

I use this script in gitlab-ci.yml

image: docker:19.03
services:
- docker:19.03.0-dind
before_script:
- $(aws ecr get-login --no-include-email --region eu-central-1)

... and provide AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY variables into Settings/CI-CD section.
Ensure your region with images is the same where are you logging in.


Question on NGINX reverse proxy when there are services on the host and on containers I have the following architecture, which consists
of an nginx server on the host machine, and 3 urls, app1.com, app2.com, app3.com:

app1 and app2 are both in the host machine, lets say they are an HTML file on the host,
on the container, there is app3 and app2.com/example url.

My question is, how can I reference the container in the nginx of the host?
Since I cannot expose port 80 and 443 of the container, because the host
is already using these ports, I am not sure what is the best approach to this.

I though about putting the container on a not used port, like 8999 for example,
and proxying the request to this port, and blocking these ports on the firewall,
so no one outside can access them. If I go with this option, does the nginx
configuration inside of the containers will need the server_name still? In the
case of the app3.com/example

The other option I thougth about is using the docker socket, or even the docker0
network, but I don't know if this is viable.

An example of the nginx configuration file of the host:

server {
  server_name app1.com;
  location / {
    root /var/www/app1/index.html;
  }
}

server {
  server_name app2.com;
  location / {
    root /var/www/app2/index.html
  }

  location /example {
    proxy_pass http://localhost:8999;
  }
}

server {
  server_name app3.com;
  location / {
    proxy_pass http://localhost:8999;
  }
}

Example of the nginx configuration file of the container:

server {
  server_name app2.com
  location /example {
    root /var/www/app2/index.html;
  }
}

server {
  server_name app3.com;
  location / {
    root /var/www/app3/index.html;
  }
}

I though about putting the container on a not used port, like 8999 for example, and proxying the request to this port, and blocking these ports on the firewall, so no one outside can access them. If I go with this option, does the nginx configuration inside of the containers will need the server_name still?

You're thinking along the right lines. You do need the server_name directives in the container, since it can be accessed both by app2.com/example and app3.com.

Here's an example setup. I'll emulate an nginx on the host machine with a docker container running on the host network.

Scroll down past the code and console log for some explanations.

Here is the listing of the files I used:

lllamnyp@lllamnyp:~/Cloud/dev/stackexchange/devops/q11420$ tail -n +1 *

==> app1.html <==

I am app1!

==> app2_example.html <==

I am app2 in example subpath!

==> app2.html <==

I am app2!

==> app3.html <==

I am app3!

==> container_nginx.conf <==

server {
  server_name app2.com;
  location /example {
    root /var/www/app2/;
  }
}

server {
  server_name app3.com;
  location / {
    root /var/www/app3/;
  }
}

==> host_nginx.conf <==

server {
  server_name app1.com;
  location / {
    root /var/www/app1/;
  }
}

server {
  server_name app2.com;
  location / {
    root /var/www/app2/;
  }

  location /example {
    proxy_pass http://localhost:8999;
    proxy_set_header Host $http_host;
  }
}

server {
  server_name app3.com;
  location / {
    proxy_pass http://localhost:8999;
    proxy_set_header Host $http_host;
  }
}

==> nginx.conf <==

user  nginx;
worker_processes  1;

error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;

events {
    worker_connections  1024;
}

http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" "$http_host" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile        on;
    #tcp_nopush     on;

    keepalive_timeout  65;

    #gzip  on;

    include /etc/nginx/conf.d/*.conf;
}

==> run.sh <==

docker run --name host_nginx --network=host --rm -d \
  -v `pwd`/nginx.conf:/etc/nginx/nginx.conf \
  -v `pwd`/host_nginx.conf:/etc/nginx/conf.d/default.conf \
  -v `pwd`/app1.html:/var/www/app1/index.html \
  -v `pwd`/app2.html:/var/www/app2/index.html \
  nginx

docker run --name container_nginx -p 8999:80 --rm -d \
  -v `pwd`/nginx.conf:/etc/nginx/nginx.conf \
  -v `pwd`/container_nginx.conf:/etc/nginx/conf.d/default.conf \
  -v `pwd`/app2_example.html:/var/www/app2/example/index.html \
  -v `pwd`/app3.html:/var/www/app3/index.html \
  nginx

Here is the log from my console, where I am testing things out.

lllamnyp@lllamnyp:~/Cloud/dev/stackexchange/devops/q11420$ sudo ./run.sh
73ebc99b8fae1d18bcbc571adbab6b65eaa49f861d66f11a7490d61b4e1beb0f
9f3761ab0619255d9bde08eee70e1fb9b686b645f38d976de7c39d4c8f0ad70d
lllamnyp@lllamnyp:~/Cloud/dev/stackexchange/devops/q11420$ curl app1.com
I am app1!
lllamnyp@lllamnyp:~/Cloud/dev/stackexchange/devops/q11420$ curl app2.com
I am app2!
lllamnyp@lllamnyp:~/Cloud/dev/stackexchange/devops/q11420$ curl app3.com
I am app3!
lllamnyp@lllamnyp:~/Cloud/dev/stackexchange/devops/q11420$ curl app2.com/example

<title>301 Moved Permanently</title>

301 Moved Permanently

nginx/1.17.10

lllamnyp@lllamnyp:~/Cloud/dev/stackexchange/devops/q11420$ curl app2.com/example/
I am app2 in example subpath!
lllamnyp@lllamnyp:~/Cloud/dev/stackexchange/devops/q11420$ sudo docker logs host_nginx
127.0.0.1 - - [25/Apr/2020:15:11:15 +0000] "GET / HTTP/1.1" "app1.com" 200 11 "-" "curl/7.68.0" "-"
127.0.0.1 - - [25/Apr/2020:15:11:17 +0000] "GET / HTTP/1.1" "app2.com" 200 11 "-" "curl/7.68.0" "-"
127.0.0.1 - - [25/Apr/2020:15:11:20 +0000] "GET / HTTP/1.1" "app3.com" 200 11 "-" "curl/7.68.0" "-"
127.0.0.1 - - [25/Apr/2020:15:11:25 +0000] "GET /example HTTP/1.1" "app2.com" 301 170 "-" "curl/7.68.0" "-"
127.0.0.1 - - [25/Apr/2020:15:11:29 +0000] "GET /example/ HTTP/1.1" "app2.com" 200 30 "-" "curl/7.68.0" "-"
lllamnyp@lllamnyp:~/Cloud/dev/stackexchange/devops/q11420$ sudo docker logs container_nginx
172.17.0.1 - - [25/Apr/2020:15:11:20 +0000] "GET / HTTP/1.0" "app3.com" 200 11 "-" "curl/7.68.0" "-"
172.17.0.1 - - [25/Apr/2020:15:11:25 +0000] "GET /example HTTP/1.0" "app2.com" 301 170 "-" "curl/7.68.0" "-"
172.17.0.1 - - [25/Apr/2020:15:11:29 +0000] "GET /example/ HTTP/1.0" "app2.com" 200 30 "-" "curl/7.68.0" "-"

I corrected some errors in your config (missing semicolon here and there), changed the server roots to directories, rather than index.html and also the request for app2.com/example would request /var/www/app2/example/index.html so pay close attention to the way in which I mounted the files when starting up the containers in ./run.sh. Finally, it was necessary to do proxy_set_header Host $http_host; in the host nginx config, so that the requested Host header gets correctly relayed to the containerized nginx and both app2 and app3 work correctly.


How to secure appspec file? What's the best way to ensure that developers can't modify the contents of the appspec.yml file used by AWS CodeDeploy? This file executes scripts on production servers with root privileges so it would be ideal if there were a way to ensure the files can't be modified by anyone with access to the application repository.

Please note that I'm looking for technical suggestions.

Please also note I'm using CodeDeploy as part of CodePipeline with GitHub as source.

The way I handled this in the past was to have a separate repository that contains the scripts, as most of them ended up shared across multiple projects anyway.  Then your appspec.yml file just contains references to the scripts and imports them via a submodule.  It's not perfect because a developer could change the appspec.yml to call his own scripts instead, although that's a lot more obvious.

Another approach I have used on Azure is to require that scripts are signed with a specific GPG key.  I used a PowerShell and a CodeSigning certificate, but I suspect there is an equally suitable solution with GPG.

A final approach would be to automatically fail the build or reject the commit if changes are detected within appspec.yml except when those changes are made by an "approved" person.


Ansible - merging inventories to share variables across? Is it possible to merge inventories so I can share variables across them? Let's say I have two inventory files. Inventory A and Inventory B. They are both in a directory called inventories.
Variables are defined in B and I want to reference them in inventory A. Is this possible? I've heard of merging inventories so share variables across but I haven't gotten this to work.
Here's are the files in my inventories directory:
/opt/ansible/inventories/aws_ec2.yaml
/opt/ansible/inventories/secrets.yaml
contents of aws_ec2.yaml:
---
plugin: aws_ec2

aws_access_key: "{{ aws_access_key }}"
aws_secret_key: "{{aws_secret_key}} "
aws_security_token: "{{aws_security_token}}"

regions:
  - us-west-2

keyed_groups:
  - key: tags
    prefix: tag
  - key: placement.region
    prefix: aws_region

groups:
  TEST: "'prod' in (tags:list)"

compose:
  ansible_host: private_ip_address

Here is my secrets.yaml file:
---

aws_access_key: xxxxx
aws_secret_key: xxxxx
aws_security_token: xxxxx

When I run my playbook, referencing both inventories, inventory does not seem to occur and I get these errors. It looks like the variables aren't being referenced and expanded?:
me@workstation:/opt/ansible/inventories$ ansible-playbook -i /opt/ansible/inventories/aws_ec2.yaml -i /opt/ansible/inventories/secrets.yaml /opt/ansible/playbooks/test.yaml
[WARNING]:  * Failed to parse /opt/ansible/inventories/aws_ec2.yaml with aws_ec2 plugin: Failed to
describe instances: An error occurred (AuthFailure) when calling the DescribeInstances operation:
Credential must have exactly 5 slash-delimited elements, e.g. keyid/date/region/service/term, got '{{'
[WARNING]: Unable to parse /opt/ansible/inventories/aws_ec2.yaml as an inventory source
[WARNING]: Unable to parse /opt/ansible/inventories/secrets.yaml as an inventory source
[WARNING]: No inventory was parsed, only implicit localhost is available
[WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost
does not match 'all'

Or is my syntax for variable assigning and referencing wrong?
Am I on the right path here?
my /etc/ansible/ansible.cfg file, under [defaults] is as such:
[defaults]
inventory = /opt/ansible/inventory/aws_ec2.yaml

While I haven't worked with plugin: aws_ec2, here's some general Ansible information that might help you:
The order of inventory files matter; the files given first will be read first; the ones read last will take precedence.  In your execution, you're referencing secrets.yaml last, so when aws_ec2.yaml is read, the secrets are still undefined.
You're trying to define a variable based on another variable with the same name.  That's asking for trouble; rename the reference variables, as that will help you debugging.
When defining variables for the inventory, it's generally better to not do that directly on the inventory itself.  See the link below on how to do that:
https://docs.ansible.com/ansible/latest/user_guide/intro_inventory.html#organizing-host-and-group-variables
Finally, keyed_groups, regions and all these other keywords are specific to aws_ec2, and they'd work because you have plugin: aws_ec2 at the top of the first file.   You have no such definition on secrets.yaml, so Ansible would probably treat it as a normal inventory file, where that structure would not be accepted.  Your approach here is probably wrong, then.
Again, I never used that inventory plugin.  It may be the case that the inventory plugin does not allow for variables at that point.


What are known efforts to establish DevOps ontology model? Today, for many domains there is a semantic ontology for making domain data machine-readable.
There are quite many terms in the DevOps and related domains, often established by vendors but applicable to other tools, like declarative pipelines in Jenkins.

What are current efforts to establish the ontology of the DevOps domain?

References/Further reading:

Ontology Development 101: A Guide to Creating Your First Ontology
Ontology (information science)
Ontology repositories
Ontology for Media Resources 1.0

According to Wikipedia, the first reference you linked,

  In computer science and information science, an ontology is a formal naming and definition of the types, properties, and interrelationships of the entities that really or fundamentally exist for a particular domain of discourse. It is thus a practical application of philosophical ontology, with a taxonomy.

The problem with developing an Ontology Model to define the terms of DevOps is that DevOps encompasses multiple domains and these domains can often have overlapping or contradictory terminology.

In fact, sometimes even within a domain there will be conflicting terminology. So it is difficult enough to establish a standard for a single domain, let alone multiple domains.

As such, it is doubtful there will be any attempt (or at least successful ones) to establish formal ontology rules for DevOps any time soon. (And it would have to start with formally defining DevOps first, and so far we have had trouble even making that clear).


How do I get dict key from ansible playbook variable? Following is the desire output:

domain.com is in gh4.allserevers.com

The idea is to pass domain name ie domain.com in playbook and playbook will lookup its A record then check it against the predefined variable. If ip is found in variable then it will display the corresponding key.

Here is my ansible-playbook role:

roles/test/tasks/main.yml

    ---
    - name: Simple A record (IPV4 address) lookup for domain.com
      set_fact:
         IP: "{{ lookup('dig', 'domain.com.')}}"
      register: output
      tags: test

    - name: checking if ip belongs to gh4
      debug:
        msg: "domain.com is in gh4"
      when:
        - IP in hosts.gh4
      tags: test

roles/test/vars/main.yml

    ---
    hosts:
      gh4: [127.0.0.1,127.0.0.2,127.0.0.3,127.0.0.4,127.0.0.5]
      gh5: [127.0.0.7,127.0.0.8]
      gh6: [127.0.0.10,127.0.0.11,127.0.0.12]

If I understand your question correctly you want to pass in a domain to find what could be a physical location or data center. I hope this is helps you!

roles/test/vars/main.yml

---
domains:
  gh4: [127.0.0.1,127.0.0.2,127.0.0.3,127.0.0.4,127.0.0.5]
  gh5: [127.0.0.7,127.0.0.8]
  gh6: [127.0.0.10,127.0.0.11,127.0.0.12]

roles/test/tasks/main.yml

---
# Make sure we have a domain passed in
- name: Ensure domain is defined
  assert: { that: domain is defined }

# Obtain the IP
- name: Simple A record (IPV4 address) lookup for {{domain}}
  set_fact:
      IP: "{{ lookup('dig', '+short', '{{domain}}')}}"
  register: output
  tags: test

# Check if the IP belongs to any location
- name: Checking location of IP
  set_fact:
    domain_location: "{{item}}"
  when:
     - IP in domains[item]
  with_items: "{{domains}}"

- debug:
    msg: "domain.com is in gh4.allserevers.com{{domain_location}}"
  when: domain_location is defined

Example Run
With domain.com pointing to 127.0.0.12

root@DESKTOP-U4VN2CS:/mnt/c/ansible/roles# ansible-playbook ../playbooks/test.yml  -e domain=domain.com
PLAY [127.0.0.1] ****************************************************************************************************

TASK [Gathering Facts] **********************************************************************************************
ok: [localhost]

TASK [test : Ensure domain is defined] ******************************************************************************
ok: [localhost] => {
    "changed": false,
    "msg": "All assertions passed"
}

TASK [test : Simple A record (IPV4 address) lookup for domain.com] **************************************************
ok: [localhost]

TASK [test : Checking location of IP] *******************************************************************************
skipping: [localhost] => (item=gh4)
skipping: [localhost] => (item=gh5)
ok: [localhost] => (item=gh6)

TASK [test : debug] *************************************************************************************************
ok: [localhost] => {
    "msg": "domain.com is in gh6.allserevers.com"
}

PLAY RECAP **********************************************************************************************************
localhost                  : ok=5    changed=0    unreachable=0    failed=0


How to use feature toggles and frequent deployment under SOX compliance? Feature toggles (or feature flags) enable me to deploy unfinished features without comprehensive manual testing of features toggled off. We can toggle the feature on for certain users, try it out, and then choose to toggle it off or on for all users. This enables more frequent integration as suggested by version control strategies such as Trunk-Based Development, and more frequent deployment of the trunk (master) branch into user's hands.

Sarbanes-Oxley rules require that all features that are in user's hands be fully tested.

So if I have a SOX-compliant system, how do I proceed with using feature toggles without having to do full manual testing of the feature toggled ON? It makes sense to do regression testing with the feature toggled OFF, but the most common need is to integrate and deploy good code but not incur the manual testing cost of the "on" state for a feature toggled off by default.

So any suggestions on ways to have Sox-compliant feature toggles without comprehensive manual testing of the toggled-off feature on every release?

It depends on how you're managing your feature flags. Ideally, you're using a flagging system that allows you to target specific users or segments of users so that an untested feature isn't inadvertently enabled for all your users. Some feature management systems (like LaunchDarkly) allow you to target users any way you want.

For example, a common workflow (for teams who use feature flags) is to release a new feature under a flag that's only turned on for the product team that's working on that feature. So, even if the feature hasn't been tested, it's guaranteed that only the right users have access to that new feature.

This allows your product team to continue to work on the feature and test it in production (if that's appropriate). Of course, a full set of tests should cover that feature before it's released, but it's nice that product teams have a way to iterate quickly without the hangups of a process that didn't involve feature flags.


What opensource solutions can be used to implement anomaly detection based on Riemann metrics? When Riemann used as system and/or application metrics collector, what are opensource tools can be used in order to implement anomaly detection monitoring?

I can think of 3 possibility (Not exactly Riemann metrics based, but around anomaly detection):

Outdated but the most efficient in anomaly detectionI know: Shyline + Oculus from Etsy Both tools available from etsy's github as Skyline and oculus, check the network graph for more up to date forks. Best one I found is earthgecko/skyline with its doc
Graphite and checking that the Standard deviation fits in the Holtz Winter's Confidence bands
ELK stack in version 5 with timelion which, as above,  allow to compute a standard deviation and alert if it goes out of confidence bands with the proper license.


How to create staging servers for multiple git branches? I have to create a new staging process for our development and testing.

At any given time, there are only around 4 git branches actively being developed and tested. Within each git branch, there may be database evolution scripts (straight SQL) that must be run, as well as evolution scripts from the backend for heavier processing (essentially these are HTTP routes that must be called within the app using admin credentials that perform database migrations and other changes that would be too difficult/not possible to script in the aforementioned plain SQL evolution scripts).

Our live DB is moderately sized ~ 4.2 GB. We have a brand new Dell PowerEdge server that is ready to be set up and is at my disposable.

I would love advice on the following questions, as well as to know how more experienced DevOps would approach this:

How can I run several different branches on the staging server? These branches pop up and disappear frequently as they pass QA and are merged into master and are released.
How would I set up the DB evolution system to make sure it always has appropriate DB for each branch? Each branch may make modifications to the DB in different ways that are not necessarily compatible with each other until they're merged.
How do I keep these branches up-to-date? Is there a way to auto pull commits on each branch?

Would love any more input as I'm a bit lost on how to set this all up. Current workflow is difficult for everyone involved: Devs have a completely isolated local copy of the app running locally and QA has 3-4 laptops in rotation that act as their staging "servers"

1) How can I run several different branches on the staging server?

Docker

  2) How would I set up the DB evolution system to make sure it always has appropriate DB for each branch?

This depends on how much you expect your DB to scale. You can get pretty crazy with methods to clone database data but typically you will want a master copy that you do not alter until your code is released into production - and keep good backups when you do. While you infrastructure can be immutable and disposable, your data is not. You can only emulate disposability with you data. 4.2 GB really isn't that much to copy. You could make a script to spawn a new database instance for each build to operate on, then discard it once testing is complete.

  3) How do I keep these branches up-to-date? Is there a way to auto pull commits on each branch?

You could consider using something like git hooks to trigger a build, force a code checkout or begin spawning a container and cloning your database. You could make an API call to a build automation system like Jenkins and/or use it to kick off a configuration management system like Puppet, Chef, Salt Stack, Ansible or something else. This would be more like an auto-pull than an auto push.

From the wording of your question, it's clear you are thinking of a mutable infrastructure, but consider using immutable testing deployments instead.


How to send SMS through AWS? Can we send SMS through AWS for the below case?

I have an application installed on Ec2 and that application runs few rules and may send SMS to end users if the rule conditions are met.

I wanted to integrate it with some SMSC and thought of giving a shot with AWS-SNS. However, I am not able to figure out how will i integrate my application with AWS-SNS.

Going through the documentation of AWS-SNS I am not sure if this can be achieved. It talks about creating topics and assigning E164 numbers to topics but does not tell us how to integrate this with our already existing application.

I believe you will need to do this programmatically using the AWS SDK as described in Sending a Message (AWS SDKs). Here is a quote a from it:

  To send an SMS message by using one of AWS SDKs, use the action in
  that SDK that corresponds to the Publish request in the Amazon SNS
  API. With this request, you can send an SMS message directly to a
  phone number

  ...

  Sending a Message (AWS SDK for Java)

  The following example uses the publish method of the AmazonSNSClient
  class to send a message directly to a phone number:

public static void main(String[] args) {
        AmazonSNSClient snsClient = new AmazonSNSClient();
        String message = "My SMS message";
        String phoneNumber = "+1XXX5550100";
        Map<String, MessageAttributeValue> smsAttributes =
                new HashMap<String, MessageAttributeValue>();
        //<set SMS attributes>
        sendSMSMessage(snsClient, message, phoneNumber, smsAttributes);
}

public static void sendSMSMessage(AmazonSNSClient snsClient, String message,
      String phoneNumber, Map<String, MessageAttributeValue> smsAttributes) {
        PublishResult result = snsClient.publish(new PublishRequest()
                        .withMessage(message)
                        .withPhoneNumber(phoneNumber)
                        .withMessageAttributes(smsAttributes));
        System.out.println(result); // Prints the message ID.


Ansible: Backing up variable value doesn't work as expected Attempting to save a variable's value for backup before modifying it doesn't work.

I have the following inventory file (note the custom ansible_port and user):

all:
    hosts:
            test:
                    ansible_port: 11111
                    ansible_user: user_foo
                    ansible_host: 192.168.56.101

And I have the following playbook:

- name: Test
  hosts: test
  vars:
          backup_ansible_port: "{{ ansible_port }}"
          backup_ansible_user: "{{ ansible_user }}"
          ansible_port: "{{ tmp_port }}"
          ansible_user: root
  tasks:
          - name: foo
            debug:
                    msg:
                    - "{{ backup_ansible_port }}"
                    - "{{ backup_ansible_user }}"
                    - "{{ ansible_port }}"
                    - "{{ ansible_user }}"

Running the following command:

  ansible-playbook bootstrap_ssh.yml --ask-pass -e 'target=test tmp_port=22'

The output is the following:

...
TASK [foo] *********************************
  ok: [test] => {
      "msg": [
          "22",
          "root",
          "22",
          "root"
      ]
  }
...

  Expected output:

...
TASK [foo] *********************************
  ok: [test] => {
      "msg": [
          "11111",
          "user_foo",
          "22",
          "root"
      ]
  }
...

Why is this not the case?

It is a totally expected result. You define the same variables in different places and ansible applies its variable precedence

First of all, remember that variables passed as extra vars always win: you can never change them in any other place or during the run (i.e. set_fact).

Now let's decompose what happens. The schema is basically the same for all vars where you don't get the expected result. I'll only give the example for backup_ansible_port. You can follow the spaghetti yourself for the others once you got it.

When you hit the debug task to display "{{ backup_ansible_port }}"

The first jinja2 template pass finds it should be an expansion of ansible_port as defined in your play vars.
ansible_port is defined with value "11111" directly in your inventory as a host var (see #8 in above precedence doc) But it is also defined as an expansion of tmp_port in your play vars (see #12 in above precedence doc). Higher precedence wins, so the second jinja2 template pass finds it should expand tmp_port
tmp_port has value "22" in your extra vars which will be the final value returned.


Authentication Error in Kubernetes I am new to Kubernetes. I am getting below error while creating deployments in Kubernetes.

<!--
You are authenticated as: anonymous
Groups that you are in:

Permission you need to have (but didn't): hudson.model.Hudson.Read
 ... which is implied by: hudson.security.Permission.GenericRead
 ... which is implied by: hudson.model.Hudson.Administer
-->

Can anyone give me solution for this?

I am using below command to setup deployment

 kubectl run my-app --image=gcr.io/<dockeraccount>/<dockerrepository>:dockerimagetag

I found this same error in StackOverflow, problably that you problem it's about your authentication anonymous

If help you: JENKINS Authentication Fails


Chef vs Ansible: will configuration be applied if connection is lost during configuration pulling/pushing? Scenario

Configuration management tools like Ansible or Chef often have a master/controlling machine and the machines they are controlling (agents).

Question

What if the connection between master and agents is lost during pulling or pushing configuration? Will the agents themselves carry on applying the configuration or will they stop?

Please give me the answer for the above question in regard to Ansible and Chef with reference.

The safest assumption to make here is that the configuration is not applied completely and you should try again once the network is back.  You can set things up to work without the network, but why go to that work when it will try again for you in a few minutes?

Chef

Chef tends to be run in the client-server mode as you describe so the chef-client agent on the "controlled" machine would keep running.  It would get stuck when it needed to get cookbooks or figure out which roles were assigned.  It will eventually time out and the chef run will fail.  Then it would try again in a few minutes.  By default the agent tries to do a chef run every 30 minutes.

If you're using chef-zero instead of the client-server style of chef then you could have everything staged on the "controlled" machine so that any network interruption wouldn't break it because of chef itself.  You might still have network dependancies in the cookbooks used.

Ansible ssh // push

Ansible started off by doing everything via ssh.  In this style of Ansible the completion state on the "controlled" node would be unknown but from the perspective of the user running ansible that node would clearly need to be retried.  Ansible creates a file for you with all of the hosts that need to be retried.

Ansible in cron // pull

If you're running Ansible from a cron job then it will face similar issues to the classic chef-client.  The node will need to phone home to find roles and updates, but it will time out and retry later.


Kubernetes installation, configuration and verification in a local environment I am a regular user of Docker since a few years and I have the Docker service installed and running locally on my Win10Pro host. It works well. Now I want to get started with Kubernetes so I used the gcloud command-line installation to install Kubernetes and got the following message:

  WARNING:   There are older versions of Google Cloud Platform tools on
  your system PATH.   Please remove the following to avoid accidentally
  invoking these old tools:

  C:\Program Files\Docker\Docker\Resources\bin\kubectl.exe

Looks like my Docker installation brought in another version of kubectl which Google advises me to delete. Should I really just delete the file from my Docker installation?

There appears to be two versions of kubectl.exe installed now, one which came with Docker and one which I just installed from Google. Kubernetes in the Docker GUI is not enabled. Should I enable it in the Docker settings GUI?

After a while I tried to install Minikube and start it which results in the following output:

C:\WINDOWS\system32>minikube start
* minikube v1.5.2 on Microsoft Windows 10 Pro 10.0.17763 Build 17763
E1114 05:07:00.856628    3836 driver_windows.go:74] Can't find VirtualBox registry entries, is VirtualBox really installed properly? The system cannot find the file specified.
* Automatically selected the 'hyperv' driver
* Creating hyperv VM (CPUs=2, Memory=2000MB, Disk=20000MB) ...
* Preparing Kubernetes v1.16.2 on Docker '18.09.9' ...
* Pulling images ...
* Launching Kubernetes ...
* Waiting for: apiserver
* Done! kubectl is now configured to use "minikube"
! C:\Program Files\Docker\Docker\Resources\bin\kubectl.exe is version 1.14.7, and is incompatible with Kubernetes 1.16.2. You will need to update C:\Program Files\Docker\Docker\Resources\bin\kubectl.exe or use 'minikube kubectl' to connect with this cluster

C:\WINDOWS\system32>

But despite the above message, I can run and expose a service as expected:

C:\WINDOWS\system32>kubectl create deployment hello-minikube --image=k8s.gcr.io/echoserver:1.10
deployment.apps/hello-minikube created

C:\WINDOWS\system32>kubectl expose deployment hello-minikube --type=NodePort --port=8080
service/hello-minikube exposed

C:\WINDOWS\system32>kubectl get pod
NAME                              READY   STATUS    RESTARTS   AGE
hello-minikube-797f975945-tj4g6   1/1     Running   0          13s

C:\WINDOWS\system32>minikube service hello-minikube --url
http://172.17.100.11:30575

C:\WINDOWS\system32>

The service does show its output at the above url. Do you think that my installation and configuration of Kubernetes is alright or is there something that I need to fix? Did I do this correctly?

I did not remove the kubectl.exe from the Docker installation and I installed Minikube without being adviced to do it but it seems correct. What's peculiar is that the client and the server now show different versions:

C:\WINDOWS\system32>kubectl version
Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.7", GitCommit:"8fca2ec50a6133511b771a11559e24191b1aa2b4", GitTreeState:"clean", BuildDate:"2019-09-18T14:47:22Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"windows/amd64"}
Server Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.2", GitCommit:"c97fe5036ef3df2967d086711e6c0c405941e14b", GitTreeState:"clean", BuildDate:"2019-10-15T19:09:08Z", GoVersion:"go1.12.10", Compiler:"gc", Platform:"linux/amd64"}

C:\WINDOWS\system32>

You probably don't want to delete the file, just remove it from your path.  You will need to look through your System and Account environment variables and look for PATH and remove C:\Program Files\Docker\Docker\Resources\bin\.

There is a good chance it that that will just work, I have both Docker Desktop and gcloud installed on this computer and I don't have the folder above in my PATH and both products work just fine without any warnings about old tools hanging about.

If you do continue to run into problems, for example with MiniKube, it can be helpful to use the Get-Command PowerShell command to figure out which executable is getting called:

Get-Command kubectl | Format-List

Will give you the full path to the executable that would be run if you issued the command kubectl:

Name            : kubectl.exe
CommandType     : Application
Definition      : C:\Program Files\Docker\Docker\Resources\bin\kubectl.exe
Extension       : .exe
Path            : C:\Program Files\Docker\Docker\Resources\bin\kubectl.exe
FileVersionInfo : File:             C:\Program Files\Docker\Docker\Resources\bin\kubectl.exe
                  InternalName:
                  OriginalFilename:
                  FileVersion:
                  FileDescription:
                  Product:
                  ProductVersion:
                  Debug:            False
                  Patched:          False
                  PreRelease:       False
                  PrivateBuild:     False
                  SpecialBuild:     False
                  Language:


Run more pods than node's CPU with k8s Let's suppose i have 5 k8s nodes with 2 cpu on each node.

This mean i have 10 cpus.

Now let's suppose i want to deploy a very small and basic application with 20 replicas.

Is there a limitation on k8s that will allow only one pod for one cpu and reject 10/20 replicas ?

Thanks

In addition to the good answer by @therianthropie, here are useful notes to help you understand the situation better.:

You can set requests and limits very arbitrarily (and yes, as fractions of a CPU and very small amounts of memory).
Kubernetes will not do its job well if you don't get real values for these.
If you under-shoot your request, kubernetes will schedule a lot of apps without requesting enough resources and you'll have problems with the CPU/memory/disk, and so on.
If you over-shoot your request a lot, your kubernetes nodes may be doing virtually nothing.
Some apps are very  CPU bound, some memory bound, some use virtually no resources.  So, you really can't guess.
Prometheus is a popular tool that you can install into kubernetes with basically a single command.
You can chart things in grafana.
So, then you choose your arbitrary  values, then watch the charts as you use the app, and then you can adjust them to something sane / useful so kubernetes can do its job well.  You can also set alerts with something called alert manager if your values are too ridiculous (it's a good idea once you're ready).
Note - requests are what kubernetes allocates for your pod.  Limits are the max resources your pod can spike to when it needs them. There is a general assumption that not all pods will spike in usage at once / or over a long term or else this idea would not work.


Is AWS Fargate suitable for web applications? I'm making a web-app, I've tried both ECS and EKS at AWS. It's been very comfortable to use Fargate instances instead of EC2. But I'm not sure I understand how to use them since when I count at it, it's so much more expensive. Prices
In the prices it says ~$0.05/h/vCPU which would mean 37.5$/m/vCPU. Which is very expensive compared to EC2. But from what I've understood Fargate is more suitable for container that have a limited lifetime? Does this mean that they are not useable at all(or atleast price worthy) for a webapp that's running 24/7? My web app can really shutdown an start up by requests, it would be to slow. Is there something I'm missing?

Fargate is absolutely suitable for web apps. We have many customers that use it that way (and many others that use it in a more "batchy" way). Fargate is a managed compute engine and AWS takes a more responsibilities in terms of scaling, patching, security, and so forth so comparing raw Fargate costs to raw EC2 costs isn't how I would compare them. Having that said, raw EC2 capacity (like for like) is usually just 20% cheaper than raw Fargate capacity. If you factor in the cost of operations (less with Fargate) and the fact that you hardly drive EC2 clusters at full utilization, the comparison tends to skew towards Fargate most of the times. These are a couple of blogs that talk about this (blog one and blog two).


What are vertical scalability limits of JFrog Artifactory Pro? The essential feature of JFrog Enterprise over JFrog Pro (X) is that it introduces high availability and site replication support.

Obviously, operations in geographically distributed organizations with thousands developers and build processes require some planning also for this aspect.

Nevertheless, is there any data/study to show order of magnitude of load where you would say okay this is where we need not just some vertical scaling but parallel (say using some reference hardware setup)?

As @XiongChiamiov mentioned, it's a very hard question to answer, but here's a clue:

Artifactory is a network-bond server. It means given enough storage, CPU and memory, it will max out your network interface first. Network interface is considered maxed out at half its rated bandwidth limitation. That can help you do the math.


How to deploy a Kubernetes service using NodePort on AWS? I have created a cluster on AWS EC2 using kops consisting of a master node and two worker nodes, all with public IPv4 assigned.

Now, I want to create a deployment with a service using NodePort to expose the application to the public.

After having created the service, I retrieve the following information, showing that it correctly identified my three pods:

nlykkei:~/projects/k8s-examples$ kubectl describe svc hello-svc
Name:                     hello-svc
Namespace:                default
Labels:                   app=hello
Annotations:              kubectl.kubernetes.io/last-applied-configuration:
                            {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"labels":{"app":"hello"},"name":"hello-svc","namespace":"default"},"spec"...
Selector:                 app=hello-world
Type:                     NodePort
IP:                       100.69.62.27
Port:                     <unset>  8080/TCP
TargetPort:               8080/TCP
NodePort:                 <unset>  30001/TCP
Endpoints:                100.96.1.5:8080,100.96.2.3:8080,100.96.2.4:8080
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   <none>

However, when I try to visit any of my public IPv4's on port 30001, I get no response from the server. I have already created a Security Group allowing all ingress traffic to port 30001 for all of the instances.

Everything works with Docker Desktop for Mac, and here I notice the following service field not present in the output above:

LoadBalancer Ingress:     localhost

I've already studied https://kubernetes.io/docs/concepts/services-networking/service/, and think that NodePort should serve my needs?

Any help is appreciated!

More often than not you should not be using NodePort.

You're in AWS so you can consume the AWS Cloud Provider. Try using ingress resources or LoadBalancer service types. NLBs can do TCP/UDP port passthrough if that's what you need.

Edit: After further review, it looks like your just exposing a web server (I noted port 8080). NodePort is 100% not what you want. Look at Ingress


kubernetes statefulset - communicate addresses of other pods when pod launches? Is there some metadata that I can apply, or some DNS that I can query, such that when a pod is started, the addreses of all the other pods in that statefulset will be somehow made available to the newly started pod?

I want to run a stateful application (Riak) where every node (pod) in the Riak cluster needs to be aware of the addresses of every other node (pod).

Using the kubernetes statefulset controller, pods are started one by one.

I also wonder how I can somehow listen for some event in each Riak pod so that as instances are started, all the preceeding instances will be made aware of the new instance?

Following on from the information provided by @kaizenCoder - I modified the service so it had a configuration of :

spec:
  clusterIP: None

I then scaled up to two replicas, exec'd to one of the containers and ran dig again as mentioned.

root@grafana-0:/usr/share/grafana# dig SRV grafana.default.svc.cluster.local

; <<>> DiG 9.10.3-P4-Debian <<>> SRV grafana.default.svc.cluster.local
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 27002
;; flags: qr aa rd; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 3
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;grafana.default.svc.cluster.local. IN  SRV

;; ANSWER SECTION:
grafana.default.svc.cluster.local. 5 IN SRV     0 50 55555 10-1-0-77.grafana.default.svc.cluster.local.
grafana.default.svc.cluster.local. 5 IN SRV     0 50 55555 10-1-0-78.grafana.default.svc.cluster.local.

;; ADDITIONAL SECTION:
10-1-0-78.grafana.default.svc.cluster.local. 5 IN A 10.1.0.78
10-1-0-77.grafana.default.svc.cluster.local. 5 IN A 10.1.0.77

;; Query time: 0 msec
;; SERVER: 10.96.0.10#53(10.96.0.10)
;; WHEN: Fri Jun 28 12:12:32 UTC 2019
;; MSG SIZE  rcvd: 372

Much better - but unlike @kaizenCoder - I don't get back a name for each of the pods - it just uses the IP address - the names don't match - I would have expected the first pod to have had a name of grafana-0.grafana.default.svc.cluster.local - that nodes hostname is correct though - grafana-0.

Any suggestions on how to get the DNS to match the hostname ?

Update again - more weirdness - the other pods don't seem to be resolvable by hostname.

root@grafana-0:/usr/share/grafana# ping grafana-1
ping: unknown host

Presumably hostname only resolves in the pod as hostname resolves to 127.0.0.1 ... Which makes it a bit worthless tbh.

I feel like I've made some progress but still missing a piece. When a container starts up, how can software within it resolve :

It's own fully qualified name
The fully qualified names of all the other statefulset pods

I could do something like this to determine the node name and the other node names but it feels ugly.

root@grafana-0:/usr/share/grafana# dig SRV *.default.svc.cluster.local | grep 0-77
*.default.svc.cluster.local. 5  IN      SRV     0 11 55555 10-1-0-77.grafana.default.svc.cluster.local.
10-1-0-77.grafana.default.svc.cluster.local. 5 IN A 10.1.0.77
root@grafana-0:/usr/share/grafana# ip addr | grep eth0 | grep inet
    inet 10.1.0.77/16 scope global eth0

Yes you can by doing a DNS SRV lookup. Here's an example:

$ kubectl run -it srvlookup --image=tutum/dnsutils --rm --restart=Never -- dig SRV jira-headless.default.svc.cluster.local

; <<>> DiG 9.9.5-3ubuntu0.2-Ubuntu <<>> SRV jira-headless.default.svc.cluster.local
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 56846
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 2

;; QUESTION SECTION:
;jira-headless.default.svc.cluster.local. IN SRV

;; ANSWER SECTION:
jira-headless.default.svc.cluster.local. 30 IN SRV 10 50 0 jira-0.jira-headless.default.svc.cluster.local.
jira-headless.default.svc.cluster.local. 30 IN SRV 10 50 0 jira-1.jira-headless.default.svc.cluster.local.

;; ADDITIONAL SECTION:
jira-0.jira-headless.default.svc.cluster.local. 30 IN A 10.244.11.152
jira-1.jira-headless.default.svc.cluster.local. 30 IN A 10.244.0.188

;; Query time: 6 msec
;; SERVER: 10.233.0.3#53(10.233.0.3)
;; WHEN: Fri Jun 28 00:18:14 UTC 2019
;; MSG SIZE  rcvd: 143

In the example above I have a headless Service called jira-headless. If you pay attention to ADDITIONAL SECTION: you'll notice it provides the SRV record in the following format:

_service._proto.name. TTL class SRV priority weight port target.

Which maps to the Pods (along with their IPs) backing the Service:

jira-0.jira-headless.default.svc.cluster.local. 30 IN A 10.244.11.152
jira-1.jira-headless.default.svc.cluster.local. 30 IN A 10.244.0.188


Using HTTPs in application vs reverse proxy Is there any difference if you use TLS in your application for example in node or java(spring), and if you use normal HTTP in your application and then set up Nginx as a reverse proxy and configure your TLS certificate throw Nginx?
What would be the benefits of using one instead of another?

You will find people from both camps (terminating TLS as soon as possible vs terminating TLS in the application code) but it's not as serious as for example tabs vs spaces.

The benefit of terminating TLS by reverse proxy is that application developer does not have to implement TLS in the application itself - possibly introducing bugs. Another benefit mentioned by jdog is the CPU cost; which is negligible these days - unless you are running 100s of micro-services and it all adds up to a large chunk of money.

Personally I always go with reverse proxy terminating the TLS unless there is a specific requirement against it (security reasons). I also make sure that service behind reverse proxy listens on the localhost / local network only - is not accessible directly form the internet.

You may find wikipedia article about TLS termination useful.


How to ensure that git subtrees are kept up to date? Problem:

We use Data Transfer Objects (DTO) to ensure consistent interfaces across multiple services using TypeScript.

We're currently using a git subtree in order to make sure that each repository has its DTO dependencies up to date. These DTOs are changed semi-frequently.

The challenging part is that subtree pull / subtree push are outside the traditional git process for most developers.

Question:

What process patterns have people found work well to ensure each related repositories subtree is up to date on each push / pull? Post commit hooks? CI scripts? Etc. etc?

GIT hooks could work, but the trouble is that they're not automatically installed when the developers pulls a workspace. Which means they might end up being installed/used correctly or not. Unreliable.

CI scripts could work as well, but the're reactive, simply indicating that something went wrong, in most cases humans still need to identify the culprit (which might be caused by an improper DTO subtree or by something else) and fix the problem.

If you want your developers to properly pull their subtrees - offer them a scripted wrapper to do so - put the effort into that script so that it is simple to use and robust/working well such that the developer's workflow is effectively simplified. If you manage to do this (most of) the developers will see the benefit and they will use it.

For the push side - you need to take into account that mistakes in push will happen (even with the above-mentioned script, if it's not used by all developers, for example). Everyone makes mistakes.

Personally I favour preventing developers from directly pushing their changes into the integration branch and thus blocking possible mistakes from affecting the integration branch.

Instead I'd have all candidate changes (i.e. either pushed in separate branches or just as diffs) funnelled through a mandatory centralized pre-commit verification system. Automated, of course, with the proper git subtrees management made reliable through that automation. The changes would only be committed/merged into the integration branch automatically by this system, if they meet the quality checks.

This approach is applicable to any potential regression-causing mistakes, not just improper DTO subtrees. An example of such approach would be the gerrit-based development on OpenStack.


Code splitting practice using CDN I have a web frontend app uses code splitting to reduce the individual file size. And it's distributed on a CDN and works most of the time.

A problem occurs when we update the project at any time. Assume that our project consists of following files:

index.html
main.1ef1a.js(where 1ef1a, as well as bce31 in the next line, is content hash)
on-demand.bce31.js

index.html imports main.[content-hash].js and main.[content-hash].js loads on-demand.bce31.js on demand.

The error scenario is a user loads our page just before our update and he/she gets index.html and main.1ef1a.js and browser caches them. And then our update happens: we replace the files, flush the CDN, and following files are get distributed:

index.html now imports a new version of main.js
main.2a3db.js
on-demand.5231f.js

Then user does something that triggers the loading of on-demand.js, while since his/her main.js is old, he/she requests on-demand.bce31.js, which no longer exists, and then the page crashes.

He/she is a very nice user with patience and tries a refresh, but the browser serves the cached index.html and main.js again and he/she cannot get ouside the loop.

We cannot assume our users to have certain knowledge to perform a "Clean cache and force reload", all we get are disappointed users who think our website sucks.

I think code splitting and CDN are both widely used techniques and there should be some nice solution that I'm missing. And what is that ?

Your main problem is that you're flushing the CDN when you're deploying a new set of artefacts, which will effectively ruin anything still in progress (whatever its state). You should add the new artefact versions and leave the older ones in place, at least for as long as your published cache lifespan for the top index.html file is.

This way even if users look at an older (cached) version of the page which references other older version artifacts they'll still be able to download them (users have no clue that a new version is available as the cached version is immediately served as long as it's not expired). And whenever the cache expires users will get the new version of the top page, with the references to the new artefact versions.


How to ensure that removed allowed ports will be removed using Ansible's UFW module? The UFW module document has been read, but it is unclear how to let ansible remove ports that are not defined anymore.

It works to add new ports, but now 5 ports have been removed and I do not want to login to each of the more than 50 servers.

It is possible to write some custom exec, but I would like to use the Ansible's UFW module's functionality. Does it support such a scenario?

Based on:

# Allow OpenSSH. (Note that as ufw manages its own state, simply removing
# a rule=allow task can leave those ports exposed. Either use delete=yes
# or a separate state=reset task)

The following snippet was added to the top of the file:

---
- name: Reset UFW
  ufw:
    state: reset

Once ansible was run:

TASK [rolename : Reset UFW]

the undefined ports were removed


How to mange data access in a CICD pipeline for multiple users with different permissions TLDR Question;

Is it possible to setup a CICD pipeline so that it executes with the permissions of the user who triggered the build? If so which tools support this?

Full Question;

Apologies if this is a naive question as I am new to CICD.  We are trying to setup a CICD pipeline for our project which requires access to a restricted database.  The issue we are facing is that each user on the project has different levels of access (which I believe is managed via LDAP) within the database (for example user A can access datasets X, Y & Z  whilst user B can only access Z & W).

Due to the nature of our work, code is typically executed interactively (something we are trying to move away from) and such to execute any api that access the database users need to enter their username and password i.e.

get_data(dataset = A, username = $USER, password=$PASSWORD)

Our challenge is thus how to enable the CICD pipeline to access these data-assets without accidentally enabling people to access data that they shouldn't have access to.

I think the most obvious solution is to create a user account for the CICD tool itself, only giving it access to the datasets it needs and then embedding the username and password as secrets within the CICD tool. Our fear with this is that it then means that the project owner becomes responsible for ensuring that everyone with write access to the project has access to the data-assets that the project uses.  Given user permissions change fairly frequently (as does project ownership) this doesn't seem like the safest or most sustainable way of solving the problem.

Ideally we are looking for a solution where the CICD tool executes with the same permissions as the user who triggered the build. Are there any tools that support this ? Or alternatively are there any different solutions that could solve our use-case?

Any help/guidance/pointers would be greatly appreciated !

My general experience is that CI/CD tools don't support impersonation or claims based authentication well or at-all. In addition you probably want the CI/CD tool to have more access than your users, because you want them to only use the CI/CD tool to change things.

The way I have handled this in the past, however, was as follows:

Have the pipeline call the API of the CI/CD to find out who ran the pipeline
Use this pipeline to look up the group's that the user is a member of this (i.e. in LDAP)
If the member is of the appropriate group, continue. If not abort the build.

It's quite important to abort rather than fail the build because then it differentiates between an invalid user running the build, and a valid using running a build which then fails.

Another approach would be to sync LDAP groups with your CI/CD platform, and set permissions that way, although that may not give you the granularity you want.


Dockerized node.js and bind mount permission problem I have an interesting problem with node, docker-compose and local development using bind mounts.
I have a bind mount volume, called www-public, owned by my host node user (I created a node user and chowned the dir to node:node). The container (official node) is set up so it also run as node user. At some point in my node app, i need to write to this folder and i get the permission error. This seems strange at first, but it makes sense, because i figured then, that my host nodeuser has an uid of 1001 while my container node user has an id of 1000. If i just add this line to my Dockerfile:
RUN usermod -u 1001 node && groupmod -g 1001 node

The permission problem goes away.
How do i fix this in a way, that I don't hard code the container node user id to match the node user id on the host like i did with the upper RUN command.

In your Dockerfile:
ARG UID=1000
ARG GID=1000

RUN usermod -u $UID node && groupmod -g $GID node

Then using docker build:
docker build --build-arg UID=$(id -u) --build-arg GID=$(id -g) .

The ARG lines provide defaults. If you leave off the --build-arg flags, they will be used. If you do use the --build-arg flags values those will be used instead.


Are there examples of other production grade container runtimes? While we say "Docker is just one possible container runtime", are there other implementations used in production environments as of late 2019, and do they have different approaches? (while Docker being a console wrapper for LXC more or less, right).

Are there other implementations

Yes. The Container Runtime group of the Cloud-Native Computing Foundation has a list of runtimes:

They include:

Containerd
CRI-IO
Firecracker
gVisor
Kata Containers
LXD
Nabla Containers
Pouch
runc
Singularity
SmartOS
Unik

Do they have different approaches

While they cannot all be compared with each other directly, they all offer a way to provide a container runtime. In some cases, this runtime leverages the kernel, in others, they are in userspace. While some are explicitly proprietary, others are explicitly Open. Some are supported by large corporations (Google, IBM, Amazon, Alibaba), others by foundations or open source communities.

All of these are used in production, but perhaps vary in their popularity.


Automatic Docker Hub builds from GitLAB, not github As per title. I would like to configure automated builds on Docker Hub, as specified here.

My build context (the build context is a repository containing a Dockerfile) is hosted in GitLAB.

This feature is not supported as the issue in the link says.

They promised it for late 2016 but now at 2017 the issue is still open.

One comment was to look for gitlab CI, which might be my best guess of workaround  for now, waiting the Docker Hub version to come.


Can the Waterfall method leverage CI/CD+ and still be considered Waterfall? Often it seems to be assumed or implied that DevOps is simply the Agile method expanded beyond programming to things like Operations and Configuration as Code, and so forth. So, this has left me wondering: Can a classic Waterfall method incorporate CI/CD and Configuration as Code and still be considered truly a Waterfall software development method?

I don't see why not.
Sequential SDLC models, like waterfall, are about the organization of steps. The requirements are "finalized" before extensive design begins. The design is "finalized" before the implementation begins. System testing happens after the requirements are satisfied. Some waterfall-like models, such as the sashimi model with some overlap between these phases or waterfall-with-subprojects, are multiple waterfalls after the overall system requirements and architecture have been completed.
There's nothing in the organization of these steps that would prevent continuous integration or configuration of code. Continuous deployment to a development environment is also possible, but continuous delivery or continuous deployment to production may or may not be possible, depending on the usability of interim builds before the requirements are complete. These practices can still be used once system implementation begins. CI and automated tests can help developers find issues during development before system integration and system testing activities begin. Configuration as Code can help with the configuration management of the system, allowing it to be versioned alongside the software system and rolled out to different test and production environments when it's time.
The key difference between highly sequential models and iterative and incremental models is responding to feedback, not the practices that can be used. Some practices may not make as much sense to use, especially if they are about getting and responding to feedback. If you have "finalized" requirements and "finalized" designs before you can start getting feedback on the system, stakeholder feedback will likely change those requirements and/or designs. Because of the scope of the work of requirements and design and the impact on the built system, late changes are expensive.


When is it not appropriate to use serverless technology such as AWS Lambda? With serverless computing using AWS lambda, Amazon(AWS) is taking the ownership of architecture, administration & deployment aspect of AWS services(VPC/EC2/ELB/AScaling/Rtables/SGroups/etc...) for the application to run.

With this approach, AWS user losing control on designing an optimal VPC architecture that fine tunes with application

So, when not to think about serverless computing?

There is likely no general answer to this question, but some reasons to not consider Lambda (or other serverless platforms) include:

App requires a language which is not available (C++, Haskell, Erlang, etc.)
Organization has existing substantial investment in containerized (or other) app development model, including developer knowledge, build and deployment pipelines
Portability is concern; i.e. the organization wants/needs to keep app in a state which is portable across platforms (AWS, Azure, various k8s implementations, and so on).
Cost. Do costs of running the Lambda function(s), as well as associated resources (API Gateway, VPC components) exceed budget? This is unlikely if the org. is already running in AWS, but can be a factor when compared against existing (and paid-off) on-premise resources.


Best practices on how to erase everything in destination deployment folder and restart the app after deployment using Appveyor So far I have managed to get a simple deployment procedure running via SFTP. I really need to make two more steps:

Erase everything inside the folder that I am deploying to before I deploy a fresh build into it or (even better not sure this is possible) see what has changed and upload only the required and erase the not required.
Restart the app after deployment is done.

What could be the best solution for this with Appveyor? So far I see the easiest way is to possibly include a power shell script that will connect via ssh and clear out the folder, not sure how that would work with restarting the app though. Would be really great to hear some more experienced users about this solution.

So it actually only took one line of code:

1) First take the SSH public key from the settings page inside the appveyor project and make sure you put it into the authorized_keys inside user_that_you_are_going_to_login_with_from_appveyor/.ssh/authorized_keys

2) To clear the folder before deploying to it on a Ubuntu server

before_deploy:
  ssh user@host -t "rm -r -v /pathtofolder/*;exit; bash --login"

3) Now about the restart of the app I wanted to use after_deploy however there is a problem with that. After deploy fires straight after deploy and doesn't wait untill upload is finished. I wanted to use it to run

systemctl restart myapp

However the problem seemed to solved itself because systemctl restarted my app at the moment when it was substituted with a new one or maybe at the moment when I call it the first time after the files have been reloaded. So far it seems I have a new app running after I run the script.


Auto tune kubernetes requests/limits? I know of ways to set up monitoring (e.g. prometheus/alert-manger) to notify me if kubernetes pod requests/limits are too far off actual container usage.

Are there any systems built to help you actually act on this information in some automated fashion though?  This may sound odd as I realize automatically raising a limit makes it... not a limit?

Having said that, it still sounds useful to be able to automate tuning across a large k8s environment.  This may be especially true during non-prod/staging when you're doing load testing and trying to figure out what the reasonable limits are in the first place.

Automating this kind of behaviour is a quite risky since you have ResourceQuotas usually set up as a constraint for a namespace and from there on the pods must be aligned with it.
You might end up with a pod non being scheduled at all.


How can I build a Linux VM that I can use to test Rest API calls in a lab? I work in Enterprise CloudOps Support and the products we support use Rest API for notifications. Frequently we get requests to help troubleshoot solutions like Pager Duty, PagerTree, ServiceNow, etc. Obviously I can't and won't signup for trials of these products, nor do I need to know them. However, I do need to know how Rest API works and how to set it up. I've been researching it and so far all I've found is this MS DevOps write-up: https://microsoft.github.io/PartsUnlimitedMRP/pandp/200.1x-PandP-LocustTest.html

However, as of yet I can't seem to make sense of some of the steps. I thought there must be a better way, so here I am. Ultimately what I'm after is a VM that I can build in a lab and use for testing Rest API calls that relay notifications, alerts, etc.

You can test basically any rest api call from postman, a free app that runs perfectly well in windows or Linux (or Mac I’m sure).  You don’t need anything special.

If you want to use Linux specific tools, you can sign up for AWS free tier and make a Linux box in minutes following any of their tutorials (it will be free for 1 year).

You can also use docker on your computer (including on windows).  Docker will be suitable for running most non GUI based or web GUI based Linux apps.  I see a locust image here for example - https://docs.locust.io/en/stable/running-locust-docker.html.


Docker Build Ending Up `returned a non-zero code: 1` Running mktexlsr /var/lib/texmf ... done.
Building format(s) --all.
        This may take some time... done.
build.sh: line 81: -: No such file or directory
The command '/bin/sh -c bash build.sh' returned a non-zero code: 1

You can view the code here.
https://github.com/austinsonger/GreenOpenVAS

Line 81 of build.sh is trying to run python3.  But the problem is more fundamental because you aren't getting any of the earlier echos or apt-get command's output.  I would try:
chmod +x build.sh
./build.sh

to let the sh-bang line do its job.


How to make .DACPAC from Azure Sql Server using Azure DevOps release pipeline? I'm using Terraform to create my environment and I'd like to deploy Azure Sql Server along with other things. I use terraform so I can easily destroy all deployed things and reduce costs when I'm not using them.
So, I decided that I can add Azure Sql Server into my Terraform file but I need to store my information into backup file so I can restore it with "Azure SQL Database deployment" task from pipeline. Hovewer, I haven't find any indication that it is possible to create a .DACPAC from within Azure DevOps.
So, is there a way to create and save a backup of my database (preferrably, into a blob storage) from Azure DevOps pipeline? I looked into azure shell (az sql database), but haven't find anything of value

Ok, after researching some more time, I've found this gem https://markheath.net/post/backup-restore-sql-database-azure-cli. In this article author uses az sql db export and az sql db import to save database into BACPAC and restore it back on the clean database. But here comes a pitfall - Azure CLI task v1.0 uses CMD so it's not easy to translate from Bash syntax to CMD, and Azure CLI task v2.0 preview which has option to choose Bash as an interpreter doesn't have az installed (it's probably a bug and already has a post on uservoice) so one needs to rewrite the script into Powershell if they want to use it.
So, it's not completely what I was looking for but it does its job.


SVN Externals in Git I have a handful of projects that utilize SVN externals. How should these externals be handled in Git? For reference, these externals are shared by multiple repositories and are updated very infrequently, but typically the changes are very impactful (non-backwards compatible).

One could git svn clone a subversion repository and use submodules.


Restart docker container from another container I'm in a planning phase for a little tool for myself. I have to say that I have close to 0 experience with docker.

My setup

1 x Nodejs App in a Container (lets call it App)
n x Instances/Containers of the same image (lets call it Agent)

What I want

I'm looking for a way to restart one of the Agents from within my App.

The problem
I just managed to create my first docker-compose. From here I'm stuck. I just have no idea how to cross-access the container from my Apps container (by script) or if this is even possible and then force to restart it (like docker restart #hash does).

My current docker-compose.yml (if that helps in any way)

version: '3'
services:
    app:
        container_name: app
        volumes:
            - .:/usr/src/app
        restart: always
        build: .
        ports:
            - '8080:3000'
        links:
            - agent

    # no control over this as this is an existing image from docker hub
    agent:
        restart: always
        image: image/from/dockerhub
        ports:
            - '1234'

... which I would run as docker-compose up --scale agent=n.

I'm not necessarily looking for a complete solution. If someone could guide me into the right direction I would really appreciate it.

Thanks

In majority of circumstances I wouldn't recommend this in production, but to restart docker container from another container, you need to install docker in that container and then run that "master" container with

docker run -v /var/run/docker.sock:/var/run/docker.sock

as described here: https://jpetazzo.github.io/2015/09/03/do-not-use-docker-in-docker-for-ci/ (look for "The solution" part of the article).

But check your architecture and see if you really need to restart one container from another or if there is some other way.


In Jenkins, how can parameters that contain spaces be referenced? Google has failed to even lead me to a proper answer as to why spaces are permitted in names of Jenkins parameters, let alone how to reference them. Assuming it's not just a major oversight, how does one access these parameters?

For example, I have a string parameter named "IP Address". I could rename it "IP_Address" but I'm trying to avoid the underscores_as_spaces if I can.

I've tried to refer to this parameter using "${IP Address}", "IP Address" and even "IP\ Address" but nothing seems to work. Is there any way to actually reference these?

You can use params['IP Address'].

Think of params as a Map containing a key 'IP Address'. If the key didn't have a space, then you could use params.IPAddress or params['IPAddress'], but when there's a space, you can only use the latter syntax.

That syntax is also useful when you're computing the key name to look up, and have stored it in a variable:

final String myParam = "param${i}"
// can't say `params.myParam`
echo "Value is ${params[myParam]}"


How to list containers that are using redis/ubuntu image in Docker I am trying to figure out the solution for the above question. I tried the  commands docker ps -a -f "label=redis" and docker ps -f "name=redis", but that didn't gave any output.

Any help?

I think you want the ancestor filter :

From the Docker Docs on ps:

  ancestor - Filters containers which share a given image as an ancestor. Expressed as image-name[:<tag>], image id, or image@digest

$docker run -d --rm redis
c4cf36467d63c66f7d3d8271dd0f8e51ede4a627048eb5e294a2445f5b0d30ef
$ docker ps -f "ancestor=redis"
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS               NAMES
c4cf36467d63        redis               "docker-entrypoint.s…"   16 seconds ago      Up 15 seconds       6379/tcp            elated_yonath


How to communicate in a multi-organization spanning project In DevOps S-Pillar of "CALMS" says that it is also about sharing knowledge.
At a present project I am working on, I am sitting at the interface between two organizations together with another "DevOps"-guy.
A lot of our time is wasted because we have partly redundant meetings (with the organization on the left side) and we wait partially longer times until somebody from the right side can assist us in solving some issue.
Sometimes I also spend hours to find out who is reponsible for getting some information for me that I need.
What would you try to address to make more sharing possible although it is different organizations that are working on that project?

Establish a wiki (such as Confluence, DokuWiki, etc).
Establish a chat/communications channel (such as through Slack or Mattermost).
Share some code repositories (such as through GitLab, Bitbucket, GitHub).
Share some binary repositories (such as through Artifactory, Nexus, Pulp).
Occasionally send out a "newsletter" email that showcases what your team are trying to accomplish.


What is the difference between packages and artifacts? I would like to understand the difference between package and artifact.

Also In what context we should use these terms.

Mainly an artifact is the result of of a build phase, this mean a package is an artifact of a kind.

A package is usually a way to install a software or application, it includes the software itself and some intelligence to setup and configure the software.

Calling an artifact a package usually comes when the artifact (whatever it is, from a .deb to a simple zip file) contain more than just the application, it could be a simple script to help configuring the software and setting proper permissions or other softwares needed.


In Azure Devops and TFS, is it possible to create a build artifact and release definition in a different collection than the original code repository? I have code repo held in Collection 1 and, using this code repo, I am trying to create a Build artifact and Release definition in a different Collection 2.

Is this even possible?

Since on-premise TFS and the newly renamed Azure DevOps Server 2019 still have the concept of collections, which is the highest level of separation of data you can have (separate physical databases), you'd need to treat a code repo in Collection 1 as an external repository rather than try to discover it as a known internal repository.

You can do this in the Build pipeline of a build in Collection 2 by selecting External Git (if your repo is Git, TFVC is not supported), then clicking Add Connection and entering the Git repository URL and login credentials like in the screenshot below.  Once connected, your pipeline, if it produces artifacts, will store them in Collection 2, which can be used for a Release pipeline.


Is it possible to host an ECS cluster with p2 or other GPU instance types? It seems from the GUI that it is not possible (the GPU instance types are simply not listed when you go to create an ECS cluster and look at the list of instance types). However, there is an official AWS blog article here indicating that Amazon itself uses ECS in this way.

I see that there is a an ECS CLI command register-container-instance that could theoretically be used like so:

Manually create an EC2 instance with a GPU type.
Use register-container-instance to register your GPU EC2 instance to an ECS cluster.

However, the documentation notes that this command is supposed to be internal use only, and also it is a bit of a hack and a lot of extra work.

You can use custom instances of any type with ECS -- the key is running the ECS container agent on the instance of your choice. So, in general, I followed this documentation on that process with success -- http://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-agent-install.html -- but do note that you probably want to also specify the ECS cluster into which the instance goes (otherwise it will go to "default" cluster).

To specify the cluster you specify env var ECS_CLUSTER, typically in /etc/ecs/ecs.config prior to running the container agent. See here.


Looping an Array with Missing Keys in Ansible Tasks I'm attempting to create an Ansible Role to create users... I want it to be able to accept a list of dicts and if a key is missing from the dict, it will fall back to Ansible's default.
For example, given the following data:
---
users:
 - { name: foo, group: bar, create_home: no }
   { name: goo, group: bar }

I'd normally loop over the list with something like:
- name: Add users
  ansible.builtin.user:
    name: "{{ item.name }}"
    groups: "{{ item.group }}"
    create_home: "{{ item.create_home }}"
    state: present
  loop: "{{ users }}"

However, because create_home doesn't exist for the second user, it'd throw an error. How could I get it to fall back to Ansible's default for keys that don't exist?

Q: "How could I get it to fall back to Ansible's default for keys that don't exist?"
A: Omit such parameters, e.g.
    create_home: "{{ item.create_home|default(omit) }}"


Can you replicate Active Directory from a Corporate Intranet to an AWS VPC where there is an outbound-only link? In our work environment we have a standard Corporate Intranet with Active Directory.

We've been granted limited access to an AWS VPC. Our connection allows outbound (from the Intranet to the VPC) but not inbound.

That is - if we run a webserver in the AWS VPC, then a client in the Corporate Intranet can connect and browse to it. But a client in the AWS VPC cannot connect to a webserver in the Corporate intranet. Note that this 'outbound connections only principle' applies to all ports, not just http port 80.

An associate has suggested we need to replicate our Active Directory down to the AWS VPC AD. I think it is not possible to do a one-way replication.

My question is: Can you replicate Active Directory from a Corporate Intranet to an AWS VPC where there is an outbound-only link?

Yes, you just need a Read Only Domaine Controler (search for RODC) in the vpc and you'll be able to have a one way replication as the read only controler won't need to send datas back.

This bring limitation for clients in the vpc, as they won't be able to reach a writable DC they can't change passwords and you'll need a careful site architecture in your forest.


How to run Dockerized graphical applications on a headless server with packages required for display remotely on clients as part of a separate image? In this first section I detail some of the terms I have used:

Dockerized graphical applications: A Docker container run from a Docker image which runs an application which (traditionally) uses x11 (or similar) to display the application on the host machine. More details in second section.

packages required for display remotely on clients: For example, a VNC server which allows a user of a local client (e.g. a laptop with desktop Ubuntu 18.04 installed) to navigate through the browser to an particular host-name and port.

In this second section I give an example of the usual way to run a Dockerized application, using gedit as an example:

Example Dockerfile:

FROM ubuntu:18.04
RUN apt-get update && apt-get install -y gedit
CMD gedit

Example Docker build command:

docker build -t gedit:latest .

Example Docker run command:

Once-off (can potentially be a harmful command, use with caution):

xhost +local:docker

Subsequently:

docker run --rm --name=gedit -e DISPLAY=$DISPLAY -v /tmp/.X11-unix/:/tmp/.X11-unix gedit:latest

Please note: There are some security concerns running xhost +local:docker so to reverse this run xhost -local:docker

Sub-question / curiosity: The above installs x11, etc into the image. Is there any way to have a separate image with all of those dependencies and have multiple containers share folders in order to satisfy the x11 dependencies?

In this third section I detail an example in which my question is relevant:

I want to keep the Dockerized app as explained in section 2, which runs well locally on a client's laptop but have it as part of a cluster of docker containers (using docker-compose for example) so that the image built from the dockerfile in section 2 is unmodified.

I want to include an additional container which has all the VNC and x11 packages required so that the client can run the application remotely.

Which mechanisms allow the above to take place?

You can't run a program running on x11 without a x11 buffer somewhere, be it vnc, xvfb or whatever. You will always need x11 libraries on the system running the program or it couldn't work at all. That's akin to  trying to run a C program without libc on the system.

Using system libraries from a different container with a volume is a bad idea, as it create a hard dependency on both containers using the same libraries, and is likely to crash somewhere when both systems try to acquire a lock on the library to load it.

I assume with a lot of tweaks it should be doable, but the added complexity is a real footgun and create more dependencies than should exist.


What is the cleanest branching strategy to use when creating reusable artifacts? If I have 3 environments : integration, staging, and production, what is the cleanest branching strategy to use, assuming I want to reuse my deployed artifact?

For example, should feature branches be merged into master and then built to be deployed to integration? Or is there another strategy that allows you to have 1-to-1 mapping between your releases and SCM?

Some more details ... Say I merge into master, and that kicks off a build, which produces an artifact. That same artifact should be deployed to all environments to ensure consistency. Is it best to start that process with integration? In that case, there is a rather large potential window where untested code is in master before verification can occur. Is it better to use an integration branch, deploy that to integration, then merge to master and use that artifact for staging and production? Or does that just increase the need for duplicate testing?

The simplest/cleanest branch strategy is IMHO the one used in continuous deployment: a single/main integration branch which is also your release branch. From What is Your Branching Model?:

  Commits can go all the way to production from one trunk/master, if the
  automated build says the commit was good. It’s the turbo-switch for
  TBD, where no humans can hold up a release by taking their sweet time
  testing it and giving a belated seal of approval. Github, Etsy,
  Netflix (and many more startups) are all here.

You'll effectively use the CI build artifacts from that main branch across all environments. Well, as long they build and pass the respective QA verifications, obviously.

Note: there are no feature branches in this strategy, at least not long-lived ones - in order to eliminate/minimize the integration delays and costs. Practically you can eliminate branch merges from your development.


Installing Helm charts using Terraform I have been trying to install Helm charts using Terraform in a cluster which operates in GKE.

My question is in two parts:

Is it a good practice to use Terraform for installing Helm charts?
I have been getting this error when Terraform tries to install a Helm chart:

  helm_release.release_name: Get
  https://XX.XXX.XX.X/apis/extensions/v1beta1/namespaces/kube-system/deployments/tiller-deploy:
  dial tcp XX.XXX.XX.X:443: connect: connection refused

Here is how I am configuring the Helm chart:

provider "kubernetes" {

  host     = "${var.cluster-host}"
  username = "${var.cluster-username}"
  password = "${var.cluster-password}"

  client_certificate     = "${base64decode(var.cluster-client-cert)}"
  client_key             = "${base64decode(var.cluster-client-key)}"
  cluster_ca_certificate = "${base64decode(var.cluster-ca-certificate)}"
}

resource "kubernetes_service_account" "tiller" {
  metadata {
    name      = "tiller"
    namespace = "kube-system"
  }
}

resource "kubernetes_cluster_role_binding" "tiller" {
  metadata {
    name = "tiller"
  }

  role_ref {
    api_group = "rbac.authorization.k8s.io"
    kind      = "ClusterRole"
    name      = "cluster-admin"
  }

  # api_group has to be empty because of a bug:
  # https://github.com/terraform-providers/terraform-provider-kubernetes/issues/204
  subject {
    api_group = ""
    kind      = "ServiceAccount"
    name      = "tiller"
    namespace = "kube-system"
  }
}

provider "helm" {
  install_tiller  = true
  service_account = "tiller"
  namespace       = "kube-system"
  kubernetes {
      host     = "${var.cluster-host}"
      username = "${var.cluster-username}"
      password = "${var.cluster-password}"

      client_certificate     = "${base64decode(var.cluster-client-cert)}"
      client_key             = "${base64decode(var.cluster-client-key)}"
      cluster_ca_certificate = "${base64decode(var.cluster-ca-certificate)}"
  }
}

you can use terraform to install helm charts, you dont have to obviously
something is probably wrong with your kubernetes connection, check if its actually working