Unable to ssh in a multi-machine vagrant environment I created a three node multi-machine vagrant environment and am having issues ssh'ing from one vagrant vm to another.

Here is the Vagrantfile:

Vagrant.configure("2") do |config|
  config.vm.box = "centos/7"

  config.vm.define "master" do |master|
    master.vm.hostname = "master.local"
    master.vm.network "private_network", type: "dhcp"
  end

  config.vm.define "node1" do |node1|
     node1.vm.hostname = "node1.local"
     node1.vm.network "private_network", type: "dhcp"
  end

  config.vm.define "node2" do |node2|
    node2.vm.hostname = "node2.local"
    node2.vm.network "private_network", type: "dhcp"
  end
end

The hosts file (same on each node):

$ cat /etc/hosts
172.28.128.3    master.local    master
172.28.128.4    node1.local     node1
172.28.128.5    node2.local     node2

I can ping back and forth all day from any machine to the other but I cannot ssh from one vagrant vm to the other.  The typical error message is (from node1 to master):

[vagrant@node1.local] $ ssh vagrant@172.28.128.3
Permission denied (publickey,gssapi-keyex,gssapi-with-mic)

SSH is running and the port is open.

The firewall is not running.

I am sure this has to do with ssh keys.  I readily admit I am not an expert.
What am I doing wrong here folks?

Following Vagrant file address this problem.

You can get all supporting key files along with this vagrant file at https://github.com/malyabee/IaaC/tree/master/ansible_lab

$commonscript = <<-SCRIPT
sudo yum update -y
sudo yum install python2 epel-release -y
sudo yum install -y ansible
sudo echo "192.168.22.10    ansiblecontroller.example.com ansiblecontroller" >> /etc/hosts
sudo echo "192.168.22.11   node01.example.com   node01" >> /etc/hosts
sudo echo "192.168.22.12   node02.example.com      node02" >> /etc/hosts
SCRIPT

$nodescript = <<-SCRIPT
cat /vagrant/ansible_lab.pub >> /home/vagrant/.ssh/authorized_keys
SCRIPT

$ansiblescript = <<-SCRIPT
sudo yum install ansible -y
sudo cp -r /vagrant/ansible_lab /home/vagrant/.ssh/id_rsa
sudo chmod 400  /home/vagrant/.ssh/id_rsa
sudo chown vagrant:vagrant /home/vagrant/.ssh/id_rsa
SCRIPT

Vagrant.configure("2") do |config|
  config.vm.provision "shell", inline: "echo Hello"

  config.vm.define "ansiblecontroller" do |ansiblecontroller|
    ansiblecontroller.vm.box = "centos/7"
    ansiblecontroller.vm.provider "virtualbox" do |v|
          v.memory = 512
          v.cpus = 1
       end
    ansiblecontroller.vm.network "private_network", ip: "192.168.22.10", virtualbox__intnet: "mynetwork01"
    ansiblecontroller.vm.hostname = "ansiblecontroller.example.com"
    # Installing required packages for ansible controller node
    ansiblecontroller.vm.provision "shell", inline: $commonscript
    ansiblecontroller.vm.provision "shell", inline: $ansiblescript
  end

  config.vm.define "node01" do |node01|
    node01.vm.box = "centos/7"
    node01.vm.provider "virtualbox" do |v|
          v.memory = 512
          v.cpus = 1
       end
    node01.vm.network "private_network", ip: "192.168.22.11", virtualbox__intnet: "mynetwork01"
    node01.vm.hostname = "node01.example.com"
    # Installing required packages for  node01
    node01.vm.provision "shell", inline: $commonscript
    node01.vm.provision "shell", inline: $nodescript
  end
  config.vm.define "node02" do |node02|
    node02.vm.box = "centos/7"
    node02.vm.provider "virtualbox" do |v|
          v.memory = 512
          v.cpus = 1
       end
    node02.vm.network "private_network", ip: "192.168.22.12", virtualbox__intnet: "mynetwork01"
    node02.vm.hostname = "node02.example.com"
    # Installing required packages for  node01
    node02.vm.provision "shell", inline: $commonscript
    node02.vm.provision "shell", inline: $nodescript
  end
end


Upload data into docker container from docker volume I wish to create a volume and mount it to a docker container in order to collect some artifacts, and I want to reuse these artifacts into another docker container later on. Can i just mount the same volume to the new container and upload the artifacts? Is there an upload command for this?

You can mount a volume e.g. at /app/artifacts then set up your application to place the artifacts on that folder.

docker run -d \
-v artifacts-vol:/app/artifacts \
artifact-creator:latest

then after that, you can mount the same volume to a separate container and specify on the new container to read the artifacts from the volume location

docker run -d \
-v artifacts-vol:/app/artifacts \
artifact-reader:latest


Is it possible to downgrade Jenkins container plugin thought CLI? I have a job that creates a container from Jenkins image. I would like to add to the script one more option:
Currently, it's installing the image with DSL plugin version 1.77. is there a command I can run from CLI to downgrade the plugin version to 1.57?

Check out Config as Code for installing plugins.

If you're using the jenkins image, just use an older version of the image that has the proper plugin version


How to update pods if version hasn't changed? During development we are continually building a Docker image with the same version (say 0.0.1-development) as there is no need to bump for each minor change. We do eventually bump when additional features are added.

We've discovered that kubectl set image deployment/ourdeployment doesn't force the updated container to be pulled from Dockerhub and run if the version in the image hasn't changed.

In the deployment we have imagePullPolicy: Always but this doens't seem to have any effect.

Is there a way we can achieve the desried behaviour of pulling down the image even if the version hasn't bumped, replacing the old containers each time?

The main problem here is, that from a Kubernetes point-of-view nothing has changed to your deployment. So nothing gets updated.

The best solution here would be, that you add something to your image tag, like the git short hash, for example. The your update will just work. Also you will make your deployments re-creatable, when you do it this way.

If you really want to keep it the way it is, you could add some other information to the deployment, that changes on each deployment, but is not used, like an annotation. But I would not recommend that, since I saw this leading to strange behavior with old pods no being shutdown.

On a side note: The imagePullPolicy is just evaluated, when there is change to your deployment. So basically this flag says: If you do deploy something (e.g. bc there is a new deployment), no matter if you have the image already locally, reload it from the registry.


How to set up WordPress CI/CD pipeline I am planning to create 3 environments for my website i.e. dev, stage, and production. I am using GitHub to manage the code. So for URLs dev.example.com, stage.example.com, and example.com, the branches are dev, stage, and main. My knowledge says dev is for the developer to play around with new codes, tests, and do whatever he wants, the stage is like production-ready where QA personal can test codes, functions, etc.
I have modified wp-config.php to take environmental variables. For example, I have changed
define( 'DB_PASSWORD', 'password' );

to
define( 'DB_PASSWORD', getenv('DB_PASS') );

Questions
Database

How should I move a database from one environment to another? Should I let the developer do anything from the dashboard in production and later move the database from production to dev/stage if require?

Plugins

I know I can avoid including plugins inside the wp-content/plugins folder and use wpcli to install the plugins which are free but what about premium plugins? How can I integrate them?

Update

What should be the update strategies? Should I let the developer update from the dashboard? I think it will make changes in the codes and later will conflict with git merge. what is your thought on this? Also, do you take a backup of all environments or just production?

Tags

How should we configure the pipeline for new releases in GitHub? Should I manually configure the pipeline to deploy on each release?

For now, I only have these challenges. If I will have more questions then I will add.

I am planning to run WordPress in the docker container, alpine as a base image. I am thinking of separating Nginx and php-fpm container. Doing so, we need to copy WordPress files in both containers, how can I overcome this? or should I use php-fpm and Nginx in the same container? is it a good idea?
what about the cronjobs?

It has been a while since I have run a few Wordpress based sites. The database stores some of the content of the site. The file system stores media that is uploaded into the site. More significantly Wordpress is a Content Management System. The intention is that your site users can add content and media to the live site. When someone uses the WYSIWYG editor to add or edit the content you will find that the sites domain name leaks into the database. So if you move the Staging database into Live you would get links appearing on the Live that that point back to the Staging site.
I quick bit of research to see if things have improved since I last used it leads to this article on putting Wordpress into git. Please read the first article in that series that explains the challenges. You will need to understand a little of how Wordpress works to understand the recommendations I make below.
That article talks about turning off automatic updates so that you have to manually security path. I am pretty sure that I remember that updates can write things into the database. I would be very worried about turning off updates as there are a huge amount of attacks against Wordpress as it is so widely deployed. You really should avoid deploying many third party plugins as they might not get security patches in time. This isn't a theoretical problem I had quite a time dealing with Wordpress being used as a spam relay. Yet even if you follow that article you won't have solved the database migrations issue.
My recommendation is that you take a step back and try to figure out what problems you are trying to solve and how to adapt the tools and process:

What is the developer developing? A custom Theme and/or Plugin? If so then this article covers that. Put the Theme/Plugin into its own repo.
Who actually writes the site content or uploads media? The general public or members of staff? If it is staff then maybe they need minimal training in a Test environment and can happily edit Live after that.

Given that Wordpress does not let you aim for perfection due to the way that it is built, yet it is hugely successful, carefully pick which of the following you are aiming to optimise:

A Specific Service Level Agreement
Meantime to recovery
Mean time between failures
Reducing the commercial impact of errors
Something else?

There are quite a few other specifics that may affect what set of compromises might work best for your situation.
Based on my experiences without actual knowledge of your specifics:

Give up the idea of promoting changes through Dev, Staging and Live. I would simply have a Test site and a Live site. I would periodically refresh the Test site from the Live site. To do this I would dump out the entire Live database a run a sed -i over it to replace every instance of "www.mysite.com" with "test.mysite.com".
Give up on using Docker as the software distribution mechanism for the actual site. I would use Docker to supply the PHP+Apache runtime and mount a volume that contained the actual site php+content. This because the disk is a combination of the code and the content of the site so you need to be able to backup and restore it independently of changing the Docker image.
Try "git checkout vX.Y" to update the live site in a persistent volume mounted into your generic PHP+Apache docker image. I would investigate the ideas in this series of articles.
Automate the backup of the media content volume every night and keep as many backups as you can. This is why it needs to be a volume mounted into Docker.
Automate the backup of the database every night and keep as many as you can.
Automate the restore of the site as a script that restores the database to the last midnight backup then restores the last nightly snapshot of the media content folder.
Avoid any party plugins that are not commercially supported. They likely won't work at the next major upgrade of Wordpress and will pin you onto a version that will become obsolete.

Then they workflow that I would go for would be the to add "experimental features" into the Test site before applying them to the Live site. This would be based on git checkouts in the best case. Yet I would also consider doing Wordpress security updates via the CLI. Those can update the database and the state on disk underneith Docker hence I do not recommend you try to mange the state of your site as a Docker image.
You then have to worry about the Test site drifting away form the Live site. So I would on a periodic base (at least monthly) overwrite the Test site with the Live site. This would involve:

Backing up the Lite sites media content on the persistent volume and restoring it over the Test sites media content on its persistent volume.
Backing up the Lite sites database, doing a sed -i to rewrite any content links pointing at the Live to point at the Test site. Then restoring it over the Test database.

A key part of this approach is that you would be running your Live site like every other Wordpress site in the world. You don't do "brain surgery" on the Live site. You do frequently do "brain surgery" on the Test site. If the brain surgery goes wrong it's annoying and painful to fix but it is only the Test site that needs some new step to bring it into sync with Live. It is likely that this only happens when you move to a new Wordpress version. If you try a custom approach of moving change into a Live site they might break at a future Wordpress release and leave you with a multiple-day outage in Live.
I completely recognise that the approach outlined above isn't at all how "normal" software is developed. Yet it is the approach that I have scripted in the past. Even if you did the standard dev/test/live approach you do not guarantee 100% perfection and 0 bugs but you might have a longer mean time between failures. If you accept that perfection isn't possible then what you want is very fast rollback to reduce the impact of bugs. You do this by focusing on reducing the meantime to recovery to the speed of a restore script and the time taken to get permission/agreement to run it.
Being able to rehearse things in a Test environment should mostly avoid a total outage on the live site. Being able to do a fully automated restore can then bring you back to a good state at the loss of some recent edits to the site. That is normally completely acceptable compared to having an extended outage. Consider this scenario:

Staff are continually editing content in production including uploading new media into the content folder on disk (e.g., uploading a PDF and linking to it within content where the link is in the database).
You have run some Wordpress updates in Test a day ago and they seemed good. Today you run them at 9 am in the Live and something odd happens (maybe a timeout, whatever) and the site becomes corrupted.
You have automated the rollback. You run the roll-back script which computes the last nightly full backup point then restores the database, then restores last night's media content folder backup over the live folder to revert any lost or corrupted files.

To my knowledge that is as good as you can get. Your meantime to recovery should be the speed of your database restore plus the time take to spot the problem and agree to do the restore. This is exactly why you shouldn't have to speak to editing Staff about whether or not they are happy with the roll-back as the site is broken so of course, they would agree to restore the site automatically, rather than have the site down for hours while a possible point fix is investigated and attempted. You can send out a prepared email telling Staff that there was an outage at 9 am such that the site has been restored to the previous nights backup and that they should double-check and repeat any work that they have undertaken in the past twenty-four hours and redo their work as necessary. IMHO that is as close to perfection as an opensource PHP content management system such as Wordpress allows.


How to solve "E405: Remote package tasks should have a retry" in Ansible? E405 v4.0.0   Remote package tasks should have a retry    Package
  operations are unreliable as they require network communication and
  the availability of remote servers. To mitigate the potential
  problems, retries should be used via register: my_result and until:
  my_result | success

This shows up when running ansible-lint as the following error:

[405] Remote package tasks should have a retry
plays/sample.yml:31
Task/Handler: Install list of packages

Given the sample code:

- name: Install list of packages
  apt:
    name: [jq, vim, curl, unzip, dnsutils]
    state: present

How to solve this?

ansible-lint warning 405 can be quite irritating as in certain cases, i.e. when running locally retrying is very unlikely to make any difference.  Actually resolving it is normally simply a case of adding the following:

register: task_result
until: task_result is success
retries: 10
delay: 2

When this is added to task Ansible will retry ten times with a delay of two seconds between each retry until the returncode is 0.


Founded argument to have no sandbox CI/CD? CI/CD environment is the production carrier for DevOps teams.

An error here is a show stopper for possibly many teams.

Additionally, it is a product offered to teams which needs maintenance (quartely new features sometimes and security patches anyway) and room for experiment (new metrics plugins for example).

How to allow this? The only argument against a sandbox I can imagine is its cost. But then, isn't a transition to IaC, aka Pipelines as Code providing a "serverless" CI/CD like in Gitlab or Travis, not the next step?
Otherwise, CI/CD is the new "pet" system?

A CI/CD system is, after all, just another software product/system. Regular upgrades and at least some changes in the plugin installations could be treated as changes of the software product itself.

I see no founded reason for which such software product can't also use as part of its own development process a CI/CD solution. Maybe even itself - the most recent production version, for example - if it's flexible enough (I'm still investigating such possibility for the CI system I'm working on).

Now, a CI/CD system deployed/actively used in a certain software development project is effectively a software product deployed in a specific environment.

Changes to the configurations of the CI/CD system aren't actually software changes, they're changes of the environment in which the software system is deployed. These are trickier to address in a CI/CD pipeline. Some even impossible. You'd have to replicate the environment as close as possible. But it's unlikely you'll be able to do that at 100% - just like no dev/QA/etc environment is exactly as the production one.

Some environment changes could be tested in a CI/CD pipeline. For example changes to a test script: both the old and the new versions can be executed as parallel pipeline jobs with the same set of artifacts for a period of time, long enough to determine the test quality and stability before making the new version the official one and retiring the old one.

Finally, don't forget that most CI systems are reactive: they detect breakages, but human action is required to identify the culprit and/or make repairs before the development process can be resumed. Applying such CI system to an actively used CI/CD system one would allow the show-stopper problems you mention. Only a non-blocking CI system can completely prevent them (disclaimer: I'm the founder of the company owning the referenced page and offering such system).


How to fetch resource group name( with pattern matching) using Azurecli I set my az account to current subscription. And I am trying to fetch the resource name ending with (example "***kafka" ). How can i do it.

I tried with az group list -n '***kafka'

  validation error: Parameter 'resource_group_name' must conform to the following pattern: '^[-\w\._\(\)]+$'.

There are at least two ways to achieve this.

First is to go with transforming json output to proper object for PowerShell as described in this thread:

how do you select just one property from an array of “results” from the Azure command 'az group list in powershell'?

Second it to use --query global parameter for Azure CLI:

az group list --query "[?contains(name,'kafka')].name"

But remember to use at least PowerShell 6.2 as previous versions have an issue:

powershell --query error #9047


Terraform error Reference to “count” in non-counted context for resources not necessarily data here's my part of my code that I am having issues with. Unfortunately I could not understand the relationship between the last answer, despite being almost what I need, and my situation.

data "aws_internet_gateway" "zyz_igw" {
  depends_on = [aws_internet_gateway.this]

    filter {
    name   = "attachment.vpc-id"
    values = aws_vpc.this[count.index]
    #values = [aws_vpc.this[count.index]]   #based on https://www.terraform.io/docs/providers/oci/guides/filters.html
  }

  filter {
    name   = "tag:Name"
    values = ["ZYZ_Mirror_ZYZ-tp-zyz-IGW"]
  }
}

The error I am getting is the following:

Error: Reference to "count" in non-counted context

  on modules/xyz/main.tf line 292, in data "aws_internet_gateway" "zyz_igw":
 292:     values = aws_vpc.this[count.index]

The "count" object can be used only in "resource" and "data" blocks, and only
when the "count" argument is set.

I get confused since I believe my issue may be with the resource and not sure how to change the data. I also saw something related to 'for_each' to use that instead of 'count' on 0.12. But the in the very beginning of my code I have the normal 'aws_vpc' resource:

resource "aws_vpc" "this" {
  count = var.zyz_create_vpc ? 1 : 0

looks like your new to stack exchange. You need to form a clearer question to get a good answer....

As far as your error goes.... count is a reserved name. It's only accessible when the block you're writing has a count set!

You have You just need to specify the count of aws_internet_gateway data resources you're trying to get. Your code implies there is more than one.

data "aws_internet_gateway" "zyz_igw" {
   count = 1000000000
   ...


How can we prepare GitLab runners using Ansible deploys Given
We run an in-house GitLab server, with many projects (aka "repositories")
We provision all servers using Ansible (and deploy projects using pip)
We deploy to

dev.project.example.com
test.project.example..com
beta.project.example..com
(and eventually to) www.project.example..com

When
We add .gitlab-ci-.yml to all projects' repository roots
We want to use dev.project.example.com as a runner for that GitLab project
Then
What Ansible playbooks could we write to provision dev.project.example.com as a GitLab Runner for that project?

A gitlab runner is a process that f.e. tests, compiles your project, pushes the artefact to you artefact store etc.
A Gitlab runner can be tagged inside Gitlab or via config.toml - so - you can assign runners via tags to Gitlab projects (shared).
Gitlab runner are not designed to run an application 24x7. So - you assign a project to a runner (or a tag or all) to build you application). If you want to deploy the project, use the K8S deployment inside Gitlab (that must be connected to a specific Kubernetes environment based on you Git-branches etc). Or (just an example) run an AWX task that is able to push your new artefact via the artefact storage (docker registry). Also you could just call a SSH remote command on a script (via Gitlab runner) to start a deployment - but don't use the runner to "host" your environment. They will stop (per default after 1h) without a return code.


AWS or Azure with usage auto shutdown occasional usage I'm looking for recommendations on which cloud service provider we should be looking at.

The situation:

I'm part of a small team looking to leverage more powerful analytics. Most of our day to day can be done on local machines but we want to explore using more powerful hardware through a cloud platform before committing to anything in house. As far as software we use Excel, MSSQL, and R, although these cloud instances only need to run R.

Our ideal use case would be uploading data to the cloud, spinning up an instance to run whatever we need to run, then shutting down the instance while keeping the data and software (eg R packages) stored. This way (I think) we would only be charged for the storage while we aren't actively running a project.

Additionally we'd like an auto shutdown feature where the computing instances are automatically shut down if they aren't used for a certain period of time or if that isn't possible, at a given time each day.

We're really outside of our area of expertise here so user friendly is a big plus.

Any advice?

I can think of a few different ways to achieve this.

First and easiest way is to create a new CloudWatch Events Rule 1, set its "Event Source" to run on a schedule and "Target" to EC2 StopInstances API call with Instance IDs provided. Look at this screenshot for clarity:

With this example, An event will start at every night at midnight and it will stop Instances by their IDs. You can change the schedule with cron expressions. See: https://crontab.guru

You can implement any kind of custom logic with AWS Lambda functions and schedule them to be run with a variety of AWS services (including CloudWatch Events).

For example, with a Lambda function you can implement logic to find instances by a tag, like "GoodBefore=2018-08-10", and stop these instances. In fact you could even make use of API Gateway to expose an API for requesting instances, only to be terminated after a few days by their "GoodBefore" tag.

Of course, while this way is much more customizable and sophisticated, it may prove hard especially if you don't have experience in programming.

If you don't want to implement custom logic with Lambda, but also need more than what CloudWatch Events can provide, you can make use of Instance Scheduler solution provided by AWS. It's a similar solution to what I described in previous method, but this time AWS already did the hard work for you.

1 URL for us-east-1 region: https://console.aws.amazon.com/cloudwatch/home?region=us-east-1#rules:action=create

Note: While I haven't used Instance Scheduler myself, I'm pretty sure it wouldn't decide on such things, and I think it only makes sense if it wouldn't do much more than simply making a stop call for the instance. So, on any stop event, either made by an automated solution or you, if your instances' root volume was created "EBS-backed", it will retain the data. If it is "instance store backed", it will lose the data. You can find this information from AWS EC2 console. If you want to retain data after stopping instances, make sure all volumes are EBS-backed when creating a new instance.


kubernetes allocating pods with the wrong IP I'm running kubernetes with the supposed pod network cidr of 10.244.0.0/16. Everything seems to agree with that:

# history | grep kubeadm
kubeadm init --control-plane-endpoint "guide:6443" --upload-certs --pod-network-cidr=10.244.0.0/16

$ kubectl get nodes -o kubectl get nodes -o jsonpath='{.items[*].spec.podCIDR}'
10.244.0.0/24 10.244.1.0/24 10.244.2.0/24 10.244.3.0/24 10.244.4.0/24 10.244.5.0/24

# cat /run/flannel/subnet.env
FLANNEL_NETWORK=10.244.0.0/16
FLANNEL_SUBNET=10.244.0.1/24
FLANNEL_MTU=1450
FLANNEL_IPMASQ=true

However all the pods I create get IPs in the 172.17.0.0/16 range.

$ kubectl get pods -o wide
NAME                                   READY   STATUS             RESTARTS   AGE   IP            NODE         NOMINATED NODE   READINESS GATES
busybox1                               1/1     Running            1          13m   172.17.0.11   k8s-node-1   <none>           <none>
debain-shell-547f8fc54b-v8p99          0/1     ImagePullBackOff   0          27m   172.17.0.3    k8s-node-1   <none>           <none>
nginx-deployment-ex-54f57cf6bf-k55nn   1/1     Running            1          27m   172.17.0.5    k8s-node-1   <none>           <none>
nginx-deployment-ex-54f57cf6bf-q2khx   1/1     Running            1          27m   172.17.0.9    k8s-node-1   <none>           <none>
nginx-deployment-ex-54f57cf6bf-r8b7v   1/1     Running            1          27m   172.17.0.7    k8s-node-1   <none>           <none>

I can't talk to coredns (which is how I found this condition) and I can't seem to use my ingress service anymore (though I've restarted enough things by now I may have just temporarily broken that).

What is causing this, or alternatively, where should I look for more info?

This was caused by missing --network-plugin=cni on the end of kubelet. Added that to the args and it all works now.


Error passing arguments to ansible playbook I am trying to create a playboook that executes a python3 script with 2 parameters, name and password in localhost.
However, when i try to execute the paybook with this command ansible -b --become-user=ras createdeploy.yaml --extra-vars "user=ras pass=ras"
 I get this error message:

[WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all'

ERROR! Invalid variable name in vars specified for Play: 'pass' is not a valid variable name

The error appears to have been in '/home/ras/createdeploy.yaml': line 3, column 5, but may
be elsewhere in the file depending on the exact syntax problem.

The offending line appears to be:

  vars:
    user: "{{ user }}"
    ^ here
We could be wrong, but this one looks like it might be an issue with
missing quotes.  Always quote template expression brackets when they
start a value. For instance:

    with_items:
      - {{ foo }}

Should be written as:

    with_items:
      - "{{ foo }}"

Here you are the .yaml file:

- hosts: all
  vars:
    user: "{{ user }}"
    pass: "{{ pass }}"

  tasks:
    - name: Stop services
      shell: "bash stopservices.sh"

    - name: Edit real hosts file
      shell: "bash editrealmachinehosts.sh"

    - name: Create dirs and cloning
      command: python3 user pass

I'd suggest that you first put some default values within the playbook (if you explicitly want your variables there, for the purpose of .. reducing interactivity/command line args lets say):

- hosts: all
  vars:
    user: greatuser
    passa: greatpassword

Second, I'm pretty sure that pass is a reserved word somewhere, although I'm failing to find it in the documentation. That's why I've changed your variable to passa. The final version of your playbook should look like this:

- hosts: all
  vars:
     user: greatuser
     passa: greatpassword

  tasks:
     - name: Stop services
       shell: "bash stopservices.sh"

     - name: Edit real hosts file
       shell: "bash editrealmachinehosts.sh"

     - name: Create dirs and cloning
       command: python3 {{ user }} {{ passa }}

Note the way used to reference to the username and password variables in the third task - {{ variable }} instead of variable.

Execute your playbook with the following command:

ansible-playbook createdeploy.yaml --extra-vars "user=ras passa=ras" -b --become-user=ras

As your target host is actually localhost (if I did understood you correctly), and since you're not showing your inventory file, I'd like to suggest that you might have to add -c local to the list of arguments of your ansible-playbook command.


How to get the ARN for an arbitrary Amazon Resource? I am creating an AWS IAM policy for a Serverless Framework project, I am applying the Principle of Least Privilege to ensure the policy is as tight as possible.  To restrict the scope of the policy I need to be able to identify the ARN for each of the resources, i.e.:

arn:aws:logs:us-east-1:123456789012:log-group:my-log-group:*

Looking through the CloudWatch and CloudFormation UIs I can't actually see any mention of the resulting ARN for a created log group.

Question: For any resource within Amazon Web Services how can I obtain the ARN?

https://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html lists the schema for (I believe) all of the ARNs that Amazon supports.  Text in italic red indicates where you should substitute in your own value, rather than the literal string, and there are examples in each section to help figure this out.

Be warned that IAM is less usefully restrictive than it appears on first glance.  To start with, most services have only partial IAM support.  Secondly, often the things that are supported at a resource level are supported only in very specific ways; for instance, granting all permissions on a subnet ARN does not grant the attached users the ability to fully manage instances in that subnet, as AttachVolume only supports restricting by volume id, etc.  Thirdly, if the users are going to use the Amazon web console at all, they must have broad viewing permissions, as the console was not written with PoLP in mind and issues non-resource-restricted API calls (and often merely dumps errors in the javascript console!).


Does TFS Build have something like a Jenkinsfile? I've seen some custom settings files used in combination with TFS build, but nothing native.  Does Team Foundation Server or Visual Studio Team Services have a Jenkinsfile-like, declarative method for defining a build process?

The strict answer is no, in that there is no direct equivalent to a Jenkinsfile. However, there are Build Process Templates and you can use the Psake Build Automation tool in combination with templates to reduce the barriers to entry for creating a new build, test and deployment build definitions.

It is nowhere near as fluid as Jenkins' Pipelines as Code, AppVeyor's appveyor.yml, CircleCI's circleci.yml and Travis' .travis.yml.  There is a wealth of information on the ALM Rangers Library of tooling and guidance solutions.


I need a good tool for running a series of Spring Boot based tools on Docker, where each needs a web page and a database connection These are pretty much just tools our support and ops departments need to solve small tasks which crop up as a routine sideeffect of our main business operation. They might need to find all instances of Entity with both Field A and Field B, or change some property in some database without having to go through Ops.

Anyway, I write them in Kotlin on Spring Boot just for speed and I use a different Spring Boot instance on each to avoid dealing with conflicting datasources in a single Spring Boot app. I use Docker just because we all use Docker, so why change now. And Docker-compose for easy deployment.

The problem I have is that I want a single command to git update, rebuild image and restart containers. Or, even better. When one of the tools gets updated, just recreate it and replace the container live.

Obviously I could script this, but it seems like something that would exist already. Is there something out there that's "like" CD, but which doesn't have all the overhead?

I tried Dokku. Didn't like it. Way too complicated for what I needed.

According to me a CI like Jenkins or gitlab should be used to build the docker images when the git code changes. Once this works well, you could start to run some scripts that deploy to an environment (CD). First of all, I would like to emphasize that implementing CI is the highest priority as the docker images seems to be built manually at the moment.


What is the best strategy for running an ansible playbook within an ansible playbook? I have playbook that I've been hacking on for a year and it works OK, but I know it's not great, part of it runs ansible on the target machine with a "command" task.  Because of that, I don't see the output of this task in my ansible playbook.

I need to run this play in the middle of the playbook because it is dependent on a prior task and a subsequent task depends on it (otherwise I'd break it up differently)

I'd like to be able to see the output of all my plays, is there a better way to branch off plays in the middle of a play.

For instance: given a playbook running on HOST_A and HOST_B at TASK 3, suspend running plays, run next plays on HOST_C and HOST_D then HOST_E and HOST_F?

- name: update other machines
  command: "/usr/bin/ansible-playbook /etc/ansible/playbooks/other-machines.yml -e 'ansible_user={{ item.ansible_user }}' -l {{ item.ansible_hostname }}"
  with_items: "{{ other_machine_connections }}"

If I've understood your problem correctly:

the ansible shell command in the middle could be converted to a simple playbook using the command task - let's call this middle.yml
your existing playbook could then directly import this middle.yml play (or even a more complex playbook).  In Ansible 2.4+, you would use import_playbook, specifically import_playbook: middle.yml, or in earlier versions, just include: middle.yml.

This processes the whole middle.yml playbook as if it was part of the main playbook - you have all the features of Ansible plays (hosts etc) and the output appears as part of main playbook.

This answer may need updating depending on your response to my comment above.


What are your companies policies for S3 data durability? AWS says that their systems are designed for 99.999999999% durability, but apparently there is no guarantees in their SLA.

Yes, this can (hypothetically) be improved by using cross-region replication, but this comes at the cost of paying twice for storage, and also for data transfer costs.

How are your companies handling this? Do they go with pure trust in AWS, or do they spend (a lot) of money to (try to) improve this?

The durability design objective is not especially relevant above some number of 9's, because it is simultaneously both extremely unlikely and non-zero.

The 99.999999999% durability is easily calculated by reading between the lines as the product of the failure probabilities of 6 copies each with an AFR between 1% and 2%.  Older versions of the documentation suggested 3 AZs × 2 devices per AZ was relevant to the durability calculation.  Assuming these "devices" were RAID arrays, these AFRs are well within reason.

This is raw durability.  It does not take into account natural disasters or acts of war or terrorism or other sabotage (any of which would need to wipe out multiple availability zones within a region) or serious bugs in the S3 system code... all of which are also extremely unlikely but similarly difficult to quantify.

And we need to factor in bugs and human errors in your code and systems, compromise of your credentials, and malicious insiders in your organization.

The bottom line is that the risk of loss is extremely low, difficult to quantify, but still non-zero.

I have many millions of objects (dozens of terabytes) in S3 and have never lost one.  A majority of them have been stored with copies of both their md5 and sha256 checksums, they are all cataloged in an external database, and extensive audits have never revealed a corrupt or missing object.

But if I lose one critical object in the future, all of that still becomes irrelevant.

Can you live with the loss of an object?  If so, no problem.  It's really unlikely that you will lose one.

But if the data is critical, there is a compelling argument that it should still be backed up, and not exist in only "one place."  An object in an S3 bucket is physically in multiple places but logically that is still "one place" because whether the object exists without corruption ultimately resolves to a single true or false answer from your perspective.

Just as using most classes of RAID arrays reduce the likelihood of data loss but do not remove the need for backup, I would argue that S3's durability -- as high as it is -- does not elminate the requirement that critical data be backed up elsewhere.

Cross-region replication is one option, and apparently one important motivation for the introduction of the 1-Zone-Infrequent-Access storage class -- similarly durable except for the case where the availability zone is lost -- was as a very low cost target for cross-region replication.  It seems to meet that objective -- it's more vulnerable to the loss of a single AZ (and you don't get to pick which AZ) but that's only relevant when data loss has occurred in another bucket, in a totally different region.  A replicated bucket could also migrate its copies of objects to Glacier, further reducing long-term storage costs. (Note that very small objects may encounter storage cost increases since 1ZIA has a minimum billable object size of 128K, and with the Glacier storage class, you will be paying for 40k¹ of metadata storage in addition to the object size in bytes).

Depending on the two regions involved, cross-region data transfer charges can be as low as $0.02/GB (or even $0.01/GB for the special case of us-east-1/us-east-2) which hardly seems burdensome if you need that additional level of assurance -- and note that since CRR also supports object versioning and can be configured not to replicate object deletion, it provides some protections against the risks that you pose to your own data -- risks that arguably exceed the systemic risks in S3 itself.

Other options are cross-cloud-provider replication, copying objects to GCS, for example, which would be more expensive due to data transfer costs but then can also fit into your "beyond worst case" disaster recovery plan for when AWS (or your entire AWS account?) is completely inaccessible.  The cost of cross-cloud can be partially mediated by using aggressive compression where applicable, and services like Direct Connect, which reduces the cost of exporting data from AWS, particularly if the presence of the interconnection is justified by other purposes.

Or, consider the extreme opposite: a managed local backup strategy that makes physical, tangible backups on DVD or Blu-Ray is still better than no backup at all.  I take a fair amount of grief for suggesting this "low tech" approach, but as long as the lifespan of the media is factored in and aggressive checksumming and integrity measures are in place, this is a viable protection against the extremely unlikely event where data is lost from S3.  The only problem with this solution is that it appears to cost less than it really does -- it can be somewhat labor-intensive, requiring frequent touching of disks even when everything else is automated -- and of course if you compare the costs of this "simple" solution to the relative set-and-forget of cross-region replication, CRR becomes a pretty obvious choice.

¹ 40k is an oversimplification, based on two values,  8k + 32k, mentioned in the Amazon S3 Developer Guide. "For each archived object, Glacier adds 32 KB of storage for index and related metadata" is straightforward enough, but the explanation of the remaining 8k is a bit less intuitive: "For each object archived to Glacier, Amazon S3 uses 8 KB of storage for the name of the object and other metadata. Amazon S3 stores this metadata so that you can get a real-time list of your archived objects by using the Amazon S3 API... You are charged standard Amazon S3 rates for this additional storage."  When S3 migrates an object to the Glacier storage class, you are still able to access informarion about  the object in bucket listings and access all the object's metadata in real time, just as any other S3 object, even though the object payload itself is stored in Glacier and not accessible in real time. This metadata is stored at the S3 STANDARD storage class rate, rather than the Glacier rate.  It is unclear whether this is a fixed 8k, or actually a maximum, depending in part on the actual size of the object metadata, which includes but is not limited to the optional x-amz-meta-* user-defined metadata for each object.  As such, the importance of this 8k as a cost consideration is not entirely clear.  The 32k is noteworthy whenever it is significant in comparison to the total size of a small object.


how can I make jenkins find 2 jenkinsfile in a repo with multibranch? I am quite new to DevOps and Jenkins in particular. I have played around and created some pipelines jobs. I wanted now to start practicing with a jenkins in local, and apply CI-CD to an small project I had. I have created a multibranch pipeline. which will have 3 repos (front, back, integration). The first 2 repos have 2 jenkinsfile each in the root folder (Jenkinsfile.ci and Jenkinsfile.cd) but when I scan the multibranch pipeline it says no Jenkinsfile has been found. How would be the right way to do this?

You need to update your multibranch pipeline build configuration to use Jenkinsfile.ci and Jenkinsfile.cd instead of Jenkinsfile. And you would have to create two multibranch pipelines, one for Jenkinsfile.ci and other for running Jenkinsfile.cd.


Docker image, containers and layers I want to use docker to compile a custom PHP executable.
I plan to use alpine as a base image, installing the required tools (build-essentials, etc).
The goal is to have the most minimalist resulting image for later use in child images.
Would it make sense to tar the resulting php binary and publish to a host, and later have the child layers just pull in the php binary?
Or does it make sense to keep the original image? What happens to all the build tools and libraries? Do they become part of the image?

This is what I would to better handle the base docker images lifecycle:

Create a git repo to save your Dockerfile + configs
Create a Dockerfile with alpine + all the configs/libraries you want
Add your php.ini + php-cli.ini configs
Have a Jenkins job to start on each push to that repo:

Build your docker image
Tag it incrementally
Push to your registry

Use that base image to create your final docker image


What are some effective questions to ask someone who is considering me as their DevOps person? So these days, DevOps has been reduced to the guy who uses Puppet, tick, he uses Jenkins, tick and so on.

What effective questions can I ask to ensure the client and I would be on the same page as to what I would be doing as opposed to just looking for the guy who does Puppet/Jenkins and/or Kibana/Logstash, etcetera.

In other words, on paper they will not see "DevOps" anywhere. I do have some "DevOps" accomplishments on paper, the work that was done, but my title was still that of a Developer. To many of you, that makes perfect sense, but to the business world, its like if you don't have "customer service" as a past title, then you don't have "customer service" experience. Your resume wont appear on Indeed for the person looking for that.

How do most of you navigate those waters in terms of a presentation? What do you say?

In my opinion since the DevOps culture is derived from, or you could say appeared because of, the Agile software development methods

I would ask if the company is following any agile software methods or else you would be just the release guy
Also I would ask how I will be working as part of the dev team like I will be assigned to one project team or I will be working with all the teams
Also it would be great to know the current situation like they are currently facing urgent problem and need fix now , what's their current stack , and also ask about their current workflows in git and in development and try to find a way to solve the issue to make everything go smoothly
Also what really appears on paper is the workflows like git workflows, how often you release cause automating parts like delivery, and integrations,......etc really means less time and better quality and that will appears on paper even for the business guys

But at the end if the day every company is different than any other and every situation is different than the other so it's all back to your judgement to be comfortable in your daily job.


Where to state CFT dependencies With AWS, why can't I just state in the cloudformation template which dependencies I need? Right now I need to add the dependencies automatically in an accompanying script during our gitlab pipeline. It would seem more convenient to do like Google Appengine, and state the dependencies in the top-level template. Here's an example adding dependencies from my google appengine app.yaml, just stating what I need and it seems AWS doesn't have something similar:

libraries:
- name: webapp2
  version: latest
- name: jinja2
  version: latest

This is a matter of implementation details and it's actually coming from the AWS Lambda product's implementation (which the doc you linked refers to), not from the Cloud Formation itself.

Presently the AWS Lambda implementation requires you to include the actual dependent packages in the deployment package (per the doc you referenced). Hence the CFT doesn't really need a way to specify such dependencies - it'd be useless as it can't be passed to Lambda.

Should Lambda infra end up supporting a way to just list the dependent packages, Cloud Formation might be enhanced to support specifying the list right in the template and passing it somehow to Lambda. But this is just speculation, it all depends on the AWS feature roadmap :)

Side note: the Google App Engine (GAE) example could be seen as a bit questionable as it's specific for just a small subset of that product's capabilities (applicable only to the GAE-supplied libraries for the python 2.7 standard environment). Maybe a better reference would be to the requirements.txt or package.json (or other language-specific) files used to specify external/third party package dependencies for GAE environments, as well as other Google Cloud Platform (GCP) products, including Cloud Functions (CFs) (which would be the GCP product closest matching AWS Lambda). See Specifying dependencies for CFs in Python, NodeJS 6 and NodeJS 8.


What is the difference between Tasksel and CMTs like Ansible/Salt/Puppet/Chef? What is the difference between Tasksel and Configuration Management Tools (CMTs) like Ansible/Salt/Puppet/Chef?

I'm not sure Tasksel should be considered a "CMT". Maybe "Package Installation Automizer" or similar term is the correct one.

The purpose of this question is to find out the "software type" of Tasksel so I could search data on similar software (given Tasksel currently doesn't support LEMP, I'd like to look for a similar tool that might does indeed support them).

From ubuntu documentation:

  Tasksel is a Debian/Ubuntu tool that installs multiple related
  packages as a co-ordinated "task" onto your system.

  This function is similar to that of meta-packages, and, in fact, most
  of the tasks available from tasksel are also available as
  meta-packages from the Ubuntu package managers (such as Synaptic
  Package Manager or KPackageKit).

So in brief, tasksel is just a helper to select and install a bunch of package with one command (acting as a meta-package).

Ansible/Chef/Puppet/Salt are configuration managers, they are able to install packages but that's only a small part of their goal, they aim at converging the system into the state you describe in their respective DSL (Domain specific Language), being package to be installed as well as the content of services configuration files and even up to cloning a git/svn repository somewhere and running install/configuration/database migration tasks for an application.

Building a LEMP stack is just installing nginx, mariadb and php on a linux system, configuring it is another step but that's a common enough task such that there's already existing tutorials about it for every CMT.

Quick list taken from the first page of a google search with 'product nginx mariadb php':

Ansible Tutorial
Chef Official Tutorial (using apache, but moving to nginx is a matter of changing the web server cookbook used)
Puppet tutorial
Salt tutorial


Traefik configuration issue with local proxy Edit: I have updated my configuration to reflect Henrik Hoegh's tutorial. I am still running into issues.

I am setting up a local development environment with lots of docker containers running applications. I am using Traefik to manage and locally proxy these containers (each container having a named local URL). Right now my setup is as follows

Traefik.toml

[web]
  address = ":8088"
  [web.statistics]

################################################################
# Docker configuration backend
################################################################
[docker]

endpoint = "unix:///var/run/docker.sock"
domain = "local.io"
watch = true

# Expose containers by default in traefik
exposedbydefault = true

# Use the IP address from the bound port instead of the inner network one. For specific use-case :)
usebindportip = true

# Use Swarm Mode services as data provider
swarmmode = false

Docker-compose.yml file for container

...
container_name: mcserviceface
labels:
  - "traefik.port=8080"
...

I also have added local.io and mcserviceface.local.io (both pointing to my local machine) to my ect/hosts file.

When I start up the Traefik container and the mcserviceface container, it shows up on the Traefik dashboard, but it does not show the URL as mcserviceface.local.io rather something like http://185.28.0.3:80. I also hit a ERR_CONNECTION_TIMED_OUT when I follow that link.

How do I resolve both of these issues?

Edit: I have made sure that both containers on are on the same docker network via docker network connect .... It does not resolve the issue.

Make sure you connect your traefik instance to the client docker-network (as specified on the bottom of this page):

When running inside a container, Træfik will need network access through:

docker network connect <network> <traefik-container>


What are the advantages of dockerizing nginx and php in different containers? I just started working with Docker and Kubernetes and I've been watching a lot of stacks, in which some people build nginx+php in a single image and some build an image with nginx and another one with php (mounting the same path and enclosing both containers in the same deployment in Kubernetes).

What would be the advantages of building two docker images instead of installing both nginx+php in the same one?

PHP with nginx is usually done using php-fpm which is a separate processus.

Keeping the core idea of docker of one process (see end of answer for more details on this point) per container this makes sense to have the nginx process and php-fpm process in separate containers.

As the communication between nginx and php-fpm arise through fastcgi the php-fpm container can also be on a separated host and this allows using a cluster of php-fpm containers behind nginx.

After the wall of comment here's a little more background, docker documentation have paragraph about the idea that a container should have only one concern.

The main idea of a linux container (lxc) is to run a process in an isolated namespace at the cpu and memory level, docker add on top of this an isolation at the filesystem level.
The advantage is that the compromission of a process within this namespace won't allow to read memory of other processes and as such should prevent other compromission on the host.

While talking about nginx and php-fpm, they work in pair but each has it's own concern, nginx will do the HTTP part, routing, headers validation, etc. and php-fpm will do the code interpretation and return the html part to nginx. While it's usual to have both together serving a single application that's not mandatory.

Depending on context it may be easier to have a container including the whole stack for an application, on a developer workstation for exemple. But ideally for production use, try to keep the fewer interaction inside the container, having separated processes in the same container with supervisord brings its share of problem in term of zombie process and log handling (exemple story here for illustration purpose only).

So finally I'll quote the docker page with some emphasis:

  While “one process per container” is frequently a good rule of thumb, it is not a hard and fast rule. Use your best judgment to keep containers as clean and modular as possible.

There's no "silver bullet rule" which apply to everything, it's always a balance between the complexity within the container and the complexity orchestrating the containers themselves.


How to gain root access to an existing docker container? Able to get a response from the docker hub basex container:
nicholas $
nicholas $ lynx http://localhost:8984/ --dump
No function found that matches the request.
nicholas $

Ran the container as:
root $
root $ docker run -ti     -d     --name basexhttp     --publish 1984:1984     --publish 8984:8984     --volume "$(pwd)/basex/data":/srv/basex/data     basex/basexhttp:latest
71cbb84cf53143d00e6009d1ea5caa9de96c094c5d6aefc11cad95188b976776
root $
root $ docker container list
CONTAINER ID        IMAGE                    COMMAND                  CREATED             STATUS              PORTS                                                      NAMES
71cbb84cf531        basex/basexhttp:latest   "/usr/local/bin/mvn-…"   26 seconds ago      Up 23 seconds       0.0.0.0:1984->1984/tcp, 0.0.0.0:8984->8984/tcp, 8985/tcp   basexhttp
root $
root $ docker run -ti basex/basexhttp:latest bash
bash-4.4$
bash-4.4$ whoami
basex
bash-4.4$
bash-4.4$ pwd
/srv
bash-4.4$
bash-4.4$ sudo su -
bash: sudo: command not found
bash-4.4$
bash-4.4$ su -
su: must be suid to work properly
bash-4.4$

but how do I gain root access?

Solution is to use exec as:
root@doge:~#
root@doge:~# docker run -ti     -d     --name basexhttp     --publish 1984:1984     --publish 8984:8984     --volume "$(pwd)/basex/data":/srv/basex/data     basex/basexhttp:latest
5699af1dc04aa9c78066222e4b0b8e5621aa2ab9dd2bd46bcd87c6a0377c1960
root@doge:~#
root@doge:~# docker container list
CONTAINER ID        IMAGE                    COMMAND                  CREATED             STATUS              PORTS                                                      NAMES
5699af1dc04a        basex/basexhttp:latest   "/usr/local/bin/mvn-…"   33 seconds ago      Up 30 seconds       0.0.0.0:1984->1984/tcp, 0.0.0.0:8984->8984/tcp, 8985/tcp   basexhttp
root@doge:~#
root@doge:~# docker exec -ti --user root basexhttp bash
bash-4.4#
bash-4.4# whoami
root
bash-4.4#
bash-4.4# su basex
~ $
~ $ whoami
basex
~ $
~ $ exit
bash-4.4# exit
exit
root@doge:~#
root@doge:~# docker container list
CONTAINER ID        IMAGE                    COMMAND                  CREATED              STATUS              PORTS                                                      NAMES
5699af1dc04a        basex/basexhttp:latest   "/usr/local/bin/mvn-…"   About a minute ago   Up About a minute   0.0.0.0:1984->1984/tcp, 0.0.0.0:8984->8984/tcp, 8985/tcp   basexhttp
root@doge:~#
root@doge:~# docker ps -a
CONTAINER ID        IMAGE                    COMMAND                  CREATED              STATUS              PORTS                                                      NAMES
5699af1dc04a        basex/basexhttp:latest   "/usr/local/bin/mvn-…"   About a minute ago   Up About a minute   0.0.0.0:1984->1984/tcp, 0.0.0.0:8984->8984/tcp, 8985/tcp   basexhttp
root@doge:~#


Billing my clients for CDN usage My clients have the ability to download pictures I uploaded to a CDN network.
The CDN network is a PaaS service I found online.

I want to be able to distinguish between clients who download those pictures, so I can bill them by their usage.

Does anyone know such CDN service, that allows to bill (or at least measure) the clients of their (the CDN's) users?

I did something similar with Akamai a few years ago that might work for you and I have a Plan B in case this doesn't fit your needs.

Plan A

assign each client a unique hostname.  So you might have client0001.example.com and so on.
optionally, you can let the client create a CNAME within their domain.  I was doing this for ad creatives so I wanted blame for content to go the originators of the content.  Since it looks to end users like they were serving it from their own domain and my org was just in the middle of a CNAME chain that most people wouldn't notice I'm guessing the grief went where it belonged.
then within the CDN have the host headers map to your client ID's
get regular reports from the CDN for hits/bandwidth used per client ID
bill accordingly

Plan B

If you don't want to do all of that mucking with DNS or if your provider won't do the sort of mapping described there's still a way to make this work:

give each client their own directory like /cli0001/, /cli0002/
write your own reporting that downloads the raw logs from the CDN provider and then pulls the client ID's out of each directory name and totals it all up for you.

Billing plans

There are different ways you can go about billing folks, but common choices are:

create tiers so that the first megabyte is $X/byte and the next gigabyte is a little cheaper per byte and so on
give everyone a quota for a fixed fee or call it "free" within a broader service.  Then charge for going over the quota at a reasonable rate.


What is the difference between feature flags and feature toggles (if any)? Part of an answer to my (meta) question "Rename feature-flags to feature-toggles?" is like so:

  ... There's could be valid questions about feature flags (formalism, documentation about them, process of definition), which may not be related to feature toggles but to feature ramp up which carry a less binary meaning. ...

My question: In the context of DevOps, what's the difference between feature flags and feature toggles (if any)?

PS: Looking at https://martinfowler.com/articles/feature-toggles.html , I don't get the difference between such flags and toggles.

If you take it on an objective only point of view, there's no difference.

But feature toggles carry a heavy binary "on/off" for the whole application idea, whereas feature flags are a more relaxed term which can encompass ramp up testing more easily.

When you launch a ramp up testing and turn the new feature on for 10% of your traffic or user base and move up to 20% the next week and so on, the toggle term become counter productive.
Some people (managers/commercials/some client support people) may not really understand only a part of the clients get the new feature.

Ramp up can be defined as (quote from wikipedia on software testing technology):

  The usual sequence is to ramp up the load: to start with a few virtual
  users and increase the number over time to a predetermined maximum.
  The test result shows how the performance varies with the load, given
  as number of users vs. response time.

I like etsy's feature API (even is not maintained) as source for explanation, it's an easy to undertand framework which has a good readme around feature flags and their uses.

So the difference can be done in the usage of the terms to just help avoiding confusion on what happens on the application.


Docker container can't reach host services I am running a nodejs app in a docker (v 20.10.2) container on a Ubuntu 20.04 LTS (this is for dev purpose only)
My docker-compose config
  test:
    container_name: forge_test
    restart: always
    image: "node:lts"
    volumes:
      - ../src:/home/node/app
    command: "node test.js"
    user: "node"
    working_dir: /home/node/app
    extra_hosts:
      - "host.docker.internal:host-gateway"

In the docker, I can ping host.docker.internal (which does not work without extra_hosts), however I can not reach any services that I have running on any port (the port I am actually trying to reach is 7545 but none work) on the host.
Searching for a few hours now, no solutions I have found (adding docker0 to iptable, adding docker.host.internal to /etc/hosts) work.
Am I missing something? Could there be a firewall blocking access somewhere (II have ufw installed but tbh I don't really know how to use it)?
I can't find any similar issue where host.docker.internal can be pinged but the posrts not accessed

It happens that the problem comes from ufw not allowing connections from docker containers on host ports.
Deactivating it helped for now, since it's only for dev purpose.


ArgoCD Autosync based off of Docker Image `latest` Tag Using ArgoCD, I'd like to have the App AutoSync Update/Reploy when the docker image is updated on a specific tag such as latest or stable.
For example, if I have a deployment in my git repo like this:
apiVersion: apps/v1
kind: Deployment
metadata:
  name: k8s-nginx-example
spec:
  selector:
    matchLabels:
      app: k8s-nginx-example
  replicas: 2 # tells deployment to run 2 pods matching the template
  template:
    metadata:
      labels:
        app: k8s-nginx-example
    spec:
      containers:
      - name: nginx
        image: my-repo/k8s-nginx-example:latest
        ports:
        - containerPort: 80
        resources:
          requests:
            memory: "64Mi"
            cpu: "250m"
          limits:
            memory: "128Mi"
            cpu: "500m"

I'd like ArgoCD to update the deployment when a new docker image is pushed to latest tag or stable... etc. I know that ArgoCD polls the git repo of the app every 3 seconds so it is not aware of the changes to the Docker Image Tag. Another option could be (but I'm not sure how to do it)  would be to have ArgoCD Update via a webhook that notifies ArgoCD that there is a new image to pull with the latest tag, although even after pushing a Docker Image to the repository with the latest and then manually syncing the app in ArgoCD, Argo does nothing. I suspect that Argo is only reaching out to the git repo to check for changes and is not concerned with the Docker image in the deployment.
The only way I can think of doing this currently with ArgoCD is to version the Docker images with version tags 1.2.3 and then script the build process to update the deployment in the git repository with the image tag version each time.
...
containers:
  - name: nginx
    image: my-repo/k8s-nginx-example:1.2.3
    ports:
...

Is there a way to do this in ArgoCD? Or would a better option be using Argo Workflows with Argo Events? I was hoping that there might be a plugin or something that can either trigger the event or monitor the docker image.
Thanks
Meta: Need tags Argo and ArgoCD

This isn't really an ArgoCD thing. Keel is a good tool to help keep your images updated across the cluster and is super easy to implement. Basically, you'd just add some annotations to your manifest.
metadata:
  annotations:
    keel.sh/policy: force
    keel.sh/trigger: poll
    keel.sh/pollSchedule: "@every 10m"

It may be worth noting that if you are moving to run this in production, avoid the use of the latest tag. This is considered bad practice as it can burn you when breaking changes are pushed. Keel can handle that through an innate understanding of semver tags allowing you to update on minor version changes, but avoid major without testing, etc.


Understanding Docker layers We have the following block in our Dockerfile:

RUN yum -y update
RUN yum -y install epel-release
RUN yum -y groupinstall "Development Tools"
RUN yum -y install python-pip git mysql-devel libxml2-devel libxslt-devel python-devel openldap-devel libffi-devel openssl-devel

I've been told that we should unite these RUN commands to cut down on created docker layers:

RUN yum -y update \
    && yum -y install epel-release \
    && yum -y groupinstall "Development Tools" \
    && yum -y install python-pip git mysql-devel libxml2-devel libxslt-devel python-devel openldap-devel libffi-devel openssl-devel

I'm very new to docker and not sure I completely understand the differences between these two versions of specifying multiple RUN commands. When would one unite RUN commands into a single one and when it makes sense to have multiple RUN commands?

A docker image is actually a linked list of filesystem layers. Each instruction in a Dockerfile creates a filesystem layer that describes the differences in the filesystem before and after execution of the corresponding instruction. The docker inspect subcommand can be used on a docker image to reveal its nature of being a linked list of filesystem layers.

The number of layers used in an image is important

when pushing or pulling images, as it affects the number of concurrent uploads or downloads occuring.
when starting a container, as the layers are combined together to produce the filesystem used in the container; the more layers are involved, the worse the performance is, but the different filesystem backends are affected differently by this.

This has several consequences for how images should be built. The first and most important advice I can give is:

  Advice #1 Make sure that the build steps where your source code is involved comes as late as possible in the Dockerfile and are not tied to previous commands using a && or a ;.

The reason for this, is that all the previous steps will be cached and the corresponding layers will not need to be downloaded over and over again. This means faster builds and faster releases, which is probably what you want.  Interestingly enough, it is surprisingly hard to make optimal use of the docker cache.

My second advice is less important but I find it very useful from a maintenance view point:

  Advice #2 Do not write complex commands in the Dockerfile but rather use scripts that are to be copied and executed.

A Dockerfile following this advice would look like

COPY apt_setup.sh /root/
RUN sh -x /root/apt_setup.sh
COPY install_pacakges.sh /root/
RUN sh -x /root/install_packages.sh

and so on.  The advice of binding several commands with && has only a limited scope. It is much easier to write with scripts, where you can use functions, etc. to avoid redundancy or for documentation purposes.

People interested by pre-processors and willing to avoid the small overhead caused by the COPY steps and are actually generating on-the-fly a Dockerfile where the

COPY apt_setup.sh /root/
RUN sh -x /root/apt_setup.sh

sequences are replaced by

RUN base64 --decode … | sh -x

where the … is the base64-encoded version of apt_setup.sh.

My third advice is for people who wants to limit the size and the number of layers at the possible cost of longer builds.

  Advice #3 Use the with-idiom to avoid files present in intermediary layers but not in the resulting filesystem.

A file added by some docker instruction and removed by some later instruction is not present in the resulting filesystem but it is mentioned two times in the docker layers constituting the docker image in construction. Once, with name and full content in the layer resulting from the instruction adding it, and once as a deletion notice in the layer resulting from the instruction removing it.

For instance, assume we temporarily need a C compiler and some image and consider the

# !!! THIS DISPLAYS SOME PROBLEM --- DO NOT USE !!!
RUN apt-get install -y gcc
RUN gcc --version
RUN apt-get --purge autoremove -y gcc

(A more realistic example would build some software with the compiler instead of merely asserting its presence with the --version flag.)

The Dockerfile snippet creates three layers, the first one contains the full gcc suite so that even if it is not present in the final filesystem the corresponding data is still part of the image in same manner and need to be downloaded, uploaded and unpacked whenever the final image is.

The with-idiom is a common form in functional programming to isolate resource ownership and resource releasing from the logic using it. It is easy to transpose this idiom to shell-scripting, and we can rephrase the previous commands as the following script, to be used with COPY & RUN as in Advice #2.

# with_c_compiler SIMPLE-COMMAND
#  Execute SIMPLE-COMMAND in a sub-shell with gcc being available.

with_c_compiler()
(
    set -e
    apt-get install -y gcc
    "$@"
    trap 'apt-get --purge autoremove -y gcc' EXIT
)

with_c_compiler\
    gcc --version

Complex commands can be turned into function so that they can be fed to the with_c_compiler. It is also possible to chain calls of several with_whatever functions, but maybe not very desirable. (Using more esoteric features of the shell, it is certainly possible to make the with_c_compiler accept complex commands, but it is in all aspects preferable to wrap these complex commands into functions.)

If we want to ignore Advice #2, the resulting Dockerfile snippet would be

RUN apt-get install -y gcc\
 && gcc --version\
 && apt-get --purge autoremove -y gcc

which is not so easy to read and maintain because of the obfuscation. See how the shell-script variant outs emphasis on the important part gcc --version while the chained-&& variant buries that part in the middle of noise.


OpenShift configuration management As far as I can tell, reading the OpenShift documentation, including https://github.com/openshift/training/, the officially published "standard" way to manage OpenShift configuration (about their objects like services, build config etc.) is to simply use oc or the web console and "do it". They do not seem to document/suggest a standard way to keep this configuration, i.e. the YAML or JSON representation that you see in the web console or in oc edit in, say, a git repository.

Would you say that most projects (that you witnessed or know about) work this way, i.e., just keeping the config inside OpenShift? Or are there established standards, to have this outside of OpenShift itself?

I noticed that there is a 3rd party OpenShift ansible module on github, but am not quite sure how proven/established that is.

AFAIK neither OpenShift nor Kubernetes itself allow you to externalize services configuration. You should have a bunch of yml/json files somewhere and control their deployment/configuration using CLI tools.

But there is Rancher which in version 2.0 allow us to import external Kubernetes clusters and manage them as Rancher's own environments.

Description of how this could be done with Rancher you can find here.

FYI: Rancher 2.0 is slowly moving to Beta so it might take time for you to use it in your activities.


Best solution for code develop What is the best git solution for java code development? Make my own git server? Developers (Linux, Mac, Windows) are working on their local computers, then they are using my own git server.

Would Jenkins with Maven be a better solution? What is the benefit?

Thank you.

Update

I would like to find the best, lean, and simplest solution for our develop team. Here is some info:

Team of about 10 developers.
Everybody has their own LOCAL (windows, mac, linux), and their own IDE.
Project could be developed in java, php, javascript, css, html. I think this is not important for now.
My conception is, that I have 3 servers, 1st for DEV, 2nd for TEST and 3rd for PRODUCTION.
Every server has own git environment, as I mentioned above.
Process should be from LOCAL -> DEV (check by lead developer), then from DEV -> TEST (check by customer), then from TEST -> PRODUCTION.
And this is question if is Jenkins / Maven useful, or only git is sufficient.

Thanks.

The Git Server

It's best practice to setup a Git server for the team to access.  This way you have a central server that can be maintained, backed up, and focus on serving the source code.

While you could create your own, there are also several popular commercial options available that often have free private repositories (non-exhaustive list):

Bitbucket
GitHub
AWS CodeCommit

The Git server will just be a repository for your code.  It will allow for workflows like Pull Requests to better manage the changing code base.  However, the Git Server will not build your code or deploy it anywhere.

Jenkins

Jenkins will not actually store your source code.  Instead it is a tool used for Continuous-Integration (CI) and Continuous Deployment (CD), a big part of DevOps culture.  The basic idea is that the Jenkins server will take the source code put into your Git server, ensure it compiles, automated tests pass, and possibly even deploy to your test environment.  Tools like Jenkins work with multiple programming languages.

Maven is a tool is specific to building Java projects.  Tools like Jenkins often have options to use Maven to build the project.

Setting up a CI/CD tool like Jenkins would be a great next step after you have a consistent place for everyone to store their code.

Best Solution

Gitlab, Bitbucket Pipelines, Azure DevOps are just three examples of tools that both house Git repositories and can do CI/CD. There are many options available for Git and CI/CD, and you can even combine them if you choose not to select an all-in-one solution.  Which tools are best is opinion based, and the best practice is to use a tool that helps manage your Git repositories and a tool for CI/CD.


Entering to docker container of my elasticsearch google kubernetes pod - CONTAINER ID is changing I have a kubernetes cluster on GCP, in which I have a elasticsearch service.
I  have a elasticsearch-0 pod which is running a docker container of elasticsearch.

~ ❯❯❯ kubectl describe pod elasticsearch-0
Name:           elasticsearch-0
Namespace:      default
Node:           gke-jaeger-persistent-st-default-pool-xxxxx-phvx/10.166.0.4
Start Time:     Mon, 07 Jan 2019 14:21:19 +0100
Labels:         app=jaeger-elasticsearch
                controller-revision-hash=elasticsearch-8684f69799
                jaeger-infra=elasticsearch-replica
                statefulset.kubernetes.io/pod-name=elasticsearch-0
Annotations:    kubernetes.io/limit-ranger: LimitRanger plugin set: cpu request for container elasticsearch
Status:         Running
IP:             10.36.2.27
Controlled By:  StatefulSet/elasticsearch
Containers:
  elasticsearch:
    Container ID:  docker://5212b8c223401355bd29a5639caea5097074a8a8101ceb10300f76465e4a6536
    Image:         docker.elastic.co/elasticsearch/elasticsearch:5.6.0
    Image ID:      docker-pullable://docker.elastic.co/elasticsearch/elasticsearch@sha256:f95e7d4256197a9bb866b166d9ad37963dc7c5764d6ae6400e551f4987a659d7
    Port:          <none>
    Host Port:     <none>
    Command:

And I need access to the node which manage my pod in order to get the container ID of my elasticsearch container.

I get the name of my instances with gcloud compute instances list and enter via  gcloud compute ssh <node-name> --zone=<zone-name> command and I've enter to the elasticsearch docker container with root privileges docker exec -it -u root containerID bash (following these steps)

I need perform this because I need add a new mountPath named  /jaeger-data in order to associate it to a persistent volume claim that I have prepared.

I enter to my docker container and create the mountPath

[root@elasticsearch-0 elasticsearch]# mkdir /jaeger-data
[root@elasticsearch-0 elasticsearch]#

But I can see that I am disconnecting continously from my docker container because the containerID is changed of a continous way.
Each time that I enter to my node to get my containerID with docker ps -a command, this containerID is different and when enter to the new container making use of the new ID, my /jaeger-data mountPath is not found, this mean that there isn't, may be because I am entering to a new container ...

What is the reason to my containerID change all time?
Is this related with Kubernetes node that contain my pod which have my elasticsearch docker container?

Is related with the Service situation?

  Kubernetes Pods are mortal. They are born and when they die, they are not resurrected. ReplicaSets in particular create and destroy Pods dynamically (e.g. when scaling out or in). While each Pod gets its own IP address, even those IP addresses cannot be relied upon to be stable over time. This leads to a problem: if some set of Pods (let’s call them backends) provides functionality to other Pods (let’s call them frontends) inside the Kubernetes cluster, how do those frontends find out and keep track of which backends are in that set?

  Enter Services.

How to can I address this situation?

I need create my mountPath in order to associate it to my elasticsearch-0 pod configuration

If I understand you question clearly, you're trying to mount /jaeger-data inside the elastic-search container or pod. To achieve this, what you can do is put this mount details inside the statefulset yaml of the Kubernetes pod for elastic-search. Using the volumeMounts inside the statefulset, you will not need to ssh or exec inside the container or bother about the change or container ID for each pod restart/recreation. You can use the below example as a reference for your PVC and volumeMounts, just edit your statefulset using kubectl edit sts elasticsearch-0 -n default, this will open an editor and you can replace you PVC details in the below yaml and append and save and exit will recreate your pod and attach the mountpoint:

          volumeMounts:
            - mountPath: /es-storage
              name: jaeger-data
  volumeClaimTemplates:
  - metadata:
      name: es-storage
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 100Gi

Container ID will change with each recreation of containers;
If in case you need to exec into the container, the better way is to use kubectl exec -ti <pod-name> -n <namespace> bash to exec into the container from any of the master node(similar command as docker exec, just replace pod-name and namespace) or from your local laptop if you have kubeconfig configured.
Using Service, you can expose this application on a particular port, Kubernetes creates an abstraction and itself takes care of the change of pod IP


What are Pros and Cons of various VCS and CI/CD tools to store, change and deploy configuration snippets of more than 1000 customers in a VCS? Follow-up to this question.

We are in the middle of a transition to Configuration as Code (CaC). At the moment a lot of sites are configured manually. A side effect of this is that so now and then configuration gets broken and nobody knows what happened. By applying CaC the Version Control System (VCS) could be consulted when an environment is broken and then it is easier to figure out the cause of the issue. Another advantage is that people cannot change the configuration directly on the production systems.

One of the comments to this Q&A was whether the configuration is shared between customers. At the moment this is not the case and this is something we would like to prevent as sharing too much could also be a disadvantage as this could cause tangled configuration, i.e. spaghetti.

In summary, there are multiple configuration files per customer site.

Process

When the configuration is changed and merged into master or trunk then the configuration has to be deployed.

There are more than 1000 customer environments that have to be migrated. One approach is to create 1000 VCS repositories. An advantage of this is that a configuration change will result in a deploy to a specific environment, e.g. code pushed to VCS customer0001 will deploy the configuration to customer0001.

Discussion

As there are more than 1000 customers, I wonder whether 1000 repositories will be a good idea in terms of costs, maintenance and so on. What about creating a separate VCS for the configuration and create a folder per customer, i.e. a monorepo. E.g.:

VCS repository

customer0001

configFileA
configFileB
configFileC

customer0002

configFileA
configFileB
configFileC

customer0004

configFileA
configFileB
configFileC

customer0005

configFileA
configFileB
configFileC

customer0998

configFileA
configFileB
configFileC

customer0999

configFileA
configFileB
configFileC

customer1000

configFileA
configFileB
configFileC

An advantage is that it is clear that all configuration resides in one repository, but how to trigger a certain build if a certain customer folder change and deploy it to the environment of the customer?

It is possible to create a custom groovy script in Jenkins that does some filtering or use branches in certain VCS repositories like bitbucket, but I wonder whether there are more efficient solutions.

Preferred answer

A table that gives an overview of the pros and cons:

| VCS | Monorepo | CICD     | Pros               | Cons                  |
|-----|----------|----------|====================|=======================|
|     |          | Circleci |                    |                       |
|-----|----------|----------|--------------------|-----------------------|
|     | true     | Jenkins  |                    | groovy script required|
                                                 | to filter customer    |
                                                 | folders               |
                            |--------------------|-----------------------|
|     | false    | Jenkins  | code change results|more than 1000 jenkins |
                            | in deploy to       |jobs required          |
                            | associated customer|                       |
                            | environment        |                       |
|                           |--------------------|-----------------------|
| git |          |          |                    |                       |

Personally I'd keep them in separate repositories, for clear customer isolation:

minimal/no risk of unwanted interference between customers
different access control for different customers is possible
different CI/CD pipelines and/or configs for different customers is possible
simpler/standard CI/CD configurations
clean per-customer repository history
easier calculation of per customer resource usage
lower maintenance risks - better to have just a few customers down than all of them at once

I see some cons as well, but IMHO not enough to tip the balance:

more resources needed (one CI/CD pipeline per customer), but these costs can be reflected in the service pricing. And they should get a lot smaller due to automation.
sharing of configs/procedures across customers becomes difficult

If a unified VCS vision/presentation is desired I'd still keep separate per-customer repositories (thus keeping some/all of the pros above), collectively managed under a single umbrella repository:

the umbrella repository would contain just a manifest file listing the per-customer repositories, their versions and maybe some other relevant info
changes in one or more of the individual repositories need to be accompanied by corresponding manifest changes in the umbrella repository to advance the overall project
a small set of wrapper tools would be used to operate at the umbrella repository level but controlling the individual repositories


Does Oracle have an API that shows the latest stable JDK version? There is an ansible role that installs jdk8, but is incompatible with jdk9. The role parses a couple of html pages and the commands are incompatible with java9. It was quite hard to make it work for jdk9, and now the PR is broken for jdk8.

https://github.com/ansiblebit/oracle-java/issues/63

In order to prevent that a lot of work has to be done over and over again, it would be great if there is an API that shows the latest jdk version. Then this version could be used in the ansible role and then this will be more stable.

http://api.releasesoftwaremoreoften.com/latestversion?name=oraclejdk8
http://api.releasesoftwaremoreoften.com/latestversion?name=oraclejdk9

Running:

curl -s http://releasesoftwaremoreoften.com/latestversion?name=oraclejdk9 | jq .latestVersion

returns:

9.0.4

and

curl -s http://releasesoftwaremoreoften.com/latestversion?name=oraclejdk8 | jq .latestVersion

returns:

8u162


Are CI and CD DevOps team tasks or are these separate professions? DevOps is about preventing silos and let teams work more together then let them throwing things over the wall. In an ideal situation every individual in a DevOps team is able to create programs, test them and make them available to the customer. According to me an important competence of a DevOps engineer is communication as this is required to get acceptance in a team and among teams  for new tools and processes.

The more I think about it the more I get doubts regarding the structure and responsibilities in a team. From a DevOps perspective, every individual should be able to do dev, qa and ops activities, but if you compare it with for example football, all players need to be able to kick a ball, but some of them are specialized in goalkeeping and others in scoring goals. So a DevOps team could exist of QA, Ops and Dev engineers, but all of them are able to test, create and deploy software and are a basically able to kick a ball like in football.

As DevOps does not mean specialzed in dev, qa and ops, like goalkeepers will normally not score as much as a striker, is CD a profession or is it one of the DevOps tasks, or should it be everybody's responsibility, but why are Dev, QA and Ops engineer part of a DevOps team as well? From that perspective there should be a CD engineer as well, but CI is not part of CD. Should two professions, i.e. CI engineer and CD engineer be part of DevOps team as well?

DevOps is a combination of two roles Development and Operations, CI & CD are the main role a DevOps must be doing. Continuous Integration (CI) is a part of developer's role and Continuous Deployment/Delivery is a part of operation/infra/release-management role. In the agile methodology of development we've combined these two different role to a new position called DevOps who'll be responsible for release management or deployment. Thus it is a mandatory for a DevOps to know about CI & CD.


Jenkins and Kubernetes CLI plugin and GKE I'm trying to use Jenkins Kubernetes CLI plugin from https://plugins.jenkins.io/kubernetes-cli/ to let deploy artifacts from Jenkins to GKE cluster.
Expected Behavior
Get cluster info on kubectl cluster-info
Actual Behavior
Error out as Unable to connect to the server: x509: certificate signed by unknown authority
Steps to Reproduce the Problem
Output of generated ./kube/config
+ kubectl config view
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: DATA+OMITTED
    server: https://xxx.xxx.xxx.yyy
  name: my-cluster
contexts:
- context:
    cluster: my-cluster
    user: jenkins-pipeline
  name: my-cluster-jenkins
current-context: my-cluster-jenkins
kind: Config
preferences: {}
users:
- name: jenkins-pipeline
  user:
    username: jenkins-pipelin

When I run from a command line I need to do
export GOOGLE_APPLICATION_CREDENTIALS=path/to/key.json
Where that part goes in that plugin?

Put it inside the sh?
withKubeConfig(...) {
  sh “””
    export ...
    kubectl ...
  “””
}


Best method of retrieving environment information via Salt Stack? I have been tasked with using Salt Stack to find out what files/services exist in each of our environments (Test, UAT, Production).

The Salt master can speak to all servers in Test and UAT (Production Salt master is yet to be configured) so I was thinking of creating a state in 'base' which can target all minions just doing simple file.exist queries. However I quickly came to the realisation that certain files won't exist on certain servers due to the different nature of them. E.g. Our API server has a lot of specific scripts and configs which won't exist on our LDAP server.

What is the most efficient and scalable way of implementing this? Could I define roles for each type of server (e.g. webserver, ldap server, api server) and therefore a specific state for each role?

SaltStack provides a method for modifying grains and adding additional information to the grains dictionary in several different ways by either setting them in the /etc/salt/minion config and/or through the /etc/salt/grains file.

For example:
1. add grains to the minion config. Note, simply include the grains key here:

id: minion-07
grains:
  roles:
    - production
  region: nl-amsterdam

add custom grains within the /etc/salt/grains file. Note, add grains without prefacing them with grains:

   roles:
       - production
     region: nl-amsterdam

Alternatively you may set roles via command line shell from master or from salt minion:

  salt 'your minion' grains.setval roles [production, dev, qa]

  salt-call grains.setval roles [production, dev, qa]

For matching grains I usually use Jinja2. Or you may refer to commandline/config file approaches:

  salt -C 'G@roles:production' test.ping

By using sytnax in the top.sls file:

base:
  'roles:production':
    - match: grain
    - salt.master
  'roles:dev':
    - match: grain
    - api-dev

Alternatively you may consider using environments in SaltStack, see here for more details: https://docs.saltstack.com/en/latest/ref/states/top.html#environments

However, roles are more suitable for running specific states, at least We use them in our our company. So up to you what is the best ;)


Jenkins Node Image at Kubernetes Plugin I'm working on huge pipeline with different development environment (python, node, java).
For manage this CI pipeline our teams using Jenkins at Kubernetes environment.

I want to give PipeLine node annotation at Groovy Script show as below .

 node("test-py") {
   sh "python -m -v py.test TEST_FILE.py "
 }

How I can create pod with image , with Jenkins Kubernetes Plugin .

Have a look at the documentation presented here:

https://github.com/jenkinsci/kubernetes-plugin

You need to specify a Kubernetes pod in your pipeline. An example of your use case would look similar to this for a single python container. You'll notice that containers: [] expects an array so you can include additional containers.

def label = "mypod-${UUID.randomUUID().toString()}"

podTemplate(label: label, containers: [
    containerTemplate(name: 'python', image: 'python:3.7-alpine', ttyEnabled: true, command: 'cat'),
]) {
    node(label) {
        stage('Get a Python project') {
            git 'https://github.com/oğuz/project.git'
            container('python') {
                stage('Test a Python project') {
                    sh "python -m -v py.test TEST_FILE.py "
                }
            }
        }
    }
}


Can be DevOps culture mapped to a specific type of organization as described by Laloux? In his book, "Reinventing organizations", Frederic Laloux describes a historical evolution of organizations and their systems of internal structure, management and values.

Where will an organization place on this chart when it would adopt a set of DevOps best practices and culture recommendations? Which stage of development is a target for DevOps practices and what development of the organization is being achieved? Does it correspond to this model?

Here is a picture of the Laloux Culture Model:

This question perfectly illustrates the different approaches in adopting DevOps by organizations. While both approaches target the 3rd circle on the culture model - the traditional corporations - there is a difference in aim and results.

Adopt DevOps best practices and tools for increased effectiveness.
Adopt DevOps culture and values for organizational transformation.

Established corporations often aim at (1) and either omit or fail to adopt (2). They will still gain increased effectiveness and improved organizational key performance indicators, but they will fundamentally stay at the orange level of organizational development, failing to undergo the transformation. Such transformation always requires an existential threat to the organization and those are rarely recognized and acted upon until there is too late for the organization to transform.

New organizations and organizations with new leadership have an easy way to focus on (2) and so they either are raised up at the new developmental stage or manage to successfully seed business units operating with new culture and values that either subsume the entire organization eventually or the old business units slowly decay and whither while the new ones take over a large bulk of the business. It is almost like a parent passing his family business to a child who has been raised with different values and more progressive ideas.

Either way, you can say that DevOps as it is understood today, operates between the orange and green layer and has the potential to enable the transformation from Orange to Green under certain conditions.

The core ideas behind DevOps are a different matter. They are in a way a method for an organization to pass from any developmental stage to the next, but as DevOps is being codified through specific cultural recommendations, concrete values, and actual practices and tools, the instantiation of those core ideas in form of DevOps is squarely aimed at the border between Orange and Green.


Include verbose output in terraform/terragrunt Is there any way to include verbose / debugging output in Terraform / Terragrunt?
I'm new to the language, and often struggle with knowing exactly what's going on.  It would be helpful for me to be able to inspect certain values; e.g. to confirm that locals & variables have the values I expect, to get the ids of created components, etc.
I can include such info by creating outputs and assigning values there; but if I'm debugging a module I then have to include the output in the module's output, and again in the calling code's output for it to be visible; then have to delete those entries before delivering the solution.
I'd like a solution more like C#'s Debug.WriteLine(/*...*/) or PowerShell's Write-Verbose #... i.e. somewhere I can say what value I'd like to inspect, but have that output only show in the terminal when run with verbose output enabled, so these statements can be toggled on when needed, rather than impacting the delivered solution.
Does anything like this exist in the language?  So far I've found nothing documented or by Googling.
Note: The TF_LOG environment variable documented here is helpful for providing more info on what's going on under the covers; but only allows you to see predefined output; not to push your own content to the output stream.

You might be able to do something with the local-exec Provisioner to push the information out to an external script or file which you could monitor separately during the run.
The issue you might encounter is the run order, HCL isn't processed top to bottom like a "traditional" programming language, it works out a dependency tree for the resources it is creating, so might not run the local-exec when you'd expect them to be run.