What is a good storage for storing performance results and visualizing in grafana Intro

I am a tester and need to analyze the results of automated tests (response times and etc.)

Current situation

1) I run performance tests using locust.io
2) During test execution, response times of successful results are pushed to Graphite database
3) I visualize the results in Grafana

Problem

I tried to calculate the average using Grafana to see if it matches the results generated by Locust. It doesn't.

After a while, i tried to figure out what could be a problem. Graphite has data retention. As a result, the smallest time period for a data point is 1 second.

My situation is that I could have several data points with a time difference of several milliseconds.

I thought I could use something like Prometheus as a substitute for graphite, but it doesn't seem to have an import of data. A solution would be installing Graphite Exporter, such that i can imitate "pushing" data to db. But I think installing this extension and setting everything up is an overkill. There should be a much simpler solution.

Question

Could someone suggest a solution from personal experience? I need a db/storage to which I can write in a format query_name responce_time timestamp, and then use this storage as a data source for visualizing a report in Grafana.

Thanks in advance!

Sadly nobody answered, so I will provide feedback about how i did it so far.

Graphite was not a good solution for me due to its retention policy. Performance tests sent data to the database every millisecond, and due to retention Graphite did not store all the results but only the last one per the period, meaning if you sent 100 data points in the last minute, the first 99 would be dropped, the 100th one would be stored. This fact was found by me somewhere on some forum. I do not claim I have setup/configured/used graphite correctly, yet the result was several days of debugging a bug which appeared an actual feature, lots of stress and questioning if my head and hands grow out of my shoulders.

Next option was to use InfluxDB. Sadly didn't work for the following reason:
 - If running InfluxDB in a Linux docker container, I was not able to bind mount it to a specific folder on host's OS outside of docker installation folder.
A solution would be to try and use docker named volume. Maybe that would have worked, but I was short on time and patience.

The final solution was to use simply a Postgres DB. Created a timetable with all the metrics and a timestamp field. As a result, Locust writes the final session statistics into a CSV file. CSV file is parsed by a python script and parsed data is inserted into the table. Grafana then connects to the DB and quite nicely visualizes the results using a simple query.

Next possible question would be:

  how to store the results of every single request to gain better
  precision for the results of my performance tests?

My personal suggestion would be to use influxDB with a consistent storage saved in a named volume if you are using Windows host. There the smallest time period can go down to milliseconds and data retrieval was very easily setup by manually writing a simple select query. Yet, please do not take my word for it and try this out, because i haven't done this, so my thoughts are nothing but a theory.

Happy performance testing.


Autoscaling on GoCD agents without terminating active builds? At the moment I have an AutoScaling Group (ASG) of GoCD build agents without any scaling policies. I have created some custom metrics that indicate how many build agents are currently idle and I'd like to scale based off of that. My concern is that when scaling down, the ASG may terminate instances that are in the middle of a build. This may result in failed builds and delayed builds.

How can I scale down an ASG without terminating instances that are in use?

Auto Scaling groups have a useful feature for this, named lifecycle hooks.

Worflow taken from the documentation above:

As you can notice there's a Scale in step, triggering a Terminating:Wait for the autoscaling group and notifying the instance to be terminated, the instance has now to do it's work and once done signal it is ok to be terminated.

If the task can take longer than the autoscaling group HeartbeatTimeout parameter you can reset the timeout with (quoting still from the same page):

  Restart the timeout period by recording a heartbeat, using the
  record-lifecycle-action-heartbeat command or the
  RecordLifecycleActionHeartbeat operation. This increments the
  heartbeat timeout by the timeout value specified when you created the
  lifecycle hook. For example, if the timeout value is 1 hour, and you
  call this command after 30 minutes, the instance remains in a wait
  state for an additional hour, or a total of 90 minutes.

So for your case, the lifecycle notification should startup a kind of script/program which will:

prevent this builder to receive new builds
loop periodically to check if the ongoing builds are done
if there's build still in progress and the timeout is near, reset the timer
if there's no more builds in progress, signal to proceed to termination


How to run a script or a command on EC2 instance via AWS CLI? According to New EC2 Run Command news article, AWS CLI should support a new sub-command to execute scripts on remote EC2 instances.

However I've checked in aws ec2 help, but I can't find the relevant command.

I've installed aws via apt-get:

$ aws --version
aws-cli/1.14.32 Python/3.5.4 Linux/4.12.7-64 botocore/1.8.36

Which sub-command I should look for and what's the syntax to run, let say ipconfig in PowerShell on the remote EC2 instance?

To run ipconfig from the AWS Systems Manager Run Command:

$ aws ssm send-command --document-name "AWS-RunPowerShellScript" --instance-ids "<your instance id>" --parameters commands=ipconfig

Note: If you've got the error, consider specifying the right --region.

This assumes you have your AWS credentials and CLI configured properly. See Systems Manager Run Command Walkthrough Using the AWS CLI for more information.

Here is the practical shell command example of sending and getting the command output:

cmdid=$(aws ssm send-command --instance-ids "i-ch3ng3th1s" --document-name "AWS-RunPowerShellScript" --parameters commands=ipconfig --query "Command.CommandId" --output text)
aws ssm list-command-invocations --command-id "$cmdid" --details --query "CommandInvocations[*].CommandPlugins[*].Output[]" --output text


How does Docker container orchestration work? I am familiarizing with the architecture and practices to package, build and deploy software or unless, small pieces of software.

If suddenly I am mixing concepts with specific tools (sometimes is unavoidable), let me know if I am wrong, please.

On the road, I have been reading and learning about the images and containers terms and their respective relationships in order to start to build the workflow software systems of a better possible way.

And I have a question about the services orchestration in the context of the docker :

The containers are lightweight and portable encapsulations of an environment in which we have all the binary and dependencies we need to run our application. OK
I can set up communication between containers using container links --link flag.

I can replace the use of container links, with docker-compose in order to automate my services workflow and running multi-containers using .yaml file configurations.

And I am reading about of the Container orchestration term, which defines the relationship between containers when we have distinct "software pieces" separate from each other, and how these containers interact as a system.

Well, I suppose that  I've read good the documentation :P

My question is:

A docker level, are container links and docker-compose a way of container orchestration?

Or with docker, if I want to do container orchestration ... should I use docker-swarm?

welcome to DevOps SE, @bgarcial!

Well I think there are the following options depending on the use case:

Docker Container links are a according to the vendor site a legacy feature and you shouldn't use it
Docker compose should work well but you can't scale horizontally: the stack is runnable only one a single machine
With Docker Swarm using mostly same YML you can distribute your containers across many machines
If you are going to deploy large scale production environments you might want to choose Kubernetes over Swarm, maybe have a look for example on Rancher/Kubernetes.


How to check accepted cipher suites by an Azure App Service? I am looking for a way to reliably check accepted cipher suites by an Azure App Service. On standard virtual machine I just remote to it and execute PowerShell command: Get-TlsCipherSuite which provides a list of currently accepted keys or check the system's registry.

I could pen test app service by ssllabs.com however I cross checked their results against results from Get-TlsCipherSuite for a certain machine and it's not 100% accurate: ssllabs didn't detect suites with Pre Shared Keys.

My questions:

how to query Azure App Service accepted cipher suite?
how do browsers know which cipher suites are available to it when it tries to go to a given domain?
if that's not obvious out of answers to previous questions why ssllabs doesn't detect suites using PSK?

Not sure if this is the right answer but let me give it a try with the limited knowledge i have on this.

these are the resources that I found when I am looking up for tls enabling for my app service. please see if any of this works.
msdn
how to configure ciphersuite for app service
enforce tls vrsion
The client (browser) gives a list of cipher suites it can handle to the server and the server selects one, the decision is passed on to the client during the handshake.
so all the servers won't accept all the cipher suites, it based on server config to choose.


Separation of CI and CD processes Is it appropriate to keep the DevOps resources in a different repo from the git repos where the project source codes are placed for the execution of DevOps processes?
some benefits i expected

Versioning files to a different repo such as Dockerfile, Jenkinsfile or Kubernetes deployment
Ensuring the security of deployment processes
Being able to manage the deployment processes of these projects since some projects consist of more than one repo

I know the concepts of monorepo and multirepo, but what I say is a little different. What I mean is to abstract the repos of deployment files from the source code repos.
Could this be a method of use? have you done something like this before? is this method useful? Does this method have a name or does it mean multirepo?
I am waiting for your advice
thanks everyone

You're basically describing GitOps. Here's how I've followed that for a containerized microservices build and deploy pipeline:

Each service repo includes it's own Dockerfile, Kubernetes manifests, and anything else needed to locally build and test the project. The base image for the Dockerfile will likely be managed from another repo but built and hosted by the organization.

CI steps are mostly externalized to a set of common libraries or templates. So if there's a Jenkinsfile or equivalent, it sets a few variables and calls the common template to build and generate the artifacts.

Artifacts include a docker image, pushed to a registry, and also the deployment manifest (compose file, Kubernetes manifest, Helm templates), along with any other variables for those manifests. These deployment manifests/variables are pushed to a separate deployment repository that includes all of the microservices' manifest. There is only one deployment repo for all of the microservices in a single application stack, so this commit becomes a fan-in step. The push to this repository is scripted/automated during the CI of the microservice repo when a pull request is approved by enough trusted people to a protected branch. This provides the needed security checks.

The deployment repo commits triggers it's own pipelines that are effectively CD and following something close to a GitOps design. This could also be a GitOps tool monitoring the repo and applying the changes, but presently I've been sticking with a single CI tool that does both, simplifying the workflow and allowing some scripting to run on the deploy. That scripting for me loads the variables that were committed, applies additional variables (e.g. which environment is this being deployed to), renders the templates (currently a Helm umbrella chart), and deploys.

Deploys to other environments becomes a pull request between branches of the deploy repository. It could also be separate repositories to enforce security controls. Importantly, rendering the manifest should point to an immutable image tag or sha hash to ensure what was pushed to dev and tested there is the same as what's pushed to production. I include these image tags in a values file that is committed to the deploy repo.

One big advantage of this, multiple microservices that interact with each other are tested together and get deployed together as a unit, all services are fanning into a single deploy repo that contains their deployment manifests and what image tag to use. There's never the situation where service A gets a new feature, but it's buggy, service B gets built depending on that new feature, and service B gets released to production before service A (a common issue when each service is built and deployed in isolated pipelines). If A is blocking the deployment of the stack, then a pull request can be submitted to the deploy repo reverting the service A commit. Then if service B passes tests against an older service A, service B gets deployed to production.


What reasons should I NOT use a private DNS zone and CNAME services in my AWS/K8s environment? I have a Kubernetes cluster in AWS, with apps that talk to AWS services like RDS, Elasticache, Elasticsearch, DynamoDB, etc.
For the services where it's available (Elasticache and RDS, and more recently Elasticsearch with a custom ACM certificate), I've considered making a private zone in Route53 and making CNAMEs for all the things. So instead of app-production.789yuiahsdf.redis.elastitcache.aws.com or whatever the Elasticache endpoint is, my app talks to sidekiq.redis.app. I'd use a pattern, maybe <purpose>.<service>.<app>, so I'd have sidekiq.redis.app, cache.redis.app, primary.postgres.app, replica.postgres.app, etc, etc.
The next step is having the app is configured by default to use these hosts. In development, staging, and production, they don't NEED any configuration (maybe it's still available), but the defaults should "just work" if the environment is set up properly.
What are the downsides here? What am I missing? To me, this seems sort of like simple service discovery and a reasonable idea. But reasonable ideas can sometimes bite you later on.

It seems a reasonable idea to me.
There migt be a few things you might want to check tho :
This article or this one. For example :

the fact the that a private hosted zone is associated with a VPC
there is some DNS hostnames and DNS resolution parameters on the VPC settings

enableDnsHostnames : Indicates whether instances with public IP addresses get corresponding public DNS hostnames
enableDnsSupport : Indicates whether the DNS resolution is supported.

Only VPC resolver will resolve the private records (subnet range +2 for the ip of the VPC resolver, search for reserved ip here)

take care you didn't configure custom resolver and so wont be able to access your private records
think about the fact that you won't be able to resolve those records from outside the VPC (office for example) unless you manage to plug to the VPC resolver somehow

You could also give a look at Alias record ?
Could give advantages like :

create record that has the same name as the hosted zone
not being charged for the queries (in opposite to CNAMEs)

These are few things I can think off right now. Hope this helps.
Cheers.


How does Kubernetes determine which node to use? I am researching using Kubernetes on baremetal for our application. There's an issue I can't find info about. If a worker node dies, how does Kubernetes determine which node(s) to redirect its workload to? What I'm mostly worried about is that if a node dies due to being overloaded with work, Kubernetes will redirect its workload and another node will die due to the workload increase, which leads to a domino effect of nodes dying. How does Kubernetes prevent this?

In order to avoid this kubernetes reserves resources on the nodes for its daemons. See full explanation and configuration guidelines here


Why do companies use JFrog Artifactory or other repositories instead of Maven central repository? I have usually seen that when someone uses Maven on a company network to download dependencies, then user is redirected to the JFrog Artifactory or any other, instead of downloading dependencies from official Maven Central Repository.
What's the benefits to companies/organisations using other Repositories like JFrog?

Saved bandwith and faster downloads: Artifactory stores the artifacts that are downloaded from maven central. So if another developer needs the same dependencies they don't need to be downloaded again from maven central but instead they can be delivered from the local artifactory instance.
This makes downloading faster because company networks are usualy faster then the internet connection and it also saves bandwith.
As example imagine that a developer adds a new dependency to a project. This dependency then needs to be downloaded into the local maven cache of every other developer that is working on the same project.
On large projects with a lot of developers this means that the dependency might be downloaded 30 times.
Stable builds: Dependencys might be removed from maven central. If you have them in your local artifactory instance the project who depends on it can still be build.
Easier maintenance: Another benefit that has nothing to do with maven: Artifactory provides several types of repositories like for Docker or Vagrant.
So instead of setting up a maven artifacts repository, a docker registry and some kind of vagrant store you could create three repositories in artifactory.
Thus you only need to maintain one application (artifactory) instead of three separate applications.

Some or all of these benefits might also apply to other binary repository managers but as I only know artifactory I can't say something about that.


RPM's, VM's : time consuming I am constantly delivering RPM packages upon every new version of some project.

Every version is dependent on some developing prerequisites

CentOS 7
git
Some npm packages.
Some Python PIP packages
Some shared objects

I'm dealing with this issue currently with having a virtual machine for every release, and that is very convenient to release every version - just load the ova file of the specific machine and you are ready to go.

but the problem is, the overhead of dealing with virtual machines:  cloning and other operations - taking too much time.

What do you recommend in order to do it much more easier?

Have you considered setting up a build server (i.e. Jenkins, TeamCity) with disposable build agents?

If you do it in the cloud (i.e. via ec2 integration), you can literally set up an AMI with generic dependencies, then configure it to spin up a new VM for each build agent, which can be configured whichever way, and then delete the VM once you've done publishing your artifact (i.e. RPM).


How to know which user created the GCE Instance We have several billing enabled projects in our GCP account and each of them has several Compute Engine Instances. I want to remove non-critical instances just to reduce the unnecessary billing. But before I do that, I need to know which instance is being used for what and for that I need to know which user created it in the first place.

How do I get to know the creator of each instance either by using Google Cloud Shell or Google Cloud Console?

Note: I'm not a GCE user yet, the answer is based solely on documentation.

You could be Viewing Audit Logs in the Google Cloud Console, more specifically the Admin Activity logs:

  Admin Activity logs contain log entries for API calls or other
  administrative actions that modify the configuration or metadata of
  resources. For example, the logs record when VM instances and App
  Engine applications are created and when permissions are changed. To
  view the logs, you must have the IAM roles Logging/Logs Viewer or
  Project/Viewer.

You would be looking for the User identities in audit logs:

  Audit logs record the identity of the user performing logged actions.
  The identity is held in the AuthenticationInfo field of AuditLog
  objects.

Ideally you'd be looking for the GCE instance creation events, but some/all of those may be too old for the logs retention policy. Then maybe search for VM instance start or other relevant events.

You could also check and maybe cross-reference the audit logs info with that from Usage Reports or exported billing info.

If this is a regular, on-going activity you may want to setup an automated log exporting and processing pipeline.


How does Kubernetes terminate an instance after helm upgrade I have a node.js application that is running on a Kubernetes cluster, I am using helm to manage it currently. I have ran into a bit of an issue, when the application starts it connects to a service discovery tool, when it is to shut down it should disconnect from said tool. If I do it locally then it works fine, however when I do a helm upgrade it goes sideways and doesn't disconnect which then results in the api gateway having a fit.
Is there something different than a basic helm template and using upgrade? The application supports what I need just a matter of having Kubernetes do it too.
Here is the express server and consul code, it goes at the bottom of my app.js file.
const port = 8081;
app.listen(port, () => {
    const CONSUL_ID = require('uuid').v4();
    const ip = require('ip');
    const my_IP = ip.address();
    let options = {
      name: 'auth',
      address: `${my_IP}`,
      port: port,
      id: CONSUL_ID,
      check: {
        ttl: '10s',
        deregister_critical_service_after: '1m'
      }
    };
    consul.agent.service.register(options, function(err) {
      if (err) throw err;
      console.log(`Registered service with ID of ${CONSUL_ID}`);
    });

    setInterval(() => {
      consul.agent.check.pass({id:`service:${CONSUL_ID}`}, err => {
        if (err) throw new Error(err);
      });
    }, 5 * 1000);

    process.on('SIGINT', () => {
      console.log(`SIGINT. De-Registering service with ID of ${CONSUL_ID}`);
      consul.agent.service.deregister(CONSUL_ID, (err) => {
        if(err) console.log(`Error de-registering service from consul, with error of : ${err}`);
        if(!err) console.log(`De-registered service with ID of ${CONSUL_ID}`);
        process.exit();
      });
    });
    console.log(`Auth server running express started on port ${port}.`);
});

Suppposing that you are using typical Deployments, a new replicasets is created and the old one shrunk.
Looking at the docs for 1.16, it seems that TERM is sent, and then finally KILL.

When a user requests deletion of a Pod, the system records the intended grace period before the Pod is allowed to be forcefully killed, and a TERM signal is sent to the main process in each container. Once the grace period has expired, the KILL signal is sent to those processes, and the Pod is then deleted from the API server. If the Kubelet or the container manager is restarted while waiting for processes to terminate, the termination will be retried with the full grace period.

https://v1-16.docs.kubernetes.io/docs/concepts/workloads/pods/pod/#termination-of-pods
Therefore, you'll want to accept TERM signals, not just KILL. Also, you'll need to ensure the signal is properly transmitted from the surrounding docker container into your server process.


How to automatically remove k8s namespaces every week? In our company multiple stakeholders create test systems in k8s that they forget to remove. The aim is to remove namespaces that are older than one week automatically. Is there a certain operator that is able to do this?

I recommend that you have an expire-date as label or annotation on the namespaces used for test. Then you can have a Job to clean-up expired namespaces.


Ansible: Use inventory_hostname variable in lineinfile module I'm using Ansible 2.7.9
and I'm having trouble with the regex definition in the lineinfile module. My file has the following line:

host.example.com = /path/to/a/directory

I want to remove that line. As {{ inventory_hostname }} contains dots, I'm trying to escape those with replace. The task in my playbook looks like this:

- name: Remove LE webroot definition
    lineinfile:
      path: "/etc/path/to/config/{{ inventory_hostname }}.conf"
      regexp: "^{{ inventory_hostname | replace('.', '\.') }} = /path/to/a/directory"
      state: absent

When I'm executing the playbook, the following error occurs:

[...]
The offending line appears to be:
      path: "etc/path/to/config/{{ inventory_hostname }}.conf"
      regexp: "^{{ inventory_hostname | replace('.', '\.') }} = /path/to/a/directory"
                                                      ^ here
We could be wrong, but this one looks like it might be an issue with
missing quotes.  Always quote template expression brackets when they
start a value. For instance:
[...]

As you can see, I'm using the double quotes, and all examples that I've found are using this syntax. What I'm doing wrong here? Thanks for any help.

The issue seems resolved, If I escape the backslash in the regex too:

- name: Remove LE webroot definition
    lineinfile:
      path: "/etc/path/to/config/{{ inventory_hostname }}.conf"
      regexp: "^{{ inventory_hostname | replace('.', '\\.') }} = /path/to/a/directory"
      state: absent


phpMyAdmin using dockerised database I have installed mariadb inside docker container, and mariadb is running properly inside that container. I want to get access of mariadb through phpMyAdmin from outside docker container.

What will I have to do for that?

You need to

Expose the server port in the Dockerfile
get the IP that the container is listening on (e.g. docker inspect code-rade_cache_1 |jq .[0].NetworkSettings.Networks[].IPAddress ) -- note that you will need JQ installed  on your computer to do this
Set the connection in the mysqlphpadmin


Connect to RDS Private Instance from Docker Container I am running a docker swarm cluster and I was trying to connect to a private RDS instance within the same VPC from a docker container which is running in an EC2 within the same VPC but I get an error like "unknown host". I have enabled the database port number(3306) in RDS security group for my EC2 instances.

However, I am able to connect to the database from the host level but not from the container level.

As per the error, it looks like the resolution is not happening for host xxxxx-xxxxx-xxxx.xxxxxxxxx.us-east-2.rds.amazonaws.com'. I mean the hostname is not getting resolved to the IP address from inside the container.

You may need to check if the DNS resolver is working properly from inside the container or not. For testing, you can try to use the IP instead of the RDS endpoint hostname and see if it works.


Ansible event driven capacity I've been using SaltStack a lot in my previous job, and loved its event driven model. Now, in my new job, we are using Ansible, and definitely not in an event driven model.
I was wondering if there was this capacity hidden (or not hidden :)) anywhere in Ansible ?
Also if you think that it was a good idea to try and implement this event driven model based on Ansible (or if Ansible was not designed to be used like this and it would be an mistake).
Thanks
[EDIT]
I got what Arcege is telling me, my question was if there was an event-driven sub-system in Ansible : The answer is NO.
So my second question is : What event system would you plug into Ansible in order to get a truly automated infrastructure ? (e.g. Kafka)

I can't say that I've used SaltStack, but I have a good deal of experience with Ansible.  The two are different approaches.  Event based use is not hidden, it is just not there by itself.

Ansible is agent-less (no minions), and as such it relies on a management system to be able to control.  This adds flexibility on how to manage events, where they come from, how they are raised and what to do in an automated fashion.  This means that it can be used for provisioning through CI/CD, fixing faults through a monitoring system, combined with a server agent like Telegraf.

The questions I think that should be asked is: what would trigger the tool, where it should run and what should be performed.  The rest is just getting the right tools to link up.  Generally, it will involve a CI/CD system, but there are plenty of those.

Some example uses cases:

detecting a flake in a build on a bare-metal server in a pipeline, triggering to re-provision the server.
a trigger coming from the PR system, to start a build to update test servers.
Telegraf detects a disk issue, which triggers an alert which can then start a build to resize the disk or provision a larger server and transfer services there.

Except for the Telegraf use case, the events are likely coming from sources other than the agent running.  And then with the Telegraf case, the option of provisioning a different server would be outside the scope of SaltStack.

To answer your last question of even to use a event-driven model, it's a matter of necessity and choice depending on the use cases.


Salt Stack environments ignoring saltenv I have a salt master with four environments configured (base, dev, staging, prod)

I am initially trying to set a pillar item to track which envrionment is in use, namely core:env

However when I try to switch environments using the following command I always get the dev environment (it doesn't matter what env I put)

salt 'deploy*' pillar.get core:env saltenv=base

Here are my config files

/etc/salt/master (partial)
file_roots:
  base:
    - /srv/salt/base/states
  dev:
    - /srv/salt/dev/states
    - /srv/salt/base/states
  staging:
    - /srv/salt/staging/states
    - /srv/salt/base/states
  prod:
    - /srv/salt/prod/states
    - /srv/salt/base/states

pillar_roots:
  base:
    - /srv/pillar/base/pillars
  dev:
    - /srv/pillar/dev/pillars
    - /srv/pillar/base/pillars
  staging:
    - /srv/pillar/staging/pillars
    - /srv/pillar/base/pillars
  prod:
    - /srv/pillar/prod/pillars
    - /srv/pillar/base/pillars

and my pillars

root@Salt-Master:/srv/pillar# cat base/pillars/top.sls
base:
  'deploy':
    - core
root@Salt-Master:/srv/pillar# cat base/pillars/core.sls
core:
  env: base
root@Salt-Master:/srv/pillar# cat dev/pillars/top.sls
dev:
  'deploy':
    - core
root@Salt-Master:/srv/pillar# cat dev/pillars/core.sls
core:
  env: dev

(the rest follow the same pattern)

I have other data in the base pillar that is being pulled out, but the core:env item won't change from dev.

State and pillar environments are set independently.

Specifing

environment: prod

in the minion configuration will force the minion to use the prod state, but it will still use the default pillar data. To select a pillar environment you will also need:

pillarenv: prod

This also does not work in old versions of salt and is only supported with pillar.get as of salt version 2016.3, so you may be in need of an update.


Kubernetes expose an app via a DNS name in minicube I have a Minikube installation in which I created a simple hello-world deployment like this:

kubectl create deployment hello-node \
    --image=gcr.io/hello-minikube-zero-install/hello-node

I exposed the deployment via a service in the following way:

kubectl expose deployment hello-node --type=LoadBalancer --port=8080

Now If I call: http://<local cluster ip>:8080 it prints "Hello World!" as expected.

What I want to achieve:

I want to expose different deployments in the same cluster to different sub-domains of the cluster. For instance, deployment hello1 to hello1.my-k8-cluster.com, hello2 to hello2.my-k8-cluster.com.

I want to test this locally because later I will do the same on a real cluster.

Question: How to test DNS configurations of services locally? How to define sub-domains in services?

What I tried so far:
I went through the how-to guides here and the documentation which though didn't bring me a clear picture on how to configure what I want.

The approach I've seen work for this is to use Nginx as the LoadBalancer and defining the Ingress with type NodePort for the different services.  You can see a tutorial that shows the first steps for setting this up here.

Additionally, the official docs DNS for Service and Pods and Customizing DNS Service are good resources.


Set additional required approver for pull requests at subfolder lever in Azure Devops I have a pull request policy on my repo which requires at least one member of the devs group reviews and approves any change before it is merged in to master. I'd like to be able to specify additionally that any files changes under the sql_scripts folder require a member of dbadmin group to review and approve.

Can this be done in Azure Devops using either the UI or the API?

Yes you can do this under branch policies.

From the GUI navigate to Repos-> then in the hamburger menu branches-> then select your master branch and then ... on choose Branch Policies. Like so

Then from the branch policies page scroll down and find the Add Automatic Reviewer button and set a path filter based on the folder.


Saltstack Jinja template pillar merge I have a several pillars and I am trying to merge them in the same jinja variable

{%- import_yaml "server_list_de.yaml" as servers %}
{%- import_yaml "server_list_uk.yaml" as servers_uk %}
{%- servers.update(servers_uk) %}

but I've got following error:

File "/usr/lib/python2.7/dist-packages/salt/utils/context.py", line 211, in __getitem__
return self._dict()[key]
KeyError: '_errors'

The "saltey" way of doing this is typically in a map.jinja file. This one for example...

In that, you would:

{%- load_yaml as serverlist %}
#Note that a default list could be provided here
{%- endload %}

{%- set serverlist2 = salt['grains.filter_by'](serverlist, merge=salt['pillar.get']('pillarname:server_list_de')) %}
{%- set servers = salt['grains.filter_by'](serverlist2, merge=salt['pillar.get']('pillarname:server_list_uk')) %}

Then in your sls file you would then put at the top:

{%- from "formulaname/map.jinja" import servers with context %}

Servers should then be available to you. I've never tried merging twice though, so give it a try. At a minimum this may give you some direction as the syntax for doing a pillar merge like this may get you what you are looking for.


Merging Git Database Commits from One Branch to Another without Cherrypicking We have main two Git branches at our company:
During our Sprint,

Everyone has (local branches)

Release branch: Contains prepared/developed code, ready for testing,  and

Master branch: Production- final ready code

I read in Git, Cherry picking is bad.

https://blogs.msdn.microsoft.com/oldnewthing/20180312-00/?p=98215
http://www.draconianoverlord.com/2013/09/07/no-cherry-picking.html

After items are placed in master, they are ready to be deployed.
Sometimes , we do not move all Release items into Master, many reasons:, delay schedule, conduct more testing, late issues. What is the proper Git Devops strategy to only move certain items into Master? Should we backout commits, so we can do a clean merge?

Databases are different, as we are conducting change scripts, and not overwriting binaries like applications, etc.

Example:

Release branch ------- ------------->   Master

Commit A, B, C, D, E   ------------->   Commit B,D

I read in Git, Cherry picking is bad.

Be careful of such blanket statements. They take on a life of their one, and after a while you have a lot of myths in your team (or workplace) about no cherry-picking, no re-basing, no xyz. These are all just tools; there are times to use them, and times to not use them. Nothing is generally bad.

  Sometimes , we do not move all Release items into Master, many reasons:, delay schedule, conduct more testing, late issues. What is the proper Git Devops strategy to only move certain items into Master?

Adam Dymitruk's Branch per Feature works splendidly for that. In short:

There are these branches:

1 master branch
1 qa branch
n feature-123 branches

The general principles:

Every branch starts at master.
Only feature-123 branches receive regular commits.
qa is created by starting fresh from master, and then merging any feature-123 branches that should be in testing. Whenever qa needs to be changed, it is created fresh again, there are no other commits into it. This happens very frequently.
When qa has been released/deployed, it is simply renamed to master and replaces the previous one. At this point, all remaining feature-123 branches are rebased onto the new master. The git rerere cache is used to avoid having to do repeated conflict resolution.

This scheme works great if you have developers who can stomach the rebases. You frequently find people who think they are bad, or who simply do not allow it - then you're out of look obviously. But if you get it to run in your team, it is very smooth, and ultra flexible. You literally never ever need to "back a commit out" or do cherry-picking to revert a commit, or whatever.

Read Adam's page, it explains this in more detail, but basically that's about it. You'll need to create your own tooling for the individual steps (for example, to pull the feature branches which are supposed to go into qa from your Jira instead of manually doing that), but that's not too hard.

Addendum:

Trivially, you can have more than one qa-type branch at any point in time. For example, you will likely start with one that always has all features which are deemed close to "done" (but which may not have successfully passed test yet); and maybe another one (let's call it rc - release candidate) which contains a subset of that for the very next release.
Adam also describes a ci branch; it is nice, but optional (we don't use it at all, and are not missing it).


How do you stop malicious requests in Azure APIM from repeatedly hitting your rate or quota limits? You have an open facing Azure API Gateway that can be consumed by anyone and you want to protect yourself from the API being spammed by the same IP or the same individual behind multiple IP's.
There are policies one can configure within API Management to:

Limit call rate by subscription
Limit call rate by key
Set usage quota by subscription
Set usage quota by key
Restrict caller IP's

Setting limits by subscription (specific consumers) or key (specific apps) wouldn't work because your API is open to the public, and the last option of restricting caller IP's one-by-one doesn't prevent the issue from happening in the first place.
How do you protect against the same user (IP) repeatedly crushing your rate or quota limits? Additionally, how would you protect against that same user doing this across multiple IP's?
My guess is that this is likely covered in Azure's DDoS Protection Basic but I felt that I couldn't get a concrete answer out of those docs.

You can rate limit or throttle based on the IP: https://docs.microsoft.com/en-us/azure/api-management/api-management-sample-flexible-throttling#ip-address-throttling
Also, Azure DDoS Protection Basic doesn't cover much. It won't know what is normal or not for your application without paying the ~$3,000 per month for the Standard plan.


Minikube for Ubuntu 20.04 WSL stuck on 'Restarting existing docker container for "minikube" ...' after minikube start and 'StartHost failed' error I've come across the following problem where my minikube cluster doesn't start and the command line becomes stagnant in the following way:
$ minikube start
�  minikube v1.17.1 on Ubuntu 20.04
�  Kubernetes 1.20.2 is now available. If you would like to upgrade, specify: --kubernetes-version=v1.20.2
✨  Using the docker driver based on existing profile
�  Starting control plane node minikube in cluster minikube
�  Restarting existing docker container for "minikube" ...

This all started after running minikube stop and then minikube start again after troubleshooting a kube-prometheus-stack helm release's pods not running. I've tried restarting my Docker daemon, rebooting my computer, searching online of course, and do see the minikube docker container created just after running minikube start despite the "StartHost failed" error and being stuck on updating the running docker container. I am not connected to a VPN, though I use one often.
Any help would be appreciated. Thank you!

Not sure if complete reinstall is a valid option for you, but for me this helped:

sudo minikube delete
After delete is finished, minikube start again


Should the master node's hardware be the same as the worker node's hardware for a Kubernetes setup? I'm new Kubernetes and am setting up a raspberry pi cluster at home and I want to put Kubernetes on it.  I have 4 new raspberry pi 4s with 8gb each and one old raspberry pi 2+.  There are several hardware differences between the two models but I think technically I can make the old pi2 the master node to manage the pi4s.  I'm wondering if this is a good idea.  I don't really understand what the consequences of this would be.  For instance if I was to expand my little cluster to 8 or 16 pi4s in the future, would my one pi2 be overloaded in managing the workers?  I'm really trying to understand the consequences of putting lower grade hardware in control of higher grade hardware in the master/worker node relationship in Kubernetes.
There are three main goals that I have for this hardware.  I want to recreate an entire development environment, so some VMs that would host a testing environment, a staging environment, a dev environment, and then a small production environment for hosting some toy website, and then I want to be able to host some services for myself in Kubernetes like a nas storage service, a local github repo, an externally facing plex media server, stuff like that.  I think I can use k8s to containerize all of this, but how will having a pi2 as master limit me?  Thanks in advance.

Should the master node's hardware be the same as the worker node's hardware for a Kubernetes setup?

The control plane (master nodes) and the data plane (worker nodes) are completely different workloads. You can use very different hardware or virtual machines if you want.
You can use some with x86, some other nodes with arm64 or even Windows nodes are supported as worker nodes. But you probably need groups for your nodes, since the application typically only run on a specific type of nodes and it is easier to understand e.g. metrics if the application run its instances on the same type of nodes.


How can I determine which command / step failed within a travis build So I have a CICD travis build that failed... How do I know which line / step is the culprit?

Look at the Travis CI Logs.  Often a failed build step will return "Error", "Failure", "-1", buried in a wall of text.  If you want a better way to collect and  these logs you can check out the following:

Official Github page with details on how log collection works
An official blog on how to structure your production logs.


docker | database connection refused between two containers I have a mysql database running with the following docker-compose.yml:

version: '3.3'
    services:
        db:
            image: mysql:5.7
            restart: always
            environment:
                MYSQL_DATABASE: 'demo'
                # So you don't have to use root, but you can if you like
                MYSQL_USER: 'user'
                # You can use whatever password you like
                MYSQL_PASSWORD: 'password'
                # Password for root access
                MYSQL_ROOT_PASSWORD: 'password'
            ports:
            # <Port exposed> : < MySQL Port running inside container>
                - '3306:3306'
            expose:
                # Opens port 3306 on the container
                - '3306'
            # Where our data will be persisted
            volumes:
                - my-db:/var/lib/mysql
            networks:
                - backend

networks:
    backend:
        driver: bridge
# Names our volume
volumes:
    my-db:

$ docker-compose build

$ docker-compose up

I have a basic golang rest api with the following Dockerfile:

# Start from golang:1.12-alpine base image
FROM golang:1.12-alpine

# Adding git, bash and openssh to the image
RUN apk update && apk upgrade && \
    apk add --no-cache bash git openssh

# Set the Current Working Directory inside the container
WORKDIR /app

# Copy the source from the current directory to the Working Directory inside the container
COPY . .
RUN go get -d github.com/gorilla/mux
RUN go get -d github.com/go-sql-driver/mysql
RUN go get -d github.com/golang-migrate/migrate
RUN go get -d github.com/golang-migrate/migrate/database/mysql
RUN go get -d github.com/golang-migrate/migrate/source/file

# Build the Go app
RUN go build -o main .

# Expose port 8080 to the outside world
EXPOSE 8080

# Run the executable
CMD ["./main"]

and the following function is called:

func CreateDatabase() (*sql.DB, error) {
    serverName := "localhost:3306"
    user := "user"
    password := "password"
    dbName := "demo"

    connectionString := fmt.Sprintf("%s:%s@tcp(%s)/%s?charset=utf8mb4&collation=utf8mb4_unicode_ci&parseTime=true&multiStatements=true", user, password, serverName, dbName)
    db, err := sql.Open("mysql", connectionString)
    if err != nil {
        return nil, err
    }

    if err := migrateDatabase(db); err != nil {
        return db, err
    }

    return db, nil
}

$ docker run -p 80:8080 --network=<appname>-mysql_backend <imageid>

$ Database connection failed: %sdial tcp 127.0.0.1:3306: connect: connection refused

I cannot get the api to establish a database connection with the database container?

Your golang api should be included in the compose file and added to the same network.

It also looks like when you run the golang api that you are adding it to a network other than backend, which is the network that is created in your compose file.


How do I expire users credentials for Kubernetes in Azure? I am looking for information on how to forcefully expire users credentials in Azure Kubernetes Service (AKS). The reason is to better understand Security models and protecting Kubernetes deployments from unauthorized access.

Scenario: You are an Administrator and your notebook is lost/stolen. Your Kubernetes credentials have been cached because you used az aks get-credentials .... The one who now has your notebook has been able to extract your ~/.kube/config file. They now have your Admin access!

Thankfully this has not happened to me. However, imagine the nightmare! Now, how do we prevent unauthorized access with this scenario?

One thing I have not yet found is how/where to forcefully expire credentials after X hours. Thus requiring users to get fresh credentials every day or (X hours). What should I be looking for or where can I find documentation that talks of this?

Google searches did not render results to answer this.

First How to check Check certificate expiration

You can use the kubeadm alpha certs check-expiration command to check when certificates expire on a given cluster.

Simplest remedy : you can choose to upgrade the control plance which will renew the certs and then you are good and secure as before loosing your laptop.

Automatic certificate renewal

  Note : kubeadm renews all the certificates during control plane upgrade.

This feature is designed for addressing the simplest use cases; if you don’t have specific requirements on certificate renewal and perform Kubernetes version upgrades regularly (less than 1 year in between each upgrade), kubeadm will take care of keeping your cluster up to date and reasonably secure.

  Note: It is a best practice to upgrade your cluster frequently in order to stay secure.

Another option is to perform Manual certificate renewal

  Note: The default Kubernetes certificates normally reach their expiration date after one year.

Refer more on kubeadm alpha certs renew command usage.

You probably also need to explore the beta feature for Certificate Rotation


How to cache node_modules folder when using Jenkins Multibranch Pipeline I'm using Jenkins Multibranch Pipeline to handle my CI/CD and I'd like to know if is there a way to cache my node_modules folder?

Every time I push a branch, I execute yarn to install the dependencies. when is master branch, it took 1s to execute this step because it's already cached. Otherwise, when is a new branch, yarn took 2 minutes because is a new environment. Is there a way to cache node_modules globally to be used by all branches/environments?

It is possible to do this (not with any built-in steps; you essentially have to either use a global cache or write your own caching tool), but I do not recommend it.  When I've tried this in the past, I often ran into race conditions.  For example, one build would write to the cache while another build simultaneously reads from the cache, or two builds would write to the cache simultaneously, etc.  I got a lot of builds with corrupted modules as a result.

Caches do not have these kinds of race conditions when limited to a single job (or in this case, a single branch for a multibranch job) because Jenkins can be configured not to run multiple builds of one job simultaneously.  If you take extra steps to ensure that all of the jobs which might access the cache can't run simultaneously, then you could probably get away with a shared cache.  But at that point, you're likely to lose the added speed benefit of sharing the cache anyway, since you've massively reduced Jenkins' capacity to run jobs in parallel.

The other scenario where you could get away with a shared cache is if you write to the cache extremely infrequently - and I really do mean extremely.  If the cache is rarely written to, then there simply aren't many opportunities for these kinds of race conditions.


What are best practices for creating and triggering various pipelines regards monorepos? According to this article a monorepo contains different projects. What are best practices for triggering different pipelines?

The current git repo contains the following projects:

╔═════════════╗
║ projects    ║
╠═════════════╣
║ A           ║
║ B           ║
║ C           ║
║ D           ║
║ E           ║
╚═════════════╝

There are three different pipelines, i.e. x, y and z

If a certain project inside the monorepo changes, different pipelines should be triggered, i.e.:

If a commit is done in folder a then pipeline x should be triggered (a --> x).
If a change occurs in directory b or c then pipeline y should be run (b || c > y) and if either d or dir e changes then pipeline should start running (d || e > z).

How to achieve this? Most CI's are triggered by changes in a repository, e.g. one commits to a repo and a pipeline is triggered. There is a 1 on 1 relation. If this is translated to a monorepo that means that 1 change results in 3 pipeline triggers, while in some cases only 1 pipeline should be triggered.

If this is translated to a monorepo that means that 1 change results
  in 3 pipeline triggers, while in some cases only 1 pipeline should be
  triggered.

I would definitely take this point into further consideration. One of the most compelling reasons for using a monorepo is that it does not create a technical separation of projects. There is only a logical separation, with the only technical distinction possibly being your versioning (unless you have a universal version number). This means, that you have no 100% guarantee that any one project does not depend on another project. Because of this, your final pipeline at some point should include all of the projects within your repository. By doing so, you are guaranteeing that a change to one project does not negatively impact any other projects in your repository (at the cost of the project with the longest time to build, publish, etc.).

That being said, if there are some steps in your pipeline that you would like to run separately (lint, syntax, compile, etc.), most CI's should have a way to filter your repo for a specific folder/project. If it doesn't, and you still don't want to run against all projects, you may need to consider refactoring your repository into separate repos or changing CI tools to comply with your workflow.


Connect Docker container to both host and internal bridge network I am trying to run a Docker container as a router between a private (--internal) Docker network and the predefined host network. This means the container needs to have two network interfaces: One "outside" interface, that can access all host IP addresses, and one "inside" interface, that acts as a gateway for the containers in the internal Docker network

The router container itself will then NAT network traffic from/to containers.

I have not found a way to configure Docker to run the container with those two interfaces. The closest I could get is having two bridge interfaces assigned, which is not exactly what I need.

Trying to connect manually results in an error:

# docker network connect host root_router_1
Error response from daemon: Container cannot be disconnected from host network or connected to host network

Can anybody show me how to achieve this, preferably even whith Docker Compose?

Docker does not allow to connect a container to the host network and any other Docker bridge network at the same time. I will try to illustrate the reason with an example:

Let us think of a container C1. Hypothetically, C1 would be connected to the host network (--net=host) and a Docker bridge network Br1 (--net=Br1).
A second container, let us say C2, is connected to Br1.

With the above setup, my guess is that the host network is visible from C2, and I suppose this is the reason why Docker automatically prevents us from unintentionally exposing the host network to non-host-specified containers.

That being said, if we have a set of containers, and we want all of them to be interconnected, with just a single container having access to the host network, my approach would be:

[C2,...,CN] are connected to a user-defined Docker bridge Br1 (--net=Br1)
C1 is connected to the host network (--net=host)
C1 exposes a port in order to be accessible from the rest of containers

EDIT: We would still have to adapt iptables policies in such a way that C1 can be reached from the rest of containers (see https://docs.docker.com/network/iptables/)


Connect external Jenkins Master to GKE with Kubernetes plugin I find the documentation at https://github.com/jenkinsci/kubernetes-plugin quite unclear when it comes to step by step configuration of external Jenkins master.

What are the exact steps that need to be followed?
I have installed the Kubernetes plugin and I have tried to supply some cluster info, but I have got lost there.

For Name I have taken the output from kubectl config view --raw, specifically the context for the given cluster and the cluster name part.

For Kubernetes URL I have set the IP address from the Server field in the same output from above.

Kubernetes server certificate key - for this part I have initially tried to follow the same pattern from above and take the value of the certificate-authority-data field, but I got an "Error testing connection https://XX.XXX.XXX.XX: java.security.cert.CertificateException: Could not parse certificate: java.io.IOException: Empty input", so I have attempted to convert that value with echo "the field value here" | base64 -d > ca.crt and use that, but it failed again and I have also got it form the UI for GKE show cluster ceritficate section, all of them give me failure on testing connection with the error from above.

I couldn't really go past this point, but I see everything being quite unclear forward, so I would appreciate a step-by-step recommendation or any guide that I am missing.

You could install Jenkins using helm to install the k8s plugin as well.

https://github.com/helm/charts/tree/master/stable/jenkins


Problem with Anaconda installation This is what comes up any time I start up Anaconda Prompt -

\Xtralien was unexpected at this time.
C:\Users\User Name>            @SET "PATH=C:\Program Files (x86)\QuantumWise\VNL-ATK-2016.3\bin;C:\ProgramData\Oracle\Java\javapath;C:\Program Files (x86)\Intel\iCLS Client\;C:\Program Files\Intel\iCLS Client\;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Program Files (x86)\Intel\Intel(R) Management Engine Components\DAL;C:\Program Files\Intel\Intel(R) Management Engine Components\DAL;C:\Program Files (x86)\Intel\Intel(R) Management Engine Components\IPT;C:\Program Files\Intel\Intel(R) Management Engine Components\IPT;C:\Program Files (x86)\NVIDIA Corporation\PhysX\Common;C:\Program Files\Intel\IntelSGXPSW\bin\x64\Release\;C:\Program Files\Intel\IntelSGXPSW\bin\win32\Release\;C:\Program Files (x86)\MiKTeX 2.9\miktex\bin\;C:\Program Files (x86)\LyX 2.1\Perl\bin;C:\Program Files\MATLAB\R2017a\runtime\win64;C:\Program Files\MATLAB\R2017a\bin;C:\Program Files (x86)\Calibre2\;C:\Program Files (x86)\Skype\Phone\;C:\Program Files\Git\cmd;C:\Program Files (x86)\QuantumWise\VNL-ATK-2016.3\bin;C:\ProgramData\Oracle\Java\javapath;C:\Program Files (x86)\Intel\iCLS Client\;C:\Program Files\Intel\iCLS Client\;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Program Files (x86)\Intel\Intel(R) Management Engine Components\DAL;C:\Program Files\Intel\Intel(R) Management Engine Components\DAL;C:\Program Files (x86)\Intel\Intel(R) Management Engine Components\IPT;C:\Program Files\Intel\Intel(R) Management Engine Components\IPT;C:\Program Files (x86)\NVIDIA Corporation\PhysX\Common;C:\Program Files\Intel\IntelSGXPSW\bin\x64\Release\;C:\Program Files\Intel\IntelSGXPSW\bin\win32\Release\;C:\Program Files (x86)\MiKTeX 2.9\miktex\bin\;C:\Program Files (x86)\LyX 2.1\Perl\bin;C:\Program Files\MATLAB\R2017a\runtime\win64;C:\Program Files\MATLAB\R2017a\bin;C:\Program Files (x86)\Calibre2\;C:\Program Files (x86)\Skype\Phone\;"C:\Program Files (x86)\Xtralien Python Distribution\Scripts";"C:\Program Files (x86)\Xtralien Python Distribution";c:\FullProf_Suite"
C:\Users\User Name>

I am guessing it's because of a conflict with an old Xtralien python installation that I had and forgot about. I uninstalled that and reinstalled Anaconda but it hasn't resolved the problem.

@PrestonM indicated that:

  Whenever I see an "unexpected at this time" error it's in my batch
  scripts, and it's either because I didn't use the correct escape
  character, or an absolute path is incorrect. I don't see Anaconda
  anywhere in your PATH, but I do see Xtralien. Is it possible that you
  need to add Anaconda?

OP feedback was:

  Not changing the default directory and adding it my path variable
  during the installation seems to have done the trick. Thanks a bunch
  for the advice!


What is a container? There are some questions about containers, such as:

What role do Configuration Managment tools play in immutable infrastructure?
Which problems does a container orchestration solve?
Simple CI/CD Containers in AWS

My questions:

What is actually a "container" (in the context of DevOps)?
Why are they used?

The very first thing to know about a container is:

It is, first and foremost, a process.

Once that is understood, one can start to understand how containers compare and contrast with virtual machines.  Containers and VMs both share isolation from their hosts.  The method of isolation is the critical difference.

Container processes use extensions to the OS kernel host on which they run to isolate themselves from other processes.  Other extensions also provide disk and resource isolation.  Containers share their kernel and memory with the host OS.

Virtual Machines use a hypervisor to isolate VMs from their hosts.  This is a layer of software that forwards requests for resources from "guests" (VMs) to the hardware.  Disk isolation is provided by disk virtualization.  VMs do not share a kernel with the host- they load their own kernels into memory space dedicated to the VM.

One important impact of this difference is that a container must be kernel-compatible with its host.  For example, it is not possible to run a Windows Nano Server-based container on a Linux host, or an Ubuntu container directly on a Windows host.  Virtual Machines, by contrast, can run any kernel regardless of host OS.  When running a Linux container on a Windows host, Docker runs the container in a Linux VM.

The operational differences are in agility: containers start and stop about as fast as a normal process.  VMs are "heavier," requiring dedicated resources set aside for them and take longer to start and shut down.

Containers offer a lot of flexibility to a DevOps operating model:

Containers isolate software component dependencies.  Developers can use containers to guarantee that an application module will function as well on their development machines as in QA/UA/Production
Containers use software-defined networking to communicate
Container definitions are declarative and can be source-controlled
Container management systems (Kubernetes, DC/OS, Swarm) can manage hardware resource (compute/RAM/storage) pools and dynamically scale containers


jenkins pipeline nested parallel Jenkins pipeline, declarative.

simple question, how do I create a parallel stages inside stage which is parallel by itself?

pipeline {
agent { label 'master' }
options {
    timestamps()
    timeout(time: 15, unit: 'MINUTES')
    ansiColor('xterm')
}

stages {
        stage('Main parallel') {
            parallel {
                stage('Start containers') {
                agent { label 'my-agent' }
                    stages {
                        stage('Start Linux containers') {
                            steps {
                                script {
                                    dir (""){
                                        println "running"
                                    }
                                }
                            }
                        }

                    }
                }
                stage('Build UI') {
                    stages {
                        stage('Cleanup main workspace'){
                            steps {
                                    dir("") {
                                        cleanWs()
                                    }
                            }
                        }
                        stage('Git checkout') {
                            steps {
                                dir("") {
                                        checkout()
                                }
                            }
                        }

                        stages {
                            stage('Gradle build') {
                                parallel{
                                    stage('gradle1') {
                                        steps {
                                            dir("") {
                                                script {
                                                            buildInfo = rtGradle.run switches: gradleParams , tasks: 'clean deploy'

                                                        }
                                                }
                                            }
                                        }
                                    }
                                    stage('gradle2') {
                                        steps {
                                            dir("") {
                                                script {
                                                        buildInfo2 = rtGradle2.run switches: gradleParams , tasks: 'unitTests'

                                                    }
                                                }
                                            }
                                        }
                                    }
                            }
                        }
                }
            }
        }
    }
}

I am sure it is possible, but how?

thanks

Sure, you can do this with Scripted Pipelines.  I've never tried to do it with Declarative so I'm not sure whether it's possible to do with Declarative or, if it is, if it's easy or straightforward.

Here's an example of what this might look like with a Scripted Pipeline.  I haven't tested it so there may be syntax errors or other problems.

def parallelTopLevelSteps = [:]

parallelTopLevelSteps['A and B'] = {
  def parallelNestedSteps = [:]

  parallelNestedSteps['step A'] = { echo('A') }
  parallelNestedSteps['step B'] = { echo('B') }

  parallel(parallelNestedSteps)
}

parallelTopLevelSteps['C and D'] = {
  def parallelNestedSteps = [:]

  parallelNestedSteps['step C'] = { echo('C') }
  parallelNestedSteps['step D'] = { echo('D') }

  parallel(parallelNestedSteps)
}

parallel(parallelTopLevelSteps)


how to update .env file in production while Jenkins deployment how to update the .env file in production while Jenkins deployment?
we have not committed the .env file in git for security reasons.

You can store secret file at Jenkins and use it for deployment.
It can be achieved by using Credentials and Credentials Binding plugins.

Install Credentials Binding and Credentials plugins
Select your user name in the top navigation bar.
Select Credentials in the left pane.
Under Stores from parent, select (global) in the Domains column.
Select Add Credentials in the left pane.

insert into your pipeline script

withCredentials([file(credentialsId: 'env', variable: 'mySecretEnvFile')]){
    sh 'cp $mySecretEnvFile $WORKSPACE'
}


Rename VM in VMware Fusion via command line Is it possible to rename a VM in VMWare Fusion via command line? (ie not clone and then name the new VM and delete the old VM).

For example, Parallels has the prlctl tool, and the process would be something like:

prlctl stop oldname
prlctl set oldname --name newname
prlctl start newname

Well, there is no rename, it is a multi-step process, which involves renaming a bunch of entries inside the vmx file, using vmware-vdiskmanager to rename to vmdk and in general doing a bunch of potentially problematic actions if you don't know what you are doing.

You could probably write a quick Perl script to do all that using this guide.

Another option would be to clone it and delete the original, but it is resource intensive and does way too much work for what it is worth. The code below is untested:

vmrun -T fusion stop "/Volumes/vms/old/old.vmx"
vmrun -T fusion clone "/Volumes/vms/old/old.vmx" "/Volumes/vms/new/new.vmx" full --cloneName=new
vmrun -T fusion deleteVM "/Volumes/vms/old/old.vmx"
vmrun -T fusion start "/Volumes/vms/new/new.vmx"


How to define constants in Ansible? If one writes a constant in Python, then one uses uppercase:

SOME_CONSTANT = "helloworld"

Now I wonder how to define constants in Ansible. My first impression was to use the same convention as in Python as Ansible is written in python, but when one checks the variables information page then this does not seem to be a best practice.

Current approach

https://github.com/030/ansible-firefox/blob/master/vars/main.yml

---
firefox_download: /tmp/firefox-{{ firefox_version }}.tar.bz2
firefox_bin: "{{ firefox_home }}/firefox/firefox"

https://github.com/030/ansible-firefox/blob/master/tasks/main.yml

- name: Download.
  get_url:
    url: https://ftp.mozilla.org/pub/firefox/releases/{{ firefox_version }}/linux-x86_64/en-US/firefox-{{ firefox_version }}.tar.bz2
    dest: /tmp/firefox-{{ firefox_version }}.tar.bz2
checksum: "{{ firefox_checksum }}"

Discussion

I am considering to define https://ftp.mozilla.org/pub/firefox/releases/{{ firefox_version }}/linux-x86_64/en-US/firefox-{{ firefox_version }}.tar.bz2 as a constant in the vars/main.yml, but I did not see any other ansible roles that define constants as:

FIREFOX_DOWNLOAD_URL: https://ftp.mozilla.org/pub/firefox/releases/{{ firefox_version }}/linux-x86_64/en-US/firefox-{{ firefox_version }}.tar.bz2

while this is valid python and the convention to write constants.

One could argue that an Ansible constant could be defined in the default/main.yml, but on the other hand, as a CONSTANT is immutable it should not be possible to overwrite it and should reside in vars/main.yml.

As pointed out in the other comment, there are no constants in python. It's just a convention, but defining a variable in capitals, doesn't mean you can't reassign the value.

There are many ways to pass/define variables in ansible (command line, vars in playbook, include a yaml, vars dir in a role, defaults dir, etc) however, there isn't a way to define a constant.

The closest to setting a constant in the way you ask for it is setting your variables as defaults(that is defaults/main.yml inside the role structure). The defaults of a role are the type of variables that have the lowest precedence and are being overwritten by any other way you can define a var but are a great place to put a variable that is unlikely to change and won't clutter your "important" role vars.

Regarding the way you define them - I'd say it's up to you. Just stick to it. I haven't seen anybody using capitals and I would probably make the variable lower case, however, since ansible provides tremendeous amount of flexibility and freedom to do what you like the way you like it, it really is up to you.


How not to break Jenkins Pipeline when adding new parameters (declarative pipeline) An Issue I have been struggling with is whenever I add a new parameter to a Jenkins Pipeline, the next job will not correctly get this parameter.
This is especially an issue if those parameters are required for certain pipeline procedures like determining agent.
This issue is further frustrating because on a job that already exists, when the default parameter is changed the new job will still use the old one; but when creating a new job it will take none.

In order to solve the parameter issue from the first run of a job, and also to solve the parameter for changes to the parameters for jobs that already ran we need a solution. (At the moment, I have a workaround, which I am posting as an answer, but I would like to know if there is a better solution)

Here is a solution which seems almost like a workaround that I have found which works for declarative pipelines.

First, we need to add some code at the start of our file before we enter into the pipeline{} section. The code we add will run before the pipeline section, and will check if the env/param exists, and if it does will take the value, otherwise we will give it a default and assign that to a variable that we can access during the pipeline. Then we need to change our pipeline code references from the parameter to the variable.

Below is a sample that defines an arbitrary parameter, and also an agent label to be used. As you can see, the variables can be used during stages and also during procedures such as determining the agent.

def MY_VARIABLE = null
if (env.my_parameter) {
  MY_VARIABLE = env.my_parameter
} else {
  MY_VARIABLE = "my default value"
}
echo "MY VARIABLE IS SET: ${MY_VARIABLE}"

def MY_AGENT = null
if (env.my_agent) {
  MY_AGENT = env.my_agent
} else {
  MY_AGENT = "agent_label"
}
echo "MY AGENT IS SET: ${MY_AGENT}"

pipeline {
    parameters {
        string(defaultValue: "my default value", description: 'This is my parameter', name: 'my_parameter') ;
        string(defaultValue: "agent_label", description: 'This is my agent label', name: 'my_agent') ;
    }
    agent {
      label "${MY_AGENT}"
    }
    stages {
        stage('my stage') {
            steps {
                sh 'echo ${MY_VARIABLE}'
            }
        }
    }
}


Which Chef software do we need to install for Windows? I am using Windows 8.1.  Which Chef do I need to install to cover all Chef modes like workstation, node, and chef-server?

I want all three of these in my Windows 8.1.

A node in chef terminology is a machine managed with chef-client.

Chef-client is available for various OS, windows, Linux, Aix, Mac etc.

However the chef-server is only available for linux.


How to ensure equitable distribution of SQS messages in a distributed systems setting? I have multiple servers, each having a script polling an SQS queue [all polling the same queue].

So, is there any way I can ensure an equitable distribution of messages to all those clients [i.e. my worker servers here]. Like for example, if there are 100 messages in the queue, then 20-20-20-20-20 if there are 5 workers, and so on.

Can AWS ELB (Elastic Load Balancer) help me do it?  If yes, then how?  If not, then is there an alternate service in the AWS ecosystem which can help me do it?

Or am I overthinking this? I mean, can this be solved straightforwardly in the polling script?  [Please keep in mind the race conditions involved due to multiple clients polling a single queue]

If there are 100 messages in the queue and 5 consumers, the initial distribution will be no more than 10-10-10-10-10.

A single response can never return more than 10 messages.

This seems like a non-issue.

Race conditions related to multiple consumers should be a non-issue as well.  SQS is designed for multiple simultaneous consumers.

Use long polls and a 20 second max wait timer and be amazed. (No, a 20 second wait does not delay messages by 20 seconds.  It doesn't delay them at all.  You sort of need to see it in action to really understand how it works.)

You're definitely overthinking some things, I suspect.


Why does Docker fail to start on AWS ECS instances. ECS stateful instance fails initialization. ECS stateful instance doesn't join the ECS cluster The ECS instance fails to join the ECS cluster because it relies on docker to launch its ecs agent. If in this case docker fails to start then the whole process halts.

Docker fails to start in AWS ECS instances with the following message in /var/log/docker :

Error starting daemon: error initializing graphdriver: devicemapper: Non
existing device docker-docker--pool

Notice the following.

Instance AMI is a COPY of the AWS ECS optimized AMI with the embedded ECS agent(and docker config for ECS)

The key here is the behavior of AWS AMIs in the context of ECS. By default the AWS ECS Amazon Linux AMI optimized for ECS is in charge of VOLUME management but there's a problem if you copy that AMI and try to launch a new instance -- the SECONDARY volume that ECS uses to deploy docker on can go missing.

Troubleshooting steps:

Check /var/log/cloud-init-output.log and see if you have an error message like:

  ERROR: /dev//dev/xvdcz is not a valid block device.

Check your instance mounted volumes and make sure that the Secondary Volume exists. Usually if you see the error above it means that only /dev/xvda is mounted.

FIX:

Enforce your Launch Configuration, Launch Template or script to always deploy the secondary volume /dev/xvdcz regardless of AMI configuration. Do not rely on the AMI as unfortunately the AMI flow has changed and might not preserve all the necessary configuration.


How do I run ansible on one host at a time and break on a failure I've got an Ansible playbook where I want to update a number of flaky devices in sequence. I can use serial:1, but I want to stop the playbook altogether if I get a failure so I can fix it before proceeding instead of accumulating errors.

I'd also like to restart the playbook at the same host I stopped on.  Currently using Ansible v2.0, but can also switch to a newer version if that sort of a feature is only available in newer versions.

Your playbook will stop when a failure occurs and you're using serial: 1 according to the documentation.

  By default, Ansible will continue executing actions as long as there are hosts in the group that have not yet failed.

That said there seems to be some confusion in the community over the default behavior, and it seems to have changed--or been buggy--somewhere between 1.8 and 2.1.

So, if serial: 1 doesn't suffice, use this additional setting:

max_failure_percentage: 0

  In some situations, such as with the rolling updates described above, it may be desirable to abort the play when a certain threshold of failures have been reached. To achieve this, as of version 1.3 you can set a maximum failure percentage...

==

As for retrying your playbook, you should be seeing a failure message like this:

to retry, use: --limit @/home/user/site.retry

Use that --limit flag and on your next execution of ansible-playbook and it will continue from where it failed.

Retry files will be created unless you've set retry_files_enabled = False in your configuration.

Alternatively, --start-at-task may also work.

Sources:

https://github.com/ansible/ansible/issues/1663

https://github.com/ansible/ansible/issues/16241

http://docs.ansible.com/ansible/playbooks_delegation.html#rolling-update-batch-size

http://docs.ansible.com/ansible/playbooks_delegation.html#maximum-failure-percentage

http://docs.ansible.com/ansible/intro_configuration.html#retry-files-enabled

http://docs.ansible.com/ansible/playbooks_startnstep.html#start-at-task


Is there an alternative to setting up the whole project workspace in a Jenkinsfile I have a ROS project for which I'm looking to use Jenkins to set up a CI system.

The project has a large number of dependencies which take a long time to build (they can't be installed on the build executor machines), so we typically build them once, then let Catkin (ROS's build system, based on CMake) use the cached artifacts on subsequent builds.

The Jenkins setup we currently have does this:

Clone repository at commit
Create an empty Catkin workspace
Clone and setup all our dependencies
Build everything together
Run tests

Due to building all our dependencies, this takes several hours.

What I'd like to do is this:

Clone repository at commit
Move it into an already built Catkin workspace containing all dependencies
Build just our project
Run tests

I was thinking of setting up a Docker container with all the dependencies ready to go, and then just running the build in that. Is that a typically recommended approach?

Using Docker with Jenkins in a situation like this is a great way to

Keep Jenkins worker slave nodes clean (so you don't have to build or install anything on them)
Have all dependencies and environment in an easily manageable artifact (in this case, a docker image is the snapshot of the exact OS, filesystem, env and dependencies)
Vendor / keep copies of dependencies at a certain point in time (Example: You don't want to update at the same rate as the dependency)

If you have some kind of artifact management system (Example: Nexus) you can store your docker images in the artifact management system, and then pull them into Jenkins, and create a running container that has your repository in it. From there you can build project run tests inside the docker image. Just be sure you remember to move/copy all test results and built artifacts out of the docker container if you need them or use them later.

If you need more details or specifics please edit your question and I will edit my answer accordingly.


Recommended Design - Communication among Jenkins + Vault + App Design A: CD tool communicate with Vault to fetch App secrets on its behalf

Jenkins ( CD tool ) computes the Vault path according to the app/project & according to which environment the app will be deployed  (qa, staging, prod,...).
Jenkins uses the computed path as payload to make a REST call to Vault.
Jenkins get key-value pairs as an HTTP response
Jenkins injects these values as ENV variables in the server/container where the app is deployed.
Then, Jenkins deploys the app itself ( or restart it) after having ENV vars ready.
The application is designed to read these values from ENV variables ( e.g.: process.env in case of NodeJs App ).
If the server/container is restarted/rebooted, it is configured to call the same Vault API to bring again the key-value pairs before the app is started.

Design B: App communicates directly with Vault to fetch its secrets

Jenkins ( CD tool ) deploys everything except app's secrets.
Application is environment-aware. It fetches its secrets from Vault thru API call.

Design A Concerns

In-transit Encryption of secrets while injecting them from CD tool to App environment.

Design B Concerns

High Availability of Vault is crucial.If it goes down for any reason (.e.g: Network outage), All applications will not be functional.

What's the best design according to you

My choice would be design B.

It's true that the availability of the Vault would be crucial, but Jenkins' availability would similarly be crucial in design A. So from this perspective the decision would normally be driven by the comparison of the two availability figures, if available. Based on its functionality I suspect that Jenkins is likely to be a more complex tool than Vault (but I don't have any hard data supporting it). If that's correct then Jenkins would normally have a lower availability/reliability score.

A slightly different aspect of the dependency on Jenkins in design A isn't exactly appealing: Jenkins' job normally ends once it successfully deploys a certain version of the app, it isn't really designed to be around to "support" that app version during its expected lifetime (that includes restarting the app). Other tools are better suited to perform app monitoring and management in general. The app's independence from Jenkins that design B brings is, IMHO, an advantage from this perspective.


How to use Jenkins github user credentials to access github api in my script? I have a Jenkins job foo that works perfectly well. It is authenticated with jenkins_foo_user. It is able to clone the repo and checkout the right branch.

In my codebase, I have another script - query_github.py a script used to query Github API to get information such as commits and pull request. To be able to use this API I need an access token.

I want to be able to use the jenkins user access token to authenticate and access the github credentials.

How do I do this?

Note: Using the Github Enterprise version

You want the Credentials Plugin.

The administrator of the Jenkins instance can add the plugin, then add the credential (access token) that the job needs. Then, in the job, use the credential as a variable.

See the documentation for more details.


How to use Jenkins Pipeline and choose triggers for email notifications? Before Pipeline, I used Email-ext plugin to define triggers for sending emails (on first failure, and when back to success).

But in the Pipeline version, I can't control the triggers, I can only "send always", or send based on current build result.

How do I send emails in pipeline based on more general triggers? (first failure, first success, etc)

This seems not to be possible at the moment with pipelines. See this Jira ticket.


Container for every user Please help me here based on your experiences.

We have launched a desktop application where users can download all their online data from several platforms at one place. Our motto is to provide users access to their digital footprint so that they can utilize it in their own different ways.The whole idea of desktop application is to put privacy first, Our servers only have access to the users login information and never had access to the users data, which is also encrypted at rest on users machine with user's mnemonic.

Based on the users feedback, We have realized that most of the users did like our platform but they don't want to bear the hassle of installing a desktop application. For them, Making use of their data (search, organize etc) is more important than military grade privacy.

As a result, we have decided to provide both the options to the user, Desktop application as well as web app.

Now shifting such a huge data on users machine (around 50GB to 1 TB for single user) to common database on cloud would be a cumbersome task, In terms of servers required for replication, sharding , caching etc, their costs and complexities which comes with data at scale.

Our Approach:

We are thinking to maintain docker containers for every user with their storage on Elastic file system on AWS. Data on rest on EFS for a container will always be encrypted with user's mnemonic and will only be decrypted on successful users login.

Every user docker container will have a small SQLite database to make this data searchable and indexed. In future, containers can also have their own Elastic search database with any other NoSql or Sql database.

The identities of the users can be put of AWS Cognito or Any decentralized identity system.

One other approach could be: Instead of running dockers we can keep different encrypted SQLITE DB for different users.

It would be a problem because then we have to keep track of all the SQLite db files present being accessed by the user, because SQLite doesnt support multi threading, we cannot open different instances of same SQLite database,

In case of docker, We will only spin docker when user logs in, that ways we always know that only one instance of SQLite is being accessed.

Our worry: We are still trying to figure out loopholes in this approach. Please help us in identifying the same.

Note: https://aws.amazon.com/blogs/containers/welcome-to-the-aws-containers-blog/

Based upon my understanding you essentially have a Data Partitioning problem, in that you need to maintain a degree of separation between each of your customers but also host the service centrally.

You have suggested that you instantiate a Docker container for each of your customers, I think there are a couple of issues with this:

Containers are designed to be somewhat ephemeral, you can use some of the concepts within Kubernetes to effectively create a singleton per customer, however:

Availability will likely be hurt by virtue of it being a singleton if the pod, node or cluster fails then that pod will need to be rescheduled.
The cost is likely to be quite high, as each of those containers will consume some resources regardless of whether they are being used or not.

Depending on how many customers you have, you get into the realms of too many containers:

Orchestrators such as Kubernetes, Docker Swarm and Mesosphere are generally designed to support a large number of containers, however, the expectation is that's lots of the same workload not lots of disparate workloads. Kubernetes has in the past experienced slowdown when lots of ConfigMaps, Pods and CustomResourceDefinitions are deployed.
Whilst lightweight each container will consume system resources, not just Memory, CPU but also TCP/IP pool exhaustion.

Isolation - containers are a good way to isolate workloads from each other, but it's not infallible - Read Containers are not VMs

If I were to approach your problem I would be inclined to use Kubernetes as a Microservices platform hosting your application across multiple ReplicaSets then implement your partitioning at the datastore level in AWS via DynamoDB and EFS.  You are already encrypting your data so you have cryptographic certainty that isolation can be achieved.


create entry-point script to get all env variables starting with VUE_APP_ and take it's value I would like to Dockerize my existing vuejs app for staging , production and I used multi staged approach to create my app container and I'm using Ubuntu 18.04 and this is my,
Dockerfile
# build stage
FROM node:lts-alpine as build-stage

LABEL maintainer="servers@xpl.com"

WORKDIR /app

COPY package*.json ./

RUN npm install

COPY . ./

RUN npm run build

# production stage
FROM nginx:stable-alpine

LABEL maintainer="servers@xpl.com"

COPY --from=build-stage /app/dist/ /app/dist/

COPY nginx/nginx.conf /etc/nginx/

EXPOSE 80

I don't want to create docker-compose.yml file to assign anything and my question would be like , Create .env file with hardcoded values.
i.e
VUE_APP_API_ENDPOINT_DOMAIN=VUE_APP_API_ENDPOINT_DOMAIN

1 .When running npm run build it'll put VUE_APP_API_ENDPOINT_DOMAIN in the dest files.
2 .We need to know which file. Probably index.html I believe.
3 .We create entry-point.sh script will get all env variables starting with VUE_APP_ and take it's value. And then using sed command we find and replace those hard coded variable placeholders in index.html.
And finally start Nginx.
I was stuck on step 3.

While it's not clear what your trying to do (the script you have so far for the bit you are having trouble with would probably be handy), I think you are essentially just trying to parse your environment and manipulate the variables at the start of your script?

I don't see why add something like this to the first line of your script wouldn't work:

eval $(typeset -p | grep -E 'declare -x VUE_APP_' | sed -e 's#declare -x VUE_APP_#export VUE_APP_#g')

But then I also don't know what shell you're running or anything else about the container, so I'm not sure..


How to pull a docker image from a private docker registry using Helm? When the default values.yaml is inspected it is not clear how to pull a private docker image.

# Default values for sonatype-nexus-apt.
# This is a YAML-formatted file.
# Declare variables to be passed into your templates.

replicaCount: 1

image:
  repository: nginx
  tag: stable
  pullPolicy: IfNotPresent

service:
  type: ClusterIP
  port: 80

ingress:
  enabled: false
  annotations: {}
    # kubernetes.io/ingress.class: nginx
    # kubernetes.io/tls-acme: "true"
  path: /
  hosts:
    - chart-example.local
  tls: []
  #  - secretName: chart-example-tls
  #    hosts:
  #      - chart-example.local

resources: {}
  # We usually recommend not to specify default resources and to leave this as a conscious
  # choice for the user. This also increases chances charts run on environments with little
  # resources, such as Minikube. If you do want to specify resources, uncomment the following
  # lines, adjust them as necessary, and remove the curly braces after 'resources:'.
  # limits:
  #  cpu: 100m
  #  memory: 128Mi
  # requests:
  #  cpu: 100m
  #  memory: 128Mi

nodeSelector: {}

tolerations: []

affinity: {}

Based on this Github documentation it is possible to pull a docker image from a private docker registry:

values.yaml

imageCredentials:
  name: credentials-name
  registry: private-docker-registry
  username: user
  password: pass

templates/imagePullSecret.yaml

{{- define "imagePullSecret" }}
{{- printf "{\"auths\": {\"%s\": {\"auth\": \"%s\"}}}" .Values.imageCredentials.registry (printf "%s:%s" .Values.imageCredentials.username .Values.imageCredentials.password | b64enc) | b64enc }}
{{- end }}

templates/secret.yaml

apiVersion: v1
kind: Secret
metadata:
  name: {{ .Values.imageCredentials.name }}
type: kubernetes.io/dockerconfigjson
data:
  .dockerconfigjson: {{ template "imagePullSecret" . }}

at the end of the templates/deployment.yaml add the following:

imagePullSecrets:
  - name: {{ .Values.imageCredentials.name }}


AWS Security group rule for certbot renewal I want to add a rule to my aws security group to allow for incoming traffic from the acme server (I guess it is acme-v01.api.letsencrypt.org) to issue and automate the renewal of my cert;

What is the best way of doing this, given that sg rules do not support hostnames but only IPs?

You just can not because from Let's Encrypt FAQ:

  What IP addresses does Let’s Encrypt use to validate my web server?

  We don’t publish a list of IP addresses we use to validate, because
  they may change at any time. In the future we may validate from
  multiple IP addresses at once.

You will need a reverse proxy (usually a WAF), routing /.well-know/acme to a machine and answering the challenges (from anywhere) and doing the IP filtering before forwarding to your application all other requests.


How to Connect Docker Containers? In "What is a container?" the docker docs show the following nifty image:

For my scenario I have only App A and App B; let's designate them Alice and Bob.  How do Alice and Bob communicate, generally and specifically?  Using the KISS principle, concretely I'm using MySQL and BaseX from the docker hub.
The notion being for an xquery file residing on the host, which is then sent to the BaseX container for execution.  This specific query is actually SQL, so the MySQL container will need to be queried for the result.
Does this necessitate port forwarding or other networking approaches?  Volumes?  Does this require an additional compose file composed of, in turn, MySQL and BaseX?
No doubt there are many possible approaches.  What's the most pragmatic, from a learning perspective?  Conceptually light, with a gradual learning curve.
One approach seems to utilize bridge networking:

Taking advantage of the fact that the containers can see each other.
This doesn't seem to me related to RESTful web services.

Looking at related questions, this might touch on a reverse proxy.

I'm still fairly new to the Docker game, too, but I've got a few little projects that I do exactly what you are asking about. Through my own experience with trial & error (mostly error...) I would suggest using a docker-compose.yml with your network set to static internal IPs so each container will have predictable, repeatable results and IPs that you can configure your containers to communicate across.
I'm not familiar with BaseX but I have a sample situation using docker-compose version: "3.7" here that could easily run multiple apps and/or be added to or repurposed to fit your hypothetical.
This example docker-compose has the following:

A separate container for populating my app-volume volume with the contents of a public GitHub repo.

An Apache Web Server container with PHP-FPM enabled for a "selective backend" that gets built from a Dockerfile in the ~/dockerbuild directory.

An nginx web server container used as a reverse proxy for all things PHP to be passed to Apache (but serves & caches pretty much everything else).

A MariaDB container in the mix for the app DB that I populate with a mysqldump file after the app gets built.

This is a slimmed down version of my docker-compose.yml, so note that you can add a variety of things like healthcheck: and command: to fit your needs.
You execute with docker-compose up -d --build to both build and start your app in one command (but you'll probably want to create your own ba(.sh) file that you execute afterwards, too for those important finishing touches).
Take notice of static-network. And while I am using named volumes in the example, that's just a preference of mine.
version: "3.7"

services:

  github_repo_clone:
    image: debian:latest
    container_name: github_repo_clone
    networks:
      static-network:
        ipv4_address: 172.20.0.254

    command: bash -c "
        apt-get update &&
        apt-get install git -y &&
        rm -rf examplerepo &&
        git clone https://github.com/exampleuser/examplerepo.git &&
        rm -rf /var/www/vhosts/exampleuser/examplerepo &&
        mkdir -p /var/www/vhosts/exampleuser/examplerepo &&
        cp -rapv examplerepo/* /var/www/vhosts/exampleuser/examplerepo &&
        rm -rf examplerepo &&
        tail -f /dev/null
      "
    volumes:
     - app-volume:/var/www/vhosts

  apache_php-fpm_backend:
    container_name: debian-build-demo0
    build: ~/dockerbuild
    ports:
     - "8080:8080"
    networks:
      static-network:
        ipv4_address: 172.20.0.2
    volumes:
     - app-volume:/var/www/vhosts

  nginx_rp:
    image: nginx:latest
    container_name: nginx-build-demo0
    ports:
     - "80:80"
     - "443:443"
    networks:
      static-network:
        ipv4_address: 172.20.0.3
    volumes:
     - app-volume:/var/www/vhosts

  mariadb:
    restart: always
    image: mariadb:latest
    container_name: mariadb-build-demo0
    ports:
     - "3306:3306"
    networks:
      static-network:
        ipv4_address: 172.20.0.10
    environment:
     - MYSQL_ROOT_PASSWORD=4007p@$$w04d
     - MYSQL_USER=dbusernamegoeshere
     - MYSQL_PASSWORD=@p@$$w04d
     - MYSQL_DATABASE=dbnamegoeshere
    volumes:
     - app-volume:/var/www/vhosts
    healthcheck:
      test: ["CMD", "mysqladmin" ,"ping", "-h", "localhost"]
      timeout: 10s
      retries: 5

volumes:
  app-volume:
#    external: true

networks:
  static-network:
    driver: bridge
    ipam:
      driver: default
      config:
       - subnet: 172.20.0.0/16

My clean up command when I take the app down is kind of ugly, but not terribly difficult to read and goes something like this:
docker-compose down -v ; docker rmi -f dockerbuild_apache_php-fpm_backend:latest mariadb:latest nginx:latest debian:latest ; docker rmi -f $(docker images -q --filter "dangling=true") ; docker rmi -f $(docker images -q --filter label=stage=intermediate) ; docker volume prune -f;
I think each "independent app" could (and probably should) have its own docker-compose and/or Dockerfiles, but you may find synergies with certain Dockerfiles that you can create custom command: instructions for (such as the git_repo_clone example) to be used & shared across multiple "apps".