creating private registry, id and password with -Bbn option Any one knows what does Bbn stands for in the below command

I am aware that,If the bbn option is not provided its not possible to write in the htpasswd file,but what is the meaning of it

docker run --entrypoint htpasswd registry:2 -Bbn testuser password > auth/htpasswd

From docs:

-B  Use bcrypt encryption for passwords. This is currently considered to be very secure.
-b  Use batch mode; i.e., get the password from the command line rather than prompting for it. This option should be used with extreme care, since the password is clearly visible on the command line. For script use see the -i option. Available in 2.4.4 and later.
-n  Display the results on standard output rather than updating a file. This is useful for generating password records acceptable to Apache for inclusion in non-text data stores. This option changes the syntax of the command line, since the passwdfile argument (usually the first one) is omitted. It cannot be combined with the -c option.


How to add shards to a Redis cluster managed by Terraform without loosing all state? I have set up a Redis cluster with Terraform. The setup looks like this:

resource "aws_elasticache_replication_group" "instance" {
  replication_group_id = "test"

  node_type                     = "cache.t2.micro"
  port                          = 6379
  parameter_group_name          = "default.redis3.2.cluster.on"
  subnet_group_name             = "${aws_elasticache_subnet_group.instance.name}"
  security_group_ids            = ["${aws_security_group.instance.id}"]

  cluster_mode {
    replicas_per_node_group = 0
    num_node_groups         = "${var.cluster_size}"
  }

  automatic_failover_enabled = true
  apply_immediately = true
}

When I change var.cluster_size from 2 to 3, I would have expected that Terraform will perform the update in place. However, that is not the case:

-/+ module.groupsign_redis.aws_elasticache_replication_group.instance (new resource required)
  id:                                              "test" => <computed> (forces new resource)
  apply_immediately:                               "true" => "true"
  at_rest_encryption_enabled:                      "false" => "false"
  auto_minor_version_upgrade:                      "true" => "true"
  automatic_failover_enabled:                      "true" => "true"
  cluster_mode.#:                                  "1" => "1"
  cluster_mode.3760271746.num_node_groups:         "" => "3" (forces new resource)
  cluster_mode.3760271746.replicas_per_node_group: "" => "0" (forces new resource)
  cluster_mode.3784625311.num_node_groups:         "2" => "0" (forces new resource)
  cluster_mode.3784625311.replicas_per_node_group: "0" => "0"

This is also confirmed in the documentation:

  num_node_groups - (Required) Specify the number of node groups (shards) for this Redis replication group. Changing this number will force a new resource.

Currently, I see only the option to use Terraform for the initial deployment and then add or remove shards using the AWS UI, which will perform the update in place.

Do you know of a way to execute the in-place update with Terraform?
Or is adding shards without loosing the state of the database simply not possible with Terraform at the moment?

Maybe it is possible to add/removing shards by hand (via the AWS UI) and then import the new state back into Terraform. I tried to run terraform refresh, but it does succeed in fully synchronizing the state. terraform apply still believes the resource has to be changed.

Update:
What does work though is to simply ignore the change. Updating other Terraform resources still works. However, it is a bit confusing as the cluster size that is specified in the Terraform file does not match the actual size that is deployed. You then loose the ability to look at the Terraform setup and see exactly what is currently deployed.

Currently, that is the best option that I am ware of.

You can create aws_elasticache_replication_group resource and use data source aws_elasticache_cluster to retrieve an actual amount of nodes. This way you should not write ignore_changes.


How do I package a jar and a Linux VM to run in Windows? I'm new to DevOps so please forgive my ignorance. I have inherited a Win10 Java app that is currently run by pressing play in NetBeans. It makes API calls to a server running in a Linux VM.

What I'd like to know is how can I bundle this together and create an application that can be installed and run'out of the box' in win 10? Also, if possible, as a web service.

I'm just after pointers on how to start.

Generally people go the opposite direction and try to get more to run on Linux as it is generally cheaper (e.g. license free).  Also, Windows 10 is more of a personal workstation (vs Windows Server).  So, it seems like it may be a short term strategy (unless you're just learning on your own).

Having said that, the most DevOps standard way to do this would be to:

Install Docker on your Windows 10 PC.  Docker (in layman terms) lets you run apps on top of slimmed down operating systems of various kinds in a portable manner.
Dockerize the application(s) running on the Linux server.  You can start off with some slim linux variant like Alpine and just add what you need.
While you're at it, just Dockerize your windows 10 Java app at the same time and start running it from command line rather than NetBeans.
Now, you can run it all on Windows or Mac Or Linux (or kubernetes) with ease.

This assumes the server-side app is not a Java or otherwise portable app.  If it is portable, you can just run it locally and forget all this (although all of this is still a good idea).


Git repo setup for multiple dot net project I have a dot net solution containing a WPF application, MS-SQL database, and 2 dlls.  Each is their own product in the sense that each can be shipped independently (dlls go to nuget to be consumed by other applications to talk to the database).  They are independant of each other but are developed in conjunction with one another (changes made in one typically results in changes in another).  The repos are used by VSTS build and release pipelines.

Should I set up a single git repo that contains all the projects or should i do a repo for each project?  Should I use submodules?  Are there any alternative solutions?

Only put a library in a separate repo when it is used in two different repos. If that is not the case then put the code in the same repo. Otherwise one could run into the anti-pattern nanoservices.


Creating ansible host_vars and group_vars files dynamically Is it possible to populate host_vars and group_vars with results from a database like we do in dynamic inventories?

Actually I'm doing it inside my playbook either with lookups, either by using shell module and registering queries result.

If anyone has a more elegant way of populating variables like making an executable host_var file printing JSON to stdout or so ..

The most elegant way would be to use dynamic inventory, rather than static inventory and dynamic group variables.

However, if for some reason you don't want to do that, one other option is to use custom facts; these are scripts that live in /etc/ansible/facts.d on the client machines.  Like dynamic inventory, they be executable scripts (and therefore, query redis or whatever else), but in this case they define variables just for that particular host.  This isn't quite what you were looking for, since you'd have to query the database on each individual server instead of once for the entire group, but it can have a similar effect.


Can I run Chef/Puppet server from Azure and manage my on-premise infrastructure? I would like to use Azure Chef server (or Puppet, or any other solution) from Azure to manage my infrastructure (around 25 servers and 20 desktops) wich is on-premise behind a corporate firewall.

Is that possible? I tried searching for a similar solution but all I can find are articles how to manage Azure infrastructures.

The prerequisites for a chef server are here and common to any installation.

You only need to allow https to port 443 from your inner infrastructure to the cloud server.

If you have a proxy with SSL interception I'd recommend adding this proxy certificate to each client cacert.pem and set an environment variable SSL_CERT_FILE=<chef_install_path>embedded/ssl/certs/cacert.pem so other call from different ruby tools using openssl will also work.


Do I need an orchestration tool to autoscale/scale on demand my web service? I have a stateless web service running in a Docker container. The web service is single threaded.

We would like to span new instances of the service when the load reaches a given threshold. Is that a job for an orchestration tool?

Yes. And there are tools out there covering this piece of functionality.

Probably the most popular one is kubernetes (see also kubernetes):

  Kubernetes (k8s) is an open-source system for automating
  deployment, scaling, and management of containerized applications.

Depending on the provider for your deployment environment the actual orchestration tool/service may be already included and all you'd provide would be your desired scalability configuration. For example when deploying on the Google App Engine Flexible environment the service scaling settings could look like this:

automatic_scaling:
  min_num_instances: 1
  max_num_instances: 15
  cool_down_period_sec: 180
  cpu_utilization:
    target_utilization: 0.6


Jenkins: store build information for later reference Is there a way to store custom information, for example my staging environment ID. I need this information for cleanup purposes later after the staging environment is no longer needed.
Ideally, I want a database table where I have these columns, buildName, buildCommit, pullRequest.
Then when the pull request is closed/merged I can simply find all buildNames and buildCommits
for that pull request and remove them.

If you want to use a database, you can always shell out to a database:
sh "mysql -u my_username -p my_password -h my_hostname -P 3306 -e \'INSERT INTO my_table (column1, column2, ...) VALUES (value1, value3, ...);\'  my_database"

You can also write out your data to a JSON or YAML or whatever type of file and then store it in Jenkins' built-in artifact storage with archiveArtifacts.


Apache Mesos - cases pro and contra containers? My current impression is, Apache Mesos was introduced before the LXC container technology became easily available (via Docker).

So you could implement "your own Apache Mesos Java framework" for example, or use Marathon to do that.

Now, Marathon seems to support containers as well.

Question - when would you deploy Java nowadays on/with Mesos/Marathon without contauners? And if you use containers, why should you opt for Mesos instead of for example Kubernetes?

For example, here they say in my understanding, k8s ecosystem deployment is easier while with Mesos you would have to opt for the enterprise version of DC/OS.

Your impression is false, linux containers (LXC) exists since 2008, mesos and docker have started later. All three makes use of cgroups available since kernel 2.6.28.

For your overall question, mesosphere have a blog post about it.

But mainly you're looking at it from the wrong point of view, choosing the underlying orchestration system should not be done from 'how to deploy this application?' point of view, but more on 'What would be the best orchestration for what I have to deploy as a whole ?'. Making the choice between swarm/kubernetes/mesos/k8s on mesos/swarm on mesos/whatever is very dependent on your environment, your existing knowledge, your team skills etc. and we can"'t really help you choose without an extended description on what you're willing to manage.


How can I get a list of encrypted disks for instances that are running on the AWS CLI? There is a great question here about how to get a list of encrypted disks. I'm currently using the describe-volumes command.

For the record I'm currently using:

aws ec2 describe-volumes --region us-east-1 --filters Name=encrypted,Values=false Name=attachment.status,Values=attached --query "length(Volumes[])"

What I want is to filter that down further to just the instances that are running. (ie exclude the ones that are stopped).

My question is: How can I get a list of encrypted disks for instances that are running on the AWS CLI?

Here is what I ended up using. Enjoy.

for instance in $instances;
do
  count=`aws ec2 describe-volumes --filters Name=attachment.status,Values=attached Name=attachment.instance-id,Values=$instance --query "Volumes[]"  | jq  -r '. | length';`
  name=`aws ec2 describe-tags --filters Name=resource-id,Values=$instance Name=key,Values=Name --query Tags[].Value | jq -r '.[0]'`
  if [ $count -gt 0 ]; then
    START=0
    END=$count
    for ((i=START; i<END; i++))
    do
       #echo "i: $i"
       encrypted=`aws ec2 describe-volumes --filters Name=attachment.status,Values=attached Name=attachment.instance-id,Values=$instance --query "Volumes[]"  | jq  -r ".[$i].Encrypted";`
       volumeid=`aws ec2 describe-volumes --filters Name=attachment.status,Values=attached Name=attachment.instance-id,Values=$instance --query "Volumes[]"  | jq  -r ".[$i].VolumeId";`
       echo "$instance $name Volumes: $count VolumeId: $volumeid Encrypted: $encrypted "
    done
  fi
done


How to check available volume provisioners in Kubernetes cluster? Is there a command or script that one can use to see or list enabled/available volume provisioners in a Kubernetes cluster?
We have an on-premise cluster and I came to know the hard way that 'local' provisioner has already been enabled in the cluster. So I am trying to see if there is something we can do upfront and set up a few of the provisioners for use by the containers/applications.


Google Container Registry giving 'Access Denied' when trying to push Docker container So I'm trying to push my docker image to gcr.io/docker-url, but it's not working. I've tried the documentation that Google has, but that's still failed (https://cloud.google.com/container-registry/docs/advanced-authentication#standalone_docker_credential_helper). I also tried:

docker login gcr.io

and

docker login us.gcr.io

Here's the command I run and my error:

docker push gcr.io/docker-url/jenkins/container
The push refers to repository [gcr.io/docker-url/jenkins/container]
d7ab52d2e8be: Pushing [==================================================>]   2.56kB
fae569eb21a4: Pushing  1.536kB
7a20d9096c74: Pushing [====>                                              ]  735.7kB/7.854MB
a26d457fd72d: Pushing [>                                                  ]  60.93kB/6.055MB
36f2fac55108: Pushing [>                                                  ]  502.8kB/49.16MB
4cd936335289: Waiting
f1b5933fe4b5: Waiting
denied: Access denied.

For reference, I used the json file to login. I even tried all of the StackOverflow answers I could find, but no luck there, either.

In case anyone else is running into this issue and can't figure it out, I fixed it by running this command:

gcloud auth activate-service-account --key-file key-file.json

And then ran my docker push command and it pushed through.


CORS and subdomains - do subdomains matter? Will CORS ever be a problem when you have x.example.com going to y.example.com or only if the root domain is different?

CORS is not allowing subdomains, so you need to specify them in your server configuration.

If you are using NGINX (or you could use it as a proxy and solve your problem) by providing dynamic cors header response

server {

    root /path/to/your/stuff;

    index index.html index.htm;

     set $cors "";

    if ($http_origin ~* (.*\.yoursweetdomain.com)) {
        set $cors "true";
    }

    server_name yoursweetdomain.com;

    location / {

        if ($cors = "true") {
            add_header 'Access-Control-Allow-Origin' "$http_origin";
            add_header 'Access-Control-Allow-Methods' 'GET, POST, OPTIONS, DELETE, PUT';
            add_header 'Access-Control-Allow-Credentials' 'true';
            add_header 'Access-Control-Allow-Headers' 'User-Agent,Keep-Alive,Content-Type';
        }

        if ($request_method = OPTIONS) {
            return 204;
        }

    }
}

Source here


Azure DevOps Artifact.zip folder structure for .NET Core 3.0 application I have an ASP.NET Core 3.0 application for which I have a full CI/CD setup in Azure DevOps. The Application is hosted on a Ubuntu 18.04 server machine.

In the Build Pipeline, the solution is built and afterward published to the ArtifactStagingDirectory.

- task: VSBuild@1
  inputs:
    solution: '$(solution)'
    msbuildArgs: '/p:DeployOnBuild=true /p:WebPublishMethod=Package /p:PackageAsSingleFile=true /p:SkipInvalidConfigurations=true /p:PackageLocation="$(build.artifactStagingDirectory)\MyProject.$(build.BuildNumber).$(Build.BuildId).zip"'
    platform: '$(buildPlatform)'
    configuration: '$(buildConfiguration)'

- task: PublishBuildArtifacts@1
  inputs:
    PathtoPublish: '$(Build.ArtifactStagingDirectory)'
    ArtifactName: 'drop'
    publishLocation: 'Container'

This zip contains the complete folder structure of the server, which has IMHO nothing to do with the project. For me this is noise.
For example:

  /drop/MyProject.20191028.2.49/Content/d_C/a/1/s/MyProject/obj/Release/netcoreapp3.0/PubTmp/Out

Is it possible to reduce this folder structure to something like this or even less?

  /drop/MyProject.20191028.2.49/Release/netcoreapp3.0/

At the moment, I am working around this in the release stage with a bash script.

drop=$(find /MyProject/Release/drop/MyProject.$(build.BuildNumber).$(Build.BuildId)/ -name "Out")

ln -s $drop /MyProject/Release/Current

In general, I am fine with this, but a little cleaner solution would be nice :)

*The application will be ported to .NET Core 3.1 as soon as it gets released. If this is relevant.

I'm looking this fact:

  I have an ASP.NET Core 3.0 application for which I have a full CI/CD setup in Azure DevOps. The Application is hosted on a Ubuntu 18.04 server machine.

Therefore I can safely assume that you are developing ASP.NET Core 3.0 app to be hosted in Ubuntu. Any .NET Core 3.0 (or later) application means that you should rely on the dotnet build instead of using VSBuild.

Also you had stated that you will host the app on Ubuntu 18.x, then you should also run the build on Azure DevOps agent that runs on Ubuntu. This means you should only use dotnet build in DotNetCoreCLI@2 task, because VSBuild task only runs on Windows based agent, not Ubuntu and it is intended to compile .NET Framework and other platform other than .NET Core.

Please consult the official doc of DotNetCoreCLI@2 task at https://docs.microsoft.com/en-us/azure/devops/pipelines/tasks/build/dotnet-core-cli?view=azure-devops

After you use dotnet build, use dotnet publish to publish your artifact. It is much easier and it's also the best way to publish .NET Core apps.

I have sample usage of dotnet build and dotnet publish of this on my repo:
https://github.com/eriawan/dotnetcore-cicd-samples/blob/master/yaml-samples/sample_pipelines_check.yml


Unable to validate certificate chain: cert.pem in Zimbra mail server I am trying to install let's encrypt SSL in my Zimbra mail server but getting below error when running below command.

/opt/zimbra/bin/zmcertmgr verifycrt comm privkey.pem cert.pem chain.pem

Error:

** Verifying ‘cert.pem’ against ‘privkey.pem’
Certificate ‘cert.pem’ and private key ‘privkey.pem’ match.
** Verifying ‘cert.pem’ against ‘chain.pem’
ERROR: Unable to validate certificate chain: cert.pem: C = US, O = Let’s Encrypt, CN = Let’s Encrypt Authority X3
error 2 at 1 depth lookup: unable to get issuer certificate

This error occurrs because you do not add the root CA in your chain.pem file:

for solutions add below chain in your chain.pem file:

-----BEGIN CERTIFICATE-----
MIIDSjCCAjKgAwIBAgIQRK+wgNajJ7qJMDmGLvhAazANBgkqhkiG9w0BAQUFADA/
MSQwIgYDVQQKExtEaWdpdGFsIFNpZ25hdHVyZSBUcnVzdCBDby4xFzAVBgNVBAMT
DkRTVCBSb290IENBIFgzMB4XDTAwMDkzMDIxMTIxOVoXDTIxMDkzMDE0MDExNVow
PzEkMCIGA1UEChMbRGlnaXRhbCBTaWduYXR1cmUgVHJ1c3QgQ28uMRcwFQYDVQQD
Ew5EU1QgUm9vdCBDQSBYMzCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEB
AN+v6ZdQCINXtMxiZfaQguzH0yxrMMpb7NnDfcdAwRgUi+DoM3ZJKuM/IUmTrE4O
rz5Iy2Xu/NMhD2XSKtkyj4zl93ewEnu1lcCJo6m67XMuegwGMoOifooUMM0RoOEq
OLl5CjH9UL2AZd+3UWODyOKIYepLYYHsUmu5ouJLGiifSKOeDNoJjj4XLh7dIN9b
xiqKqy69cK3FCxolkHRyxXtqqzTWMIn/5WgTe1QLyNau7Fqckh49ZLOMxt+/yUFw
7BZy1SbsOFU5Q9D8/RhcQPGX69Wam40dutolucbY38EVAjqr2m7xPi71XAicPNaD
aeQQmxkqtilX4+U9m5/wAl0CAwEAAaNCMEAwDwYDVR0TAQH/BAUwAwEB/zAOBgNV
HQ8BAf8EBAMCAQYwHQYDVR0OBBYEFMSnsaR7LHH62+FLkHX/xBVghYkQMA0GCSqG
SIb3DQEBBQUAA4IBAQCjGiybFwBcqR7uKGY3Or+Dxz9LwwmglSBd49lZRNI+DT69
ikugdB/OEIKcdBodfpga3csTS7MgROSR6cz8faXbauX+5v3gTt23ADq1cEmv8uXr
AvHRAosZy5Q6XkjEGB5YGV8eAlrwDPGxrancWYaLbumR9YbK+rlmM6pZW87ipxZz
R8srzJmwN0jP41ZL9c8PDHIyh8bwRLtTcm1D9SZImlJnt1ir/md2cXjbDaJWFBM5
JDGFoqgCWjBH4d1QB7wCCZAA62RjYJsWvIjJEubSfZGL+T0yjWW06XyxV3bqxbYo
Ob8VZRzI9neWagqNdwvYkQsEjgfbKbYK7p2CNTUQ
-----END CERTIFICATE-----

OR you can use below script for install Let's Encrypt SSL in your Zimbra server:

#!/bin/bash

# SSL certificate installation in Zimbra
# with SSL certificate provided by Let's Encrypt (letsencrypt.org)
# Author: Subhash (serverkaka.com)

# Check if running as root
if [ "$(id -u)" != "0" ]; then
   echo "This script must be run as root" 1>&2
   exit 1
fi

read -p 'letsencrypt_email [xx@xx.xx]: ' letsencrypt_email
read -p 'mail_server_url [xx.xx.xx]: ' mail_server_url

# Check All variable have a value
if [ -z $mail_server_url ] || [ -z $letsencrypt_email ]
then
      echo run script again please insert all value. do not miss any value
else

# Installation start
# Stop the jetty or nginx service at Zimbra level
su - zimbra -c 'zmproxyctl stop'
su - zimbra -c 'zmmailboxdctl stop'

# Install git and letsencrypt
cd /opt/
apt-get install git
git clone https://github.com/letsencrypt/letsencrypt
cd letsencrypt

# Get SSL certificate
./letsencrypt-auto certonly --standalone --non-interactive --agree-tos --email $letsencrypt_email -d $mail_server_url --hsts
cd /etc/letsencrypt/live/$mail_server_url
cat <<EOF >>chain.pem
-----BEGIN CERTIFICATE-----
MIIDSjCCAjKgAwIBAgIQRK+wgNajJ7qJMDmGLvhAazANBgkqhkiG9w0BAQUFADA/
MSQwIgYDVQQKExtEaWdpdGFsIFNpZ25hdHVyZSBUcnVzdCBDby4xFzAVBgNVBAMT
DkRTVCBSb290IENBIFgzMB4XDTAwMDkzMDIxMTIxOVoXDTIxMDkzMDE0MDExNVow
PzEkMCIGA1UEChMbRGlnaXRhbCBTaWduYXR1cmUgVHJ1c3QgQ28uMRcwFQYDVQQD
Ew5EU1QgUm9vdCBDQSBYMzCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEB
AN+v6ZdQCINXtMxiZfaQguzH0yxrMMpb7NnDfcdAwRgUi+DoM3ZJKuM/IUmTrE4O
rz5Iy2Xu/NMhD2XSKtkyj4zl93ewEnu1lcCJo6m67XMuegwGMoOifooUMM0RoOEq
OLl5CjH9UL2AZd+3UWODyOKIYepLYYHsUmu5ouJLGiifSKOeDNoJjj4XLh7dIN9b
xiqKqy69cK3FCxolkHRyxXtqqzTWMIn/5WgTe1QLyNau7Fqckh49ZLOMxt+/yUFw
7BZy1SbsOFU5Q9D8/RhcQPGX69Wam40dutolucbY38EVAjqr2m7xPi71XAicPNaD
aeQQmxkqtilX4+U9m5/wAl0CAwEAAaNCMEAwDwYDVR0TAQH/BAUwAwEB/zAOBgNV
HQ8BAf8EBAMCAQYwHQYDVR0OBBYEFMSnsaR7LHH62+FLkHX/xBVghYkQMA0GCSqG
SIb3DQEBBQUAA4IBAQCjGiybFwBcqR7uKGY3Or+Dxz9LwwmglSBd49lZRNI+DT69
ikugdB/OEIKcdBodfpga3csTS7MgROSR6cz8faXbauX+5v3gTt23ADq1cEmv8uXr
AvHRAosZy5Q6XkjEGB5YGV8eAlrwDPGxrancWYaLbumR9YbK+rlmM6pZW87ipxZz
R8srzJmwN0jP41ZL9c8PDHIyh8bwRLtTcm1D9SZImlJnt1ir/md2cXjbDaJWFBM5
JDGFoqgCWjBH4d1QB7wCCZAA62RjYJsWvIjJEubSfZGL+T0yjWW06XyxV3bqxbYo
Ob8VZRzI9neWagqNdwvYkQsEjgfbKbYK7p2CNTUQ
-----END CERTIFICATE-----
EOF

# Verify commercial certificate
mkdir /opt/zimbra/ssl/letsencrypt
cp /etc/letsencrypt/live/$mail_server_url/* /opt/zimbra/ssl/letsencrypt/
chown zimbra:zimbra /opt/zimbra/ssl/letsencrypt/*
ls -la /opt/zimbra/ssl/letsencrypt/
su - zimbra -c 'cd /opt/zimbra/ssl/letsencrypt/ && /opt/zimbra/bin/zmcertmgr verifycrt comm privkey.pem cert.pem chain.pem'

# Deploy the new Let's Encrypt SSL certificate
cp -a /opt/zimbra/ssl/zimbra /opt/zimbra/ssl/zimbra.$(date "+%Y%m%d")
cp /opt/zimbra/ssl/letsencrypt/privkey.pem /opt/zimbra/ssl/zimbra/commercial/commercial.key
sudo chown zimbra:zimbra /opt/zimbra/ssl/zimbra/commercial/commercial.key
su - zimbra -c 'cd /opt/zimbra/ssl/letsencrypt/ && /opt/zimbra/bin/zmcertmgr deploycrt comm cert.pem chain.pem'

# Restart Zimbra
su - zimbra -c 'zmcontrol restart'

# setting auto https redirect
cd /opt && touch https-redirect.sh && chown zimbra:zimbra https-redirect.sh && chmod +x https-redirect.sh
cat <<EOF >>/opt/https-redirect.sh
zmprov ms $mail_server_url zimbraReverseProxyMailMode redirect
EOF
su - zimbra -c '/opt/https-redirect.sh'
rm /opt/https-redirect.sh
if

read this link for more details: https://github.com/SubhashPatel/Install-Zimbra-mail-server-ubuntu/blob/master/configure-ssl-zimbra.sh


Control order of container termination in a single pod in Kubernetes I have two containers inside one pod. One is my application container and the second is a CloudSQL proxy container. Basically my application container is dependent on this CloudSQL container.

The problem is that when a pod is terminated, the CloudSQL proxy container is terminated first and only after some seconds my application container is terminated.

So, before my container is terminated, it keeps sending requests to the CloudSQL container, resulting in errors:

could not connect to server: Connection refused Is the server running on host "127.0.0.1" and accepting TCP/IP connections on port 5432

That's why, I thought it would be a good idea to specify the order of termination, so that my application container is terminated first and only then the cloudsql one.

I was unable to find anything that could do this in the documentation. But maybe there is some way.

But maybe there is some way.

Order of termination (or start) as such is not something you can easily define, but you can try a workaround with Container Lifecycle Events.

General idea is that on a PreStop handler for proxy container you can check if app container finished working (say localhost:appport is not active or whatever api check you can muster to determine when app pod is down) and then continue with proxy pod shutdown normally.


How can I add a when clause for a specific upstream trigger in a Jenkins declarative pipeline? I have a job defined as a declarative pipeline that gets triggered by a number of different upstream jobs. Depending on which upstream job triggered my job, I don't necessarily need to run all of the stages in my pipeline, so I want to specify a list of triggers for each stage.
I was hoping to use a when clause and triggeredBy conditions to determine when each stage runs, but the documentation isn't very detailed:

triggeredBy
Execute the stage when the current build has been triggered by the param given. For example:
when { triggeredBy 'SCMTrigger' }
when { triggeredBy 'TimerTrigger' }
when { triggeredBy 'UpstreamCause' }
when { triggeredBy cause: "UserIdCause", detail: "vlinde" }

Is there an 'UpstreamCause' equivalent to the example for 'UserIdCause' that would allow me to specify which upstream cause should trigger a specific stage? If so, what is the exact syntax? Basically I want to do something like:
when {
    anyOf {
        triggeredBy cause: 'UpstreamCause', detail: 'Job A'
        triggeredBy cause: 'UpstreamCause', detail: 'Job C'
    }
}

I'm also open to other approaches for accomplishing this.

Is there an 'UpstreamCause' equivalent to the example for 'UserIdCause' that would allow me to specify which upstream cause should trigger a specific stage?

Unfortunately, not in native Declarative syntax.
As you can see from the source code, UserIdCause is special-cased, but no other cause is:
    static boolean shouldRunBeAllowed(Cause causeClass, String cause, String detail){
        if( causeClass instanceof Cause.UserIdCause && Cause.UserIdCause.simpleName == cause){
            return detail == null || causeClass.userId == detail
        }else {
            return causeClass.class.simpleName.matches("(?i)\\.*${cause}.*")
        }
    }

So if you want to trigger conditionally based on which job triggered the build, you'll need to use Scripted syntax. For instance (I haven't tested this so no guarantees it will work without debugging):
script {
  def myJob = 'put-your-job-name-here'
  def causes = currentBuild.rawBuild.getCauses()
  def triggeredByMyJob = causes.any { cause->
    cause.class.toString().contains("UpstreamCause")) && cause.upstreamProject == myJob
  }

  if (triggeredByMyJob) {
    // put steps to execute when triggered by your job here
  } else {
    // put steps to execute when NOT triggered by your job here
  }
}

However, there's potentially a much easier way, although it's not strictly what you asked for:  Give your downstream job a boolean parameter, for the sake of example let's call it runMyStages, and use a when clause to conditionally execute stages based on whether runMyStages is true or false.  Then in your upstream jobs, you can set that parameter in your build step: build(job: 'downstream-job-name', parameters: [booleanParam('runMyStages', false)]).


How to check Jenkins build executor free memory? In Jenkins when clicking on Build Executor Status I can only see free disk space related statistics (URI: /computer). How I can monitor free system memory (RAM) in Jenkins?

I'm asking, because sometimes when I had too many executors (despite having swap space configured, but not the one below), Jenkins was freezing or crashing a lot.

Have a look at the Monitoring plugin. Here are some more details about it (from the linked page):

  Charts of memory, cpu, system load average, http response times by day, week, month, year or custom period
  Statistics of http requests with mean response times, mean cpu times, mean response size by request and by day, week, month, year or custom period
  Errors and logs
  Current http requests
  Threads
  Heap histogram (instances and sizes by class)
  Http sessions
  Process list of OS
  MBeans
  Actions for GC, heap dump and invalidate session(s)
  Report in html or pdf
  In English, German, French, Portuguese or Chinese
  Jenkins security
  For Jenkins nodes (slaves in general):

  The report for the nodes is available at http://yourhost/monitoring/nodes
  Charts aggregated for all nodes of memory, cpu, system load average, number of running builds, build queue length, build times by period
  Detailed statistics of the build times and of the build steps by period
  Threads, process list and MBeans for each nodes
  Heap histogram aggregated for all nodes

  For each individual node (each node in http://yourhost/computer), reports and actions are available from the "Monitoring" page in the contextual menu or in the detail of the node:

  Threads, process list, MBeans of that node only
  Heap histogram of that node
  Actions for GC, heap dump

  And more...

Credits: Assaf Lavie (who first mentioned it in a comment, but for some reason didn't post that as an actual answer ...).


What is the impact of using Helm Deployments instead of StatefulSets for Databases like MongoDB or MySQL? The definition of a stateless app seems to apply to database servers like MongoDB or MySQL. I found Helm Charts as a kind of template-repository for Kubernetes. However, the stable builds all use deployments. Only in the incubator, which are not provided as stable.

What is the impact of using Deployments for DB Servers?

Are there any issues like sync/replication issues, when running multiple instances in a cluster, for example?

Background: I want to run MongoDB in a failover-cluster, so at least 2 instances are running.

Back when Kubernetes announced the new StatefulSet feature with K8s v1.5 (converting it from the old PetSet name), they put out a really good blog post walking through an example of its usage: https://kubernetes.io/blog/2016/12/statefulset-run-scale-stateful-applications-in-kubernetes/

The first paragraph has a really good description of differentiating features that always stuck out in my mind:

  ...“at most one pod per index” for deployment of the Pods in the set. Along with ordered deployment, ordered termination, unique network names, and persistent stable storage ...

Break down the important ones:

Unique network names

In a deployment, it doesn't matter the network name as any pod should be able to pick up the work. However, when you are running things like a stateful cluster, you might have the concept of 'masters' and 'slaves'. This unique network naming will allow you to keep the distinction by referencing a specific pod, with a predictable incremental name.

Ordered Termination

This is important for re-deployments in a stateful cluster as you could have critical machines that need to stay up longer. If you have coordinator or 'leader' nodes, those should be the last to be restarted. StatefulSets allows you to do this. The linked blog post even shows an example of this.

Persistent stable storage

Arguably, this is the most crucial difference. By default, StatefulSets retain their storage. Kubernetes does not automatically delete the storage that was attached to a StatefulSet. The same can be achieved with deployments, but its not the default.


Does K8s ClusterIP service supports custom timeout? Report from Stack Overflow due to off-topic

Base knowlege

Since this question deals particularly with various timeouts configuration in Nginx, I thought giving their references here would make it easier for everyone to help me. These are Nginx time out directives I use:

proxy_connect_timeout: Defines a timeout for establishing a connection with a proxied server. It should be noted that this timeout cannot usually exceed 75 seconds.
proxy_read_timeout: Defines a timeout for reading a response from the proxied server. The timeout is set only between two successive read operations, not for the transmission of the whole response. If the proxied server does not transmit anything within this time, the connection is closed.
proxy_send_timeout: Sets a timeout for transmitting a request to the proxied server. The timeout is set only between two successive write operations, not for the transmission of the whole request. If the proxied server does not receive anything within this time, the connection is closed.

Scenario

Machine from Internet (A) – opens HTTP connection – > NginxPod (B) – reverses proxy – > a ClusterIP service (C).

    location /abc {
        proxy_pass http://service1.test.svc.cluster.local:3000;
        # This is a dns name given to a `ClusterIP` service by K8s.

        proxy_connect_timeout 2m;
        proxy_read_timeout 60m;
        send_timeout 60m;
    }

K8s will do some magic internally (e.g. load balancing my requests to specific pods which are serving the service1).

Because there's another layer of load balancing imposed by K8s which I'm not in charged of, I wonder if my timeout settings for Nginx will be honored by K8s.

All cookies and headers are a construct of the HTTP protocol. Kubernetes Services operate on TCP/UDP packets therefore it will pass the payload as-is.

It is not that there isn't an nginx.conf file but rather the configuration is managed via annotations. See this link
https://github.com/kubernetes/ingress-nginx/blob/master/docs/user-guide/nginx-configuration/annotations.md


docker container for running automation scripts using jenkins For my test environment, I am trying to setup the following setup.

Create scripts in Frisby

Have a jenkins job, which creates a slave which is a docker container.
The docker container will checkout my latest scripts from my git repository, and run the tests in the container.  Since its api tests, I do not need any product code, my scripts is all that is required which will be connecting to API to test.

My question about the dockerfile. When I create the image for spinning the container. What is the best way to create the image file.

My assumption is as following

use evagra/jenkis slave image and add on it,
installation for Frisby
do a checkout of my scripts
run command for my scripts.

Is this the right approach.

Yes, you are doing the correct approach. Or rather, "a" correct approach, in your specific circumstance.

You are proposing to build the image beforehand, not as part of each individual Jenkins job. Then your Jenkins job will consist of running the image; and in that ephemeral state (i.e., inside the container) you are checking out the test script, and running it.

The more usual alternative would be to leave the actual git checkout to Jenkins, and pull the script into your container with a bind volume. Jenkins has to be able to access your git repository anyways, so you get this checkout for free. There are two sides to the coin here: if you do the git checkout from inside the container, then you can run your test outside of Jenkins with no change whatsoever - say, during development, on your own machine. In any case, if you do the git checkout from inside, and if you need credentials to access the git repository, be sure to handle those securely. Don't put cleartext passwords in the image, or ssh keys with an empty passphrases.

The benefit you are getting from this is that your test environment (libraries, 3rd party test drivers and so forth) is well defined, so you can run it on any Jenkins node you have available. By not creating new images whenever your code changes, you avoid creating and distributing new images for every job.

  I do not need any product code, my scripts is all that is required

So you would never actually deploy those images anyways - hence it is perfectly fine to not build them each time. You still should be automatically building your images - no matter if you do it as part of the actual job, or as a separate job which only does it when the Dockerfile (and whichever files it pulls in with ADD / COPY) changes. Not for technical reasons, but to avoid a manual step.


Pattern(s) to scale node.js based ETL I have a node.js ETL application that transforms data from 1..n locations and from each location there will be 0..m transactions.
The application is started via cron on a configurable interval, currently 15 minutes.
Currently, the pattern employed is to use Promise.map({concurrency:x}) at both the location and transaction level to obtain a combined degree of concurrency to achieve the maximum throughput (constrained by DB I/O.

Right now the app runs on a single Ubuntu VM, processing 6 locations and ~ 1000 transactions/location.  So minimal data processing overhead.  The pattern works.
But when there are 1000+ locations and possibly more transactions per interval, the pattern will still work of course, but the time to complete all locations will grow.
Being node, the process is single threaded and so the pattern is not leveraging the available cores.

So I’m thinking about how to change the pattern to scale.  Options I’m considering (not mutually exclusive)

containerise the app, start one for every n locations - based on some manual determination of # of locations to process per container.
remain non containerised but use fork() to leverage additional CPU’s and partition on n locations

Based on my reading to date, my approach is:

forget “fork"
containerise the app

spin up x apps, x being determined based on manual determination since the parameters are known vs dynamic
each container would be responsible for n locations
each app would set its own “cron” to orchestrate its batch processing

Not being an expert in docker (but reasonably familiar) and certainly a newbie to orchestration, I would appreciate any guidance on how containers / orchestration might help.

I ended up following this path
Rather than use cron, i used a scheduling framework (in my case Bree.js) which allowed me to "jobify" the tasks and also execute these tasks in separate processes - this allowed me the vertical expansion needed (ie: using more processor cores)
A docker container contains the dispatch (scheduling) and jobs - so once the container is started, jobs (and associated processes) are created/managed within that container
Eventually, when I need to scale horizontally, the "dispatch" function will morph to running a job whose sole purpose is to post an event  (eg: Azure service bus).  I will have an orchestrator running various containers who read from the queue and process accordingly.


What are off-the-shelf options for creating a microservice that consults a REST interface and shows the outcome in a graph? I am planning to create a microservice that reads a database and shows a graph. Are there any people who have created such a service? What are things to consider before starting? I am thinking about django and Springboot. On the other hand, perhaps it is from a "do not reinvent the wheel" perspective better to choose something as graphite, but then still has to configure it.

I am basically seeking for a tool that is able to read a REST interface and shows the results in a graph or otherwise a framework preferably written in Kotlin, golang or java.

The aim is to show how fast something is increasing in a graph. At a certain moment I also would like to show the R2, i.e. consistency of the points, slope and so on.

While writing this question, at a certain moment I would also like to predict whether something will be increased in the near future, i.e. applying data science.

A cron job runs every night and inserts data to the Postgres database. A part of the table looks as follows:

ID = 61
PACKAGE_NAME = FIREFOX
DATE = 2018-04-29
VERSION = 59.0.2

ID = 62
PACKAGE_NAME = TOMCAT8
DATE = 2018-04-29
VERSION = 8.0.51

ID = 63
PACKAGE_NAME = TOMCAT85
DATE = 2018-04-29
VERSION = 8.5.30

ID = 64
PACKAGE_NAME = TOMCAT9
DATE = 2018-04-29
VERSION = 9.0.7

ID = 65
PACKAGE_NAME = FIREFOX
DATE = 2018-04-30
VERSION = 59.0.2

ID = 66
PACKAGE_NAME = TOMCAT8
DATE = 2018-04-30
VERSION = 8.0.51

ID = 67
PACKAGE_NAME = TOMCAT85
DATE = 2018-04-30
VERSION = 8.5.30

ID = 68
PACKAGE_NAME = TOMCAT9
DATE = 2018-04-30
VERSION = 9.0.7

This data has to be displayed in a graph like Tiobe:

I am currently looking into http://www.chartjs.org/. It looks like that this is able to create graphs. I will add more to this answer if I have created a graph using this library.


AllowedPattern for emails in Cloudformation template I am struggling with specifying a regular expression for email parameter of Cloudformation :

Parameters:
  MyEmail :
    Type: String
    Description: Email for notifications
    AllowedPattern: /[^\s@]+@[^\s@]+\.[^\s@]+/
    ConstraintDescription: You should enter a valid email

As you see, I tried /[^\s@]+@[^\s@]+\.[^\s@]+/ and it won't work.

What is the regular expression for emails in Cloudformation template ?

aws regex aws-cloudformation

Important
Since AWS CloudFormation templates use the JSON syntax for specifying objects and data, you will need to add an additional backslash to any backslash characters in your regular expression, or JSON will interpret these as escape characters.
For example, if you include a \d in your regular expression to match a digit character, you will need to write it as \\d in your JSON template.

http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/cfn-regexes.html
The very same applies to your \s expressions. Use \\s and it works.
EDIT:
Looking back the question I started to wonder the missing quotation marks on your definition. I found out the expessions are yaml format that is the second way to define things in aws.
If you should follow the yaml syntax, the right way to give string values is like this:
AllowedPattern = '/[^\s@]+@[^\s@]+\.[^\s@]+/'

https://aws.amazon.com/blogs/aws/aws-cloudformation-update-yaml-cross-stack-references-simplified-substitution/
EDIT:

Use quotes if your value includes special characters, (e.g. :, {, }, [, ], ,, &, *, #, ?, |, -, <, >, =, !, %, @, ).

https://stackoverflow.com/questions/19109912/do-i-need-quotes-for-strings-in-yaml


Extended Choice Parameter: How to load params dynamically from any file for jenkins build job Though below code works fine individually:
extendedChoice description: '', multiSelectDelimiter: ',', name: 'Capablities', quoteValue: false, saveJSONParameterToFile: false, type: 'PT_CHECKBOX',value: 'Chrome_macos10_15, Chrome_win10', visibleItemCount: 30

I need Chrome_macos10_15, Chrome_win10 need to be passed from external file.
How can it be achieved?

Use the Active Choice Parameter plugin which allows you to run a Groovy script to dynamically load values for your parameters.


Retrieve files deleted after Docker service rm I had to, so I removed a service with docker service rm but I forgot to get the log files of the service. Now I am not able to find those. Can anyone help me or suggest to me if and how can I get those log files. I was using json-files driver for logging via yml.

There's no special feature to undelete files on the host provided by docker. So if you delete your files using docker, you are left with the same option as when you delete on Linux without using docker:

Learn from your mistake and don't do it again (I'd recommend forwarding logs to an external log aggregator).
Restore from backup.
Do a forensic recovery of the hard drive contents.


replacing values in shell script using sed My shell script looks like this:

FAILED_REQUEST_DIRECTORY=/bla/bla2 \
java -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=1234 \
     -jar /xyz/abc-service-1.0-SNAPSHOT.jar \
     --server.port=1111 \
     --server.address=127.0.0.1 \
     --kafka.bootstrap.servers=10.111.11.11:3197,10.112.12.111:3197,10.123.44.25:3197 \
     --topic.pushcrew.encryptionKey=abc-def-egh \
     --spring.application.name=name-sevice \
     --fb.verification.token=token \
     --management.trace.include.payload=true \
     --topic.pushcrew.hits=activity-logs \
     --max.block.ms=1000 \
     --pager.duty.api.key=apikey \
     --dsn=dsn

If I have to replace values of keys in the command, for ex: replacing value of key --server.address with the value to shell script through a variable, how would I do it?

I understand it would look like following if I had to replace value for key --server.address:

sed -i "s#--server.address=.* #$SERVER_ADDRESS#g;

But how do I maintain the space that has to be there after the replacement and how do I make sure that part int the key only after =(equals to) and before   (space) gets replaced.

Negation to the rescue:

sed -i "s/server.address=[^\ ].*\ /server.address=${SERVER_ADDRESS} /g"

This is telling to search zero or more characters (.*) NOT matching whitespace ([^\ ]) up to the first whitespace.

A good pattern is not to use real values to be replaced (10.111.11.11:3197) but to use a mark like ###SERVER_ADDRESS###. If something goes wrong you can easily post-check with

grep "###" && exit -1

This can avoid continuing with dangerous values (mixing dev/pre/pro values)


Waiting for privilege escalation prompt I'm new with Ansible so hope to find answers here.
I'm trying to run some playbook but getting an error:

fatal: [10.65.30.117]: FAILED! => {
  "msg": "Timeout (12s) waiting for privilege escalation prompt: "
}

Here are the steps which I performed:

I installed Ansible 2.6.17 on RHEL 6.8 using root user.
I update /etc/ansible/host with new group of IPs (with root user)
I created playbook file (with some other user) with the following content:

  name: sample_playbook
  hosts: [ansible_clients]
  remote_user: ronenal
  become: true
  tasks:
     - name: create file
       file:
         path: /tmp/ansible.log
         state: touch
         mode: u=rw,g=r,o=r

When I execute ansible-playbook -sample_playbook.yml I'm getting the above error.

I set ANSIBLE_DEBUG=1 and it looks like it related to sudo command.
The user which I use is not in the sudoers list.

My questions:

Is it required to be a privileged user in order to execute Ansible playbook?
if no, what am I doing wrong?
if yes, is that mean that I need to add this user to the sudoers users list?

You don't need to be a privileged user (in your local machine) in order to execute Ansible playbook.
When using become, Ansible allows you to ‘become’ another user, different from the user that logged into the machine (remote user).
For more details see Understanding Privilege Escalation
In your inventory of hosts /etc/ansible/host you have specified a group of IPs with root user while you are specifying a remote_user: ronenal in your playbook.
So you need to run ansible-playbook with --ask-become-pass or -K.


Why does the Jenkins bitbucket-build-status-notifier-plugin post the status to the shared pipeline library repository? I am attempting to implement the Jenkins bitbucket-build-status-notifier-plugin, found here:
 https://github.com/jenkinsci/bitbucket-build-status-notifier-plugin

We used the shared-pipeline code functionality of Jenkinsfile, along with multibranch pipelines- all our Jenkinsfile code lives in one repo and each application makes only a parameterized call to a function in this repo.
The problem comes when the plugin posts status- it posts to the repo of the shared pipeline code, not the application being built!  It is otherwise working correctly, authenticating and posting status quickly.

The plugin is not parameterized to allow me to pass in a commit ID- meaning that this is being derived from an environment variable, presumably in a novel way that other plugins are not using.  The plugin specifically cites changes to facilitate Jenkinsfile and multi-branch pipelines.  I suspect I may have a race condition with the scm checkout command, perhaps the bitbucket plugin is being initialized before any call to scm checkout is made?

Does anyone have any advice on debugging this?  jenkins.err does not contain any errors, since nothing is throwing.  Is there an environment variable I can change in order to force a context switch from the shared pipeline code repo to the application code repo?

This is a known bug.  See JENKINS-42878 and JENKINS-41996.  This bug has been resolved upstream, which means you should be able to fix the bug by upgrading the plugin to the latest version.


How to move disk image in the command line? As answered in this questions: Does Docker on MacOS support tmpfs? In order to run in memory docker, I need to move docker volumes to ramdisk. In macOS I can do it through docker preferences. However, I would like to write a script that could do it for me everytime I run docker on my machine. Is it possible to specify this setting using cmdline in mac?

Docker is running in a VM on MacOS. One of your options is to put the entire VM on a tmpfs. You could use this shell script to create and mount the ramdisk: https://gist.github.com/koshigoe/822455

Here is a guide for resizing the machine. https://gist.github.com/stefanfoulis/5bd226b25fa0d4baedc4803fc002829e

The trick is to stop docker first, then you can make changes and move the

  ~/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/Docker.qcow2

wherever you like and symlink there from this location.


Understanding how Docker volumes work with compose In the Docker Compose example for WordPress, the Ycode listing [1] says:

volumes:
   - db_data:/var/lib/mysql

What is exact meaning here? To compare, while using the Docker client, you can map volume to a host folder [2]:

-v /src/webapp:/webapp

  This command mounts the host directory, /src/webapp, into the container at /webapp.

[1] https://docs.docker.com/compose/wordpress/#define-the-project

[2] https://docs.docker.com/engine/tutorials/dockervolumes/#locate-a-volume

db_data:/var/lib/mysql simply means that the db_data volume that was previously created will be made available in the container at /var/lib/mysql.

In you example it's created at the very end of the compose file.
From the Docker documentation: "The docker volume db_data persists any updates made by Wordpress to the database."

Compose volumes works the same way as the Docker engine.

In you case db_data is just a named volume instead of a path like your second example.


Service backend/frontend in one or two containers? Imagine in your stack you have RESTful services which also provide some rudimentary frontends, mostly for admin/other tech user use.

Do you include the UI inside the container or are these two containers? Why?

Are they separate projects (from a logical or technical perspective)? Would one ever need to be updated/recreated independently of the other one? Would they need to scale independently of each other? I think if any of those answers is a yes, then they should be separated. If not, you should be fine bundling them together. Bundling them together will reduce discrepancy or compatibility issues between the two containers.


Clarification about Operational Support System I just started my career as a  BA. As the first task, I have to study about operational support systems like ansible tower. I could not find any references to study about this. What are the similar operational support systems to ansible tower? Are there any references to study about them.

The Wikipedia definition of Operational Support System (OSS) makes it sound like a term of art from telecommunication companies.  I haven't seen this come up in systems administration over many years of working in this area.  The top vendors in the OSS space are Ericsson, Nokia, and Huawei.  I never see those guys trying to help me manage cloud deployments, so I don't feel there's much overlap with what Ansible Tower does.
Ansible Tower provides a GUI for working with the anisble configuration management tool.  While this provides similar functionality to what it seems an Operational Support System to be, I doubt Ansible Tower is being used as an OSS by anyone.  It wasn't mentioned in any of the product comparisons I googled for OSS vendors.
Tools similar to Ansible Tower would be in categories like deployment tools or infrastructure visualization.  A few examples of these:

rundeck
DeployHub
Foreman

There's also overlap with CI/CD tools.


How do I use Jenkins to maintain versioning on a Python project? An independent Jenkins server monitors my develop branch in Team Foundation Server's Git repository and builds a new package when it's updated. The problem I'm having is that it uses the same version, specified in setup.py. I could configure pypi-server to allow overwrites, but prefer to maintain build history.
Theoretically, there could be a build step that checks the version and increments it, but I'm hoping that someone has already done this work and published it somewhere.
Has anyone seen something like this?

If you follow Semantic Versioning you could achieve this as a step in the pipeline using Semantic Release.
This and the original tool it's based on are designed for automated releases as part of a continuous integration and delivery pipeline. From the docs:

The key point with using this package is to automate your releases and stop worrying about version numbers. Different approaches to automatic releases and publishing with the help of this package can be found below. Using a CI is the recommended approach.

In your case the workflow would be to do all the testing in pull request, and then when they are merged, a step in the develop branch would run semantic-release publish. Again, referencing the docs, this does the following actions:

Publish will do a sequence of things:

Run semantic-release version.
Push changes to git.
Run build_command and upload the created files to PyPI.
Run semantic-release changelog and post to your vcs provider.
Attach the files created by build_command to GitHub releases.

Part of the benefit of having this included in the pipeline is maintaining a clear and consistent changelog, with the help of a commit message linter like committizen.


Groovy script to show only selected item I used jenkins Active choices plugin and Mavenmetadata plugin to build above UI. When I click a radio button particular version only should be display. How can I write a groovy script to fulfill above requirement?

Below images showing my groovy scripts.

You can use the Jenkins pipeline syntax to get the groovy script.  For example below code will generate radio buttons for active choice parameter. you can generate scripts like this from your jenkins instance at  http://localhost:8080/pipeline-syntax/. More info is here https://jenkins.io/doc/book/pipeline/getting-started/#snippet-generator

def createInputParameters(){
([[$class: 'RebuildSettings', autoRebuild: false, rebuildDisabled: false],
 parameters([[$class: 'ChoiceParameter', choiceType: 'PT_RADIO', description: '',
 filterLength: 1, filterable: false, name: 'Select_snapshot_or_release',
  randomName: 'choice-parameter-1683871426502398',
   script: [$class: 'GroovyScript', fallbackScript: [classpath: [], sandbox: true,
    script: 'return "Error"'],
     script: [classpath: [], sandbox: true,
      script: '''return[
            \'snapshots\',\'release\']''']]]]),
             [$class: 'ThrottleJobProperty', categories: [],
              limitOneJobWithMatchingParams: false, maxConcurrentPerNode: 0,
               maxConcurrentTotal: 0,
               paramsToUseForLimit: '', throttleEnabled: false, throttleOption:
               'project']])
      }


For storing avatars in s3, which approach has the lowest cost? On my site I have the ability for users to upload avatars (which they upload to s3). They can only have one avatar at the moment, so I currently just have the filename set to their user id.

However, they may replace their avatars, which causes issues. Because the s3 is propagating to CloudFront, I need to invalidate the cache for the filename when they upload their new avatar (because it replaces the old avatar file). But with CloudFront invalidations are expensive.

It almost seems like Amazon would prefer I just not delete the old files, and instead just give the new avatar a new name and leave the old one in there, even though it's no longer being used by anyone. That way I can avoid the invalidation.

The other option is not to use CloudFront at all, and just upload to s3, and make the s3 bucket public. But the problem with this is that CloudFront is much more generous with their GET request costs and amounts (2,000,000 per month with CloudFront and only 20,000 per month with S3 buckets).

Basically, I have these 3 options, and I'm not sure which to pick, given the above:

1) Replace the files in s3 and then invalidate the file in CloudFront

Problem: invalidations are expensive

2) Just dump more files in the s3 bucket and use CloudFront

Problem: the old, unused avatars just sit there wasting s3 bucket space

3) Don't use CloudFront, and just replace the file in s3 and then serve it from the bucket

Problem: CloudFront has cheaper GET requests and better user experience with latency

Edit: In this case, I'm asking about AWS with s3 and CloudFront. I'm looking to most effectively automate the process by which a user can upload an avatar to my backend service. Specifically, the overwriting of old avatars has me confused. The general process flow of "PreSigned URL -> Upload to s3 -> Propagate via CloudFront" seems an adequate one, but I'm not sure how to accommodate the overwriting of old, no longer used files in this plan, considering the choice seems to be between an expensive invalidation, or the keeping of stale files.

For the best performance naming many small files in an s3 bucket may not be the best idea unless they user_id is a random string. See how the S3 hashing algorithm works https://aws.amazon.com/blogs/aws/amazon-s3-performance-tips-tricks-seattle-hiring-event/

So replacing it by a random hashed value may not be that bad. Of course you need to store the hash in the user profile or you need to calculate it from something. If you add something like profile id or last changed timestamp you can do sha1/md5 of (userid+lastchanged) and if you update timestamp you has the hash of the current avatar.

The older one you can simply delete. CloudFront will remove it from its cache after expiration https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/Expiration.html

This will more complex algorithm, but if the costs are the concern, a little bit of extra work seems to be a reasonable approach. Not sure how do you handle the uploads, but it may be a good opportunity to play a bit with step functions which would make sure that the new avatar is uploaded correctly and then the old one is removed as well.

It's kind of a programming question, but I get your point - it is very specific to AWS environment and requires a particular knowledge about how the infrastructure works.


How do you change a secret that half of your production applications use right now? The context of the question is Secrets management.

As the title says, how do you change a secret that half of your production applications use right now?

This question ties in heavily with your other question with where the secrets are set and where they retrieved from.

I assume your question is asking about 'canary deployments' where you change the config for only a small portion of your apps, to test things out before deploying everywhere.

If you follow the best practice of using the 12 Factor app's recommendation of 'environment variables', then changing secrets becomes very easy. At the very worst, you could log into the production server, update the ENV VAR, and then restart your app manually. In Kubernetes, this is even easier, as you can edit the ENV variables from the UI, and then restart the pod.

However, you don't want to be doing that manual process if you can avoid it. If your Configuration Management tool sets the ENV VAR, then you can tell it to update a specific group of servers only. If your build server injects the ENV vars, and it supports canary deployments, then you can easily just do a canary deployment like normal.


Jenkins Cancel (jobs) from Queue by Label I'm freshman in Jenkins. I would like to cancel jobs by it's own label (hundreds) from my Queue of tasks given to slaves: jenkins.model.Jenkins.getInstance().getQueue()
so i then could do: items[i].doCancelQueue()

Currently I only know how to do it, by its name: items[i].task.getFullName().

I'm looking for something like this: (when I iterate over Slaves that's just give me list of available labels for a single slave)

for (aSlave in hudson.model.Hudson.instance.slaves) {
    println('Name: ' + aSlave.name);
    println('getLabelString: ' + aSlave.getLabelString());
}

My whole code for deleting jobs by name

import hudson.model.*

def q = jenkins.model.Jenkins.getInstance().getQueue()
def items = q.getItems()

if (job_name_fragment == '' && revision == ''){
    println("At least one param is requied")
    return 1
}

def count = 0
for (i=0;i<items.length;i++){
    if (items[i].task.getName().contains("trunk") || items[i].task.getName().contains("dev")|| items[i].task.getName().contains("Windows")){
        println("Task: " + items[i].task.getName() + " has been skipped")
        continue
    }
    if(job_name_fragment == '' || items[i].task.getFullName().contains(job_name_fragment)){
        if(revision == '' || items[i].params.contains(revision)){
            println("Task: " + items[i].task.getFullName() + " has been deleted from queue")
            items[i].doCancelQueue()
            count++;
        }
    }
}
println("")
println("Removed total: " + count + " tasks from queue")

Thanks

Found it by myself, if anyone will be interested check:

getAssignedLabel()

in.
https://javadoc.jenkins-ci.org/hudson/model/Queue.Item.html


Tool to collect developer environment parameters I would like to start collecting parameters from local developer environments on my team, such as to see how they configure their properties files (not in source control), what directory they use as their application root, where they have this and that software installed etc.

So I was envisioning having a script that each would run daily and which would post that data to some web service within some application that does this kind of stuff within the DevOps realm -- team management etc. Is there such an app for local developer management, just like Jenkins does builds or Chef does deployments?

Ideally, I would go into this console and get an idea of how each developer on my team has their environment configured without that developer letting me see their laptop, provided that, of course, my data collection script includes all the relevant parameters.

Instead of trying to watch, how about working together to define a scripted way of defining environments that is scm managed.  Having the configuration for deployment in the SCM insures they are always consistent and allows the collaboration and communication between team members about the ways they want to work.

There are many tools that can be used to script the configuration of the system level as well as the application level definitions.  Using this kind of definition based environment configuration not only helps with the development level, but then also helps with the deployment to later stages capabilities.

This would also facilitate the communication between team members a fundamental principle of DevOps.


How to determine actual cost of specific spot instance Is there a way to know how much a specific spot instance is actually costing? My only idea is to give it a specific tag and wait to see it show up in the billing report. But what I really want is to to know in "real time" how much each spot instance is costing.

From Amazon EC2 Spot Instances Pricing:

  The following table displays the Spot price for each region and
  instance type (updated every 5 minutes).

So one (crude) option would be to parse that page to get the price matching your particular instance type and region.

I don't know if the equivalent info is available via the AWS Price List API:

the Bulk API clearly doesn't support it:

  Offer files do not include information about expiring free tier offers
  or Amazon EC2 Spot Instances.

the spot instace pricing isn't explicitly declared as unsupported in the Query API, so there might be a chance for another option here, that'd be better than parsing pages. But I didn't try - I'm not an AWS user yet.


create resource name based on terraform.workspace I want to have different s3 names based on different environments using terraform.

If I define the name in terraform.tfvar, it won't work:

s3_bucket_name          = "tf-${terraform.workspace}"

If I hardcode s3 bucket name in s3.tf, it works:

resource "aws_s3_bucket" "s3" {
   bucket        = "tf-${terraform.workspace}"
   ....
}

how to define the workspace related name from terraform.tfvar? I do not want to hard code it in tf file.

Variables are static, literal inputs to your configuration that come from either the command line or from tfvars files. It is not possible to use interpolations here, since the variable values are processed before interpolation begins in order to make them available to other expressions.

If the goal is to define the S3 bucket name in one place and re-use it several times in the same configuration, the Local Values feature can do this if you are using Terraform v0.10.4 or newer. You can declare the named value in one of your .tf files:

locals {
  s3_bucket_name = "tf-${terraform.workspace}"
}

You can then interpolate this value in various places in your configuration using the name local.s3_bucket_name, like this:

resource "aws_s3_bucket" "s3" {
  bucket = "${local.s3_bucket_name}"
  ...
}

Local values are conceptually similar to module outputs but they are exposed within the same module they are defined in, rather than passing a value up to the parent module.

If the goal is to allow this generated default name to be overriden optionally by a variable, then that can be achieved with some conditional logic:

variable "override_s3_bucket_name" {
  default = ""
}

locals {
  s3_bucket_name = "${var.override_s3_bucket_name != "" ? var.override_s3_bucket_name : "tf-${terraform.workspace}"}"
}

resource "aws_s3_bucket" "s3" {
  bucket = "${local.s3_bucket_name}"
  ...
}

The above will generate a default name based on the selected workspace if override_s3_bucket_name is not set, but if that variable is set then its value will be used instead.


Why has Hudson been renamed to Jenkins? There are two similar CI: Hudson and Jenkins. The Jenkins project was forked from Hudson after a dispute with Oracle. Why was it renamed back to Jenkins and what was the dispute about?

The Jenkins website has a blog post on this issue:

But one issue, which we feel is the most significant issue of all, one for which we now believe no resolution is possible: the rights to the name Hudson.
Oracle has told us that they have trademark applications filed in both the EU and US for Hudson, based on Hudson's creation by Kohsuke while working at Sun. The problem is that this trademark ownership gives Oracle the ability to revoke the Hudson project's right to call itself Hudson at any time, and while Oracle has made an attempt to offer some guarantees (most notably, that binary releases of Hudson, once they've been released with the name Hudson attached, will always retain the right to the name), they are not offering any binding guarantee that the Hudson project will be able to retain its use of the name in perpetuity.
Therefore, to continue using the name Hudson means ceding some of the project's independence to Oracle - if the project and its governance board opted to go in a direction Oracle disapproved of, Oracle would be able to take away the naming rights. Or, in a less dramatic scenario, Oracle could insist on certain changes to the code, infrastructure decisions, process, etc, regardless of opposition from the Hudson development community, in order to retain the rights to the name.

The Jenkins team were concerned that using the name 'Hudson' would give Oracle far too much control over the project, because there was a constant threat that if the project didn't do what Oracle wanted, they'd simply yank the rights to the trademark.
In order to retain independence, the developers of Hudson voted to rename the project Jenkins. However, here's where it gets confusing. Oracle chose to continue developing Hudson, essentially making Jenkins a fork because the code started to diverge. There is some dispute about whether this is really a fork or a rename, although it's more an argument over semantics than constructive discussion.
As of now:

Jenkins has become the predominant version, and interest in Hudson has collapsed

Hudson is now managed by the Eclipse Foundation, after Oracle gave up development and donated the code to Eclipse.


How to deploy Envoy EDS/SDS This is a micro services deployment question. How would you deploy Envoy SDS(service discovery service) so other envoy proxies can find the SDS server hosts, in order to discover other services to build the service mesh. Should I put it behind a load balancer with a DNS name( single point of failure) or just run the SDS locally in every machine so other micro services can access it? Or is there a better way of deployment that SDS cluster can be dynamically added into envoy config without a single point of failure?

Putting it behind a DNS name with a load balancer across multiple SDS servers is a good setup for reasonable availability. If SDS is down, Envoy will simple not get updated, so it's generally not the most critical failure -- new hosts and services simply won't get added to the cluster/endpoint model in Envoy.

If you want higher availability, you set up multiple clusters. If you add multiple entries to your bootstrap config, Envoy will fail over between them. You can either specify multiple DNS names or multiple IPs.


How to deploy my Express/React app to server with docker-compose and Dockerfiles Hi, I'm trying to deploy my app to production with Docker, but I struggle a lot. Here is the structure from the root folder :

a server folder : runs with Express on port 4000
a web folder : a React app running on 3000 (locally), that needs to run on 80 on my server
a docker-compose.prod.yml file in the root folder

What I'm currently trying :

the docker-compose.prod.yml is running a Postgres service (port 5432), as well as web and server's respective Dockerfile
a Dockerfile in both web and server folders
I yarn build both server and web folders locally and send everything on my server
on my server, I run docker build -f web/Dockerfile.prod -t app_web ./web and docker build -f server/Dockerfile.prod -t app_server ./server to build both images
finally, I run docker-compose -f docker-compose.prod.yml up

My problem :
The server container throws Error: Cannot find module '...' although I run yarn install --production in its Dockerfile (it may be related to volumes?)
Also, when I run docker run -it -p 80:80 app_web and docker-compose up, the web container acts differently when visiting my server IP address : in the first case, the app displays well, in the second case, a page is listing my web folder content (all files and folders)
I'd really appreciate help as I'm a bit new to Docker in production (should I use docker machine? are my folders structure / docker-compose / Dockerfiles wrong?)
docker-compose.prod.yml :
version: "3"
services:
  db:
    image: postgres:latest
    restart: on-failure
    environment:
      POSTGRES_DB: ${DB_NAME}
      POSTGRES_USER: ${DB_NAME}
      POSTGRES_PASSWORD: ${DB_PASSWORD}
    ports:
      - 5432:5432
    volumes:
      - ./.db/data:/var/lib/postgresql/data

  server:
    container_name: app_server
    build:
      context: ./server
      dockerfile: Dockerfile.prod
    volumes:
      - ./server:/app/server
    ports:
      - 4000:4000
    depends_on:
      - db

  web:
    container_name: app_web
    build:
      context: ./web
      dockerfile: Dockerfile.prod
    volumes:
      - ./web:/app/web
    ports:
      - 80:80
    depends_on:
      - db

server/Dockerfile.prod :
FROM mhart/alpine-node:11 AS builder
WORKDIR /app/server
COPY package.json ./
RUN yarn install --production
COPY . .

FROM mhart/alpine-node
WORKDIR /app/server
COPY --from=builder /app/server .
CMD ["node", "/app/server/dist"]

web/Dockerfile.prod :
FROM mhart/alpine-node:11 AS builder
WORKDIR /app/web
COPY package.json .
RUN yarn install --production
COPY . .

FROM mhart/alpine-node
RUN yarn global add serve
WORKDIR /app/web
COPY --from=builder /app/web/build .
CMD ["serve", "-p", "80", "-s", "."]

I found the solution, eventually it was just a volume issue, I had to add a specific line for modules :

volumes:
      - ./server:/app/server
      - /app/server/node_modules

After that I no longer got the "Can't find module" error
Still not sure what the impact of this is though


How can DevOps help to improve Software Escrow procedures? Consider a software vendor and a licensed customer of some software of this vendor, wheres the software being licensed is either used on premise (at the customer's location), or in the format of SaaS solution (hosted by the vendor). However the customer only gets access to what's needed to use/run the software (executables and similar things), so not the source components and anything related to that to create the executables.

To protect the customer's business continuity in scenarios where something may go wrong with the vendor (eg: bankruptcy), both parties may agree to some sort of Software Escrow (SE ... 'also') agreement (also called Source code escrow). With such agreement both parties agree to get a 3rd party involved (= a "Software Escrow agent"), trusted by both parties. These are the highlights of such SE-agreement (= specs of the actual SE-process):

ALL pieces of software components (artifacts related to the licensed software) get deposited by the software vendor, at some agreed location related to the SE-agent. Such deposits include the executables, but also the source components and anything related to that to create the executables (even documentation, instructions, etc to create the executables).
Since the software vendor may create multiple releases for the duration of the software license, and the customer has the right to receive such new releases (as per the license agreement), part of such SE-agreement is that "with every major new release" (whatever "major" may mean ...), the deposit delivered to the SE-agent will also be updated/refreshed.
If specific conditions are satisfied (e.g: bankruptcy of the vendor), then the Software Escrow agent will deliver to the licensed customer, upon request of such customer, of copy of everything that was deposited, so that customer will be able to continue using the software, and where needed even adapt the source code to continue to use it for the customer's business.

A common practice for such SE-agent to get involved, is some sort of a legal person/entity, such as a lawyer. But to actually "process the SE-deposits" (by the SE-agents), all sorts of release management and/or software delivery tasks need to be performed by somebody or something (the poor SE-agent) who probably doesn't know at all what the licensed software is supposed to do ... fun guaranteed!

My question:

How can DevOps help to improve Software Escrow procedures as described above? Like what kind of toolchain-tools would you recommend to be used for the fulfillment of which part of the SE-agreement? And where appropriate using which (preferably open source) software solutions for it?

Notes:

To not further complicate things, just assume that it is agreed between all parties involved, that the SE-agent does NOT have to do any type of "verifications" about the deposits being done. That is: whatever is deposited is assumed to be complete, up-to-date, documented, etc.
About "major new release": assume there are between 1 to 3 every year, which means that the licensed customer only expects to be able to get access (via the SE-agent) to those releases. Even though if there have been intermediate deliveries (like fixes or beta versions) to the licensed customer, those types of deliveries are considered out-of-scope. Even if it was only because:

the SE-agent charges "for each deposit to be processed by the SE-agent".
the licensed customer rarely changes releases, and is only interested in being able to use the SE-agreement if things go wrong, for the very release they are running at the moment things go wrong.

A very interesting question. On the assumption that the goal of a Software Escrow process is to allow for a 3rd Party to take over or nominate an additional party to fulfil the responsibility of the software vendor, I would suggest the following elements of a DevOps operating model that would support software escrow:

Infrastructure-as-code - effectively documenting through an executable specification of dependant infrastructure, stored and versioned in source control provides the environments that the source code is developed in.  Unlike static documentation in text files because it is executed on a regular basis by the software vendor to build their own environments it does not go out of date of suffer from "bit rot".  This will always hold the most value when the entire development pipeline is built and maintained in source control applying Infrastructure-as-code principals.
Continuous Integration - the goal of continuous integration is to execute a set of steps that integrate the solution on a regular basis, ideally upon each change.  Typically this means upon check-in and push to a central repository a set of tests are executed that validates the process.  From a software escrow perspective I would expect this to also push a working version to a secondary "backup" repository owned and operated by the 3rd parties.  It is important to note that this does need to be both legally and financially "unlinked" from the software vendor.
Continuous Deployment - the goal of continuous deployment is to deliver software in a working state on a frequent basis.  In this sense the 3rd party is just another deployment target to deploy the outputs to, albeit probably not actively spinning up infrastructure on the 3rd parties fabric.

Some other considerations to bring into the equation are:

the move from static documentation to infrastructure as code significantly reduces the toil involved in updating documentation that describes how to install, configure and recover software.
don't forget secrets management, such as X.509 Certificates, symmetric keys, passwords and licence keys, these could be stored in source control however this does have it's own drawbacks.

From a tools perspective this will depend very much upon the development environment, I don't believe there are any tools specific to software escrow however:

.NET Development

GitHub or BitBucket for source control
AppVeyor, TeamCity or Jenkins for the build and test elements of the pipeline.
Ansible, Puppet or Chef for the configuration management elements of the pipeline.
ProGet for the package management elements of the pipeline
Octopus Deploy for the deployment aspects of the pipeline

Java Development, and to some extend Open Source platforms

CircleCI, TravisCI, TeamCity or Jenkins for the build and test elements of the pipeline.
Ansible, Puppet or Chef for the configuration management elements of the pipeline.
JFrog Artifactory or Sonartype Nexus for the package management elements of the pipeline
Capistrano for the deployment aspects of the pipeline

At the end of the day if you are able to employ Infrastructure-as-code, Continuous Integration and Continuous Deployment principles to a software package it can be used to fulfil your obligations under a Software Escrow contract.


AWS CodePipeline frozen on Source I have created a very simple pipeline. I have a git repo in CodeCommit, and the pipeline, when triggered, simply copies the files to an existing S3 folder which is set up to serve a static website. Once I finished creating the pipeline, it started an execution. That execution is sitting on the first step, Source. Its been 50 minutes now and it's still stuck on the Source phase.

Any thoughts as to why this might be happening? I don't see a way to cancel the execution. Are there logs somewhere I can look at to see if there were any errors?

Thanks!

There should be a log group available in CloudWatch that you can refer to for information on the error. Also, there is an Execution History tab available on the left hand side that will let you see any errors that are not logged to CloudWatch.

From the sounds of it there may be a problem with the source repo you have selected. It should only take a few seconds to read the repo. Very generally with AWS, many of the problems can be traced to IAM permissions.

You can cancel the execution by selecting 'edit' while it is running.


How to specify a filter for Kubectl -o flag=custom-column I want to use kubectl get -o=custom-columns to extract information with specific columns. The cropped yaml response is like this

{
    "apiVersion": "v1",
    "items": [
        {
            "apiVersion": "v1",
            "kind": "Pod",
            "metadata": {
                "creationTimestamp": "2018-11-27T14:37:46Z",
                "generateName": "kubernetes-bootcamp-5c69669756-",
                "labels": {
                    "pod-template-hash": "1725225312",
                    "run": "kubernetes-bootcamp"
                },
                "name": "kubernetes-bootcamp-5c69669756-cxhwf",
                "namespace": "default",
                "ownerReferences": [
                    {
                        "apiVersion": "extensions/v1beta1",
                        "blockOwnerDeletion": true,
                        "controller": true,
                        "kind": "ReplicaSet",
                        "name": "kubernetes-bootcamp-5c69669756",
                        "uid": "950aac8e-f24f-11e8-bf5d-065b6ba59aa8"
                    }
                ],
                "resourceVersion": "118035",
                "selfLink": "/api/v1/namespaces/default/pods/kubernetes-bootcamp-5c69669756-cxhwf",
                "uid": "0190c234-f252-11e8-bf5d-065b6ba59aa8"
            },

I want to extract the name from items[0].metadata.name, but I don't know the syntax.

My questions:

What is the query syntax used in this situation. Is it JSON path? I don't see it documented.
What is the specific query in my case?

Since the result is JSON, pipe the result to jq and go from there.
kubectl get | jq '.items[0].metadata.name'

yields
➜ cat abc.json | jq '.items[0].metadata.name'
"kubernetes-bootcamp-5c69669756-cxhwf"


Handle OS and Software maintenance/updates on Hardware distributed to Customers In the last weeks I have tried to find a solution to the following scenario assuming that someone else had similar requirements and documented his solution but all my searches were unsuccessful.
So now I'll just ask away and hope that someone with enough experience in this field can give me directions so that I don't take the wrong turn too many times.

The simplified scenario:

Let's say I have written a piece of software with a web interface and REST API for my customers which absolutely has to run in the local network at the customer's premises and will be provided to them as a "black box" hardware appliance. Apart from that I will essentially provide it "as a service" to them, meaning that maintenance and updates are my responsibility. I come to them with a small headless pre-installed and pre-configured PC, connect it to their network, perform only networking configuration and I'm done. The appliance is ready to communicate with any other software components running in the customer's network.

To be a bit more precise my software product runs under Linux (and also Windows), it should also be possible to build it as a Docker container (if that makes any sense for the proposed solution, although my know-how with Docker is still limited). I expect to set up CentOS 7 on the appliances.

Now I have two challenges:

Keeping CentOS packages up-to-date
Keeping my software package up-to-date

To prevent all customers from experiencing the same bug at the same time I want to be in full control which customers update which packages at which point in time on the appliance (essentially a staged rollout).
In regards to my own software package this could very well be in form of different update channels (beta, stable, etc.).
Ideally I am also able to fetch and push configuration and other files from/to the appliance.
It would be great if I could use existing solutions for everything instead of rolling my own mechanisms.

Also, the solution should scale to hundreds or even thousands of customers.

What I have thought of until now:

Spacewalk, Puppet, etc.

For managing the CentOS systems. However, I have no idea whether any one is good at managing machines that are not running inside a single company network.

RPM packaging my software component
or
My software as a Docker container

I don't know which one of these would be better regarding updates. Docker may be overkill, it's just a single software application on a single machine for each customer, I only thought that it would ease application updates. Maybe an RPM package would be preferable, that way I could handle update channels as different RPM repositories and control the updates themselves with Spacewalk/Puppet.

An OpenELEC/LibreELEC JeOS approach

I don't know what this is called exactly or whether there are pre-built solutions for this but I have first encountered it with OpenELEC. Essentialy the whole OS and application are packaged in a single non-modifiable and updatable (kernel) image which is installed on the target machine and there's a separate partition for user and local configuration data.

I know this is a pretty big question, but any pointers would be very appreciated.

I can provide some personal experience here. I worked at a company that had the same need. The product was a BeagleBone (like an Arduino or Raspberry Pi) that ran our software and sent data back to our SaaS. It had to be installed inside customer networks, and be a little black box that they didn't touch.

When I walked into this company, management of the systems was a nightmare. Networking was a nightmare. OS updates, keeping them online, shipping new versions of the software, etc... it was a nightmare.

The path we ended up choosing ended up working really well: The entire application and supporting packages was shipped as a Docker container. That allowed us to easily ship application code changes and control versions of any supporting software (ffmpg, gcc, etc...).

With the application abstracted, that simply left making the OS as minimal as possible. All we needed was a dead simple OS that could run docker. One of our low level engineers actually made our own in-house fork of Debian (if I remember) and we shipped that embedded directly into the devices.

We already had a service contract with all of our customers to service these machines yearly, so when our field technician went on-site, the procedure was to replace the entire unit with a fresh hardware that came with the latest embedded OS. We didn't have to make many OS changes just to run Docker, so often times swapping wasn't even necessary. But out of procedure, we always swapped hardware whether it was an emergency service call or the yearly maintenance.

I can't say if this would be a standard approach or not, but just that it worked really well for us and was a life saver. It sounds like your idea of shipping the OS as an immutable image is along the same lines.

Edit: When I left, we were still using home grown bash scripts to automate the Docker container deployments, but the idea was to move toward a config management tool (Like Chef, Salt, or Ansible). The basic premise is that you have a private Docker registry somewhere, then on the devices you would simply run a couple commands:

docker pull my-private-registry/my-custom-image:latest
docker run my-custom-image:latest

If you are going to always be using the 'latest' tag, you could then have your devices ship with a cronjob that has then always shut down their local running Docker container, and re-pull the 'latest' tag every week during a scheduled maintenance window.


What is the best approach to deploying to multiple servers using Jenkins? Trying to figure out the best approach for deploying to multiple servers using Jenkins Blue Ocean pipeline.  We will have six servers in development, six in QA, six in Staging, and six in production.  Should I put the list of the servers in a config file and store in git or is there a plug-in or something within Jenkins that stores host names?  All feedback is greatly appreciate.

Have you looked at the Matrix project plugin?

Multi-Configuration Projects
A multi-configuration project is useful for instances where your
builds will make many similar build steps, and you would otherwise be
duplicating steps.
Configuration matrix
The Configuration Matrix allows you to specify what steps to
duplicate, and create a multiple-axis graph of the type of builds to
create.

Welcome to the Matrix is an excellent reference doc.
Example (from the blog) - Pipeline for multiple platforms and browsers

Jenkinsfile

pipeline {
    agent none
    stages {
        stage('BuildAndTest') {
            matrix {
                agent any
                axes {
                    axis {
                        name 'PLATFORM'
                        values 'linux', 'windows', 'mac'
                    }
                    axis {
                        name 'BROWSER'
                        values 'firefox', 'chrome', 'safari', 'edge'
                    }
                }
                stages {
                    stage('Build') {
                        steps {
                            echo "Do Build for ${PLATFORM} - ${BROWSER}"
                        }
                    }
                    stage('Test') {
                        steps {
                            echo "Do Test for ${PLATFORM} - ${BROWSER}"
                        }
                    }
                }
            }
        }
    }
}


macOS testing on Desktop: Docker containers vs. Homebrew installation Often I test some stuff on my MacBook. Mostly database and data store related stuff (MongoDB, MariaDB, Postgres). I install the software with Homebrew and test.

I wonder if I should switch to docker? What are pros and cons Homebrew vs. Docker for testing things on local Desktop?

One of the advantages of using Docker would be that you can easily run tests with different versions of the (DB) apps, which could be quite difficult with Homebrew (some apps don't easily support multiple versions installed on the same system).


What branching strategies/workflows are available for Git repositories? I'm looking for a high-level, summary description of the recommended branching strategies/workflows primarily applicable with Git, which would serve as a guide for navigating this environment.

Ideally a summary of pros/cons and maybe intended usage of such strategies/workflows should be mentioned as well.

The Atlassian website includes a comparison of different workflow strategies available when concerned with Git. More strategies exist, like the one used by Linux Kernel team, but are not relevant to most organisations.

The workflow types most commonly used are -

Centralized - every change is added using a single branch.
Feature Branch - every change is added on its own branch and considered for merging into a central branch for integration.
Gitflow - based on feature branch workflow, but also adds another branch for stable releases, and another bunch of branches for hotfixes on that stable release branch.
Forking - in OSS projects often the whole project is copied over to an additional repository and changes are merged between multiple public repositories, or not merged, but still available to the public.

Depending on the organization, one of these might be more suitable.

Centralized - non-restrictive, allows everyone to shoot anyone to "break the build" by pushing changes, least overhead and process involved. The advantage is a continuous integration of changes, even when those don't work.
Feature branch - allows anyone to add changes by adding some overhead process, but integration into main collaborative work is a controlled process. Has the disadvantage of integrating changes late, when requests to merge are forgotten or not attended.
Gitflow - huge overhead in managing the various branches, causes a lot of confusion for regular people who just want to get their changes into the branch. Can work well when most developers only see the "feature branch" part of this. Merging between the development and stable branches often happens late, and integration is not trivial. Adds a lot of extra overhead and terminology that everyone must understand well.
Forking - the wild west, requires a server that supports managing multiple repositories and repositories per-user (GitHub, GitLab, BitBucket Server, etc...). Allows most freedom for each contributor to do whatever he wants, but does not force integration. Demands that integration is done piecemeal in small bits and very controlled, usually by using pull requests across repositories. Works well in unorganized teams such as OSS project contributors.


Running Docker images as Jenkins slave I have a Jenkins Master running in Windows as Local system. I have Jenkins slaves running as Docker containers in a VM in the same Machine.

Everything is working OK, I just want to know are there any gains in memory consumption or ressources if I'm going to run parallel jobs in the master and slaves or should I just use parallel steps using pipeline in Master?

In general, you can go with the assumptions that running something inside docker does not add a significant amount of resource consumption, neither CPU nor RAM, compared to starting the same process directly.

Also, it is usually good to optimize only if necessary.

So pick the Jenkins configuration that seems cleanest/nicest/most elegant to you. If you run into trouble, then look for a solution. As you are running everything on a single physical machine, thinking too much about performance is a moot point anyways, it's not like the solution will scale in any direction, anyways. But getting to know Jenkins, and finding good configuration solutions which you may be able to apply in a "real" environment later is certainly possible in this way.