Can I use AWS Step Functions to launch many different fargate 'runs'? I am building a service that has two independent steps. The first step is a data transformation step that has to happen and is common to all downstream processes, then the second is starting a few dozen independent parallel processes that use the same input data, save for a few different runtime specified parameters.
I know I can use a step function to set off the data transformation step and wait for it to complete before starting off a single fargate task, but can I get it to start a few dozen (or more) in parallel using a common docker object but pass each of those runs slightly different parameter values.
I want to use fargate because we are a small team and I don't want to have to spend team resources on maintaining compute resources, and I can't use AWS Lambda because we expect the run times will often exceed 15 minutes, and there isn't an easy or obvious way to break the problem down any further to be certain the jobs will be able to execute within 15 minutes.

Yes, you can start ECS Fargate tasks from Step Functions in the way you've described.
Step Functions allows you to pass Parameters to the underlying service API.
https://docs.aws.amazon.com/step-functions/latest/dg/connect-parameters.html
[sidebar: that doc also specifies how to pass state inputs, which you've stated is a requirement]
For ECS Fargate, a single state machine with no unique configuration could look something like this:
 "States": {
    "${name}": {
      "Type": "Task",
      "Resource": "arn:aws:states:::ecs:runTask.sync",
      "Parameters": {
        "LaunchType": "FARGATE",
        "Cluster": "${cluster}",
        "TaskDefinition": "${task_definition}",
        "NetworkConfiguration": {
          "AwsvpcConfiguration": {
            "Subnets": ["${subnet_1}", "${subnet_2}"],
            "SecurityGroups": ["${security_group}"],
            "AssignPublicIp": "DISABLED"
          }
        }
      },

Now, you've asked about passing each task slightly different parameter values...
Thankfully, the ECS RunTask API has a helpful overrides: { containerOverrides: [] } parameter.
https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_RunTask.html
Overriding the command (for task execution) or environment (for OS variables) should provide the customization you require without needing multiple ECS Task Definitions.
So, add each distinct task to your state machine, each ingesting state inputs and with its own unique override, and that should do it!


Keeping AWS RDS Instances up to date We are running databases in AWS RDS (MySQL Databases), EC2 instances etc. We keep the EC2 instances up-to-date with the latest AMI based on the latest security patches etc, we do this through terraforms.
I assume that, AWS RDS being a managed service does not require the kind of updates that we are following for EC2. Please correct me if I am wrong and suggest the best practice.

Your assumption is correct: you will not need to perform patching on an RDS instance, AWS handles that for you. There is a caveat, though: the majority of updates are not forced by AWS, but require an automated maintenance window to execute.

Periodically, Amazon RDS performs maintenance on Amazon RDS resources. Maintenance most often involves updates to the DB instance's underlying hardware, underlying operating system (OS), or database engine version. Updates to the operating system most often occur for security issues and should be done as soon as possible.

Some maintenance items require that Amazon RDS take your DB instance offline for a short time. Maintenance items that require a resource to be offline include required operating system or database patching. Required patching is automatically scheduled only for patches that are related to security and instance reliability. Such patching occurs infrequently (typically once every few months) and seldom requires more than a fraction of your maintenance window.

Source: Maintaining a DB Instance
That document has a lot of useful information about the types of maintenance, the maintenance windows, and the manual actions you can take.
Database engines need patching, too, and AWS refers to this as a minor engine upgrade. When necessary, these upgrades happen during the aforementioned maintenance windows, but you need to ensure your RDS instance has Auto Minor Version Upgrade enabled to take advantage of this feature.
Source: Automatically Upgrading a Minor Engine Version


What are best practices for running Hashicorp's Vault in terms of high availability? The Production Hardening document indicates how to harden Vault for production purposes, but what if Vault is run on for example Digital Ocean and the platform is unavailable or the disk gets corrupted?

According to this documentation two vault servers have to be run. One of them has to be active and the other passive. In order to run such a setup both vault servers need to be unsealed.


How to re-enable RDP on AWS-EC2 instance Accidentally i disabled RDP on EC2 windows machine and now I am not able to access it via RDP, Is there any way to re-enable RDP service on EC2 instance?

Yes, You can able to re-enable RDP using run the AWSSupport-ExecuteEC2Rescue Automation (AWS Systems Manager)

Please follow below steps carefully:

Open the AWS Systems Manager console at https://console.aws.amazon.com/systems-manager/.
In the navigation pane, choose Automation.
Choose Execute automation.
In the documents list, choose AWSSupport-ExecuteEC2Rescue. The document owner is Amazon.
In the Document details section verify that Document version is set to the highest default version. For example, 13 (default).
In the Execution mode section, choose Simple execution. Leave the Targets and Rate Control option disabled.
In the Input parameters section, specify the following parameters:

For UnreachableInstanceId, specify the ID of the unreachable instance.

For LogDestination, specify an Amazon S3 bucket if you want to collect operating
system-level logs while troubleshooting your instance. Logs are automatically uploaded
to the specified bucket.

For EC2RescueInstanceType, specify an instance type for the EC2Rescue instance. The
default instance type is t2.small.(Please leave as it is if you are confused and go
ahead)

For SubnetId, specify a subnet in an existing VPC in the same availability zone as the
unreachable instance. By default, Systems Manager creates a new VPC, but you can
specify a subnet in an existing VPC if you want. (Please leave as it is if you are
confused and go ahead)

Note

If you don't see the option to specify a bucket or a subnet ID, verify that you are using the latest Default version of the document.

For AssumeRole, if you created roles for this Automation by using the CloudFormation procedure described earlier in this topic, then specify the AssumeRole ARN that you copied from the CloudFormation console.

Choose Execute automation.

It takes sometimes (5-10 minutes) and you are able to access your old instance with new IP.

Here is the AWS documentation for this issue:
https://aws.amazon.com/premiumsupport/knowledge-center/troubleshoot-remote-desktop-connection-ec2-windows/


Docker update with persistent storage and 0 downtime I have a docker container running with persistent storage. When I try to update now I kill the container, and rerun the updated image all over again.
I've considered setting up the new container first, then switching traffic over via NginX, but how would this work with persistent storage? I can't bind the same data to two different containers can I?
Most questions and googling I've done suggest moving over to kubernetes, but for a single-host, 3 container setup, it doesn't seem worth it for such a small application.

Docker itself does not provide such rolling update feature. You have to do it by hand which is not an easy task. Even if you want to use a one node Swarm cluster, the scheduler does have downtime.
The scheduler applies rolling updates as follows by default:

Stop the first task.
Schedule update for the stopped task.
Start the container for the updated task.

If you only have one machine, you can try something like k3s, which is a lightweight Kubernetes distribution. You'll be able to use a Deployment, a Kubernetes object, which provide zero downtime rolling update feature.


Building a Docker container using Gitlab CI Docker runner I am trying to build a docker image using Gitlab Docker runner.

For this I have configured the runner with a shared volume in /etc/gitlab-runner/config.toml called /builds:

concurrent = 1
check_interval = 0

[[runners]]
  name = "My Docker Runner"
  url = "http://host"
  token = "xxxxxxxxxxxxxxxxxxxx"
  executor = "docker"
  [runners.docker]
    tls_verify = false
    image = "docker:latest"
    privileged = false
    disable_cache = false
    volumes = ["/var/run/docker.sock:/var/run/docker.sock", "/cache", "/builds"]
  [runners.cache]

I created a .gitlab-ci.yml in my repo which loosely looks like this:

image: alpine:latest

variables:
  GIT_STRATEGY: none

test_ci:
  script:
    - apk update
    - apk add docker
    - apk add git
    - docker info
    - git clone http://gitlab-ci-token:${CI_JOB_TOKEN}@host/user/repo.git /builds/user/repo
    - docker build -t database .
    - docker run database

I have also created a Dockerfile in the repo which looks like this:

FROM postgres:9.6

ADD /builds/user/repo/script.sql /docker-entrypoint-initdb.d/

Everything works alright except for the Docker ADD instruction which fails because the /builds/user/repo/script.sql cannot be found:

ADD failed: stat /var/lib/docker/tmp/docker-builder886348850/builds/user/repo/script.sql: no such file or directory

So I imagine that the /builds volume I am trying to share between containers is not working as I am expecting it to ... Could someone help me with this? I would like to be able to share the code which is cloned on the runner container to the docker build command and then to the newly created container.

Weird Docker. I changed the WORKDIR in the Dockerfile and it worked:

FROM postgres:9.6

WORKDIR /builds/user/
ADD repo/script.sql /docker-entrypoint-initdb.d/

EDIT
The /builds folder does not exist on the host system. I thought that adding it to the runner's config would create a volume container that would be shared between containers. But no. So I removed it from the runner config, /etc/gitlab-runner/config.toml:

[[runners]]
  name = "My Docker Runner"
  url = "http://host"
  token = "xxxxxxxxxxxxxxxxxxxx"
  executor = "docker"
  [runners.docker]
    tls_verify = false
    image = "docker:latest"
    privileged = false
    disable_cache = false
    volumes = ["/var/run/docker.sock:/var/run/docker.sock", "/cache"]
  [runners.cache]


How to run a build from a specific old tag in gitlab ci? I can build from a specific old tag in teamity.
Screenshot:

How to run a build from a specific old tag in gitlab ci?

variables:
  MAVEN_OPTS: "-Djava.awt.headless=true -Dmaven.repo.local=.m2/repository"
  MAVEN_CLI_OPTS: "-s maven_settings.xml --batch-mode --errors --fail-at-end --show-version"

stages:
  - build
  - release
  - build-tag

build:
  stage: build
  script:
    - mvn $MAVEN_CLI_OPTS -Dmaven.test.skip=true clean install -PRPM -U

release:
  stage: release
  script:
    - eval $(ssh-agent)
    - ssh-add <(echo "$SSH_PRIVATE_KEY")
    - git config user.name "$PUSH_USER_NAME"
    - git config user.email "$PUSH_USER_EMAIL"
    - git checkout master
    - git reset --hard origin/master
    - mvn $MAVEN_CLI_OPTS clean release:prepare -Dresume=false -DautoVersionSubmodules=true -DdryRun=false -Dmaven.test.skip=true
  when: manual

build-tag:
  stage: build-tag
  script:
    - git checkout master
    - git pull
    - git reset --hard $(git describe --abbrev=0 --tags)
    - mvn $MAVEN_CLI_OPTS -Dmaven.test.skip=true clean install -PRPM -U
  when: manual

build-tag i dont see in pipeline

How to run a build from a specific old tag in gitlab ci?

Within the project

go to CI / CD -> Pipelines
press Run Pipeline
under Create for choose the branch or tag you want to run the pipeline for
press Create Pipeline

Now to trigger a manual Job you can use the jobs of the recently created pipeline.


Jenkins DSL - Pipeline Job (Unexpected token '$') I have the following DSL code and I´m getting error on line 13, $class. I guess than everything inside script("""...""") is parsed as plain text to the pipeline definition but I don´t know what´s wrong or how to escape the $ character.

My code:

pipelineJob('DSL-Test') {

parameters {
    choiceParam ('version', ['1.2.1', '1.2.0'], 'Docker Version')
}

definition {
    cps {
        sandbox(boolean sandbox = true)
        script("""
            node ('master'){
                stage('Git Checkout'){
                    checkout([$class: 'GitSCM', branches: [[name: '*/master']], doGenerateSubmoduleConfigurations: false, extensions: [], submoduleCfg: [], userRemoteConfigs: [[credentialsId: 'mycrednetial', url: 'myurl']]])
                }
            }
        """.stripIndent())
         }
    }
}

This is the error I get:

  Processing provided DSL script ERROR: startup failed: script: 13:
  unexpected token: class @ line 13, column 24.
                            checkout([$class: 'GitSCM'])
                            ^

  1 error

  Started calculate disk usage of build Finished Calculation of disk
  usage of build in 0 seconds Started calculate disk usage of workspace
  Finished Calculation of disk usage of workspace in 0 seconds Finished:
  FAILURE

Does anyone know how to make that character not display error?

I have just managed how to escape characters.

The solution is to add: "\" before the character.

checkout([\$class:....


Docker prunes image and networks by itself I have a script which performs backup of several docker volumes during the night.
It starts by stopping the containers that use these volumes, then it creates a new container with all the volumes mounted and back them up with borg. Finally it restarts the containers.
Sometimes the scripts runs into weird issues, like a container can not start because its image or a network it uses is missing. Is it possible that docker daemon prunes them while my backup script is running? How can I disable it?
I could not see anything in daemon logs.

I noticed that you have the docker run command with --rm option. In the Docker docs it says:

If you set the --rm flag, Docker also removes the anonymous volumes associated with the container when the container is removed. This is similar to running docker rm -v my-container

Perhaps this is why the volumes are being deleted?


Log/Data rotation in Docker Container My question is more on the concept or best practices to follow.
I have two docker containers running InfluxDB and Grafana (monitoring applications). I have mounted external volume to each.
now, the problem is, these containers are running for quite some time (1 year approx), i am little bit worried about Host VM resources on which these containers are running. specifically storage resources as these are monitoring applications, which may be storing some data for historical purpose as well.
So, my ask is, what is the best practice to manage logs/data stored in docker volume. I don't want my Host VM to go out of storage and crash my containers.
Any possible solution, if we can rotate stored data so that my container won't stop and i have the historical data as well. or if possible from application point of view (InfluxDB and Grafana) to archive old data.
Thanks in advance... Please let me know if you need more details on this.
below is my docker-compose.yml file
version: "2"
services:
  grafana:
    image: grafana/grafana
    container_name: grafana
    user: root
    privileged: true
    restart: always
    ports:
      - 3000:3000
    networks:
      - monitoring
    volumes:
      - grafana-volume:/var/lib/grafana
  influxdb:
    image: influxdb
    container_name: influxdb
    restart: always
    ports:
      - 8086:8086
    networks:
      - monitoring
    volumes:
      - influxdb-volume:/var/lib/influxdb
networks:
  monitoring:
volumes:
  grafana-volume:
    external: true
  influxdb-volume:
    external: true

You can update the daemon.json config to specify your normal syslog driver (assuming linux) to use the normal syslog logrotation method for keeping log files or shipping off-box
https://docs.docker.com/config/containers/logging/syslog/
{
  "log-driver": "syslog",
  "log-opts": {
    "syslog-address": "udp://localhost:514"
  }
}


Are Route 53 DNS records aliasing ELBs guaranteed for their lifetime? Given a Route 53 DNS record aliasing a (classic) load balancer, and the potential for the load balancer ip to change:

Is it possible for the DNS record to guarantee that the provided ips are valid for the record's lifetime? If so, is there a specific TTL value I need to set?

If you’re talking about using a Route 53 ALIAS record, you cannot set the TTL on it, for this reason. It defaults to a low value, and is managed by AWS for you (to the world it appears as an A record, and it will change often).

If you’re using a CNAME, then you can set whatever TTL you like. The TTL of the DNS record for the load balancer’s domain will be low, but you don’t need to worry about that. Your only concern in this circumstance is whether you think you’ll be changing the load balancer name or repointing the name elsewhere. If you don’t plan to do that, you can set a high TTL to save a bit of money and milliseconds at load time.

Either way, the IP addresses themselves will change.

For help choosing between an ALIAS and CNAME record, see https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/resource-record-sets-choosing-alias-non-alias.html. I’d usually recommend a CNAME unless you’re pointing your root domain, in which case you have to use an ALIAS.


How to create databases, users and grants with terraform mysql and output the created credentials? For a new deployment I want to create databases, users and grants on a previous created MySQL database on azure.

The following code used inside a module creates the environment as expected and I still struggle how to create the outputs of the module and so I get something like the following, to give it to the operations team:

module.test_cluster.mysql_users[cat].password = randompassword1
module.test_cluster.mysql_users[dog].password = randompassword2

Code example:

mysql_databases = [
  "foo",
  "bar"
]

mysql_users = [
  "stage-cat",
  "stage-dog",
  "stage-snake"
]

mysql_grants = {
  "cat-0" = {
    name     = "stage-cat"
    database = "foo"
    grant    = ["ALL"]
  },
  "cat-1" = {
    name     = "stage-cat"
    database = "bar"
    grant    = ["SELECT", "EXECUTE", "SHOW VIEW"]
  },
  "dog-0" = {
    name     = "stage-dog"
    database = "bar"
    grant    = ["ALL"]
  }
}

resource "mysql_database" "test" {
  for_each = var.mysql_databases
  name     = each.key
}

resource "mysql_user" "test" {
  for_each           = var.mysql_users
  user               = each.value
  host               = "%"
  tls_option         = "SSL"
  plaintext_password = random_password.test[each.value].result
}

resource "mysql_grant" "test" {
  for_each   = var.mysql_grants
  user       = mysql_user.test[each.value["name"]].user
  host       = "%"
  database   = each.value["database"]
  privileges = each.value["grant"]
}

resource "random_password" "test" {
  for_each = var.mysql_users
  length   = 32
  special  = false
}

To ensure separation and use in other environments, make sure to have an outputs file and set each password as an output.  TF Output Docs.  I haven't done much with for each as it was just released last time I used terraform.  I would imagine a way to iterate over passwords and assign them as outputs.

once you have the output set for the module, I would then have my consuming terraform put those values in a password vault of some kind.  You can use Azure's Key Vault.  TF Azure Key Vault, or any other solution.  Just ensure that the ops team has access to the destination.  This way there isn't anything introduced between when/where the keys are generated and them being stored, reducing the chance for human error.


How much memory/cpu to allocate per pod? This is a question that's been bothering us quite a while.
How can you determine how much to allocate per pod?
What if I don't know the program, maybe it needs a lot of memory at peak times.

What are some general strategies to finding out how much to allocate? cpu/memory

It's a severe problem to solve if you don't know the performance characteristics of the process you are containerising.  There are a couple of approaches you could take that will get you an answer, although you will need to monitor it carefully in production and an adjust the resource limits as required.

Approach #1: Educated Guess - take a look around at what other people have done to host a specific process, GitHub is an excellent resource for finding Dockerfiles and their associated Helm Charts.  Equally, if someone has done some tests for Virtual Machine sizing, you may be able to derive an answer that is acceptable in the short term.
Approach #2: Testing - my personal preference is to test the component in question:

The Apache Bench is excellent for quick performance tests against HTTP(S) endpoints
Tools such as the MySQL Benchmark tool allow you to test MySQL, many applications have dedicated benchmarking tools.
If the software is more complicated you can use tools such as Artillery or Gattling to create a performance test suite that you can use to slam the container and find out how it performs.

Whichever approach you take, you will need to make sure that you have a proper monitoring and dashboarding solution working so that you can actually analyse the behaviour of the containers under load and adjust accordingly.


How to add label to Pull Request via Github API? According to GitHub API user can add label to an issue using this POST request.
I tried using this request but with PR number instead to automatically set label on PR. Here is the function I wrote:
def setLabel (String repository, List labels) {
    try {
        def response = httpRequest httpMode: 'POST',
                    url: "https:[Repo_URL]/${repository}/issues/${CHANGE_ID}/labels",
                    authentication: 'ors_git_service_account',
                    requestBody: """{
                        "labels": labels
                    }"""
    } catch (e) {
        wrap([$class: 'AnsiColorBuildWrapper', 'colorMapName': 'XTerm']) {
            echo "\u001B[31magdSlack.setLabels: '" + e.toString() + "'\u001B[0m"
        }
        throw e
    }
}

And the response I get is HTTP/1.1 400 Bad Request
So I get that this request can't be used to labeling PRs.
My question: Is there other method which could be triggered from Jenkinsfile, to add or change PR label?
Closest thing to the question I asked was this question. Commenting API works on issues and PRs but that's not a case for labels.

I made it work by changing the way labels argument is passed into requestBody.
Turns out the list representation while interpolated into string is missing quotation marks around each list element.
def setLabels (String repository, List label) {
    try {
        String labelsAsString = "[\"${label.join('", "')}\"]"
        def response = httpRequest httpMode: 'POST',
                        url: "[REPO_URL]/${repository}/issues/${CHANGE_ID}/labels",
                        authentication: 'ors_git_service_account',
                        requestBody: """{
                            "labels": ${labelsAsString}
                        }"""
    } catch (e) {
        wrap([$class: 'AnsiColorBuildWrapper', 'colorMapName': 'XTerm']) {
            echo "\u001B[31magdSlack.setLabels: '" + e.toString() + "'\u001B[0m"
        }
        throw e
    }
}


mariadb docker on raspberry I've tried countless docker images, but all of them have the same behaviour.

When I open up a console and try to import my database structure via the mysql/mariadb command. But I always get the same error:

ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: YES)

This is my dockerfile:

version: '3'

networks:
    mynet:
        driver: bridge

services:
  mariadb:
    container_name: mariadb
    image: webhippie/mariadb # this one doesn't work (at least for me)
    networks:
        mynet:
            aliases:
                - database
    environment:
        - MARIADB_ROOT_PASSWORD=topsecret
        - MARADB_DATABASE=mydb
    ports:
    - 3306:3306

Do you have any suggestions on how I can login via the console?

Does the following work?

docker-compose exec mariadb /usr/bin/mysql -u <your_db_username> --password=<your_db_password> < your_db_backup.sql

Note: mariadb is the name of your container.

Edit: I realize I'm using docker-compose. So I'm not entirely sure if this would work with just a Dockerfile.


Other tools to provision virtual machines I've been using vagrant for provisioning virtual machines (Linux only) and so far I've been happy with it. The two things I dislike about it - the hardcoded dependency on the user vagrant as well as making the very first interface a virtual box NAT - this causes some routing problems when communicating outside my LAN - it has eth0 created by vagrant and eth1 in bridged mode. As far as I understand, this was a design decision so there is no way around it.

I currently use vagrant

To create a Virtual Box virtual machine,
Set Static IP/MAC addresses
Create my own secondary user, and then treat it as a standalone "physical" host. I then use this user for any additional provisioning.

I can delete the user vagrant - the problem is that if I do vagrant up (or vagrant halt I believe to gracefully shutdown) it tries to ssh and the command hangs while it retries. The other problem is that I need to delete it each time I spin up a VM and I cannot when doing the first time provisioning because I am logged in as user vagrant.

I've been wondering if there is any other similar software/tool that just does a one time provision, sets up a static IP/MAC address, and allows for easy start/resume/shutdown when testing/deploying/maintaining.

I've been looking at Hashicorp's packer but I don't think it is possible to automatically configure IP/MAC addresses.

I am mainly looking for creating a virtual machine from scratch as if it was installed on a bare metal desktop computer by hand using a ISO on a CD.

In the end, using packer to provision and then using VirtualBox command line tools to recreate some of the functionality that vagrant does will suffice (sett IP/MAC address) - but I wonder if there are any other tools used for creating secure (no vagrant user and ssh only with private key) vms.

Have you considered to use Ansible?

You have some resources about using Ansible with VirtualBox and also vagrant


Testing docker image builds when low level libraries change Background
In our organization we have many small repos for shared libraries used among our applications. Particularly, we have many Python libraries and Python applications. We have a structures like:

myorg/python-lib1: produces package myorg.lib1
myorg/python-lib2: produces package myorg.lib2
myorg/python-app: produces a package myorg.app, depends on myorg.lib1 and myorg.lib2

In our CI for myorg/python-lib1 and myorg/python-lib1 we run unit tests when PRs come in and publish wheels to PyPI on merges to the main branch.
In our CI for myorg/python-app1 we will build test application images when PRs come, deploy them to a test kubernetes cluster, and allow developers to run integration tests against it. It has a Dockerfile like:
# Builds image reistry.myorg.com/python-app
FROM reistry.myorg.com/python-base:latest

COPY . /app

# Implicitly gets the latest `myorg.lib1` and `myorg.lib2` from PyPI
RUN pip install /app

Note: We also have some projects using build-packs to achieve a similar goal.

Issue
Because we only run the integration tests for myorg/python-app1, we can't get as much confidence when a change to myorg/python-lib1 and myorg/python-lib2 happens. We have to wait for the libraries to get published to PyPI and then rebuild the application images. When there's an issue in the library, this usually involves us git reverting and fixing the PyPI version. This is super cumbersome when we have a lot of images that pull in the dependency before we detect it needs to be rolled back.
I'm looking to restructure the way we build images in myorg/python-app1 (and other apps that share this pattern) to support pulling in dependencies based off of the PR branches.
I've looked at a lot of tools for containerization but I haven't seen any strategies for building application containers given a change in a lower level dependency. I'm interested in seeing what strategies exist for this. Hopefully the strategy could be applied to other languages as well, where we have a similar strategies for applications written in languages like Java, Golang, and npm.

So I believe I came up with a somewhat re-usable solution for this problem using devpi. It allows you to use the same Dockerfile for the testing and production image.
Devpi allows me to create my own index which can shadow my production index with versions of software that haven't been officially released.
Steps to make this work:
1. Base image from ARG
I first edited my Dockerfile to allow overwriting the base image used. This will allow me to create a layer in the middle that configures pip to talk to devpi:
# Builds image reistry.myorg.com/python-app
ARG BASE=reistry.myorg.com/python-base:latest
FROM $BASE

COPY . /app

# Implicitly gets the latest `myorg.lib1` and `myorg.lib2` from PyPI
RUN pip install /app

2. Upload test versions of library
For each python change:

Clone the repo
python -m build the repo
devpi upload the wheels

3. Create a layer with devpi pip conf
Make a pip conf that can access your devpi instance and configure to your index name:
[global]
index-url = http://<some ip>:<some port>/root/dev/+simple/
trusted-host = <some ip>
disable-pip-version-check = true

[search]
index = http://<some ip>:<some port>/root/dev/

And add that to your base images pip conf:
FROM reistry.myorg.com/python-base:latest

COPY pip.conf /etc/pip.conf

4. Chain the images together
docker build -t devpi-base -f devpi.Dockerfile .
docker build -t my-app -f Dockerfile --build-arg=BASE=devpi-base .
docker tag my-app reistry.myorg.com/my-app:test1
docker push reistry.myorg.com/my-app:test1

Now when pip installs all of your dependencies you can have it pick up lower level changes to python packages. This concept should be re-usable for a lot of languages I imagine!


The right way to Ansiblically deploy environments directly from Github: Can one deploy an environment for some Linux OS (say, Ubuntu) from a raw Github playbook in a way similar to this (I wonder if it's the correct syntax)?

bash <(curl -s https://raw.githubusercontent.com/user/repo/master/nginx-cms-addons.yml | tr -d '\r' | ansible-playbook)

Edit: This is my own Github account so there isn't a risk.

In use a similar pattern to download and execute bash scripts but I want to know if the principle is identical for Ansible playbooks.

If it matters, the environment I want to deploy directly from the Github raw yaml configures Ubuntu a bit (firewall), installs some trivial software like zip, a server environment (Nginx/Postfix) and that's basically it.

My purpose to execute my playbook with ansible as a substitute for the long Bash script I currently use, made of this code, basically:

#!/bin/bash
ufw --force enable && ufw allow 22,25,80,443,9000/tcp
apt-get update -y && add-apt-repository ppa:certbot/certbot -y && apt-get update -y
apt-get upgrade zip unzip tree unattended-upgrades sshguard nginx python-certbot-nginx mysql-server php-fpm php-mysql php-mbstring php-mcrypt -y
DEBIAN_FRONTEND=noninteractive apt-get upgrade postfix -y
sed -i "s/post_max_size = .M/post_max_size = 250M/ ; s/upload_max_filesize = .M/upload_max_filesize = 250M/" /etc/php/*/fpm/php.ini
sed -i "s/;cgi.fix_pathinfo=1/cgi.fix_pathinfo=0/" /etc/php/*/fpm/php.ini
# sed -i "s/# gzip_/gzip_/g" /etc/nginx/nginx.conf
/etc/init.d/php*-fpm restart && systemctl restart nginx.service

There is no "right" or "wrong". If the commands work when you type them in, then they work, and we won't keep you from doing it.

Everything else is opinion. Sure, there are some best practices, for example some people find it unwise to directly fetch scripts from a public (3rd party) website and execute those locally without having a look inside first.

It goes without saying that your approach is in fact a rather significant risk. You are effectively giving root access on the target machine (and any other machine your local user has ssh keys for...) to anyone who can push into your repository. How earnest that risk is is something only you can decide.


how to start a docker container which dies immediately after startup? On the understanding, from #docker on IRC, that the basex container from docker hub died after starting up, how do I now make it run something which doesn't immediately exit?
I'm only trying to get a bash prompt at this point:
docker run -ti     --link basexhttp:basexhttp     basex/basexhttp:latest basexclient -nbasexhttp

results in:
docker: Error response from daemon: Cannot link to a non running container: /basexhttp AS /charming_villani/basexhttp.

per the basex official doc's.
In trying to add docker logs I've tried:
sudo journalctl -fu docker.service

but will have to look into that a bit as there was no output.

trying a network as suggested:
root $
root $ docker network create -d bridge basexhttpnetwork
74c94db9195cf2b671123d016a9082f4aeacbbcef6ed91e4252744afba45d529
root $
root $ docker run -ti --name basexhttp --network=basexhttpnetwork basex/basexhttp:latest basexclient -nbasexhttp
docker: Error response from daemon: Conflict. The container name "/basexhttp" is already in use by container "2473f440e93855f967482aed405c30796a779d3592bd66a00fe48c2131b43a5e". You have to remove (or rename) that container to be able to reuse that name.
See 'docker run --help'.
root $

The proper way to run this image, according to the README, is:
docker run -d \
    --name basexhttp \
    --publish 1984:1984 \
    --publish 8984:8984 \
    --volume "$HOME/basex/data":/srv/basex/data \
    --volume "$HOME/basex/repo":/srv/basex/repo \
    --volume "$HOME/basex/webapp":/srv/basex/webapp \
    basex/basexhttp:latest

No weird linking to itself is necessary.
If you wan to run a shell in an already running container (started as above) use:
docker exec -it basexhttp sh

or you can try bash
docker exec -it basexhttp bash

but the container is Alpine-based, so bash might not be installed.
If you want to run a new container and execute shell inside run
docker run -it --rm basexhttp sh
# or bash if it's installed
docker run -it --rm basexhttp bash

The above will remove the container once you exit bash (but the 3 volumes it creates
will remain, you need to find and remove those by yourself).
Edit:
It seems README is wrong about how to start the container. See my answer to this question for more details.


Jenkins - Running code in Windows agent but script says we are in Linux I have the following pipeline (Jenkinsfile):

@Library('mylib@master')

import com.mylib.*

pipeline {

    agent { label 'WindowsSlaveWS2016' }

    stages {
        stage('Demo') {
            steps {
                echo 'a'
                determineOS()
                echo 'b'
                // sh 'echo "[Version 1.0.0]" | tee -a changelog.txt'
                sayHello()
                //getServiceVersion('.', 'changelog.txt', 'myservice')
                // cat service.properties
                // sendEmail('failed', 'test@test.com')
                script {
                    def utils = new Utils()
                    if (utils.isWindows()) {
                        echo '<-- Is windwos -->'
                    }
                    else {
                        echo '<-- Is linux -->'
                    }
                }
            }
        }
    }

}

determineOS() groovy script is a simple script that does the following:

#!/usr/bin/env groovy

def call() {
    if (System.properties['os.name'].toLowerCase().contains('windows')) {
        println "it's Windows"
    } else {
        println "it's not Windows"
    }
}

and utils.Windows() does the following:

#!/usr/bin/groovy

package com.ciplatforms;

class Utils implements Serializable {

    def isWindows() {
        if (System.properties['os.name'].toLowerCase().contains('windows')) {
            println "it's Windows"
            return true
        } else {
            println "it's not Windows"
            return false
        }
    }

}

I am running the job in a Windows machine as you can notice in the pipeline but the log displays the following:

Running on WindowsSlaveWS2016 in C:\Jenkins\workspace\test-pipeline-shared-library
[Pipeline] {
[Pipeline] withEnv
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Demo)
[Pipeline] echo
a
[Pipeline] echo
it's not Windows
[Pipeline] echo
b
[Pipeline] echo
Hello, human.
[Pipeline] script
[Pipeline] {
[Pipeline] echo
<-- Is linux -->
[Pipeline] }
[Pipeline] // script
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
Finished: SUCCESS

Any ideas why this is happening?
Thank you.

I got the answer.
As Mark Waite answered here:

https://groups.google.com/forum/#!topic/jenkinsci-users/aGYt-OEYds8

Use the isUnix pipeline step rather than writing your own version of that step.  The System.properties that you are using will execute on the master, not on the agent.  When it executes on the master, it will report the os.name value of the master.


Structuring Ansible playbooks for low volume, purpose-built images I hope this question is not too vague / opinion-based:

I need to create and maintain a handful of virtual machines that get packaged up and deployed out to thousands of users, depending on specific requirements. Everyone is doing roughly the same job, but different people need different tools. This is important because I don't think I can structure things around roles (though I'm happy to be told I'm wrong).

They way I have things structured now is:

vagrant-directory
|-Vagrantfile # Multi-host setup, with one machine per "use-case"
|-scripts/ # Shell scripts run by Vagrant (mostly used to install Ansible prereqs)
|-ansible/ # All Ansible-related files
|----Playbook1.yml # One playbook per use case
|----tasks/ # Contains task files for everything I want to do
|------install/ # All task files related to installing software
|------apt-update.yml # For example, a script to run apt update
|------iptables-flush.yml # Disable iptables

With this approach, it basically means that when I get a requirement for a new machine, I have to create a new Playbook42.yml, use import_tasks for the tasks that I need, create any machine-specific tasks, and then provision and export.

Is there a more "Ansible official" or industry standard approach to doing this?

Unless I totally missed your question, roles are exactly what you need in this case. I'll use examples from my real world. My code below is just to illustrate my point (so it is not fully runnable, might contain errors and is not bullet proof rocket science: you will probably have to adapt/overcome tools limitations). But at least you'll be able to tell me if I got you wrong.

Let's say your servicing a large community of developers on different techs: java, php, nodejs, scala, python, ruby... needing some tools as well like apache, nginx, postgres, mysql, redis...

You have a role for each of these techs/tools, and possibly use existing ones from galaxy that you don't even have to write and that you can download at provisioning time. These roles can possibly be controlled with vars to choose e.g. versions. (see vagrant provisionner config below)

Now, one way you can activate/deactivate roles in a play is with the use of tags

Your unique playbook could look something like this

- name: Vagrant provisionning playbook
  hosts: all

  pre_tasks:
    # whatever you need to do before roles

  roles:
    - role: php
      tags: [php]
    - role: java
      tags: [java]
    - role: nginx
      tags: [nginx,php] # if you always deploy php with nginx
    - role: redis
      tags: [redis]
    # more roles here

    tasks:
      # Whatever tasks are needed to be played after roles

With such a structure, you can now configure your ansible provisioinner in vagrant defining the tags and specifying the vars needed to provision with your single playbook.

Vagrant.configure(2) do |config|

  config.vm define "php_application" do |php_application|

    # Some config for vm
    php_application.vm.provision "ansible" do |ansible|
      ansible.playbook = "playbook.yml"
      ansible.tags = "php,nginx,untagged"
    end

  end

  # Host vars declared globally. You might be able to do
  # it on host by host basis but the doc does not specify it
  config.vm.provision "ansible" do |ansible|
    ansible.host_vars = {
      "php_application" => {"php_version" => "7.2"}
    }
  end
end

Does this fit your needs ?

Note: if a single playbook is too much of a hassle, you can still use the roles and assemble in each playbook the techs/tools you need and reuse things that you will need in several places.


how to automatically create a subdomain for every pull request Background

I got a team of untechnical QA's who have to do testing on iOS/android apps for every Pull Request (PR) that gets created by my backend team.

Question

This is what I want to do: everytime a backend engineer creates a PR on bitbucket, I would like a script to automatically deploy the code of that PR git branch into a subdomain of our dev server that matches the JIRA issue created.

For example suppose the jira issue that the PR addresses is BAC-421, then as soon as the engineer creates a PR, the script deploys the code they created into AWS EC2 so that the QA can point their apps to www.bac421.mydevdomain.com

What is the best way to do this? I'm a devops technical nube.

Update - Environment Specs

so here is a breakup of our env
- the backend uses laravel 5.3
- it's deployed on AWS EC2
- we use forge for auto deployment (nothing fancy.. we just run this script:

cd /home/forge/default
git fetch --tags
git pull origin master
git describe
composer install --no-dev --no-interaction --prefer-dist --optimize-autoloader
echo "" | sudo -S service php7.1-fpm reload

if [ -f artisan ]
then
    php artisan migrate --force
    php artisan config:cache
    php artisan queue:restart
fi

that we run as soon as we merge dev unto master branch)
- besides that we don't use any CI/CD tools although I'm open for recommendations
- DNS provider is GoDaddy
- our application server is nginx
- our database is in a separate RDS instance

We do this at work.

We have a small server, let's call it the receiver, it's the target of the GitHub webhook events. It runs a small application that parses the payload and incorporates logic around how to proceed e.g. create a new server on the infrastructure provider, update the load balancer, deploy to an existing server, destroy the server etc. This could be a traditional web application serving up an API or it could be a serverless application, up to you how you want to approach it.

The receiver is relatively straightforward to handle, the other necessary supporting systems you'll need are an approach to configuration management/provisioning (how does the server have the packages necessary to run the application), secrets management (how does the server get access to sensitive information) and routing  (how do the subdomains get updated and route to the correct server).

It could be worthwhile looking at preparing an AMI with the necessary services configured, a CloudFormation template with the infrastructure provisioning logic and CodeDeploy could handle deployments for you.

Configuration Management

This is really up to you and your team, there are a multitude of tools you can use or you can simply rely on shell scripting. At what point in the servers lifecycle you apply the changes is what's discussed in the AMI design article I linked.

Secrets Management

This is a challenging topic to address with the information at hand, in the interest of brevity I'll leave that up to you and your team.

Routing

There are a few ways you can handle routing, the Application Load Balancer (not to be confused with the ELB/NLB) offered by AWS supports host based routing. Alternatively you could use a reverse proxy like NGINX or HAProxy, when you provision a new environment you will need to update this routing (ideally automatically) irrespective of what approach you take.

Don't forget to consider how you'll handle the database/persistence layer & zero down time deployments. The question to ask with the persistence layer is whether the team will share a database and how that will interact with things like migrations. On the topic of zero downtime deployments, CodeDeploy should handle that nicely for you. One more thing, you mentioned a single mobile application pointing to different environments, how will you point these applications to the environments.


Where should we keep application properties file while doing build promotion using Jenkins? We are using spring boot to develop Java backend services. application.properties file has the database configurations. We are deploying in 4 different environments (dev, test, UAT, and production). The properties files will be different for different environments. Where should we keep these files and how should do build promotion using Jenkins? We are using Apache Tomcat to deploy our WAR file.

Now we are keeping the properties file in C:\Program Files\Apache Software Foundation\Tomcat 7.0\conf directory and accessing using the following code.

@PropertySource(value = {"file:${catalina.base}/conf/application_dev.properties"})

Each time we deploy in different environments, we need to replace the respective file name application_test/uat/production.properties, commit the code and let Jenkins build and deploy. Now manually we have to keep the files in different environments too.

Please suggest how should we do it.

I disagree with @levi, you should stick to the 12 factor app method. Leaving configs in the repo scusk. Simple changes to a password or a small setting requires a commit and a pipeline rebuild. That's too much work for something so simple.

The last 3 companies I've worked for are using 'single branch' repos. There is only the master branch, and developers create 'feature' branches off of that. All the different environments are built from master.

In that case, we have a static config file, but all the values are injected by the build pipeline. You can pull them in from different sources: build server env variables, an external tool (Hashicorp Vault), or even another repo that has all the configs (this last one is least desirable).


Why does Jenkins Pipeline script returnStatus value not match status examined in shell? This is my little newbie Groovy script, trying to cobble together a very beginner understanding of Jenkins Pipelines:
node {
 stage("hello") {
  def var = "val"
  echo "${var}"
  def stdout = sh( script: 'pwd', returnStdout: true ).trim()
  echo "${stdout}"
  def ret_status = sh( script: 'cd subdir', returnStatus: true )
  echo "${ret_status}"
 }
}

The output is:
Started by user unknown or anonymous
Running in Durability level: MAX_SURVIVABILITY
[Pipeline] Start of Pipeline
[Pipeline] node
Running on Jenkins in /home/user/workspace/newbie_dont_know_what_hes_doin
[Pipeline] {
[Pipeline] stage
[Pipeline] { (hello)
[Pipeline] echo
val
[Pipeline] sh
+ pwd
[Pipeline] echo
/home/user/workspace/newbie_dont_know_what_hes_doin
[Pipeline] sh
+ cd subdir
/home/user/workspace/newbie_dont_know_what_hes_doin@tmp/durable-2fa119e0/script.sh: 1: cd: can't cd to subdir
[Pipeline] echo
2
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
Finished: SUCCESS

The failure to cd subdir is intentional -- to demonstrate the mechanics of capturing shell stdout and status.
Why does the failed cd subdir report a returnStatus of 2 when cding to nonexistent directories in bash returns status-code 1? The Groovy documentation I have as reference says sh runs a Bourne shell script.
$ echo $SHELL
/bin/bash
$ cd subdir
-bash: cd: subdir: No such file or directory
$ echo $?
1
$

There's a few things to note. First is that in the absense of a shebang being supplied, the actual interpreter that will be used is actually "sh", not "bash".
Taking a look at the Jenkins source for the BourneShellScript task, what happens is that Jenkins does not actually return the exit code from the subshell - it monitors a file called "jenkins-result.txt for changes (using a FileMonitoringTask) and when that file has been updated, it merely assigns the content of it as the exit status.
From what I can see, when you call "sh" it does not just run your script code, but actually wraps it in a fairly substantial block of boilerplate code (refer line 272 of the above linked BourneShellScript). In effect, something closer to this:
sh -c ({ while [ -d '/home/user/workspace/newbie_dont_know_what_hes_doin@tmp/durable-2fa119e0' -a \\! -f '/home/user/workspace/newbie_dont_know_what_hes_doin@tmp/durable-2fa119e0/jenkins-result.txt' ]; do touch '/home/user/workspace/newbie_dont_know_what_hes_doin@tmp/durable-2fa119e0/jenkins-log.txt'; sleep 3; done } & jsc=durable-2fa119e0; JENKINS_SERVER_COOKIE=$jsc sh -xe '/home/user/workspace/newbie_dont_know_what_hes_doin@tmp/durable-2fa119e0/script.sh' > '/home/user/workspace/newbie_dont_know_what_hes_doin@tmp/durable-2fa119e0/jenkins-output.txt' 2> 'jenkins-log.txt'; echo $? > '/home/user/workspace/newbie_dont_know_what_hes_doin@tmp/durable-2fa119e0/jenkins-result.txt.tmp'; mv '/home/user/workspace/newbie_dont_know_what_hes_doin@tmp/durable-2fa119e0/jenkins-result.txt.tmp' '/home/user/workspace/newbie_dont_know_what_hes_doin\@tmp/durable-2fa119e0/jenkins-result.txt'; wait) >&- 2>&- &

This is likely necessary to support scenarios where the node on which the script is executed is not the node on which the pipeline is executing, however it does leave a lot of room for error in returning the result, depending on the idiosyncracies of whatever interpreter "sh" is aliased to (not necessarily bash) and the OS on which it is running.


unable to save register value even if the play is skipped! I have situation where i fetch some ip address on the fly with this play,  when env variable is passed as prod, i have to fetch ip in a diff way then when i am in QA env.

If env is prod the 1st play would get ip address and store in register "kafkaip" and i have put a when condition on the 2nd play it gets skipped which is fine as the env is diff.

But although its skip the 2nd play, it wipes out the register value and my set_fact fails,, any way to overcome below scenario.

    - name: Get the Kafka InstancesPrivateIps
      local_action: shell aws cloudformation describe-stacks --stack-name "{{ stack_kafka }}" --query "Stacks[*].Outputs[?OutputKey=='KafkaLink'].OutputValue"
      register: kafkaip
      when:
      - stack_kafka is defined

    - name: Get the Kafka InstancesPrivateIps New Automation.
      shell: |
        aws  ec2 describe-instances --filter "Name=tag:StackName,Values=KafkaCluster" --query 'Reservations[].Instances[].PrivateIpAddress' --output text >  /tmp/kafka_ip.txt
      register: kafkaip
      when:
      - stack_kafka is defined
      - ( deployment_env == "QA" )
      delegate_to: localhost

    - name: setting kafka_host.
      set_fact:
        kafka_host: "{{ kafkaip.stdout }}"
      when: stack_kafka is defined

register will always populate your registered var even when the task is skipped, e.g. you will end up with something like the following:

"kafkaip": {
    "changed": false,
    "skip_reason": "Conditional result was False",
    "skipped": true
}

Since only the commands differ depending on your env, I would store the command in a variable and then execute it, something like the following.

- name: "Set command for env <> QA"
  set_fact:
    get_ip_command: >-
      aws cloudformation describe-stacks
      --stack-name "{{ stack_kafka }}"
      --query "Stacks[*].Outputs[?OutputKey=='KafkaLink'].OutputValue"
  when:
    - stack_kafka is defined
    - ( deployment_env != "QA" )

- name: "Set command for env = QA"
  set_fact:
    get_ip_command: >-
      aws ec2 describe-instances
      --filter "Name=tag:StackName,Values=KafkaCluster"
      --query 'Reservations[].Instances[].PrivateIpAddress'
      --output text
  when:
    - stack_kafka is defined
    - ( deployment_env == "QA" )

- name: Get the Kafka InstancesPrivateIps
  shell: "{{ get_ip_command }}"
  register: kafkaip
  when: stack_kafka is defined

- name: Register kafka_host
  set_fact:
    kafka_host: "{{ kafkaip.stdout }}"
  when: stack_kafka is defined

Note1: there is an output redirection in your QA command which will prevent capturing the output as you wish, I left it out in my example.
Note2: I used set_fact to create the command variable but you can put that in your vars or your inventory to have a leaner playbook.


Is it possible to install bitbucket container on Centos? I'm trying to run the below composefile on centos7:
version: "3.3"
services:

  bitbucket:
    image: atlassian/bitbucket:latest
    ports:
      - 7990:7990
      - 7999:7999
    networks:
      - my-app-net
networks:
  my-app-net:
    driver: bridge

But it fails to install it. I took the info from here:
https://hub.docker.com/r/atlassian/bitbucket-server/
did someone manage to install it as a container on Centos?

Since I do not currently have access to a server with Centos installed, I tested my soultion on Ubuntu 20.04. That being said, the whole point of using Docker is to make applications distribution-agnostic, so it should work on Centos as well.
You're getting an image does not exist error, because the image you specified does not, in fact, exist. There is a typo in one line of the README file provided on Docker Hub and you happened to use that line when writing your Compose file.
That being said, incorect image name is not your only issue. You have not specified a volume for your Bitbucket installation, meaning all of your data will be lost if the container is destroyed. Furthermore, the only way to backup a dockerized Bitbucket installation is to use method described here, which requires using external database.
With that in mind the minimal Compose file for Bitbucket is:
version: '3.3'
services:
  bitbucket:
    hostname: bitbucket
    image: atlassian/bitbucket-server:latest
    environment:
     - JDBC_DRIVER=org.postgresql.Driver
     - JDBC_USER=bitbucketuser
     - JDBC_PASSWORD=jellyfish
     - JDBC_URL=jdbc:postgresql://db:5432/bitbucket
    volumes:
      - bitbucket_data:/var/atlassian/application-data/bitbucket
    ports:
      - 7990:7990
      - 7999:7999
    depends_on:
      - db
  db:
    hostname: db
    image: postgres:11.2-alpine
    command: -c 'max_prepared_transactions=64'
    environment:
     - POSTGRES_USER=bitbucketuser
     - POSTGRES_PASSWORD=jellyfish
     - POSTGRES_DB=bitbucket
    volumes:
     - db_data:/var/lib/postgresql/data
volumes:
  bitbucket_data:
  db_data:

That can be further extended with httpd or nginx service used as a reverse proxy, which could also be used to enable HTTPS connections to your Bitbucket instance and an Elasticsearch service to replace the embedded one as well as an email server for Bitbucket to use to send various notifications to users.
I would definitely recommend the SSL-enabled reverse proxy and email server for production. Elasticsearch server is only necessary if you want to run Bitbucket cluster rather than a single instance.


How to execute an ansible playbook using a jenkins pipeline script? We  developed declarative pipeline code to execute an Ansible playbook.
We tested the playbook from CLI and it is working properly.
My playbook actually performs some shell commands on remote server.

SSH configuration between two servers has been done: I have
generated an ssh key and copied it to the remote server, and I am able to ping the server.

The Error msg:

ansible-playbook /var/lib/jenkins/workspace/test1/playbook2.yml -i
/var/lib/jenkins/workspace/test1/inventory.ini -b --become-user root
[WARNING] Ansible is in a world writable directory
(/var/lib/jenkins/workspace/test1), ignoring it as an ansible.cfg source

fatal: [17.14.69.21]: UNREACHABLE! => {"changed": false, "msg": "Failed to
connect to the host via ssh: Permission denied (publickey,gssapi-
keyex,gssapi-with-mic,password).\r\n", "unreachable": true}
[17.14.69.21]:
FATAL: command execution failed
hudson.AbortException: Ansible playbook execution failed.

I have given sudo access to Jenkins user and Jenkins is running as Jenkins user.

If you have Jenkins user public key, on your target hosts authorized keys file, for the user you want to login as, ssh connection authorization should work.

This is an example of Jenkinsfile step.

ansiblePlaybook colorized: true, installation: 'Ansible', inventory: 'inventory', playbook: 'playbook.yml',tags: "" , skippedTags: "", extras: "--extra-vars ' hst=targetHost "

This is Ansible Inventory example. You can define the args at the host level as well.

[targetHost]
17.14.69.21 ansible_ssh_user=sshuser

[all:vars]
ansible_connection=ssh
ansible_ssh_common_args='-o StrictHostKeyChecking=no -oHostKeyAlgorithms=+ssh-dss'


How to customize a Jenkins job dsl method I want to customize the powerShell() method in jobDSL

a .groovy file with some job dsl defined jobs

def betterPowerShell(def cmd) {
    powerShell("$cmd; exit \$LastExitCode")
}

Since I am out of the

job {
  step {
  }
}

context this is not working...

Maybe I need something like...

def betterPowerShell(def cmd) {
    javaposse.jobdsl.dsl.helpers.step.StepContext.powerShell("$cmd; exit \$LastExitCode")
}

What you want is a Jenkins shared library which will allow you to (among other things) define custom Pipeline DSL steps in a repository you own.  See the "Defining custom steps" section in that document.

Here's the example custom step from that document:

// vars/sayHello.groovy
def call(String name = 'human') {
    // Any valid steps can be called from this code, just like in other
    // Scripted Pipeline
    echo "Hello, ${name}."
}


How can I check when a subnet was created? There's an unexpected subnet that isn't anywhere in my project I think it was created manually but I can't find when it was created to help descover what it is for and if it's still needed?

On CloudTrail, select Event history, choose Event name from the first drop-down, and CreateSubnet from the second. Then fill in your timeframe.
If you don't have CloudTrail enabled, you will probably be out of luck, as I don't see a timestamp field available via the EC2 API.
References:
https://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-create-a-trail-using-the-console-first-time.html
https://docs.aws.amazon.com/cli/latest/reference/ec2/describe-subnets.html


how to run an Ansible command for specific number of hosts I am new to Ansible and i have a question about limitting hosts in a host group at running commands.I have these host groups in my inventory:

[primary]
host1
host2
[standby]
host3
host4

i have thses configuration in my .yml:

  ---
  - hosts: primary
  roles:
     - automation

for example i want to run this command ansible -playbook -i /path  test.yml --tags automation on host2 only. what should i do?

Use the --limit option to select a subset of hosts:

ansible-playbook -i /path  test.yml --tags automation --limit host2

Its description from ansible-playbook documentation:

  further limit selected hosts to an additional pattern


Pros and cons of different deploy strategies I am not sure about the best way to deploy my program.
Lets assume i have two relevant branches release(tested and ready to deploy code) and master.
I have two options

1.Merge the release to master and deploy master to production.
2.Deploy release to production and then merge to master.

Which strategy you use and why.

Assuming there are no changes in Master that are not in your release branch and you don't rebuild after you merge the code then you could deploy first and then merge to master.  If either of those are not true, then merging first would be more common.

The process depends on how you handle your branching strategy.  If other things could be merged into master that are not part of the release branch then you need to merge first and rerun the pipeline of tests.  If master represents production and all new work is now in the release branch then yes, you can deploy then push, or do them as a step in the pipeline assuming the deploy is successful then do the merge.


How do I automate deployments with Kubernetes? Assuming I have Kubernetes deployed via Rancher and Jenkins is building new images and pushing them to DockerHub upon check-in of new code to GitHub, how do I automate getting new images deployed?

Another way to ask the question might be, “I used to use Octopus to manage my deployments. Is there something similar built in to Kubernetes or Rancher?” Ultimately, it’s this last gap that I’m struggling with.

As of you are using Rancher the easiest way would be to register custom Rancher Catalog and create an item for each stack/service you want to deploy.

Rancher Catalog is a Git repository with the following structure

Then in Jenkins you can create a job that would call Rancher REST API to deploy/update stack/service.

In its turn Rancher would pull latest version of the Docker image for this service and deploy it corresponding to Docker Compose file from the catalog.

Pros:

generic approach that might be used for almost every app
Jenkins itself might be deployed into Rancher environment, agents might be created in Kubernetes cluster

Cons:

development team should follow solid release strategy to be able to use generic builds
storage drivers are still under the question in alpha release of Rancher 2.0


What is, or could be the formal and/or officially established definion of a "stage"? Many of us use this term in daily business quite intuitively: "build stage", "test stage" and used long before the rise of DevOps.

But is there a formal and/or well-established definition of this term?

Additional question: Who and where used it for first time?

Often in technology we co-opt English words and give them alternate meanings. However, in this case it's really just the standard definition:

  A level, degree, or period of time in the course of a process


What are HiveQL limitations compared to SQL? I'm aware that Apache Hive provides SQL-like interface to query stored data. So I would like to ask, what are the main limitation of Hive-based SQL-like compared to other relational SQL query languages (such as MySQL)?

The Wikipedia page for HiveQL provides a brief summary of the differences:

While based on SQL, HiveQL does not strictly follow the full SQL-92 standard. HiveQL offers extensions not in SQL, including multitable inserts and create table as select, but only offers basic support for indexes. Also, HiveQL lacks support for transactions and materialized views, and only limited subquery support. Support for insert, update, and delete with full ACID functionality was made available with release 0.14.
Internally, a compiler translates HiveQL statements into a directed acyclic graph of MapReduce, Tez, or Spark jobs, which are submitted to Hadoop for execution.

One of the biggest limitations is with transactions, as documented in their wiki - it appears that ACID semantics have been added recently, so the support is not quite as mature as it would be in a typical RDBMS like MySQL.
Hortonworks have also published a cheat sheet for converting SQL to HiveQL - it looks pretty useful for determining the differences between Hive and a normal RDBMS, although it does appear to be slightly outdated (it talks about v0.10 and v0.11, when the latest release as of March 2017 is 2.1.1).


Monitoring tool to microservices on Kubernetes I'm deploying an instance with kubernetes on Azure public cloud, and I need a tool to monitor the microservices and pods inside this instance, the question is, what tool are you using to do that?
I've been searching in a couple of places but without success.

AKS

https://github.com/MicrosoftDocs/azure-docs/blob/master/articles/aks/tutorial-kubernetes-monitor.md

Looks interesting. I have not tried it myself, but based on the more than 300 github stars this looks interesting to me. The readme contains screen shots how to configure it in Azure.


How deploy public key to newly created VM on GCP? I create VM instance using ansible module gcp_compute_instance
My code:

---
- name: Create jenkins node-1
  hosts: localhost
  connection: local
  gather_facts: no
  vars:
    address: <my ip>
    project_id: geocitizen-app
    machine_type: f1-micro
    machine_name: jenkins-node-1
    image: https://www.googleapis.com/compute/v1/projects/centos-cloud/global/images/centos-7-v20191014
    zone: europe-north1-a
    service_email: log@project-app.iam.gserviceaccount.com
    startup_script_url: gs://project/startup.sh
    service_account_scope: https://www.googleapis.com/auth/devstorage.read_only
  tasks:
   - name: Launch instances
     gcp_compute_instance:
       auth_kind: serviceaccount
       name: "{{ machine_name }}"
       machine_type: "{{ machine_type }}"
       project: "{{ project_id }}"
       zone: "{{ zone }}"
       network_interfaces:
       - network:
         access_configs:
         - name: External NAT
           nat_ip:
             address: "{{ node_1_ip }}"
           type: ONE_TO_ONE_NAT
       disks:
       - auto_delete: 'true'
         boot: 'true'
         initialize_params:
           source_image: "{{ image }}"
       tags:
         items:
         - jenkins-server
         - http-server
       service_accounts:
       - email: "{{ service_email }}"
         scopes:
         - "{{ service_account_scope }}"
       metadata:
         startup-script-url: "{{ startup_script_url }}"

At this point I have startup-script. This script deploy my public key to newly created VM instance:

#! /bin/sh
iniFile=/var/opt/inited.ini
if test -f "$iniFile"; then
    exit 0
else
    useradd ssh
    runuser -l ssh -c "mkdir -p ~/.ssh"
    runuser -l ssh -c "echo 'ssh-rsa <my public key>' >> ~/.ssh/authorized_keys"
    runuser -l ssh -c "chmod -R go= ~/.ssh"
    touch /var/opt/inited.ini
fi

Can I add public key when I create VM without using startup script? I read
https://cloud.google.com/compute/docs/instances/adding-removing-ssh-keys But i don't understand how I can use it in my playbook.

As per the link, You can add keys via metadata. You can try the following.

   metadata:
     ssh-keys: "[USERNAME]:ssh-rsa [NEW_KEY_VALUE] [USERNAME]"

Replace [USERNAME] & [NEW_KEY_VALUE] with actual key value.


When to get rid of '-SNAPSHOT' I'm working on a Git workflow (not git-flow) for a Java project that uses Maven and Artifactory for artifact storage. On develop and feature/* branches the version number contains '-SNAPSHOT'. It is not desirable to have release/* or master branches with '-SNAPSHOT' so I'm trying to figure out when (and how) to get rid of it.

I came up with a partial workflow, as below, but it doesn't handle removing 'SNAPSHOT':

                v1.0-snap    v1.0-snap
feature/a       * ---------> *
                ^            |
                |            |
                |            v
develop         *            * v1.0-snap
                v1.0-snap    |
                             |
                             v
release/1.1        v1.0-snap * -----> *
                                    v1.1-snap

I came up with 2 options, but both might be wrong.

Option 1

                v1.0-snap    v1.0-snap
feature/a       * ---------> *
                ^            |
                |            |
                |            v             v1.1-snap
develop         *            * v1.0-snap  *
                v1.0-snap    |           /
                             |          /
                             v         /
release/1.1        v1.0-snap * -----> * v1.1-snap
                                       \
                                        \
                                         \
master                                    * --------> *
                                          v1.1-snap   v1.1

Option 2

                v1.0-snap    v1.0-snap
feature/a       * ---------> *
                ^            |
                |            |
                |            v             v1.1-snap
develop         *            * v1.0-snap  *
                v1.0-snap    |           /
                             |          /
                             v         /
release/1.1        v1.0-snap * -----> * ----- >* v1.1
                                   v1.1-snap    \
                                                 \
                                                  \
master                                             *
                                                  v1.1

Both options seems weird to me, though.

Is there any best practice for this kind of flow?

I have a similar setup.

For feature/* and develop, I use something like 18.08-SNAPSHOT in my pom.
When we're ready to start release hardening, I'll cut a release branch (release/18.08) and change the version in the pom/s to 18.08.

I'll also bump the version on develop to 18.09-SNAPSHOT, for example.

From there, bugfix and finalise the release, merge to master when appropriate.

We're not properly CI/CD yet, but the issue I bumped in to with release builds was that I still needed unique builds in Artifactory.  So, at build time (for release branches) Jenkins will change the version in the pom to something like 18.08-326 (326 being build number).  It won't commit this change, but the artefact that gets built and pushed to Artifactory will be unique (foobar-18.08-326.war)


How to find the PHP version that is used by the latest `phpdoc/phpdoc` docker image? How do I get php version used by docker image phpdoc/phpdoc?

The Dockerfile provided in the Q&A indicates that the image uses the latest php:7 as a base image. One could inspect the version as follows:

docker run -it php:7 php -v

Result

PHP 7.3.11 (cli) (built: Oct 25 2019 02:24:51) ( NTS )
Copyright (c) 1997-2018 The PHP Group
Zend Engine v3.3.11, Copyright (c) 1998-2018 Zend Technologies

How to use an older version of php in conjunction with this container?

It is possible to use a specific version by checking the tags of php on dockerhub. According to this list, 7.2.25 is currently the latest 7.2. If one changes the from of the Dockerfile to FROM php:7.2.25 and build the image again, then 7.2.25 instead 7.3.11 will be used inside the container.


How to use comments to trigger rebuilds of Bitbucket pull-requests on Jenkins multi-branch pipeline I have successfully configured a Jenkins multi-branch pipeline to build my project which is hosted on Bitbucket. I have separate pipeline requirements for the "develop" and "release" branches, and for merge requests. These are now working correctly (except for a surplus "PR-***" job whenever a Bitbucket PR is triggered).

My remaining issue is:
In old-style Jenkins I used Bitbucket Pull Request plugin, which had a feature "Comment phrase to trigger build". So in the event of a flaky unit test etc I could issue the trigger phrase as a comment in the pull request, and it would be rebuilt.

How can I achieve this behaviour on a multi-branch pipeline?

I personally use the Pull Request Notifier plugin for Bitbucket to trigger my Bitbucket Multibranch Pipeline builds in Jenkins.  Using the GUI configuration tools, you can configure the Pull Request Notifier plugin to trigger a build on any arbitrary PR action, including adding comments.

However, you don't really need to use comments to trigger builds - there is a build trigger button in the PR view in Bitbucket, and you can also use the "Build Now" button in Jenkins to trigger a PR build.  Both of these seem a little bit easier to me than a magic comment.


Share variables across instances I'm writing TestInfra test scripts for mongo cluster instances using Molecule Ansible. I want to share variables across all the instances. I created a python class, and initialise and update the variables. Every time molecule verify, each test will run on each instance, so variables are independent on single instance.

Is it possible to share variables across instances?

Q: Is it possible to share variables across instances?

A: Yes. The dictionary hostvars keeps the variables. Quoting from Accessing information about other hosts with magic variables

  hostvars lets you access variables for another host, including facts that have been gathered about that host. You can access host variables at any point in a playbook. ...

{{ hostvars['test.example.com']['ansible_facts']['distribution'] }}

For details see Scoping variables and Caching facts.

With “Fact Caching” disabled, to share information among Ansible playbooks, it's possible to store all hostvars in a file. For example with this template

  $ cat my_hostvars.json.j2
  my_hostvars_all:
  {% for my_host in ansible_play_hosts_all %}
    {{ my_host }}:
      {{ hostvars[my_host]|to_nice_json }}
  {% endfor %}

the playbook below stores hostvars of all hosts in the dictionary
my_hostvars_all and put it into the file
{{ inventory_dir }}/my_hostvars.json at localhost(master)

  - hosts: test_jails
    tasks:
      - set_fact:
          test_var: "test_var_in_{{ inventory_hostname }}"
      - template:
          src: my_hostvars.json.j2
          dest: "{{ inventory_dir }}/my_hostvars.json"
        delegate_to: localhost
        run_once: true

The dictionary can be included in the next playbook. For example the playbook below

  - hosts: test_jails
    tasks:
      - include_vars: my_hostvars.json
      - set_fact:
          my_hostvars: "{{ my_hostvars_all[inventory_hostname] }}"
      - debug:
          var: my_hostvars.test_var

gives

  ok: [test_01] => {
      "my_hostvars.test_var": "test_var_in_test_01"
  }
  ok: [test_02] => {
      "my_hostvars.test_var": "test_var_in_test_02"
  }
  ok: [test_03] => {
      "my_hostvars.test_var": "test_var_in_test_03"
  }


Multiple Microservices in CI environment hey we have a GitLab stage and inside that we need to start an environment with several containers we pull from a local registry.

We use compose and a docker in docker solution for that at the moment.

The problem is, that the compose.yml now hold all the configuration for each service.

So if we change it for one service, we need to update it in each docker-compose.yml in each project.

What is the standard approach for such a setting?

Thanks

So if we change it for one service, we need to update it in each docker-compose.yml in each project.

That's sounds like duplicate code and there are many ways to avoid that. But without more detailed information about how your setup looks like it's difficult to help.

Within a pipeline you can access other projects and their source code. Maybe Git Submodules is an option. Or you might be interested in the include keyword of GitLab CI.


Jenkins Artifactory plugin not downloading in multi-thread In my company we're having a bad time trying to download with the Artifactory plugin in Jenkins because we have a lot of tiny files in our repository and apparently the plugin works in a single thread, so it takes a REALLY long time. We tested the JFrog CLI where we can set the number of threads for the download and that actually works great. Still, we really need to use the original plugin.

Is there any workaround for this? Maybe it has something to do with the Artifactory server configuration?

Note: We have so many small files because that's how the development is being done in the company I'm afraid. We thought about archiving all the files before we store them but that would probably be non-economical in terms of storage, since every uploaded zip file would probably have a different checksum.

You could try to run the artifactory file upload in parallel if you are using the Jenkinsfile syntax: https://github.com/jenkinsci/pipeline-examples/blob/master/pipeline-examples/parallel-from-list/parallelFromList.groovy

Here is a simpler example to run things in parallel if you only have a fixed number of things you want to do in parallel:

parallel (
    "task1" : {
        //runTask1
    },
    "task2" : {
        //runTask2
    },
    "task3" : {
        //runTask3
    },
)

This can be enclosed inside a node, in which case they would all share the same workspace or each command can define node inside the task code to run into another node in which case it would have a different workspace for each task.


How do I delete a default AKS node pool when I get an error saying it's the "backend address pool of the basic load balancer" I'm attempting to replace a node pool in Azure Kubernetes Service with a new pool (that has a larger VM size).  The current pool was deployed using an ARM template that uses version 2019-06-01 of the resource manager API.  As per the Microsoft documentation I should be able to add a new System node pool and delete the original, default node pool.

I can successfully deploy an additional node pool that has it's mode set to System using this template, that template uses version 2020-03-01 of the resource manager API so that I can set the mode of the new pool.

When I attempt to delete the original, default node pool using az aks nodepool delete -g gandt-testshared-rg --cluster-name gandt-testshared-aks -n agentpool I get an error saying Operation failed with status: 'Bad Request'. Details: The agent pool cannot be deleted because it is used as the backend address pool of the basic load balancer..

Why am I getting this error and what can I do to get the backend address pool of the basic load balancer to use the new node pool?

Update

I've deleted the cluster and redeployed using the original ARM template then added the 2nd node pool using az cli command az aks nodepool add -g gandt-testshared-rg --cluster-name gandt-testshared-aks --name poolb2ms01 --node-count 1 --node-vm-size Standard_B2ms --mode System and got the same result when deleting the default node pool

If my assumption is correct that's related to having a loadbalancer that wasn't correctly deleted.
Which could be resulted by redeploying a service (of type: load balancer) in a short time before Azure cleans up the previously associated loadbalancer resource.
To my knowledge, that could be fixed with the rolling of Kubernetes 1.17 and the Loadbalancer Finalizer feature, through this pull.
please update the question if that showed up with the new cluster version.


Is there a way to use a Jenkinsfile from a git-submodule in a multibranch pipeline? Background:
The nature of our project's source involves multiple long-living branches, each one has its own Jenkins job (manually created).

Multibranch limitation:
Unfortunately we do not use a Multibranch Pipeline to organize the project, because when (for example) a change is needed in the Pipeline, it would need to be added (manually) to each of the long-living branches' Jenkinsfiles.

Current implementation:
We store the Jenkinsfile in a separate repository (let's call it ci-repo), which is added as a submodule to the project's repository (let's call that dev-repo).

What still hurts:
We need to manually monitor branches of the dev-repo and manage job creation.
Multibranch pipelines support submodule checkout for source files, but branch discovery requires presence of a Jenkinsfile under the parent repository (in our case dev-repo).

The question:
Jenkinsfile from git-submodule of Multibranch Pipeline - Is there a way to do this?
Perhaps using an initial/mock Jenkinsfile in the dev-repo, with one stage in which is calls the Jenkinsfile from the submodule?
Can a Jenkinsfile be over-written or replaced somehow during runtime?

I agree with Omri that you are heading the wrong way. I would recommend using a shared library which implements the pipeline. You would then have a simple Jenkins file which looks all the same. See https://jenkins.io/blog/2017/10/02/pipeline-templates-with-shared-libraries/

Let's say you have a shared library which contains a vars/commonpipeline.groovy which contains something like this:

def call(body) {
    // evaluate the body block, and collect configuration into the object
    pipelineParams= [:]
    body.resolveStrategy = Closure.DELEGATE_FIRST
    body.delegate = pipelineParams
    body()

    pipeline {
        agent {
            label pipelineParams.nodes
        }
        options {
            buildDiscarder(logRotator(artifactNumToKeepStr: '1', numToKeepStr: '10'))
            disableConcurrentBuilds()
            timestamps()
            timeout(time: pipelineParams.timeout, unit: 'MINUTES')
        }

        stages {
            stage('Build') {
                steps {
                    //TO SOMETHING
                }
            }
        }
    }
}

Then you can have a Jenkinsfile in your dev project which loads the library and uses the common pipeline like this:

import org.apache.commons.io.FileUtils
library identifier: 'shared-pipeline-library@master', retriever: modernSCM(
  [$class: 'GitSCMSource',
       remote: 'https://bitbucket/jenkins/hared-pipeline-library.git',
       credentialsId: 'bitbucket.service.user'
  ])

commonpipeline {
    nodes     = 'BUILD' /* label of jenkins nodes*/
    timeout   = '60'
}


Pod object and Pod’s health Couple of questions here: a) what the concept of Pod object satisfies? There is the  pause container which holds the namespace for the actual ‘app’ container(s) that will get added later and use that namespace. So, what operations in real life qualify to have more than one containers running in a Pod ? Can you provide to the point examples? What are considered sidecar operations after all?

b) what is the threshold for e.g. N containers in the Pod that master will be able to tell that Pod is not healthy? So basically, how many containers in a Pod need to be down to mark the Pod down?

the most common example would be some sort of logging sidecar.  Depending on your implementation or your service you may want a service that manipulates the files for the pod etc.  Some databases can use sidecars for log shipping in replication, or you might see some process to clean up log files as well.  Although available, those options are not suggested specifically because there are more alternatives.


How to use same Helm dependency in many charts? For example, I have this charts structure:

├── parentChart
│   ├── charts
│   │   ├── childChart1
│   │   │   ├── charts
│   │   │   │   └── postgresql-3.11.6.tgz
│   │   │   ├── Chart.yaml
│   │   │   ├── requirements.lock
│   │   │   ├── requirements.yaml
│   │   │   ├── templates
...
│   │   │   └── values.yaml
│   │   ├── childChart2
│   │   │   ├── charts
│   │   │   │   └── postgresql-3.11.6.tgz
│   │   │   ├── Chart.yaml
│   │   │   ├── requirements.lock
│   │   │   ├── requirements.yaml
│   │   │   ├── templates
...
│   │   │   └── values.yaml
│   └── Chart.yaml
...

In the childChart1 and childChart2 I have this dependency:
Child1:

dependencies:
  - name: postgresql
    version: 3.11.6
    repository: alias:stable
    alias: child1-postgres
...

Child2:

dependencies:
  - name: postgresql
    version: 3.11.6
    repository: alias:stable
    alias: child2-postgres
...

First problem: I expect that after deploy parent chart I will have 4 deployment:

childChart1
childChart1 Postgresql
childChart2
childChart2 Postgresql

Am I right?

If I don't do anything wrong with a first problem, why I see this message when I trying to deploy this charts:

$ helm install $opts --name $NAME --namespace $NAME $package --wait --timeout 9999
Error: release test failed: secrets "test-postgresql" already exists

P.S. I have this helm version:

$ helm version
Client: &version.Version{SemVer:"v2.11.0", GitCommit:"2e55dbe1fdb5fdb96b75ff144a339489417b146b", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.11.0", GitCommit:"2e55dbe1fdb5fdb96b75ff144a339489417b146b", GitTreeState:"clean"}

Finnaly, I found the solution, thanks to @simbo1905.

All that I need to fix that problem:
values.yaml Child Chart 1 (or two)

postgresql:
  nameOverride: chart-1-postgres

In the Child Chart 2 nameOverride became to chart-2-postgres.


How can I install Jenkins on Azure? Currently I have Jenkins installed on Windows Server as a Service but I am looking at installing Jenkins on Azure but the MSDN docs talk about creating a Jenkins server on an Azure Linux VM from the Azure portal, outlined in the following link:
https://docs.microsoft.com/en-us/azure/jenkins/install-jenkins-solution-template.

Is this the only way or can I install it on a Windows VM and run it as if it was installed on a local Windows machine ?
Thanks

If I understand your question correctly then yes, you can create a windows VM in azure and install jenkins on it the same way as on a local machine.

They're might be some differences in the configuration of the environment for example about security, user access, routing etc


Jenkins Declarative Pipeline: Can Jenkins execute a dynamic list of jobs? Context: I have a long dynamic list of builds calls that I am currently executing in parallel successfully. Example below.
Problem: In some specific moments it is required to run that list of builds but in a sequential fashion.
Question: Is there any way to reuse the list of jobs that I already have generated for the parallel execution, but to execute them in a sequential way?
stage ('Execute jobs') {
    steps {
        node('node1') {
            script {
                parallel_jobs = [:]
                if (params.includeJob1 == "true") {
                    parallel_jobs['ci'] = {
                        build job: job1, parameters: [], propagate: true
                    }
                }
                parallel_jobs['second'] = {
                    build job: job2, parameters: [], propagate: true
                }

                // ...
                // and more jobs dinamically added to the parallel_jobs list
                // ... parallel_jobs['etc'] = ...

                parallel parallel_jobs
            }
        }
    }
}

I am thinking to replace the 'parallel' call for any other Jenkins command, but cannot find the correct one at docs (https://www.jenkins.io/doc/book/pipeline/syntax/#parallel).
Thank you.

I'm not really sure I understand your question, but you can execute arbitrary Groovy inside of a script block, so you can do just about anything within script.
For example, here is my interpretation of what you are asking for:
stage ('Execute jobs') {
    steps {
        node('node1') {
            script {
                parallel_jobs = [:]
                if (params.includeJob1 == "true") {
                    parallel_jobs['ci'] = {
                        build job: job1, parameters: [], propagate: true
                    }
                }
                parallel_jobs['second'] = {
                    build job: job2, parameters: [], propagate: true
                }

                // ...
                // and more jobs dinamically added to the parallel_jobs list
                // ... parallel_jobs['etc'] = ...

                if (params.parallel) {
                    // execute jobs in parallel
                    parallel(parallel_jobs)
                } else {
                    // execute jobs in sequence
                    parallel_jobs.each { name, closure->
                        closure.call()
                    }
                }
            }
        }
    }
}

In Groovy, Maps (i.e. associative arrays) are ordered.  So if you use this to execute the builds serially rather than in parallel, the builds will be executed in the order they are defined.


Mesos-slave not picking up attributes I am running a x-mesos-slave container with these attributes:

"Cmd": [
    "--attributes=farm:app,region:eu-west-1,zone:eu-west-1b"
],

When I run an application in Marathon with farm:UNIQUE, this works fine.

However when I try to use zone:UNIQUE, the application gets stuck in waiting.

This makes it seem there is no region/zone configured.

As per https://mesos.readthedocs.io/en/0.22.2/attributes-resources/ states, attributes are separated by ';' not by ','.


How to set git-lfs as an additional checkout behavior using the JCAC plugin? We are using the Jenkins Configuration As Code (JCAC) plugin to automate the jenkins configuration. One of the challenges is to set git-lfs as an additional checkout behavior.

GitLFSPull has to be configured, but it does not seem to be included in the DSL:

The README of the JCAC github repository indicates that support questions should be asked on Gitter

According to this Q&A, GitLFSPull should be an extension:

extensions: [[$class: 'GitLFSPull']]

However, querying for this on the Job DSL page shows that GitLFSPull seems to be omitted:

It seems that it is only possible by using a workaround as depicted in this answer.

After setting gitLFS in the UI, subsequently inspecting the /var/lib/jenkins/jobs/some-job/config.xml file it became obvious that the XML looked as follows:

<traits>
   <jenkins.plugins.git.traits.GitLFSPullTrait>
      <extension class="hudson.plugins.git.extensions.impl.GitLFSPull"/>
   </jenkins.plugins.git.traits.GitLFSPullTrait>
</traits

After defining jenkins.plugins.git.traits.GitLFSPullTrait:

jobs:
  - script: >
      multibranchPipelineJob("example") {
        branchSources {
          ...
        }
        configure { node ->
          node / sources / data / 'jenkins.branch.BranchSource' / source / traits {
            'jenkins.plugins.git.traits.GitLFSPullTrait'()
          }
        }
      }

Git-lfs was enabled.


Managing state (e.g. presence/absence) in Ansible, any established best practices? Ansible lists a nice set of best practices in its documentation. One of the items is:

Always mention the state
For many modules, the state parameter is optional. Different modules have different default settings for state, and some modules support several state settings. Explicitly setting state=present or state=absent makes playbooks and roles clearer.

In my case the main point isn't so much about making playbooks or roles clearer, instead my goal is to manage the presence/absence of items throughout the lifetime of my servers.
Probably many of you have had the issue of installing package xyz as a "bread'n'butter" package on your machines, but a few months later it turns out your workflow has changed and you don't really need xyz anymore. Or it was replaced by xyz-ng ... or what have you ...
So now it's time to express somehow to remove that xyz package consistently across your infrastructure. In my case with Debian/Ubuntu I'd use the apt module with state=absent for the respective packages. No surprise.
However, I'd like to factor this into my roles from the outset. As an example it would be nice to have a dictionary variable (as an example):
packages:
  present:
  - elinks
  absent:
  - lynx

This way the presence and absence would dynamically be managed by variables and I can use when: to cover for an empty list and so on; potentially even by including a task dynamically and passing along a state variable that coincides with the key in the dictionary.
Another case where this must have come up for someone before is the creation of local users, or has everyone moved to directory services? When someone leaves the company I'd like to be able to put that someone on a list populated with usernames to be put into a locked state.
However, since there are established best practices for Ansible in general, I was wondering:
Q: are there best practices about managing the state of "entities" on the configured infrastructure?

This can be done with two tasks with loops:
# tasks/packages.yml
- name: pkgs | Ensure Good
  package:
    name: "{{ item }}"
    state: present
  loop: "{{ packages['present'] }}"

- name: pkgs | Ensure Bad
  package:
    name: "{{ item }}"
    state: absent
  loop: "{{ packages['absent'] }}"

This will loop over all of the items in the list that you define and ensure that they are either present or absent.
You could also use a shorthand:
# tasks/packages.yml
- name: pkgs | Ensure Good
  package:
    name: "{{ packages['present'] }}"
    state: present

- name: pkgs | Ensure Bad
  package:
    name: "{{ packages['absent'] }}"
    state: absent

This would run faster, since you don't have a list of tasks, but a task with a list, but you would lose some visibility into which packages you're actually ensuring the state of.
Caveats:
Ensure that there is no overlap between the two lists. One way would be to make an assertion on the lists before you run the task:
- name: ensure no overlap between good and bad
  assert:
    that: "{{ packages['present'] | intersect(packages['absent'] }} == [] }}"

Also ensure that the lists are not empty.
To answer your final question:

Q: are there best practices about managing the state of "entities" on the configured infrastructure?

It is usually my practice (I wouldn't call it "best", but take it for what it's worth) to declare the explicit state of a resource. It's quite difficult to do this exhaustively for packages that should be absent - this is something akin to maintaining a "blacklist". One approach might to be start from a policy and implement specific controls in it. IE for ssh config ensure that protocol v1 is absent, etc.


is there a standard or common approach to defining Service Level Indicators? I've come to the point where I am starting to adopt principles from Site Reliability Engineering to operate Cloud Native applications in a modern way.

From my reading to date, I have established that one of the core practices is monitoring. The way that SRE approaches monitoring is by defining Service Level Indicators (SLI) which measure an aspect of the end-user experience, then identifying a threshold around the SLI which is called a Service Level Objective (SLO).

is there a standard or common approach to defining Service Level Indicators?

Whilst not strictly a standard approach, Google has published an SLI Menu and a process for developing SLIs for user journies:

For each User Journey/Data Flow identify from the SLI Menu suitable types of SLI:

Make a decision about how to measure good and valid events,
Decide where to measure the SLI From out of the following: End User, Client-side Instrumentation, Synthetic Clients, Front-end Metrics, Application Metrics or Server-side Logging.

The document then goes on to describe, how you collate all of your SLIs for all of your user journies then walk the journey to look for coverage gaps.  Then finally set SLOs based upon the SLIs either based upon business need or past performance.


Is it useful to track dev time with CI/CD? Estimating is hard, but it is made easier with good record keeping.
Is there a known solution to tracking dev time on a ticket by making use of CI/CD to update JIRA tickets or the like with logged worked time, based on the time difference between first commit and final CD?

You can use the rich jira rest API for this kind of automation work. Jira API provides mechanism to update logged hours using PUT /rest/api/2/issue/{issueIdOrKey}/worklog/{id}

You can use this rest API endpoint to update the corresponding issue with the difference of first commit timestamp and first successful CD.

Find more about it here.


Tips for separating DevOps Git repos? We currently have a Git repo with automation code (with tests), iaas-code and lamba-code.

The automation code is mostly Fabric, Python and unittests.
The Iaas-code is mostly CloudFormation, Docker-compose and some Terraform.
The AWS lamdas are Serverless code and some Fabric.
There is also some code with Dockerfiles that builds a Docker image from this Git repo.

We're considering separating git repos and using git submodules as it seems much cleaner. Do you have any tips for this process?

My concern is that most tips regarding mono-repos vs. separate repos are mostly for programmers that build micro-services and not for DevOps / SysAdmins.

When considering mono-repo vs many-repos you might look at the following questions, which should also affect the composition of the code directory tree structure. There is a slight general trend toward mono-repos.

Discoverability

How easy it is for developers and ops people to discover the code for some feature. Are there clear guidelines on what goes where? Are they enforced somehow through automation? Are new repos automatically linked as submodules into correct location? Does the build or deploy process depend on the tree structure making sure people need to put it there in order for it to be even noticed? What is the size? How long it will take to grep code in certain place? Are there generic lib/ directories where devs and ops end up sharing libraries that create new unexpected inexplicit dependencies?

Deployability

Does any of the code need to be present on machines that you deploy to? Are you expecting to do such deployment through syncing repository or do you have other established deployment mechanisms. In the first case, will other code creep in during the deployments that does not need to be there?

Durability

Will the concept you are using stand the test of time? Are you a one product shop, but maybe in your future there are new products? How would adding a new product affect the code? Will there be new mono-repo for the new product? What about shared code between products? Looking at your company roadmap, how will whatever you do fare 6m, 2y, 5y down the road? Choices like this tend to be hard to change in practice (even though they shouldn't - in theory).

DevOps

You really want to take down barriers between Devs and Ops. So even just having separate mono-repo for ops could create such barrier. The operation and deployment of code, standing up environments, setting up services, it should all be in reach of developers and with explicit expectations to develop such code in coordination with the ops team, not being dependent on someone doing it for them. If things go well your developers follow through with code for months after it has already been in production and have hand in all aspects of the deployment. How easy it is to make commits that contain both product and deployment changes? How can those commits be grouped together? What is the visibility of ops into product runtime code and devs into product management code?