How can I deploy an AWS Lambda function from Jenkins? I have a simple Lambda function that I'm trying to deploy through Jenkins -

public String handleRequest(String input, Context context) {
        String output = "";
        if (input.isEmpty()) {
            output = "No input provided";
        } else {
            output = "Hello, " + input + "! Checking invocation - 1";
        }
        return output;
    }

I'm able to deploy and invoke this through Eclipse's AWS Lambda plugin without any problems.

I'm using the AWS Lambda plugin for Jenkins and following their documentation.

I'm providing my Git repository as the source.
Artifact Location - src/main/java/
Handler Name - lambda.Hello (lambda is the package name and Hello is the class name). I've also tried using lambda.Hello.handleRequest,  lambda.Hello::handleRequest and other variations.

Jenkins says that the build was successful but when I test it on the AWS console, I'm getting -

"errorMessage": "Class not found: lambda.Hello",
  "errorType": "class java.lang.ClassNotFoundException"

Where am I going wrong here? When I export the test function from AWS and unzip it, I can see that code on Git definitely got deployed but it's unable find the class.

I found the solution while trying to manually deploy the Lambda function as a jar file. These are the steps -

Create your Lambda function as a Maven project using the AWS documentation.
Create your Jenkins job as a Maven project and specify package in the goals section.
Follow the Jenkins Lambda plugin documentation for deployment and specify target/your-project-1.0-SNAPSHOT.jar as the Artifact location.

This should build your Lambda function with Maven and then deploy it to AWS.


Jenkins input pipeline step - Give submitter as a group I could find that “submitter” parameter accepts individual user IDs and external LDAP groups. But “Jenkins-local” groups defined by any authorization strategy are not working. I have a limitation not to use external LDAP groups.

The workaround that I have used is define the comma separated list of individual users as a Jenkins global variable and manage the list from there. This gives a better maintainability. However, it mandates that the stage is wrapped within node. As the pipeline waits for the approver to approve this step, it holds a node instance and this causes the rest of the jobs to wait for available instances. Is there a way to use a Jenkins global variable outside of node for the input step?

node{
stage ('Approve?')
{
    timeout(time:2, unit:'DAYS')
    {
        input message: 'Can you approve this ??', submitter: MY_APPROVERS
    }
}

}

We have moved onto using Jenkins templates and I have a workaround to keep the individual approver names list in one place. Sharing the sample code for your reference.

#!/usr/bin/env groovy
import com.mm.Constants
def call(String teamName = 'LOGIN') {
   // team name of null means UI
   teamName = teamName ?: 'UI'

   timeout(time:2, unit:'DAYS')
       {
           input message: 'Can you approve this ??', submitter: Constants."${teamName}_APPROVERS"
      }

}

#!/usr/bin/env groovy
class Constants {
   static final LOGIN_APPROVERS = 'approver1,approver2';
   static final UI_APPROVERS = 'approver2,approver3';
}

Below is how you call from the pipeline

stage ('Ship to QA?'){
    echo "Waiting for QA approval"
    shipToQA('LOGIN')
}


Creating a Separate Technical and Business Versioning System A common problem at my company is versioning. We follow a versioning system formatted very similar to semantic versioning (but not by definition). Our current versioning is not only used by development, but also by marketing/sales in their release notes, advertising, etc. Often times, there are conflicts as to what the versioning should be. What the development team defines as a major change (non-backwards compatible api change) is often not considered a "major" change by marketing. Other times, marketing will push to increment the major because their is a significant customer workflow change, but from a development perspective, the change is very insignificant (often times just an application code patch).

I am pushing heavily to separate our versioning into two distinct systems, a business versioning and a technical versioning system. This would allow both teams to define their own rules for versioning that best suites their needs. While the feedback in general is positive, some of the concerns are the confusion that this will cause, in our cross team communication and ticketing systems.

What are some of the complications of having two versioning systems? What are some of the benefits of a separation? What kind of impact will this have on the teams and the company as a whole?

It seems to me that your customers should be dictating the versioning. Both teams seem to be heading towards the same thing but only partially understanding the customer impact. For development, when they significantly re-architect components, there is risk because this could corrupt data or leave core features in a malfunctioning state - this should be reflected by a major version number change.

This doesn't mean much to marketing - everything looks the same, so why would the customer notice this change? No need to reflect this as a major change... unless you are a developer, in which case you risk a huge amount support interactions over those bugs.

Similarly marketing is looking at the user interface with the assumption that with a significant overhaul to the GUI, customers might get lost or hate the new user interface. This could lead to significant support interactions helping customer navigate the new UX, so should be reflected as a major version number change to the customer. There is a lot of risks there... Unless you are developer. Then this is purely cosmetic. Everything is still functioning the same on the back end.

So it seems you need to decide what it is that you feel version numbering is for.

Are version numbers used to signal risk to customers?
What source control system are you using
Does Development even need version numbers at all or are commit IDs sufficient?
Does Development plan features against version numbering for the purposes of setting deadlines?

Depending on the answers, it might be ideal to use a single version numbering system led by marketing - but marketing will need to do a better job of understanding the risks of under-the-hood changes. If this is needed as a designator for setting deadlines for change integrations, then two version numbering systems might be approperite:

Side note: git tags would be great for any version numbering scheme you have mentioned.


How to update nodejs on Jenkins master node? I am in the middle of trying to implement a build/deploy job in Jenkins by which the latest version of source code is pulled from git, built and then deployed to GCP. So far I am able to retrieve the lastest code from GIT but have run into issues when tyring to build. The application being built/deployed is an Angular 7 web site using npm for package management. So the normal steps would be to run npm install to make sure the packages are installed and then ng build -- prod to build the app.

My issue is that the version of npm/nodejs in the master node is out of date and not compatable with Angular 7. Even just trying the command to gert the current ng version fails:

+ /usr/bin/ng v
You are running version v8.1.4 of Node.js, which is not supported by Angular CLI v6.
The official Node.js version that is supported is 8.9 and greater.

I added a call to npm version to check the npm version is is trying to use:

+ echo current user
current user
+ whoami
jenkins
+ npm -v
5.0.3

To investigate the issue I used putty to remote onto the jenkins box and ran npm -v and ng v to see what is installed and get:

jenkins@****-int1:~$ npm -v
6.5.0-next.0

Which is different to that reported by npm -v in the shell script.

jenkins@****:~$ ng v

     _                      _                 ____ _     ___
    / \   _ __   __ _ _   _| | __ _ _ __     / ___| |   |_ _|
   / △ \ | '_ \ / _` | | | | |/ _` | '__|   | |   | |    | |
  / ___ \| | | | (_| | |_| | | (_| | |      | |___| |___ | |
 /_/   \_\_| |_|\__, |\__,_|_|\__,_|_|       \____|_____|___|
                |___/

Angular CLI: 7.1.4
Node: 11.6.0
OS: linux x64
Angular:
...

Package                      Version
------------------------------------------------------
@angular-devkit/architect    0.11.4
@angular-devkit/core         7.1.4
@angular-devkit/schematics   7.1.4
@schematics/angular          7.1.4
@schematics/update           0.11.4
rxjs                         6.3.3
typescript                   3.1.6

So it should have nodejs 11.6, I ran node -v to double check:

jenkins@****:~$ node -v
v11.6.0

BUT if I run nodejs -v I get:

jenkins@****:~$ nodejs -v
v8.1.4

I thought that node and nodejs were the same thing? so why would I have two different versions installed and how to fix this?

To double check what my Jenkins script is using I added a call to node -v in the script and as expected it reports the older version:

+ node -v
v8.1.4

I'm not a DevOps person, I'm C# back end developer who's been lumbered with sorting this out due staff changes. I know very little about Linux, Jenkins or this stuff in general.

I tried to update the node installation via the Jenkins Shell script using

sudo npm cache clean -f
sudo npm install -g n
sudo n stable

But these were refused due to access rights (no sudo pwd I presume) and I don't know if that would've fixed the issue anyway.

Any help?

I would have a look at the Tool Auto-Installation which allows Jenkins to install the tooling you need.

  Lets you configure tools so that agents will install them on demand whenever running a job that needs them. This could be especially useful when running a large farm of agents in a cloud which all start with a minimal operating system configuration.

Speaking of that also checkout the NodeJS Plugin which

  Provides NodeJS auto-installer, allowing to create as many NodeJS installations "profiles" as you want.
  The auto-installer will automatically install a given version of NodeJS, on every jenkins slave where it will be needed
  Allows to install globally some npm packages inside each installations, these npm packages will be made available to the PATH


Update volume size dynamically from inside a POD I was wondering how I could dynamically create and then mount new volumes from inside a pod ? At runtime !
I have to download a lot of files with very different sizes and would like the storage to adapt to my needs...

Use case example :
There is a decompression step. But I don't know how much will be decompressed before its completed.
If I could allocate some more memory at runtime It would not end with a DiskFull error...
I can mount an initial volume at pod startup, then either I find a way to update this volume size or at least mount a new one in the same pod !
Any ideas are welcome
Thx

In my opinion, you could create bucket (aws s3 for example) and store your data there (create simple app to make upload), thus, you don’t need to care with disk full error


Google K8s Mirror offline? Heys dudes,
More someone got error when trying update cache of the Google K8s repositories?
I have been got the error about NO_PUBKEY
Err:10 https://packages.cloud.google.com/apt kubernetes-xenial InRelease
  The following signatures were invalid: EXPKEYSIG 6A030B21BA07F4FB Google Cloud Packages Automatic Signing Key <gc-team@google.com> The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 8B57C5C2836F4BEB

and
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: https://packages.cloud.google.com/apt kubernetes-xenial InRelease: The following signatures were invalid: EXPKEYSIG 6A030B21BA07F4FB Google Cloud Packages Automatic Signing Key <gc-team@google.com> The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 8B57C5C2836F4BEB
W: Failed to fetch http://apt.kubernetes.io/dists/kubernetes-xenial/InRelease  The following signatures were invalid: EXPKEYSIG 6A030B21BA07F4FB Google Cloud Packages Automatic Signing Key <gc-team@google.com> The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 8B57C5C2836F4BEB
W: Some index files failed to download. They have been ignored, or old ones used instead.

Something was changed?

As per compute/docs/troubleshooting/known-issues:

On Debian and Ubuntu based systems, including your local workstations,
you might encounter an error similar to the following example:
[...] The following signatures were invalid: XXXX [...]

To resolve this error, get the latest valid apt-key.gpg key file from https://packages.cloud.google.com by running the following command:
curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -


Install MDB2 in docker container while PEAR is down While pear is down, the following instruction in my dockerfile is failing:

RUN pear install -o --alldeps MDB2-beta && \

I've searched on the internet how to install MDB2 without pear, successfully accomplished it by downloading the .deb file from https://packages.ubuntu.com/search?keywords=php-mdb2 and then doing dpkg -i php-mdb2_2.5.0b5-2_all.deb

My issue is that the same instructions are failing when placed in the dockerfile. Here's the instructions:

RUN wget http://mirrors.edge.kernel.org/ubuntu/pool/universe/p/php-mdb2/php-mdb2_2.5.0b5-2_all.deb -O php-mdb2_2.5.0b5-2_all.deb
RUN dpkg -i php-mdb2_2.5.0b5-2_all.deb

Here's the error message:

  dpkg: dependency problems prevent configuration of php-mdb2: php-mdb2
  depends on php-common; however: Package php-common is not installed.
  php-mdb2 depends on php-pear (>= 1.3.6); however: Package php-pear is
  not installed.

I've tried to manually search and install all dependencies, and dependencies' dependencies but failed at the end because one of them is being told to me that it'll break locales thus cannot be installed.

Also tried another solution I found in stackOverflow to install dependencies for dpkg but it also fails. Instructions:

RUN wget http://mirrors.edge.kernel.org/ubuntu/pool/universe/p/php-mdb2/php-mdb2_2.5.0b5-2_all.deb -O php-mdb2_2.5.0b5-2_all.deb
RUN dpkg -i php-mdb2_2.5.0b5-2_all.deb || true && \
apt-get -f install

Error received:

  dpkg: error processing package php-mdb2 (--install): dependency
  problems - leaving unconfigured Errors were encountered while
  processing: php-mdb2 Reading package lists... Building dependency
  tree... Reading state information... Correcting dependencies... Done
  The following packages will be REMOVED: php-mdb2 0 upgraded, 0 newly
  installed, 1 to remove and 9 not upgraded. 1 not fully installed or
  removed. After this operation, 528 kB disk space will be freed. Do you
  want to continue? [Y/n] Abort. ERROR: Service 'app' failed to build:
  The command '/bin/sh -c dpkg -i php-mdb2_2.5.0b5-2_all.deb || true &&
  apt-get -f install' returned a non-zero code: 1 Makefile:8: recipe for
  target 'build' failed make: *** [build] Error 1

Can you point me in the right path maybe?

You need to learn a bit more about Debian or Ubuntu. If you look at every single Debian or Ubuntu container that installs things, you will notice, amongst other things:

apt-get update && apt -f install [whatever]

The reason for this is that aptitude doesn't know where to find anything without the "apt update".

See for example  https://github.com/dockerfile/ubuntu/blob/master/Dockerfile

# Install.
RUN \
  sed -i 's/# \(.*multiverse$\)/\1/g' /etc/apt/sources.list && \
  apt-get update && \
  apt-get -y upgrade && \
  apt-get install -y build-essential && \
  apt-get install -y software-properties-common && \
  apt-get install -y byobu curl git htop man unzip vim wget && \
  rm -rf /var/lib/apt/lists/*

This updates all packages, then installs build-essential, then software-properties-common and then the packages in the second to last line. Of course you don't need all of this.

It is important however to have the apt-get update && apt-get install on the same line, if you are ever going to deploy your Dockerfile to any CI system. If apt-get update is on its own line, then that layer will never get invalidated as the succeeding layers will have old sources that they build consequent layers from. Keep in mind that every single command generates its own layer, that is why commands are chained in this way - so as to create one layer instead of many that will never serve any purpose.


How to push to other origins using Azure Pipelines? Inspired by Necessity of mirroring git repos for small team considering switching to Azure:

How does one use Azure Pipelines to push changes to a secondary/backup origin on push?

Possible use cases:

Need of a backup, like GitLab as backup for Azure Repos.
LAN-only Azure DevOps Server makes backups to the cloud.
Global teams wants lower latency and code hosted everywhere.

While I haven't tried this, I suppose a solution would be to have a regular pipeline (this should work in either build or release pipelines) that would check out your repo (as it's default first step) then some scripted command that adds the desired remote and pushes to it.
Some pitfalls I could see you running into:

issues with the state of the agent work folder based repo prior to a push; but that should be reconcilable with the appropriate Git commands. If the target remotes (i.e. backups) are never being committed to by anything other than the pipeline, you hopefully won't run into conflicts
arbitrary branches: I'm not a git expert, but I think there's a way to push all branches to the remote
authentication: you'll obviously need to provide appropriate authentication to the target remote(s). Credentials shouldn't be stored in the scripts in plain text. Use secured pipeline (or group based) variables
repo size: a huge repo will need to be cloned into the ADO agent workspace. Not a problem as long as you have the disk to support it.

Given the needs you listed, it seems like you should consider something like AzureDevOps services in the cloud, or private GitHub repos (recently became free), etc. Why manage your own backups if the cloud can do it for you?


Service account permissions error when provisioning instance group from template I'm trying to create a GCE Instance Group Manager using terraform

resource "google_compute_region_instance_group_manager" "frontend_group_manager" {
  name = "frontend-igm"

  base_instance_name = "frontend"
  instance_template  = "${google_compute_instance_template.frontend_instance_template.self_link}"
  region             = "${var.region}"

  depends_on = [ "google_compute_instance_template.frontend_instance_template" ]
}

When I try I get the following error:

  Error: Error waiting for Creating InstanceGroupManager: The user does not have access to service account 'One of [serviceAccount:frontend@vpc-challenge-lab-245114.iam.gserviceaccount.com]'.  User: 'terraform@vpc-challenge-lab-245114.iam.gserviceaccount.com'.  Ask a project owner to grant you the iam.serviceAccountUser role on the service account

When I validate the IAM policy on the account it appears to be configured correctly:

> gcloud iam service-accounts get-iam-policy frontend@vpc-challenge-lab-245114.iam.gserviceaccount.com
bindings:
- members:
  - serviceAccount:terraform@vpc-challenge-lab-245114.iam.gserviceaccount.com
  role: roles/iam.serviceAccountUser
etag: BwWMpQvtA3w=
version: 1

The template is configured as follows:

resource "google_compute_instance_template" "frontend_instance_template" {
  name = "frontend-template"

  instance_description = "Frontend Instances"
  machine_type         = "f1-micro"
  can_ip_forward       = false

  scheduling {
    automatic_restart   = true
    on_host_maintenance = "MIGRATE"
  }

  disk {
    source_image = "${data.google_compute_image.compute_image.self_link}"
    auto_delete  = true
    boot         = true
  }

  network_interface {
    network = "${google_compute_network.frontend_network.name}"
  }

  service_account {
    email  = "serviceAccount:${google_service_account.frontend_service_account.email}"
    scopes = ["default"]
  }

  depends_on = [
    "google_compute_network.frontend_network",
    "google_service_account_iam_member.frontend_service_account_permission",
    "google_project_iam_member.frontend_service_account_role_binding"
  ]
}

And the Service Account is configured as follows:

resource "google_service_account" "frontend_service_account" {
  account_id   = "frontend"
  display_name = "Frontend Service"
}

resource "google_service_account_iam_member" "frontend_service_account_permission" {
  service_account_id = "${google_service_account.frontend_service_account.name}"
  role               = "roles/iam.serviceAccountUser"
  member             = "serviceAccount:${data.google_service_account.terraform.email}"

  depends_on = ["google_service_account.frontend_service_account"]
}

resource "google_project_iam_member" "frontend_service_account_role_binding" {
  role   = "projects/${data.google_project.project.project_id}/roles/${google_project_iam_custom_role.base_compute_role.role_id}"
  member = "serviceAccount:${google_service_account.frontend_service_account.email}"

  depends_on = [
    "data.google_project.project",
    "google_project_iam_custom_role.base_compute_role",
    "google_service_account.frontend_service_account"
  ]
}

Finally, the custom role is as follows:

resource "google_project_iam_custom_role" "base_compute_role" {
  role_id = "base_compute_role"
  title   = "Base Compute Role"
  permissions = [
    "logging.logEntries.create",
    "monitoring.metricDescriptors.create",
    "monitoring.metricDescriptors.get",
    "monitoring.metricDescriptors.list",
    "monitoring.monitoredResourceDescriptors.get",
    "monitoring.monitoredResourceDescriptors.list",
    "monitoring.timeSeries.create"
  ]

  depends_on = ["google_project_service.iam"]
}

Why when everything appears to be in place does that error keep getting thrown by the GCE API.

Try adding permissions on your project level:

gcloud projects add-iam-policy-binding <your-project-id> --member=serviceAccount:terraform@vpc-challenge-lab-245114.iam.gserviceaccount.com --role='roles/iam.serviceAccountUser'


Question about the new AWS RDS CA When updating the database to use the new CA, you get this popup:

  Before scheduling the CA certificate rotation, update client
  applications that connect to your database to use the new CA
  certificate. Not doing this will cause an interruption of connectivity
  between your applications and your database.

Does anyone know if this is required? We didn't have to import the CA originally, so why would we have to now? If it's a matter of the new root CA not being part of the trust chain of older distros, does anyone know if Ubuntu 18.04 has the new CA in its ca-certificates package yet?

Thanks!

I am not completely sure that Ubuntu 18.04 will have this CA in their certificates packages because they announced this CA replacement recently.

If you are using the SSL/TLS to connect to your DB instances, then only this will be required, otherwise you can simply ignore this.

You can find the complete details of this CA certificate rotation and it's impact in the AWS blog


What are the benefits of spreading the various services of a multi-tiers application over several user-define bridges? I have a simple system made of a database backend, several instances of a web application and a load balancer. Each service is running in its own Docker container, all of them being hosted on the same physical server. The load balancer listens on the port 80 which is mapped to the same port on the host. All ingress/egress traffic is supposed to pass through that load balancer.

Currently, all those containers are attached to a single user-defined bridge.

I read it would be better to connect the load balancer and the app instances through one bridge, and the app instances and the database through a second one. Instinctively, it makes sense to avoid putting the DB on the public-facing bridge. But objectively, in a dockerized environment, what are the benefits of using two bridge networks instead of only one here? Which kind of threats is mitigated by doing so?

Your updated question completely changed my understanding :)

Since the db is on a user-defined bridge in both cases, the advantages of using user-defined vs legacy/default bridges discussed in the doc section you referenced (which are applicable in both cases) aren't what differentiates the 2 cases. From that perspective they're equivalent.

Original Answer:

I think most/all of the differences between the legacy/default bridge and the user-defined ones listed in the doc section you referenced are pretty objective advantages :)

For your particular question, in User-defined bridges provide better isolation and interoperability between containerized applications we see:

  Using a user-defined bridge, only the web port needs to be opened, and
  the database application doesn’t need any ports open, since the web
  front-end can reach it over the user-defined bridge

Since in such case the db doesn't have open ports it can't be accessed from the outside, so all threats associated with having the db reachable from the outside are avoided.


CloudFront -> ELB -> Pod running Gunicorn Can anyone tell me if there's a "killer" reason to put NGINX in front Kubernetes pods running Gunicorn?

externally, I have CloudFront hitting an ELB which in turn engages the pods running Gunicorn with my Django based API server.

Since Caching is handled by CloudFront, and it's flexible enough for routing rules  -- is there any good reason to add NGINX to the mix?

Nginx will help you in performing the below items if you need some of them, Then I would say it make sense to add it to the stack

Optimize the caching (you can also cache dynamic content.
Restrict access to the services (based on ips).
Implement basic auth for some services.
Load balancing traffic between different posds or services.
Rewrite URLS.
Pass extra headers or configs to the backend services.


How to automate TeamCity installation using docker? JetBrains TeamCity docker image describe build steps for already installed TeamCity.

Is there any way to automate installation of TeamCity it-self?

For example creating the admin account, enabling standard authentication and setting up the project?

Automation of installations like this can often be done with config management tools.  GitHub and the public repos for these tools is a good place to start.

There is a Puppet installer for TeamCity here: https://github.com/haf/puppet-teamcity

Here's a Chef cookbook:
https://supermarket.chef.io/cookbooks/chef-teamcity

You can also use tools like Packer to generate your image(s).  Packer can substitute for a Dockerfile in generating a Docker image. Packer will also run Provisioners (including Chef/Puppet/Ansible) to configure your image.


Deply method for python project I have made a web application in django. Currently I'm deploying by SSH'ing into my ubuntu instance at AWS and using git pull.
However  I know this isn't a very scaleable solution but I have very limited knowledge of DevOps.
So, my question is if anyone could recomend which applications I need to use to achieve the following:

I can launch more instances with just a click (probably Docker if I've understood it?)
When I git push by project, all running servers pull and run migrations and reboot.

What you're looking for is a CI/CD pipeline.  There's lots of articles around it, but the main steps would be as follows:

Some form of build artifact (it can be a docker container, but it can also be a tarball, or just code in a git repo).

Some way to automatically deploy your code.  This is done by using a CI/CD tool (literally any of them will work fine, but for the love of god, don't use Jenkins, you'll thank me later).

For something as simple as how you describe your app, you can just use a CICD tool to SSH in and deploy to your existing server.
For example, CircleCI (one of my favourite tools and what we use at my current job), has a free tier that gives you 2500 minutes of build time a month.  It also has an SSH orb you can include in your pipeline configuration that will SSH into your server and run a few commands to pull down the latest code and restart your server.
Here is a good Getting Started guide that should get you up and running with Circle.
If you'd like to start using docker, here is a good guide on how to run your Django app with it.  From there, you would need to set up a Dockerhub account and push your docker image there, to be deployed on a server.  Note you'll have to make sure you can configure your application through environment variables.  It's never a good idea to have an artifact with "baked in" configuration, since this method doesn't scale in any real way.


Is it possible to package an application in a .nupkg when creating an application revision in AWS CodeDeploy? My build process packages my application in a .nupkg instead of a .zip.

Assuming my .nupkg contains a correctly-implemented appspec.yml and is otherwise appropriately bundled, can my deployment revision use it?

Yes. AWS CodeDeploy bundletype is specified separately from the revision location, so you could upload a .nupkg to Amazon S3, set the bundle type to 'zip', and since a nupkg is simply an archive CodeDeploy will treat it as such and upload it to your EC2 instance.

In your nuspec, make sure your appspec.yml gets placed in the root of your .nupkg. My project contains the appspec.yml and powershell scripts to manage windows services in a directory structure like this:

deploy/appspec.yml
deploy/lib/get.ps1
deploy/lib/start.ps1
deploy/lib/stop.ps1
...

You can include this structure at the root by simply recursively retrieving this directory structure (via double wildcards **) in the nuspec files element:

<files>
  <file src="deploy\**" target="" />
</files>

And then after packing and outputting the nupkg, you can create the application revision as below:


Deploy to server after merge to master branch I currently have continuous deployment set up with a Travis CI job. All pushes are sent to the staging server and merges to the master to the production server. The problem is that the deployment to the production server is done before the merge is accepted. The continuous integration tests have to all pass, but the deployment occurs before manual code review is done.

The question is, what is a lightweight method to ensure that a deployment to the production server only takes place after the merge request has been accepted?

As a reference, the following code is used for deployment:

#!/bin/bash

# exit with nonzero exit code if anything fails
set -e

# Add the SSH login key
chmod 600 veleda-deploy-key
mv veleda-deploy-key ~/.ssh/id_rsa

# Register the Veleda staging and production server SSH keys
echo '|1|BqdQKtUnA/AtCT/p2M7wgMq3wlY=|lH39cRtAE64wd6EG3ry2J9ewXic= ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBH3antqwy3D4NVVfHQX3SQc/g4wl/SAVC9w9QEry7hhQmB0SJIprwNAq8Hy2DzVCS7kTj/q7fCiiL7oAznrax+0=' >> $HOME/.ssh/known_hosts
echo '|1|+Z7oOsZ+zdL6u8o8VSWp+bRzd2g=|XMw2HyJIHoekOYlJYw1n75plL2E= ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCJ/fa/mr577/qCuRXqUNccfmhpUtmi46LSyE7nDbOgxv8kZFs7yQ/sh6TM5npR+ZIbe9I0qmdvA+cE1QfvN21E=' >> $HOME/.ssh/known_hosts

# Select the remote to push to depending on the branch
if [ $TRAVIS_BRANCH != 'production' ] ; then
    echo "Pushing to staging"
    export REMOTE=staging.veleda.io
else
    echo "Pushing to production"
    export REMOTE=veleda.io
fi

# Push to the remote server
git remote add deploy "deploy@$REMOTE:/home/deploy/repo/"
git push -f deploy $TRAVIS_BRANCH

# Unpack and update the Docker services
ssh -t deploy@"$REMOTE" "\
    mkdir -p veleda &&
    git --work-tree=./veleda --git-dir=./repo checkout -f $TRAVIS_BRANCH &&
    cd veleda &&
    ./start.sh"

One could use conditional builds https://docs.travis-ci.com/user/conditional-builds-stages-jobs/

If code is merged into master one could decide to deploy code to production, but I personally prefer a human intervention by a Product Owner.

jobs:
  include:
      if: branch = master

or

stages:
  - name: deploy
    # require the branch name to be master
    if: branch = master

At other projects I preferred to deploy only tags to enforce the git flow (tags should be created otherwise no deployment to production).

stages:
  - name: deploy
    # require the tag name to match a regular expression
    if: tag =~ ^v1

One could also try (I did not try it yet) whether the if works in combination with script sections:

https://github.com/030/ansible-firefox/blob/master/.travis.yml

Last remark: I would avoid creating such scripts (the ones you included in the question) as most CI tools contain condition checks.


ansible filter debug output I have the following output resulted from a shell command which is run on ansible on every host from my inventory:

TASK [debug]
ok: [10.240.22.44] => {
    "msg": {
        "changed": true,
        "msg": "All items completed",
        "results": [
            {
                "_ansible_ignore_errors": true,
                "_ansible_item_result": true,
                "_ansible_no_log": false,
                "_ansible_parsed": true,
                "changed": true,
                "cmd": "/opt/confluent/bin/nodefirmware smm1",
                "delta": "0:00:00.128325",
                "end": "2020-02-05 11:22:19.435049",
                "failed": false,
                "invocation": {
                    "module_args": {
                        "_raw_params": "/opt/confluent/bin/nodefirmware smm1",
                        "_uses_shell": true,
                        "chdir": null,
                        "creates": null,
                        "executable": null,
                        "removes": null,
                        "stdin": null,
                        "warn": true
                    }
                },
                "item": "10.240.18.20",
                "rc": 0,
                "start": "2020-02-05 11:22:19.306724",
                "stderr": "",
                "stderr_lines": [],
                "stdout": "smm1: SMM: 1.10 (TESM14F)\nsmm1: PSOC: 0.7",
                "stdout_lines": [
                    "smm1: SMM: 1.10 (TESM14F)",
                    "smm1: PSOC: 0.7"
                ]
            },
            {
                "_ansible_ignore_errors": true,
                "_ansible_item_result": true,
                "_ansible_no_log": false,
                "_ansible_parsed": true,
                "changed": true,
                "cmd": "/opt/confluent/bin/nodefirmware smm1",
                "delta": "0:00:00.096292",
                "end": "2020-02-05 11:22:22.847949",
                "failed": false,
                "invocation": {
                    "module_args": {
                        "_raw_params": "/opt/confluent/bin/nodefirmware smm1",
                        "_uses_shell": true,
                        "chdir": null,
                        "creates": null,
                        "executable": null,
                        "removes": null,
                        "stdin": null,
                        "warn": true
                    }
                },
                "item": "10.240.19.21",
                "rc": 0,
                "start": "2020-02-05 11:22:22.751657",
                "stderr": "",
                "stderr_lines": [],
                "stdout": "smm1: SMM: 1.10 (TESM14F)\nsmm1: PSOC: 0.7",
                "stdout_lines": [
                    "smm1: SMM: 1.10 (TESM14F)",
                    "smm1: PSOC: 0.7"
                ]
            }
        ]
    }
}

The information I need to extract and display on screen is this (for every host):

"stdout_lines": [
    "smm1: SMM: 1.10 (TESM14F)",
    "smm1: PSOC: 0.7"

The previous command with the full output is registered as a variable using ansible register module:

- name: Get smm version
  shell: some command
  register: smm_output

Then I'm trying to display it using (which displays the full output):

- debug:
    msg: "{{ smm_output }}"

If I'm using msg: "{{ smm_output.results.stdout_lines }}" it returns error

I also tried with json_query filter but with no results

So how do I parse that output?

The registered variable smm_output is a dict object and you can correctly dive into it. But you cannot use smm_output.results.stdout_lines because smm_output.results is a list. You need to iterate over it or filter the first element of results. So, if you know its only one element in the list, you should use smm_output.results.0.stdout_lines.


cannot get bash prompt: Error response from daemon: Container foo is not running A basic jetty container:
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                                                  NAMES
7c10b0ca16bd        jetty               "/docker-entrypoint.…"   21 seconds ago      Up 18 seconds       8080/tcp, 0.0.0.0:80->800/tcp, 0.0.0.0:443->8443/tcp   jetty01

can easily get a bash terminal as:
docker exec -ti jetty01 bash

created as so:
docker run -d \
    --name jetty01 \
    -d \
    -p 80:800 \
    -p 443:8443 \
    jetty

but, this approach fails with the basex from docker hub:
docker exec -ti basexhttp bash
Error response from daemon: Container 52b5337d7e6a22964310afb0be5f79ace38993173218fae8d9dcd858e4681142 is not running

created as:
docker run -d \
    --name basexhttp \
    --publish 1984:1984 \
    --publish 8984:8984 \
    --volume "$HOME/basex/data":/srv/basex/data \
    --volume "$HOME/basex/repo":/srv/basex/repo \
    --volume "$HOME/basex/webapp":/srv/basex/webapp \
    basex/basexhttp:latest

How do I get a root bash terminal?
How do I start the basex container? I tried:
docker start 52b5337d7e6a22964310afb0be5f79ace38993173218fae8d9dcd858e4681142

still nothing listed.  But, docker ps -a shows:
CONTAINER ID        IMAGE                    COMMAND                  CREATED             STATUS                     PORTS               NAMES
52b5337d7e6a        basex/basexhttp:latest   "/usr/local/bin/mvn-…"   10 minutes ago      Exited (1) 2 minutes ago                       basexhttp

although it doesn't show in docker ps because it's not running.

Whoever created basex/basexhttp:latest does not seem to know Docker very well. They copied files necessary to a directory that was later configured as a volume, which means it can only be bound to an anonymous volume, otherwise files inside will be removed. They left source code inside the image, they used a single stage build with maven:3-jdk-8-alpine as base image, which means the final image has many unnecesary things installed. They left the entrypoint from the base image intact, which I guess is why they needed to create an empty /srv/.m2 directory and set it as Maven config dir for the image to start. It switches user from root to basex, which causes permission problems with bind mounts. Generally it's a mess.
I see two ways to successfully run this image:
First:
docker run -d \
    --name basexhttp \
    --publish 1984:1984 \
    --publish 8984:8984 \
    --entrypoint "" \
    basex/basexhttp:latest /usr/local/bin/basexhttp

This uses only anonymous volumes and clears that unnecessary entrypoint.
You can stop it with docker stop basexhttp and then remove it with docker rm -v basexhttp. That -v is important - it removes the volumes - unless you memorize their very long IDs and which was mounted which you won't be able to reuse them anyway, so you may as well remove them from your system.
Second:
mkdir -p $HOME/basex/data
mkdir -p $HOME/basex/repo
mkdir -p $HOME/basex/webapp
chmod a+w basex
sudo chown -R 1984:1984 $HOME/basex
docker run -d \
    --name basexhttp \
    --publish 1984:1984 \
    --publish 8984:8984 \
    --volume "$HOME/basex/data":/srv/basex/data \
    --volume "$HOME/basex/repo":/srv/basex/repo \
    --volume "$HOME/basex/webapp":/srv/basex/webapp \
    --entrypoint "" \
    basex/basexhttp:latest bash -c "cp -r /usr/src/basex/basex-api/src/main/webapp/WEB-INF /srv/basex/webapp && exec /usr/local/bin/basexhttp"

This creates the necessary directories and sets their permissions so that both your user on the host and user.
You can stop it as in the previous example and remove with docker rm basexhttp. This time -v is not necessary, as there are no anonymous volumes.
You can verify that the contaienr is running correctly by executing:
docker exec -ti basexhttp basexclient

Username and password are both admin.


Replace/Consume Istio's Prometheus/Grafana by existing Prometheus-operator I have multiple K8S clusters that have Prometheus-operator Helm chart installed in monitoring namespace. Each operator consists of Prometheus/Alertmanager/Grafana.
Aside from it I have Istio Helm chart installed in istio-namespace. I want to integrate the Prometheus/Grafana from Istio into my Prometheus-Operator setup and remove them from istio-namespace and the helm release for Istio.
Based on this PR - https://github.com/istio/installer/pull/71
it looks like it's possible, but I can't find exact guidance on how to do it, the end result should be prom-operator prometheus scrapping exactly what it was + what was being scrapped by Istio's Prometheus and then when using the Grafana from prom-operator I should have all mixins + istio dashboards.

Followed this guide https://medium.com/@emirmujic/collect-cluster-and-istio-metrics-with-prometheus-operator-42d4499d621a and it worked out for me on Istio 1.2.2:
https://medium.com/@emirmujic/collect-cluster-and-istio-metrics-with-prometheus-operator-42d4499d621a


How to deal with authority bias? https://en.wikipedia.org/wiki/Authority_bias

  Authority bias is the tendency to attribute greater accuracy to the
  opinion of an authority figure (unrelated to its content) and be more
  influenced by that opinion. The Milgram experiment in 1961 was the
  classic experiment that established its existence.

I have experienced that some people that have a lot of influence promote silos and are anti DevOps. How to deal with this?

I bet you're not the only one with such experiences :)

One possible approach today (which wasn't available in the early DevOps days) is to counter-argument with the many research-based publications from industry-recognized influencers. Like Accelerate or 2017 State of DevOps Report to name a few.

Another approach could be to propose and implement DevOps-based pilot projects to actually measure and compare performance parameters against the existing silo'd solutions. This could be tricky, though, as those influencers could insist on requirements cornering the pilot projects. For example insisting on individual/small team development speed which could be an obstacle for optimizing and speeding up the overall larger/entire project team's development.


How to find the image name corresponding to a layer digest? Since 1.10, Docker implements a content addressable method based on a secure hash of the image and layer data:

sudo docker pull redis:5.0.2-alpine
sudo docker inspect --format '{{json .RootFS }}' redis:5.0.2-alpine | jq .
{
  "Type": "layers",
  "Layers": [
    "sha256:df64d3292fd6194b7865d7326af5255db6d81e9df29f48adde61a918fbd8c332",
    "sha256:04c8ef03e9356ae0c559ecc63ef5ef7154d5da8d4bef48786a7f00f628047d88",
    "sha256:169a281fff0fef0bfbec6873d7f0b6d2cad3a67068d0f16e2d32259582a2b95c",
    "sha256:1943100a0dc33557faa38ea52c3c6e974c0311122fa0ca9c5d8e398ab532ff2b",
    "sha256:1a49f3e37547b9141b4a5f78aaedb0ddcbfc8871466bbc3d3d8b8dba420c154a",
    "sha256:ba1d81ed872f9f31960e964cf6d00c77a668fd04ab0277ece2273dbb85003910"
  ]
}

But is there a way to match the layers digest with an image name, either locally downloaded or in some repository?

As far as I can tell, docker search does not allow searching by digest. Or does it?

The mapping between image manifest and layers goes the other way. You can have one or more image manifests pointing to a layer. And those images may not even be tagged with an image name, or a single image may have multiple tags. So I'm not aware of an easy way to do a reverse search. You would need to go through each image on the local system or registry to see what images point to that layer, and then search the tags to see what names that image has.


Troubleshooting VPC flow logs with an S3 bucket using SSE-KMS encryption with CMK I'm using Terraform and trying to set up automatic export of VPC flow logs into an S3 bucket in the same AWS account and region (ca-central-1) that has default encryption turned on with AWS-KMS (using a CMK). Even after trying many permutations of policies for KMS and the S3 bucket, the flow logger still always ends up in Access error status.

Update: When the S3 bucket is reconfigured to use AES-256 as the default encryption (instead of KMS) the VPC flow logs get written normally. So it's definitely a KMS problem.

The aws_flow_log Terraform resource is configured exactly according to the documentation.

resource "aws_flow_log" "vpc_flow_log" {
  log_destination_type = "s3"
  log_destination      = "${aws_s3_bucket.awslogs.arn}"
  traffic_type         = "ALL"
  vpc_id               = "${aws_vpc.vpc.id}"
  depends_on           = ["aws_s3_bucket_policy.awslogs"]
}

And the result of aws ec2 describe-flow-logs:

{
  "CreationTime": "2020-05-15T16:22:40.142Z",
  "DeliverLogsErrorMessage": "Access error",
  "DeliverLogsStatus": "FAILED",
  "FlowLogId": "fl-REDACTED",
  "FlowLogStatus": "ACTIVE",
  "ResourceId": "vpc-REDACTED
  "TrafficType": "ALL",
  "LogDestinationType": "s3",
  "LogDestination": "arn:aws:s3:::REDACTED-awslogs",
  "LogFormat": "${version} ${account-id} ${interface-id} ${srcaddr} ${dstaddr} ${srcport} ${dstport} ${protocol} ${packets} ${bytes} ${start} ${end} ${action} ${log-status}"
}

Curiously, it works fine in my second "sandbox" AWS account where I exclusively use the AWS web console, never Terraform. This account is configured the same way with AWS-KMS on the S3 bucket.

Both accounts seem to have the same configuration, so I can't figure out why it works in the sandbox, but fails in my terraformed account.

S3 bucket

Default encryption is enabled and and Custom KMS arn is selected.

S3 bucket policy includes statements to allow VPC flow logs delivery from delivery.logs.amazonaws.com as written in Publishing flow logs to Amazon S3.

    {
        "Sid": "AWSLogDeliveryAclCheck",
        "Effect": "Allow",
        "Principal": {
            "Service": "delivery.logs.amazonaws.com"
        },
        "Action": "s3:GetBucketAcl",
        "Resource": "arn:aws:s3:::REDACTED-awslogs"
    },
    {
        "Sid": "AWSLogDeliveryWrite",
        "Effect": "Allow",
        "Principal": {
            "Service": "delivery.logs.amazonaws.com"
        },
        "Action": "s3:PutObject",
        "Resource": [
            "arn:aws:s3:::REDACTED-awslogs/*",
            "arn:aws:s3:::REDACTED-awslogs/AWSLogs/REDACTED/*"
        ],
        "Condition": {
            "StringEquals": {
                "s3:x-amz-acl": "bucket-owner-full-control"
            }
        }
    }

KMS Key Policy

KMS key policy includes a statement that allows usage by VPC Flow logs as instructed by Required CMK key policy for use with SSE-KMS buckets.

    {
        "Sid": "Allow VPC Flow Logs to use the key",
        "Effect": "Allow",
        "Principal": {
            "Service": "delivery.logs.amazonaws.com"
        },
        "Action": [
            "kms:ReEncrypt",
            "kms:GenerateDataKey",
            "kms:Encrypt",
            "kms:DescribeKey",
            "kms:Decrypt"
        ]
    }

I'm at a loss here. What else can I do to troubleshoot this?

Turns out I was missing one very important line in my KMS key policy:

    resources = ["*"]

Now it works fine, and my full policy looks like this:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Allow VPC Flow Logs to use the key",
            "Effect": "Allow",
            "Principal": {
                "Service": "delivery.logs.amazonaws.com"
            },
            "Action": [
                "kms:ReEncrypt",
                "kms:GenerateDataKey",
                "kms:Encrypt",
                "kms:DescribeKey",
                "kms:Decrypt"
            ],
            "Resource": "*"
        }
    ]
}


What is the proper relationship between rollback/rollforward and MTTR metrics? I'm trying to understand the best way to capture data to start measuring Mean Time To Repair (MTTR) metrics, and I need to wrap my head around how "rollback" impacts MTTR positively or negatively.

Scenario 1

Assuming that solid monitoring is in place, code is deployed that causes an incident that is detected rather quickly (low MTTI). At the point of identification, there are two main possible paths forward (yes, I'm oversimplifying for purposes of discussion):

Rollback the deployment, returning stability quickly, but without the intended features in production.
Roll-forward with additional changes that resolve the incident and keep the intended features live.

In this scenaro, MTTR is pretty darn low, given that site stability can come back pretty quickly. That said, the intended outcome of the change isn't live, and thus the code/feature/change is still stuck in process. If a goal is low MTTR, it seems to incentivize roll-back as a recovery mechanism.

Scenario 2

In this scenario, MTTR is strictly measured by how long it takes the expected code/feature/change to be working properly in production. Even if I rollback, until my "fixed" code change goes into prod, the MTTR timer is still running. In this case, MTTR seems tied to business outcome stability instead of just pure "hey, things are stable."

Now, the answer may be as simple as MTTR not being used as a metric in a vacuum, but rather in conjunction with Change Failure Rate - a super-low MTTR caused by frequent rollbacks could point to a sky-high Change Failure Rate. That said, there's something that doesn't seem right to me in the idea of divorcing the MTTR measurement from business outcome.

I may be way overthinking this, but I'm curious how others are measuring MTTR and what the end point-in-time is for "recovery." Are you using it simply as stability, or do other factors come into determining what "recovered" means?

Yes, MTTR is/should always be tied to the business outcome: if things are not stable the very business is at risk.

The fact that the expected code/feature/change is still stuck in process in scenario 1 is irelevant: the feature is not stable, so it doesn't bring new business, rolling back is the best you can do at that time from the business prospective.

The rollforward is a gamble: keeps the business at risk waiting for a potential fix that in fact has statistically lower changes of success (due to the instability it will always be rushed compared to the change that caused the instability in the first place without even having such pressure on it). The rollforward is a yet another version of the code which hasn't been checked before.

If you want to keep the MTTR low you rollback immediately, without debate. This removes the business risk and gives you a chance to check that the fix is actually working before attempting to deploy it. I'd strongly suggest making it a policy as yes, almost always there will be someone asking for a fix instead of the rollback and calling a meeting to negociate/decide on it - all while business remains at risk.

Side note: if you're concerned with a high Change Failure Rate then I'd suggest checking the the actual rollback rate instead of deriving it from a low MTRR. Maybe you'd like to add a gate check before deployment for the most frequent failures.
If you have such check already automated - why not include it in the CI verification? If you don't have one - maybe its time to start thinking about it? :)


Is there a way to tell if an Azure Devops Extension is secure to use? How do I know that an Azure Devops Extension does not do anything really bad?

A company that I know of uses Microsoft's Azure Devops platform to manage its software development.

Microsoft also offers a "marketplace" where one can get extensions (plugins); third-party software that provides additional features meant to make the platform more useful.

This is helpful, but is there way a to know what such an extension actually does and does not? Can be absolutely 100% certain it does not send any information anywhere? Or changes anything?

Example: Azure Devops does a poor job of listing all your git repositories, so there are extensions which does that for you. On installing such an extension, it asks for permission for "Code (read)". From a security point of view, that looks like a yuuuge red flag it's a third-party piece of software that has read access to all the source code in the project.

Now, that's only dangerous if it can also reach the outside world, as long as it cannot send any information at all about my code anywhere, I'm safe.

But can I be certain of that? Am I guaranteed that

It cannot connect to site or service outside the project
It cannot connect to or talk to other extensions
It cannot change its permissions through extension updates.

In short: Can an Azure Devops Extension be determined to be secure?

Can't comment yet, but most extensions host their source code publicly. So you could read the source code to check if something is going on that you don't want.

If the source code is not available, you could check the company for certain certifications like ISO 27001 or the like so you know they get audited.


How to define dynamic parallel stages in a Jenkinsfile? In declarative pipelines, Jenkins allows the definition of parallel stages. It further allows scripted pipeline general purpose scripts to create and manipulate the artifacts of the declarative pipeline. For example, it can create stages dynamically.

I would like to know how I can dynamically create parallel executing stages. Please note I am not talking about steps or actions, which are solved here:

How to properly achieve dynamic parallel action with a declarative pipeline?

Below is a simple Jenkinsfile that dynamically adds stages using groovy code. These stages are sequential. I would like a Jenkinsfile that adds stages dynamically as below, but uses the parallel construct at the stage level. As a result, three stages that run parallel should be generated by the program.

pipeline {
    agent any
    stages {
        stage('Add regression tests') {
            steps {
                addStage('offline','open-source','webgoat')
                addStage('offline','open-source','juice-shop')
                addStage('online','internal','our-website')
            }
        }
    }
}

final void addStage(final String type, final String suite,final String application) {
    script {
        stage("Run ${type} application  '\n${application}' in test suite '${suite}'") {

            runTestOnApplication(type,suite,application)

        }
    }
}

final void runTestOnApplication(final String type, final String suite,final String application) {
    labelledShell label: "Run test shell script for ${type} application '${application}' in test suite '${suite}'", script: 'test.bash'
}

This is quite complicated to achieve in Jenkins. We had a similar issue and here's how we solved it:

We have a shared library where we keep our scripts ins ./vars/someScriptName.groovy
We created a groovy script that generates the stages.

Here's a working pipeline code for you to try it out:

def generateITParallelStages(body)
{
    def config = [:]
    config.stages = []

    body.resolveStrategy = Closure.DELEGATE_FIRST
    body.delegate = config
    body()

    def buildStages = [:]
    config.stages.each { String stage, closure ->
        // default settings, if you want to have any
        def settings = [:]
        settings.type = 'online'; // if you don't define the type, it will be 'online' by default
        settings.suite = 'internal'; // if you don't define the suite it will be 'internal' by default

        closure.resolveStrategy = Closure.DELEGATE_FIRST
        closure.delegate = settings
        closure()

        buildStages.put(stage, prepareStage(stage, settings))
    }

    return buildStages
}

def prepareStage(String stage, Map settings)
{
    return {
        print "Run ${settings.type} application  '\\n${stage}' in test suite '${settings.suite}'"
    }
}

pipeline {
    agent any
    stages {
        stage('Add regression tests') {
            steps {
                script {
                    parallel generateITParallelStages {
                        stages = [
                                webgoat: {
                                    type = 'offline'
                                    suite = 'open-source'
                                },
                                'juice-shop': {
                                    type = 'offline'
                                    suite = 'open-source'
                                },
                                'our-website': {
                                    type = 'online'
                                    suite = 'internal'
                                }
                        ]
                    }
                }
            }
        }
    }
}

And here's the result:

This is the only way we have been able to dynamically generate the parallel stages. Everything else will fail (at least at the time of writing this).

Other people seem to be having a hard time figuring this out as well - JENKINS-53032. If you feel the same make some noise :)


Suggestions for a good run command review console Is there a good web console or tool that would help enable us to review Shell Scripts?
Usually when maintaining a system, we would run scripts that are managed, or we could run scripts that run only once.  However, these scripts must go through a vetting process to make sure they are secure, compliant, and won't create an issue.
We could of course create a git repo and create PR each time we want to run a script, but I am looking for a way for our members to submit scripts without fussing too much about format or folder structure.

I am looking for a way for our members to submit scripts without fussing too much about format or folder structure.

It sounds like you need some combination of testing and linting, perhaps with a few utility scripts to organise and gate content from landing in the repo.
The "Vetting process" you mention sounds to me exactly like "tests need to pass on the scripts". A widely-used framework to write such tests is the Bash Automated Testing System - BATS.
It would be up to you to write the tests to check whether, as you desire, these scripts are "secure, compliant and won't create an issue", but at least you have a framework for generating and running them.
At a more superficial level, you could use the Shellcheck linter to assert that any proposed new code adheres to whatever set of common rules you adopt in your community.
Pulling these together into a GitHub action (e.g. using https://github.com/github/super-linter), or a git pre-commit hook etc could be a good way to add a quick inspection to new contributions, and help maintainers.
These would provide quantitative measures of correctness. In order to also provide some qualitative or more subjective measures of acceptance, this would depend greatly on the operating environment and which tools would fit naturally into yours.
However, some tools do attempt to address this need by design. Without further knowledge of the environment it's difficult to provide concrete answers, but here is a comparison of two options:
GitHub, GitLab requests
Assuming the code is destined for a GitHub or GitLab repo, GitHub or GitLab itself could of course be used as the "review console". A contributor would fork the repo, send a new contribution in the form of a PR against the repo, and then the checks referred to above would be executed.
The actions could take care of the organisation, based on some internal convention, e.g. using labels, i.e. "file scripts in PRs labelled admin in the admin directory".
Gerrit
Another purpose-built tool to manage code contributions is Gerrit. Gerrit is heavily backed by git workflow and may not be relevant, but I mention it since it's one of the first such peer-review systems for software.
As described on the Gerrit website, a similar workflow is followed:

Making the Change
Creating the Review
Reviewing the Change
Reworking the Change
Verifying the Change
Submitting the Change
Next Steps

Much of the actual work here is done by the reviewer or contributor, and Gerrit serves to organise their collaboration.
I suspect that neither of these corresponds exactly to what you are asking for, though, however a general purpose tool to which arbitrary contributions can be submitted is a big ask and likely a mirage.
It seems more feasible that this should be achieved by adopting shared conventions in existing code collaboration platforms rather than a dedicated service to manage ingestion.

Finally, you mention:

Usually when maintaining a system...

If you meant "maintaining a configuration state of a system", perhaps an approach with Zuul. Zuul is

... a program that drives continuous integration, delivery, and deployment systems with a focus on project gating and interrelated projects.

While quite heavy, it is a means to actually execute and evaluate the result of a proposed script. This provides more meaningful feedback, but is of course more onerous in terms of setup and maintenance.


What is a suitable strategy for provisioning Chef Server through code? Background

I am tasked with setting up a Chef server on our Azure Cloud for a new DevOps automation project.
We are using the Resource Manager model for Azure, in case it's relevant.
We will likely use free versions of Chef i.e. Chef Automate is not an option.

Objective

I am new to Chef and Azure, but I would like to try to do it the "right" way from the start. Specifically: Given the Chef server will eventually become an indispensable part of the DevOps infrastructure I want to make it provisionable through code.

To move toward achieving these objectives I have explored the following options.

Option 1: Chef to provision Chef Server

Use Chef Server cookbooks and Chef solo to provision Chef server.

In such a scenario, my main question is: how do I create the Chef server VM and make Chef Solo discover it?

Option 2: Use Azure CLI + RM Templates

Manually create Azure VM
Manually install Chef server
Save it as an image and Azure RM template
Use Azure CLI to provision from the RM Template and Image.

I am angling toward this option, but I have some anecdotal opinions from teammates, that Azure RM Templates wll not be declarative. Are they?

Option 3: Use Vagrant  + Azure Provider + Chef Solo

(This option is primarily because I am comfortable with Vagrant)
The main advantage of this option is that I think it will eliminate the question I had in Option 1. Is my assumption here correct?
However, I think this option does not work for a permanent infrastructure like Chef Server, I think Vagrant is more useful when you have disposable cloud-based virtualized environments. Is my understanding here correct?

Note: Obviously these are not the only 3 options. Please help with any strategies I might have missed.

'Suitable' can honestly be whatever works for you. I personally feel its perfectly reasonable to do a manual install of Chef server. It is the CM tool to help provision the rest of your fleet. You will not be setting up Chef server over and over again. You do it once, and then never again, in theory.

That theory breaks down however, in respect to server outages. Eventually, one day, your Chef server will go down. If you are running Chef in HA mode, or you have a very good backup system that allows you to restore the entire server, then you are fine. I don't know what it is for Azure, but with Amazon, you can take nightly snapshots of the entire server and simply restore from the AMI if the server ever goes "hard down".

Let's pretend that you don't run Chef in HA, or you aren't very confident in your backup system, or you just want to tinker... my recommendation would be to leverage Terraform w/provisioner scripts, or Packer with Terraform.

You can get the basics of Chef server setup with some simple commands that only need to be run once. This is perfect for Terraform and it's provisioner scripts. Obviously, it runs the provisioner scripts only once (during server creation) and that's all you need for Chef. If something happens to your Chef server, Terraform can rebuild it from scratch. Although, you'll have to re-key your local client and tie all your instances to the new Chef server. This is a huge headache and I would not recommend ever doing that if you can avoid it. :P


Backing up an ephemeral Elasticsearch installation I have an installation of Elasticsearch without persistence running and I need to back it up so we can move it to a persistent setup.

What are safe ways to do this and keep the data in the ES database?

It is running and I want to avoid shutting it down.

It is running in a container and I think the entrypoint starts the server. So stopping the cluster my stop my pod and get rid of all my data...

ElasticSearch can backup itself when told to do so via an API call.
You just need to configure a destination, and optionally credentials, for example you make ElasticSearch create snapshots of its database in AWS S3.

https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-snapshots.html

You can also automate these calls using a schedule with the Elastic curator tool.

https://www.elastic.co/guide/en/elasticsearch/client/curator/5.5/snapshot.html


Create User Using Ansible with Authenticate Pem File Community, I am creating a user on the client machine using Ansible but the issue is it creates a user and add authenticate file but getting an error while SSH to that machine using created, User. Following is my source code.

  ---
    # This file is used to create a single user or multiple with password
    - hosts: hostMachine
      become_method: sudo
      become_user: root
      become: true
      gather_facts: no
      vars_files:
        - ../../../group_vars/required_varlist.yml
      tasks:

        - name: Ensure group is exist
          group:
            name: "{{ item.groups }}"
          with_items:
            - "{{ users }}"

        - name: create a new user without password
          user:
            name: "{{ item.username }}"
            group: "{{ item.groups }}"
            groups: "sudo,{{ item.username }}"
            shell: /bin/bash
            createhome: yes
          with_items:
            - "{{ users }}"

        - name: Add Sudo privileges to user
          command: usermod -aG sudo "{{ item.username }}"
          with_items:
            - "{{ users }}"

        - name: Add user in  sudoders
          shell: sh -c "echo \""{{ item.username }}" ALL=(ALL) NOPASSWD:ALL\" >> /etc/sudoers"
          with_items:
            - "{{ users }}"

        - name: create directory
          file:
            path: "/home/{{ item.username }}/.ssh"
            state: directory
            mode: 0700
            owner: "{{ item.username }}"
            group: "{{ item.groups }}"
          with_items:
            - "{{ users }}"

        - name: Set Authorized key token from the file
          become: true
          authorized_key:
            user: "{{ item.username }}"
            state: present
            key: "{{ lookup('file', '{{ pem_file_path }}') }}"
            # path: '/etc/ssh/authorized_keys/"{{ item.username }}"'
          with_items:
            - "{{ users }}"

        - name: restart ssh
          service: name=sshd state=restarted

required_varlist.yml

  ---
    password: raspberry
    users:
      - username: ABC
        groups: ABC
    pem_file_path: /home/ubuntu/XYZ.pem

I suspect what is happening here is you are trying to insert the private key into the authorized_keys file, which is invalid as only the public key is required on the target machine.

What you need to do is extract the public key from the private key:

- name: Generate an OpenSSL public key with a passphrase protected private key
  openssl_publickey:
    path: /tmp/key-{{ item.username }}.pub
    privatekey_path: {{ item.pem_file_path }}
    privatekey_passphrase: ansible
    format: OpenSSH
  with_items:
    - "{{ users }}"

and then use the public key:

- name: Set Authorized key token from the file
  become: true
  authorized_key:
    user: "{{ item.username }}"
    state: present
    key: "{{ lookup('file', '/tmp/key-{{ item.username }}.pub') }}"
  with_items:
    - "{{ users }}"


Azure Equivalent for CodeDeploy Right now, using AWS services we have a CI and deployment pattern that works well for us.

For every commit pushed to github, Travis-CI builds the project and runs tests, if tests pass, Travis uses CodeDeploy to deploy to staging (an AWS EC2). On tagged commits, if tests pass, deploy to production.

We are exploring using Azure, and we can't quite seem to figure out how to get the same pattern. In particular, we can't find the equivalent to CodeDeploy for Azure.

Does anyone have a similar pattern of CI and deployment that deploys to an Azure Virtual Machine?

According to Microsoft's AWS to Azure services comparison, Visual Studio Team Services correspond to CodeDeploy, but it seems like a very different tool and it is not clear to me if it can be put in our current pipeline.

Azure has a robust CI/CD pipeline which is much easier to use than CodeDeploy on AWS. It is called Build/Release pipelines under Azure DevOps. It allows you to configure multiple environments with build and release (deployment) stages for each. It also allows for easy configuration and interoperability with Azure Webapps/VMs and other resources available on Azure. Having recently switched from Azure to AWS, I definitely miss the ease of configuration and the intuitive UI on Azure DevOps.

Azure DevOps integrates a version control repository system, a agile style work items/boards which can also be used to configure your backlogs and iteration items. There is even a dashboard to integrate and monitor your work items and deployments across the team. So it is definitely much bigger than just a devops tool or CodeBuild.


Bitbucket Webhook plugin for Jenkins - differentiate between branches We use Jenkins and have the Bitbucket plugin installed. We've got multiple repos and use webhooks for lots of jobs coming from our foo repo. No problems there, the jobs go off without a hitch. Now however we want to start running a couple jobs via webhook from our bar repo.
The Problem
The default behavior here is that as soon as we attach a webhook to bar, commits to either repo will trigger any job with polling. I know we can hardcode in logic in each job to either only build when commits match a certain pattern, or ignore any commits matching a pattern.
Given the number of jobs we have that currently use polling this would be insanely unideal to have to code in to make it backwards compatible, not to mention potentially doing this again if we start running unit tests on even more repos in the future.
The Ideal Solution
I'm hoping I might be missing some way to differentiate between commits made from foo vs bar on the Bitbucket side, rather than needing to change anything in each Jenkins job. Ideally, there's some way to change the webhook so each repo gets its own url. Does this, or some other blanket solution, exist? (I will accept "no" as an answer if that's the case).

I believe it is now possible to configure the webhook on per-repo basis:

https://mohamicorp.atlassian.net/wiki/spaces/DOC/pages/381419546/Configuring+Webhook+To+Jenkins+for+Bitbucket+Bitbucket+Branch+Source+Plugin


Undeploy or deactivate an AWS Lambda Is there a way to undeploy or deactivate an AWS Lambda or must I either implement it myself or delete and later redeploy it to achieve the desired state? What I would like is similar to just "undeploying it" so that it can be deployed again later.

you can change the trigger working process and rewrite and re schedule the process. Which the way you want.
it will work.

Hope it helps.


Getting rid of redundant Ansible roles I currently have a bunch of roles, like:

apache2
gcc
gedit
...etc.

For each of those, I have a task file that looks like this:

---
- name: "Install {{ role_name }}"
  become: true
  package:
    name: "{{ role_name }}"
    state: present

Is there a way to abstract this to a higher level and have a single "Install package" role, and then somehow specify a bunch of specific packages to install per host in a variable or something?

Is this what you're looking for?

- hosts: all
  vars:
    bunch_of_specific_packages:
      - apache2
      - gcc
      - gedit
  tasks:
    - package:
        name: "{{ item }}"
      loop: "{{ bunch_of_specific_packages }}"


Execute multiple Ansible tasks with the same list of items I am using an ansible playbook to configure Apache for a list of sites.   The playbook has to copy the virtual host configuration template for each site into place, and then enable each site using a2ensite:

- name: Install apache site conf
  template: src=apache-sites-{{item}}-conf.j2 dest=/etc/apache2/sites-available/{{item}}.conf mode=0644
  with_items:
  - sitea
  - siteb
  - sitec
  - sited
- name: Enable site apache conf
  command: a2ensite {{item}}
  args:
    creates: /etc/apache2/sites-enabled/{{item}}.conf
  with_items:
  - sitea
  - siteb
  - sitec
  - sited

I don't like having to repeat the same list for each task.   How to I configure the playbook to execute both tasks with the same list of items?

Make separate tasks file make_site.yml:

---
- name: Install apache site conf
  template:
    src: apache-sites-{{ site }}-conf.j2
    dest: /etc/apache2/sites-available/{{ site }}.conf
    mode: 0644

- name: Enable site apache conf
  command: a2ensite {{ site }}
  args:
    creates: /etc/apache2/sites-enabled/{{ site }}.conf

And in your playbook:

- include_tasks: make_site.yml
  with_items:
    - sitea
    - siteb
    - sitec
    - sited
  loop_control:
    loop_var: site


How to configure gitlab-runner to run in the same docker host as gitlab? I have a gitlab instance running in a docker container. Now I would like to setup gitlab-runner on the same host. Both are running:

docker container ls
CONTAINER ID        IMAGE                         COMMAND                  CREATED              STATUS                 PORTS                                                            NAMES
279473dceb2f        gitlab/gitlab-runner:alpine   "/usr/bin/dumb-ini..."   About a minute ago   Up About a minute                                                                       gitlab-runner
6d7af0d6b946        gitlab/gitlab-ce:latest       "/assets/wrapper"        2 hours ago          Up 2 hours (healthy)   0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp, 0.0.0.0:8022->22/tcp   gitlab

I registered a docker image as a shared runner:

concurrent = 1
check_interval = 0

[[runners]]
  name = "default shared maven runner"
  url = "http://url-of-my-gitlab.instance"
  token = "valid token"
  executor = "docker"
  [runners.docker]
    pull_policy = "never"
    tls_verify = false
    image = "maven-java-8:latest"
    privileged = false
    disable_cache = false
    volumes = ["/cache"]
    shm_size = 0
  [runners.cache]

The registered image maven-java-8 is a custom image with the correct settings.xml and it is listed in docker images.

But still the pipeline fails with:

Running with gitlab-ci-multi-runner 9.3.0 (3df822b)
  on Maven Test (5b2573e4)
Using Docker executor with image maven-java-8 ...
Using docker image  sha256:c266311d78b33f89f6eecce00f19a0428e37ba6af6734c8fced32fa11e4bd4ba for predefined container...
Pulling docker image maven-java-8 ...
ERROR: Job failed: Error response from daemon: repository maven-java-8 not found: does not exist or no pull access

Which I interpret as it is trying to use the local images inside of the gitlab-runner container.

The GitLab version is gitlab-ce 9.2.7 and the runner version is v9.3.0.

The content of my .gitlab-cy.yml is:

test:
  script:
    - mvn clean test

What am I missing?

Management Summary

Set the pull_policy to "never" in the [runners.docker] section by calling:

docker exec -it gitlab-runner \
  vi /etc/gitlab-runner/config.toml

Detailed Description

It seemed the problem was a confusion with images and mounted volumes.

The documentation says to configure runners in:

/etc/gitlab-runner/config.toml

Which I did, but on the host machine.
When running gitlab-runner in a docker container, it mounts a volume:

... -v /srv/gitlab-runner/config:/etc/gitlab-runner ...

This means, that the configuration file in /etc/... in the container is mounted to /srv/... on the host machine.

In this file the pull_policy was not set, which defaults to always, what ultimately caused my problem in the first place.


OOM when running Bitbucket pipelines as one (sometimes failing) step of our pipeline we

setup environment via installation of docker which runs 3 containers through own docker-compose. Unfortunately we could not figure out how to impose any memory limits on individual containers in docker-compose v3 config.
run tests in sbt with  SBT_OPTS="-Xmx1500m" and every second to third time step is failing with OOM

definitions:
  services:
    docker:
      memory: 1024

Explanation on the memory usage is not clear on this case.

Could somebody please explain what's going on with the memory allocation to containers, docker host, and step env?

Interesting is that running the same tests, same containers for multiple times usually gives different results. Understandable that real load on the physical server changes over time and running times can differ - but memory?

Puzzled...

I suspect that Bitbucket pipelines is enforcing this memory limit, not docker-compose.  This is nothing to do with the Linux kernel's OOM killer.

The page you linked is about memory limits in additional containers (e.g. for a database) used by the pipeline, not the pipeline itself.

Some suggestions:

Try editing bitbucket-pipelines.yml to include a size option for memory size - I would expect this is more useful than editing Docker-compose options which may not be respected.  See this doc page.
Try running the build locally under Docker Compose to get an idea of memory usage.

It would help to include the details of your bitbucket-pipelines.yml, and the Bitbucket error message about being out of memory.


What is the difference between RUN and CMD in a Dockerfile? I am going to change my permission of files to 600. The following command works perfectly.

RUN /bin/bash -c 'chmod 600 /root/.ssh/config'

But this one failed

CMD /bin/bash -c 'chmod 600 /root/.ssh/config'

In the beginning, I thought the shell environment isn't bash so caused the error. However, adding bin/bash didn't solve the issue.

Note: Don’t confuse RUN with CMD. RUN actually runs a command and commits the result; CMD does not execute anything at build time, but specifies the intended command for the image.

reference


Jenkins shows the job as failed if there is nothing to commit to gitlab I have been using one Jenkins job, that will pull repository from gitlab.
Then it will modify the files and push it into gitlab (only if any user added any new files/modify contents in gitlab). Otherwise, it simply does empty commit.
When empty commits & push happens, Jenkins job shows me like the job is failed. For only this case, I want to make Jenkins should show as successful.
Is there any way to do it?

UPDATE: Here is the log I got from the job. I am expecting SUCCESS even if the Jenkins git pushes the empty repository (as shown as the below).

[1. Extract simplified configuration] $ /bin/sh -xe
/tmp/jenkins2112147489595430303.sh
+ chmod +x script.sh
+ ./script.sh
+ git add .
+ git commit -a -m 'Changes pushed by Jenkins'
HEAD detached at ee35229
nothing to commit, working directory clean
Build step 'Execute shell' marked build as failure
Triggering a new build of 2. Erase Answers
Finished: FAILURE

In order to get a success, you need to avoid non-zero exit codes. A simple solution would be to change the last line of your script to git commit -a -m 'Changes pushed by Jenkins' || true, but better solutions would parse the output of git-add and only run git-commit when there is something to commit.


Docker: strategy-advise for a rookie I am VERY new to Docker and I looking for a confirmation of my views and my usage of Docker in my use-cases.

I'll try it with examples:

I have a T3 application. Every tier is a basic requirement (lets say persistence with postgres, application with java, presentation with php). Because I find for any tier an image on docker-hub I use them and compose them to have 3 running containers which exchange their data via network. Correct?
I have an application that is in java and needs to work on the files of centos (e.g. read logs). There is not interaction - only stdout or files. Docker-images for centos are available, for java too. There is no easy way to create a Dockerfile with a double FROM. So I build my own image from centos and do ADD/COPY/RUN to get a java JDK into it. Correct?
I can do in docker-compose.yml also actions that I can have in dockerfile too (e.g. workdir and command). Is it better to keep it in dockerfile and see docker-compose only for assembling multiple containers OR should I put as many things into docker-compose as I can?

Yes, one would usually have minimal docker images, i.e., one responsibility per image/container, with appropriate networking between them.
Yes, images are linear, there is only ever one unbroken line of FROMs. To merge two images, get the two Dockerfiles and see what they are doing; it should be pretty self explanatory. Then you can either craft a single new Dockerfile mixing and merging lines from the sources, or keep one of the original dependencies as your FROM, and just copy lines from the other original to a new Dockerfile.
You would use command in a docker-compose.yml in the same cases when you would give a command on the docker run line, in case you were running the container manually. That is, if you have an image which conveniently supports different commands, and were it would just be a waste to have a different image per command. So... if you find that you run your container with the same command in 99% of all cases, then put that CMD into the Dockerfile and remove it from the docker-compose.yml. If you find, instead, that you are using different commands all the time, then keep them in docker-compose.yml.


How to keep original and new repositories in sync during a migration? (including the commit history) We have a git repository that is currently hosted on Azure Devops and have to migrate it to our in house TFS 2017 git (the move is due to a business decision).

But during this setup we need to run both the original and new repositories in parallel for a time and synchronise them regularly until the new repository is signed off by the Dev leads. Only after the new one is approved by the team lead will the devs switch over and then we will make the original read-only. Throughout this process, we need to ensure the developers remain productive.

So how do we perform the initial migration from the Azure Devops repository to the new TFS 2017 git?
And then how do we synchronise them regularly so that the new one does not get too far out of sync with the original?

We want to include the commit message history when migrating and syncing.

Simply keep pushing stuff between the repositories. Git pushes are designed for exactly that.

So, if origin is the old repository, and newis the new one, and you want to move new stuff in the master branch from origin to new, then, locally:

git checkout master && git reset --hard    # to clean up any local changes, optional
git pull origin
git push new

That's it. master is a branch like any other, so if you have multiple branches, rinse and repeat.


What is the optimum number of people in a weekly oncall schedule? What is the optimum number of people in a typical SRE/DevOps weekly on call schedule?

too few people (less than 4) could lead to burnout
too many people would lead to members forgetting the ropes of being on call, which isn't desirable either

Is there a reasonable cap on how many could be there in a weekly shift?

well, this is not your answer but I hope it helps ...
I lived both situations when I was single and no kids.
When the staff was short my boss tried to motivate us by talking about the extra money
and giving half day off. (I liked)
When the staff of complete I was getting on-call once a month and the money wasn't good enough to pay for the trouble. (I hated it)
I recommend to assess where your team is, personally and professionally, and put a few suggestion to be voted by them. Then give them the ability to self manage their schedule.


How to downgrade Jinja for Ansible? I'm using Ansible 2.8.0 (ansible --version installed by brew on macOS) which is using Jinja 2.10.

I've tried to downgrade Jinja to 2.8 via pip3 install jinja2==2.8, and I can confirm the right version with pip3 list, but Ansible is still using Jinja 2.10 (verified by these steps). It seems it's using different Python's site-packages folder.

I'd like to downgrade Jinja due to a bug, so I'd like to test Ansible using older version of Jinja.

How do I downgrade Jinja used by Ansible, ideally without downgrading Ansible it-self?

$  ansible --version
ansible 2.8.0
  config file = /Users/kenorb/.ansible.cfg
  configured module search path = ['/Users/kenorb/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/Cellar/ansible/2.8.0/libexec/lib/python3.7/site-packages/ansible
  executable location = /usr/local/bin/ansible
  python version = 3.7.3 (default, Mar 27 2019, 09:23:15) [Clang 10.0.1 (clang-1001.0.46.3)]
$  pip3 --version
pip 19.0.3 from /usr/local/lib/python3.7/site-packages/pip (python 3.7)
$  brew info ansible
ansible: stable 2.8.0 (bottled), HEAD

Since Ansible was installed via Homebrew on macOS, the workaround is to install Ansible using Pip.

$ brew remove ansible
$ pip3 install ansible
$  pip3 list | grep -e ansible -e Jinja2
ansible      2.8.0
Jinja2       2.8
$  ansible-playbook check_jinja.yaml -v
TASK [jinja_version]
ok: [localhost] => {"changed": false, "msg": "2.8"}


Does "docker run" actually SSH into the container? If I run

docker run -it ubuntu bash

Does it actually SSH into the container, or does it use another protocol?

No, it uses an internal "attach protocol" via the Docker daemon.


Jenkinsfile handling an expected job failure I've got two types of failures to account for within a Jenkins job.

The job genuinely failed
The job failed because of a certain bit of info hasn't been published. Nothing is actually wrong here.

How do I handle the 2nd case so it shows the job did no complete, but nothing is actually wrong?

One possibility would be to place the checks for all the necessary pre-conditions (i.e. including those bits of info that would need to be published) in a separate build step
to be executed prior to the actual build step.

If the checks in this preliminary step fail the build could be marked as aborted instead of failed. The aborted build would, hopefully, be a better indication that "nothing is actually wrong" (debatable, I know).

Marking the build as aborted could maybe be done in a manner similar to these SO answers:

Abort current build from pipeline in Jenkins
Jenkins - abort running build if new one is started

If those checks cannot be pulled in a step separated from the actual build step things are a bit more complicated:

the actual build step would have to somehow be configured to not fail the job. I know it's possible for steps performed in virtual environments using the ShiningPanda plugin, but other types of steps do not have such capability.
an additional step would have to be added after the actual build step. Its job would be to somehow determine if the actual build step failed and, if so, if failure was causing by one of those unfulfilled pre-requisite conditions (maybe by parsing the build logs, for example?), in which case it would abort the build. Otherwise it would pass or fail, depending on the actual build step outcome, to properly indicate the overall run status in Jenkins.


How to mark a step as failed or unstable on timeout, instead of aborting build? I have a stage in my Jenkins declarative pipeline that looks like:

stage('Step Tests') {
  steps {
    dir('test') {
      catchError(catchInterruptions: true, buildResult: 'FAILURE') {
        timeout(time: 5, unit: 'MINUTES', activity: true) {
          sh "yarn step-tests"
        }
      }
    }
  }
}

The timeout activates if my tests don't output anything to the console for 5 minutes, and subsequent stages still execute, which is what I want. However, the build is still marked as aborted instead of failed, which I thought the buildResult parameter was for.

How do I catch a timeout and mark the build as unstable or failed, instead of aborted?

I've done something similar to this in the past:

stage('Step Tests') {
  steps {
    dir('test') {
      script {
        try {
          timeout(time: 5, unit: 'MINUTES', activity: true) {
            sh "yarn step-tests"
          }
        } catch (Exception e) {
          currentBuild.result = 'FAILURE'
        }
      }
    }
  }
}

I haven't tested this code exactly as I wrote it here, so think of this more as a kind of rough sketch rather than polished final product.


config.yaml is not setting up a new context I have a kubernetes Config file such as:
apiVersion: v1
kind: Config
preferences: {}
contexts:
- context:
    cluster: test-sim-development
    namespace: test-sim
    user: developer
  name: test-sim

When I issue a command such as:
kubectl config --kubeconfig infra_k8/config.yaml set-context test-sim && kubectl config use-context test-sim

I get back the following error output:
Context "test-sim" modified.
error: no context exists with the name: "test-sim"

Why is it not finding the "test-sim" name? It is clearly referenced according to the kubernetes docs.

it seems you’re trying to change the config by setting a context and you’re not providing the config to the change context command. So do this
kubectl config --kubeconfig=infra_k8/config.yaml use-context test-sim

The setters are used when you want to add more entries to the config file, so you don’t need that command when changing context.


Kubernetes auto scaling in cloud environment Anyone knows if there is a solution to provision (non-managed) autoscaling Kubernetes clusters in the cloud?

I am looking for a generic solution that will work in any public cloud and in private data centers.

After quite a bit of research, the known available options do not answer the requirement above -

kops which is AWS specific is using own AMI images.
kubespray (aka kargo) has no notion of autoscaling, executes Ansible on a specific inventory of hosts.
kubeadm expects a known list of hosts and does not take care of OS setup like docker, kubelet, etc...
kubernetes autoscaler controls existing auto scaling groups, does not prepare the images used by those groups.

Is there some generic solution to make an autoscaling group deploy Kubernetes cluster on itself? Possibly a way to build cloud images (like AMI in AWS) using Packer for this.

Looking for a method that is not specific to one public cloud and allows to use an elastic environment that scales automatically based on prepared images.

Recently Kubernetes started preview work on https://cluster-api.sigs.k8s.io which is meant to be the standard way to solve the way how High Availability clusters are deployed.

The kubeadm tool has also been updated with features to deploy multiple masters since this question was first asked, which help tremendously. With the exception of deploying cloud-specific configuration, the documentation for kubeadm configuration YAML file brings you quite the way towards having a properly deployed cluster.

docs for kubeadm: https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm/


exponential backoff for git pushes? We've got a ton of jobs in our Jenkins that all do git pushes, sometimes simultaneously. Unsurprisingly, if two jobs try to push at the same exact time, one will succeed while the other fails. My knee-jerk reaction is to build some kind of exponential backoff and retry for these git pushes, but I'm pretty unfamiliar with git. Does something like that exist already?

Our commit/push code generally looks like this:

git config --global user.name "username"
git config --global user.email email@example.com
git add --ignore-errors -- dir/
git commit -m "some message"
git pull --rebase origin
git push origin master

As you can see, the bandaid originally applied to the problem was to put a pull before the push to fix other jobs that pushed while the current one was running. That works for ~80 of cases, but if the two pushes fall within the same <1sec window, there are still failures.

Here's an example of a failure for the last two lines above: (Anything in <> is edited for privacy)

+ git pull --rebase origin
Warning: Permanently added '<bitbucket.org ip>' (RSA) to the list of known hosts.
From <bitbucket.org:dir>
   <commit#>..<commit#2>  master     -> origin/master
First, rewinding head to replay your work on top of it...
Applying: <commit name>
+ git push origin master
Warning: Permanently added '<bitbucket.org, ip>' (RSA) to the list of known hosts.
remote: error: cannot lock ref 'refs/heads/master': is at <commit sha> but expected <different commit sha>
To <bitbucket.org:dir>
 ! [remote rejected] master -> master (failed to update ref)
error: failed to push some refs to 'git@bitbucket.org:<team/dir>'

So my question boils down to:

Is there some form of backoff-and-retry built into git?
If not, how can I build that (assume a shell block in a jenkins freestyle job)?

Instead of trying to find a solution in git, I would recommending using the Lockable Resources Plugin, which is an official plugin developed by the Jenkins developers.  This plugin implements locks in Jenkins, which are basically the same idea as a semaphore or mutex.  Using this plugin, each of your jobs that needs to push to the same branch of the same repo will request a lock when performing the push.  If the lock is available, the job will acquire the lock, push right away, and then release the lock when it is finished pushing.  If the lock is not available, the job will wait until the lock is available (i.e., all other jobs are done pushing) before grabbing it, pushing, and releasing the lock.


Is there any Selenium-like software for executables? I have some app I want to test i.e. it has a window opening up and I want to press some of the buttons and fill in some text boxes, and I need to do the same process multiple times.

I heard about selenium but it seems it only works on web-based apps, whereas my program is an executable program (and quite a heavy one while at it).  Is there something similar to this, a module to be added to selenium or a library for python/Matlab/C?

I’d appreciate any help with this.

I'm not a Windows user, but I'd start at Comparison of GUI testing tools, looking for those supporting Windows for both testing and tested system.


Jenkins & Bitbucket; cancel previous build if new commit made to same branch? We have Jenkins running unit tests when commits are made to our repo in Bitbucket. This is controlled by the Bitbucket Plugin, ie via a Bitbucket webhook.

Currently, if a commit is made to Branch A a unit test kicks off. If while that job is running a second commit is made to Branch A a second unit test will kick off, so there are now two unit tests on the same branch but with slightly different code.

Our preferred behavior is that the first commit's test would be aborted when the second test kicked off, so that only the most recent unit test is being run. Can this be achieved?

To clarify; We have many branches so we cannot just prevent concurrent builds, cancel the last as soon as the next starts, etc. Whatever method is used must specifically check if the branch already has a job running for it, not if the job in general is already running.

I've seen some trigger controls for Git, but not Bitbucket. I also found a script to check if the job's already running and cancel it if so, but as mentioned before that doesn't suit our use case. Am I missing something?

Note: this answer comes from my background of building custom solutions, it's not a config-only one, which, if available, would obviously be preferable. But it's maybe something to consider otherwise.

You could teach the script executing the unit test job to persist a per-branch execution state.

When the script is launched it would need to obtain the branch name from its invocation parameters. It would then obtain the persisted state for the branch.

If the state is either missing or not running it would set it to running + job ID, execute the unit test and then set the state to not running (or delete it).

If the state is running + job ID it means another unit test is already running and you have the corresponding job ID. The script can then (gracefully?) terminate the already running job by its ID and take it's place, updating the state to running + new job ID.

But IMHO terminating a job in progress (which may be very close to completion) is not ideal - it leaves room for constantly killing jobs and launching news ones only to be, at their own turn, killed by subsequent jobs - all without actually completing any of them.

Instead I'd modify the logic to:

leave the job in progress alone and instead just record in the persisted state a pending job ID (either a unique value, overwritten to reflect the most recent pending job submitted or a list of all pending jobs, sorted by submission time)
when the job in progress completes it checks the pending job ID values and, if any, launches the most recent one (or the oldest one, or even follow some other selection logic, if using a list)

This approach avoids wasting unit test resources and can easily be adjusted even for more than one but still limited resources.