Ansible: What's the difference between task, role, play and playbook? Ansible has tasks, roles, plays and playbooks.

Are those just names for different levels in a playbook organization?

If not, how, exactly, do they differ from and relate to each other?

No, each has a different role and way of working.
Starting with the easy ones:
A playbook is simply a list of plays.  The highest level of a playbook YAML is a list, and on that list only two things are accepted: a play definition or the keyword import_playbook, which imports a list of plays from another playbook file, as if they were defined in that place of the calling file.
The playbook is also the only thing that can be called directly from ansible-playbook.
A task is, paraphrasing the manual, 'nothing more than a call to an ansible module'.  That call definition, however, knows nothing of on which hosts it is supposed to run on.
The play is the element that ties tasks to the servers where they'll run.  The key element here is the mandatory keyword hosts.  This is the part of Ansible that tells which hosts are being affected and how.
So, the structure and behavior is as follows:

Playbook: the highest level, just a list of plays

Play: ties tasks to host lists

Tasks: definition of a call to a module
Besides tasks, a play may have pre-tasks, post-tasks and handlers, which are all task-like, and roles.

A role is a different thing, as it is not defined within a playbook.  Instead, they each have their own subdirectory under the directory roles.  The objective of roles is to organize things.  So, under the role's subdir, you can find tasks, handlers, files and templates, variables and defaults.  All of these related to that specific role.
So, roles are an easy way to encapsulate and share Ansible information.  You could have a subdir in templates per task, for example.  But if you wanted to share that task, you'd have to pick the template as well.  With roles, you can just zip the role dir and (almost) everything should be there.
Like a play, the role defines tasks and handlers.  However, roles do not define on which hosts the role will be run.  So roles must be referenced to from a play.
Roles can also declare dependencies (that is, references to other roles that need to be run before the declaring one).

PS1. Most of this information is available piecemeal on SO (except for playbooks, I couldn't find a proper definition of that on the documentation or elsewhere in the Internet), but I haven't found a comparison between them all, and it took me some time to figure this all out, so I'm sharing that by answering my own question.
PS2. I really don't like the fact that roles are named 'roles'.  It sounds a bit counter intuitive to me.  The word 'role' alone alludes to user roles or server roles, to me.  So something more related to Tower (users) or inventory (servers).  I think the rationale for that naming is that they define 'one of the roles that a server functions'.  Same could be said of plays, but roles are more abstract as they're not tied to specific hosts.  Still, I find that naming confusing.
Edit
A handler functions for the most part just like a task.  However, it is not executed by default, with the other tasks, during the playbook run.  Instead, each individual handler will be executed only if notified by a task.
On a task definition, the keyword notify identifies the handlers the task wants to notify.  If the task is executed and reports a changed state (not ok nor failed), then the handler will be marked for execution.
After all roles and tasks have run, each notified handler will be executed, in the order they appear in the file.  Check the docs for details.
Tasks will always run (unless skipped with when), and are responsible for the actual changes you want to implement.  Handlers will be run only in response to tasks making changes, and are generally responsible for finalization steps (restart or reload a service that has had its configuration changed to make it effective, for example).
You could probably replicate handlers behavior with tasks, but it is such a recurring pattern that Ansible provides handlers as a helper.


Node-centric network metrics are very different from the sum of metrics of the Pods running on that node Context
I am trying to chart the network bandwidth usage of a node in 2 different manners:

By looking at global metrics for that node
By summing up the corresponding metric for each Pod

To achieve this, I am issuing the following Prometheus queries (example for the receive bandwidth):

For the entire node (metric from node-exporter)
sum(irate(node_network_receive_bytes_total{instance="10.142.0.54:9100"}[$__rate_interval])) by (device)

Per Pod (metric from kubelet)
sum(irate(container_network_receive_bytes_total{node="$node",container!=""}[$__rate_interval])) by (pod,interface)

The results are displayed in the following Grafana dashboard, after generating some load on a HTTP service called thrpt-receiver:

Here's what I see if I look at the raw metrics, without sum() and irate() applied:

Problem
As you can see, results are vastly different, to the point I'm almost certain I am doing something wrong, but what?
What makes me especially suspicious about the Pod metrics is the supposedly increasing received bandwidth of kube-proxy (which AFAIK is not supposed to be receiving any traffic in iptables mode), and agents such as the Prometheus node-exporter, etc.

I found out what was happening in my graphs. All the Pods mentioned above have one thing in common: they use the host's network namespace, so their network metrics are all identical, and equal to the global host's metric (just with a slightly different precision).
$ kubectl - monitoring get pod -o jsonpath='{.spec.hostNetwork}' \
    prometheus-stack-prometheus-node-exporter-jnhw7

true

$ kubectl -n kube-system get pod -o jsonpath='{.items[*].spec.hostNetwork}' \
    kube-proxy-gke-triggermesh-product-control-plane-7fc0ad24-z586 \
    gke-metrics-agent-5cv4m \
    prometheus-to-sd-tk8jv \
    fluentbit-gke-xh879

true true true true

One way to see it is to compare the host's metric to one of the above Pods:


Export CloudFormation file from CDK Since I've seen Hashicorp Terraform, I have been wondering why the "plan" it can export has binary format.
Now in the AWS world you have got a declarative configuration language called CloudFormation which can be abstracted from using AWS CDK (Cloud Development Kit) in a common programming language of your choice.
Therefore my question: imagine somebody familiar with CloudFormation would like to see what CDK code will do. Is there a way to export a CloudFormation file using CDK?

Yes: in the hello world CDK example, they call it "synthesizing":
cdk synth
^^ produces CloudFormation file.


How to fix a "heartbeat failure" in Docker Swarm? My cluster is currently located in a single data center. I've been trying to change that by adding a single worker node from another data center, but so far it hasn't worked.

I'm able to make this node join the swarm and get listed by the managers, but it is always shown as "Down". Here is what "docker inspect" shows me about this node:

    "Status": {
        "State": "down",
        "Message": "heartbeat failure",
        "Addr": "xxx.xxx.xxx.xxx"
    }

I've opened the following ports in both sides:

2377 tcp
7946 tcp+udp
4789 udp

How do I troubleshoot and fix this?

This might not be the answer to your specific cross data-center-ip setup.

I occasionally run into one or more swarm-nodes being status: Down and Availability: Active. Having the  Status.Message: "heartbeat failure". This can happen after a reboot.

What helped was to stop the docker daemon, remove /var/lib/docker/swarm/worker/tasks.db and start the docker daemon again.

from:
https://github.com/moby/moby/issues/34827#issuecomment-457678500

sometimes it fixes itself: https://stackoverflow.com/a/54126180/2087704


TFS Back-up Notifications Our company uses TFS 2018. We have an annoying issue when there is TFS back-up failed,it sends notifications to hell lot of people which are invalid or unncecessary users. This list is in the domain Admins. I can't delete them manually.

Is there a script to delete the users or modify the sending list?

I believe your primary issue is that you have this group of users in the [Team Foundation]\Team Foundation Administrators group or possibly many users added with privileges to the Administration Console Users, which is unusual.  To adjust who is part of the Team Foundation Administrators group:

RDP to the server that hosts the application tier for your TFS installation
Open the TFS Administration Console
Highlight the Application Tier
Click the Group Membership link
Double-click [Team Foundation]\Team Foundation Administrators
Adjust membership of the group

To learn more about configuring administrators from the TFS Administration console, see Microsoft's documentation on the topic.


Jenkins Pipeline "Build when a change is pushed to GitLab" I have a Jenkins declarative pipeline that is ran via a webhook from GitLab whenever a change is pushed to my repo. In Jenkins, I have the following trigger option selected:

Is there a way for me to add this trigger to my Jenkins declarative pipeline, if so how?

Looks like you are using the Gitlab plugin for jenkins?
if so this link should be of help https://github.com/jenkinsci/gitlab-plugin#declarative-pipeline-jobs


How do I automate population of known_hosts and authorized_keys? This is a question that may be easier answered on askubuntu or, perhaps, serverfault, but it arises in the context of automated provisioning, so I figured that it's more appropriate here.

I am spinning up a multi-machine environment in vagrant, e.g. n+1 ubuntu-18.04 boxes, namely

term (short for terminal)
node1
...
nodeN

They can all resolve each other by the above hostnames. It's necessary, that term can SSH into each of the nodes using its public key.

Manually I can do this with the following script:

#!/bin/bash

head='node'

ssh-keygen -N '' -f ~/.ssh/id_rsa

for ((i=1; i<=$1; i++)); do
  name=$head
  name+=$i
  ssh-copy-id $name
done

running, e.g., ./copyid.sh 3. But then I have to type yes (to confirm the fingerprint of the node) and vagrant (the password) three times.

I want to move this procedure to the provisioning of the VMs in the Vagrantfile. So I have two questions:

How can this be automated without demanding manual input from myself?
When I transfer over from virtual machines to bare metal servers, what best practices should I follow to prevent infosec guys from screaming "man in the middle" at me?

You have to execute ssh-keyscan. For example to ssh to a host (github.com here ) you have to run below script

# Add ssh key to help cloning private github repo

ssh-keygen -t rsa -N "" -f secrets/ssh/github_rsa
PUB_KEY=$(cat secrets/ssh/github_rsa.pub)
PRV_KEY=$(cat secrets/ssh/github_rsa)

echo "${PRV_KEY}" >> ~/.ssh/github_rsa
chmod 600 ~/.ssh/github_rsa
eval $(ssh-agent)
ssh-add ~/.ssh/github_rsa

ssh-keyscan github.com >> ~/.ssh/known_hosts
echo IdentityFile ~/.ssh/github_rsa >> ~/.ssh/config

echo "Paste the following public key to your host machine ".${PUB_KEY}


How to copy library source code from other Git repo to an Android Gradle application GitLab build process? There is an Android application using Grandle for build. It uses a "library" stated in build.gradle file:
dependencies {
    api project(path: ':specialproject.speciallibrary', configuration: 'default')

It is configured in settings.gradle:
include ':specialproject.speciallibrary'
project(':specialproject.speciallibrary').projectDir = new File('../specialprojectlibrary')

Developers currently have the main project and specialprojectlibrary projects side-by-side on their computers. So when they build it, Gradle fetches specialprojectlibrary and builds it when building the main app (APK.
Now, when would like to have GitLab CI/CD for the app. Main app and library app are in separe Git repositories and under separate projects inside a group in GitLab.
After creating CI/CD it occured it throws error:
FAILURE: Build failed with an exception.
* What went wrong:
Could not determine the dependencies of task ':something:xyz'.
> Could not resolve all task dependencies for configuration ':something:xyz'.
   > **Could not resolve project :specialproject.speciallibrary.**
     Required by:
         project :mainproject
      > Project :mainproject declares a dependency from configuration 'api' to configuration 'default' which is not declared in the descriptor for project :specialproject.speciallibrary.

I assume it is because it canon CI/CD Gradle of the main app does not have access to library code under .//specialprojectlibrary.
So my question is:
How to copy library source code from other Git repo to an Android Gradle application GitLab build process?

What you need is not to checkout both projects side by side, but to add the library as a Git submodule in your application repository.
Then set up GitLab CI to checkout submodules before build.
This will not only make your CI work, but as an added benefit each commit from application repository will reference a specific commit from library repository, so  nobody will have to remember what commit to checkout in library repository to ensure the application builds correctly.
Edit:
To be clear, I'm not saying that composite builds (which is a proper name for your setup) are always bad, but I believe that they should be used for optional dependencies (like JetBrains did in their kotlin-native repository) or when developers need need to have an option of replacing dependency norally pulled from some Maven repository with locally build version (eg. to test if their app works with some changes in the library before submitting pull/merge request for the library) and not for everyday use.


How to configure Docker tooling on Eclipse? The Eclipse plugin "Docker tooling" seems to expect the installation of Docker Machine, or a network connection.

But, on Windows 10 things seem different from what this plugin expects so I can't browse for the required executables.
Also, how could I find out the local network URL? docker info does not reveal this.

Under Windows 10, you'll find the Docker Machine (docker-machine.exe) under: C:\Program Files\Docker\Docker\resources\bin

This is the value you need to put in the Docker Machine field.
You don't need to change anything for the VM Driver field as VirtualBox doesn't need to be installed when you use Docker for Windows.

Then you have to edit some setting of the Docker daemon to allow your Eclipse Docker plugin to connect to it:

First of all, check if the Docker daemon is running by clicking on the Task View button on the Windows taskbar.
Mouse over the Docker icon and the tooltip should say: Docker is running.
If it's running, right-click the Docker icon to open the Docker contextual menu and go to Settings.
In the General Tab, which you will land on by default, Make sure the
"Expose daemon on tcp://localhost:2375 without TLS" checkbox is checked.
(It isn't by default)

You can now edit your Docker Connection in Eclipse and do not need to search for a connection name, simply enter how you want to name it (ie. SomeDockerConnection) and check the "TCP Connection" radio button.

You can enter tcp://localhost:2375 in the URI and click "Test Connection".
You should see a message appear to confirm that the ping was successful.

At this point you'll be able to see your containers and images in Eclipse's Docker Explorer perspective.


What's are the advantages of using CloudBees Core over Jenkins? I have tried Jenkins on my own and about to propose an enterprise grade CI/CD product to my development team which is around 40 people. CloudBees Core come at the top of my selected list since it is based on Jenkins and enterprise friendly as well.Apart from enterprise level support, I still can't really see the benefits to use CloudBees Core over Jenkins.

Is there anyone who used Cloudbees through your DevOps transformation journey ?

I would appreciate if you could discuss proprietary plugins and out of the box features that CloudBees Core provides.

I personally prefer Jenkins over CloudBees as we have enough Jenkins knowledge in our engineering department. However, if their is a lack of such knowledge or interest then it could be useful to use a Software as a Service (SaaS) product like CloudBees. In general, this means that the department will be unburden as the platform will scale automatically and the SaaS provider is responsible for keeping Jenkins up and running.

A downside of an Enterprise solution is that a fee is required. However, maintenance and in particular the number of resources that is required to apply it, is often underestimated in my opinion. Maintaining Jenkins by Engineers from the department means that they have less time to develop code. At the moment, getting a CloudBees subscription will not solve our problems as we are currently busy splitting the monolith.

Whether you should get a CloudBees membership really depends on your situation. Do you have a monolith? Has all code been untangled. Do you have microservices with clear boundaries or are they nanoservices? Long story short, do you have a robust architecture? If false, then CloudBees will also not be the holy grail.

While Googling, another post on stackoverflow was found that could be useful:

  The first difference is support (as others have mentioned). CloudBees
  offers enterprise grade support as well as a fully vetted and tested
  version of Jenkins that will be more stable under various plugins and
  deployments. You can actually purchase "Support Only" from CloudBees
  if you are satisfied with your OSS Jenkins deployment and simply want
  support during upgrades, patching, break/fix, etc.

  From a feature perspective, CloudBees brings a lot from an enterprise
  manageability, scalability, and security standpoint.

  Manageability: CloudBees comes with CJOC (CloudBees Jenkins Operations Center) built into the software. This is a single pane of
  glass management console that allows organizations or large teams to
  centrally manage the jenkins environment. Things like folders, RBAC,
  pipeline and master templates, and the ability to rapidly spin up/tear
  down a containerized jenkins master are all managed from this single
  console.
  Scaleability: CloudBees leverages Kubernetes to provide organizations with the ability to elastically scale Jenkins
  environments as needed. With CloudBees, your oganization can move away
  from a single "Monolithic"/"Frankenstein" master and into a
  multi-master and distributed pipeline architecture. This greatly
  reduces upgrade and administration complexity. This also eliminates
  the risk from a single point of failure that a monolithic architecture
  exposes.
  Security: CloudBees allows organizations to install Roll Based Access Control within Jenkins. This keeps users from accidentally or
  intentionally accessing repos that they shouldn't be allowed to
  interact with. CloudBees also provides "folders" to segregate specific
  job executions onto specific agents. Lastly CloudBees allows
  organizations to create pipeline templates and associated plugins for
  each team. These templates can be as rigid or loose as desired per the
  organizations security policies.

  CloudBees is regularly adding enhancements to further differentiate
  themselves from Jenkins Open Source and make themselves more appealing
  to large enterprise requirements.

  On top of the above, CloudBees has developed a presentation layer that
  rides on top of Jenkins for SDLC pipeline, CD monitoring, and metric
  tracking called DevOptics.


Why does the datasource "external" always fail? so my issue here is that terraform really does not like a command I'm trying to run using an external datasource.

I'm trying to hack my way around a problem where terraform wont wait for a KMS encrypt operation to complete before attempting to use the ciphertext returned by KMS. I've also raised an issue in the terraform for this.

So here is what I'm attempting to do:

resource "aws_kms_alias" "kms_alias" {
    name          = "alias/${var.lambda_name}"
    target_key_id = "${aws_kms_key.hook_url.arn}"
}

data "external" "encrypt_url" {
    program = ["bash", "export ENC_HOOK_URL=$(aws --profile $${AWS_PROFILE} \
    kms encrypt --key-id $${KMS_ALIAS} --plaintext $${HOOK_URL} --output json \
    --region $${AWS_REGION})"]
}

resource "aws_lambda_function" "lambda_function" {
    depends_on       = ["data.external.encrypt_url"]
    filename         = "deploymentPackage.zip"
    function_name    = "${var.lambda_name}"
    role             = "${var.lambda_role_arn}"
    handler          = "index.lambda_handler"
    source_code_hash = "${base64sha256(file("deploymentPackage.zip"))}"
    runtime          = "python3.6"
    publish          = "true"

    environment {
        variables = {
            HOOK_URL = "${data.external.encrypt_url.result.ciphertextBlob}"
        }
    }

    tags {
        LastUpdated = "${var.timestamp}-${var.iamuser}-Terraform"
    }
}

This will run fine without the encrypt_url datasource but will fail with the following error when trying to run:

data.external.encrypt_url: data.external.encrypt_url: failed to execute "bash":
bash: export ENC_HOOK_URL=$(aws --profile ${AWS_PROFILE} kms encrypt --key-id ${KMS_ALIAS} --plaintext ${HOOK_URL} --output json --region ${AWS_REGION}):
No such file or directory

Now no file or directory is attempting to be called here and I'm unable to find much information about this because this function is not well documented.

If anyone has any pointers that'd be great!

The command being run by the external provider here is equivalent to the following:

bash "export ENC_HOOK_URL=$(aws --profile ${AWS_PROFILE} \
kms encrypt --key-id ${KMS_ALIAS} --plaintext ${HOOK_URL} --output json \
--region ${AWS_REGION})"

Bash is failing here because when used in this way it expects its first argument to be the filename of a script to run.

The -c option changes this interpretation so that it will instead expect this argument to be an inline script to run, which seems to be what you intended here. This could be expressed in the Terraform configuration like this:

data "external" "encrypt_url" {
  program = ["bash", "-c", "export ENC_HOOK_URL=$(aws --profile $${AWS_PROFILE} kms encrypt --key-id $${KMS_ALIAS} --plaintext $${HOOK_URL} --output json --region $${AWS_REGION})"]
}

Along with adding the -c option, it also seems like this script doesn't actually do anything, instead just assigning a result to a variable. Given that you asked the AWS command to produce JSON, I think perhaps the following will do what you want:

data "external" "encrypt_url" {
  program = ["bash", "-c", "aws --profile $${AWS_PROFILE} kms encrypt --key-id $${KMS_ALIAS} --plaintext $${HOOK_URL} --output json --region $${AWS_REGION}"]
}

In the interests of readability though, I'd recommend moving this command into an external script file (e.g. encrypt_url.sh) and then referencing it from the config:

#!/usr/bin/env bash

aws --profile ${AWS_PROFILE} \
    kms encrypt --key-id ${KMS_ALIAS} \
                --plaintext ${HOOK_URL} \
                --output json \
                --region ${AWS_REGION}

...

data "external" "encrypt_url" {
  program = ["bash", "${path.module}/encrypt_url.sh"]
}

This avoids the need for escaping the dollar signs, allows you to edit it in your text editor's shell script highlighting mode, and allows the script to be easily run outside of Terraform for testing purposes.


What is good and/or bad about Accenture's Multi-speed IT approach to DevOps? Accenture are talking a lot about their "Multi-speed IT" approach.

It appears in their strategy consulting here:

https://www.accenture.com/us-en/insight-calibrating-multi-speed-it

and on blog posts in their DevOps blog:

https://www.accenture.com/us-en/blogs/blogs-support-multispeed-it-with-devops-agile
https://www.accenture.com/us-en/blogs/blogs-emily-arnautovic-devops-change

As well as in many other places that deal with various aspects of a technology organisation.

Their own "head of DevOps" is talking about how Multi-speed is a bad practice https://www.youtube.com/watch?v=yGcXLwsnDgg

Does this idea of "Multi-speed IT" has merit and can be used for continuous improvement in-line with DevOps? Or does it go against some of the common DevOps practices?

There are reasons different parts of the environment might change at different frequencies.  One example might be a calendaring service, most large organizations have one, they provide information about the current day, if it is a holiday etc.   The calendaring service does not need to change frequently because it provides a stable function.  This calendaring service may be used by many different kinds of apps that also move at different speeds.

Multi speed exists because you need to deal with making changes that deliver value.  Many people indicate multi speed is to deal with the font end system of engagements and the systems of record.  Yes, generally user facing systems change more often because of the desire to update for the end users and experiment with different ways of presenting data to get the best value.  However, the systems of record may need to be changed as fast, in order to satisfy a new business requirement.

If all parts of the application are built using a pipeline with appropriate automated testing, and deployment, and their are ways to test independently but also test together, then any part can move as needed.  It is similar to the gears thought that the different parts move but need to be integrated such they you know they work together as they are deployed.


How to improve DRP-testing of SE sites? Have a look at what's described in the question about "Brief outage planned for Wed, May 3, 2017 at 0:00 UTC, 8pm US/Eastern (like a fire drill for computers)", which is about testing of the existing "Disaster Recovery Planning" (= DRP) of the entire family of SE sites.

If you were in charge of this, what would be your recommendations to improve these kinds of DRP-testing in production?

NOTE: It's probably not worth reading too much into the outward comprehension of how good-or-not StackExchange is at managing their disaster recovery scenarios.  I suspect they are following much of the best practice below and simply testing scenarios to validate their configuration.

Depending on the environment you operate within:

A disaster recovery plan might form part of a larger Business Continuity Plan, a Business Continuity Plan would also likely consider operational risks to you people, organisation, locations, information, partners and management systems.
A disaster recovery plan might be broken down into many IT Service Continuity Plans for individual services.  The disaster recovery plan would likely incorporate people and process along with the technical aspects of the service.

Given these definitions, you might consider ways of improving the capability of the organisation as a whole to be resilient against failure:

Service Recovery:

Enable individual services to be Active-Active across two geographically dispersed data centres.  This does assume that applications are capable of replicating state between data centres, for example using BASE Semantics for data.
Create Self-Healing services, this means expecting failure and building with Resilience Engineering in mind.  An example is by using a tool such as Chaos Monkey to simulate a failure.

Disaster Recovery Plan:

Again enable Active-Active mechanisms across data centres, the difference from SRPs is that capacity needs to be carefully considered, i.e. if you have to DCs in an Active-Active pattern and one fails then a single DC must be sufficiently scaled to support 100% of the traffic.
War Games and Rehearsals are really important for disaster recovery plans as this tests the people and the process, in the most mature DevOps environments much of this can be automated as evidenced by Chaos Gorilla.

Business Continuity Plan:

On the basis that this is a DevOps site I won't go into the long process of building Business Continuity plans.  However, the rules of not keeping all of your eggs in one basket apply - have a plan for when your office is flooded:

Allow your staff to work remotely from home one day a week, this tests your BCP strategies.
If possible have to geographically and politically separate locations for your workforce.
Define and test a clear process for communicating business continuity events and practice them through fire drills.


What is the AWS user permission that allows attaching and detaching IAM Roles to instances? My coworker is trying to attach IAM roles to EC2 instances and doesn’t have permissions. I’m trying to work out which permissions to give him.

My question is: What is the AWS user permission that allows attaching and detaching IAM Roles to instances?

From Granting a User Permissions to Pass a Role to an AWS Service:

  To pass a role (and its permissions) to an AWS service, a user must
  have permissions to pass the role to the service. This helps
  administrators ensure that only approved users can configure a service
  with a role that grants permissions. To allow a user to pass a role to
  an AWS service, you must grant the PassRole permission to the user's
  IAM user, role, or group.

  When a user passes a role ARN as a parameter to any API that uses the
  role to assign permissions to the service, the service checks whether
  that user has the iam:PassRole permission. To limit the user to
  passing only approved roles, you can filter the iam:PassRole
  permission with the Resources element of the IAM policy statement.

Is this what you're looking for?

An example from the above-metioned page:

  Example 1

  Imagine that you want to grant a user the ability to pass any of an
  approved set of roles to the Amazon EC2 service upon launching an
  instance. You need three elements:

  An IAM permissions policy attached to the role that determines what the role can do. Scope permissions to only the actions that the role
  needs to perform, and to only the resources that the role needs for
  those actions. You can use AWS managed or customer-created IAM
  permissions policy.

{
    "Version": "2012-10-17",
    "Statement": {
        "Effect": "Allow",
        "Action": [ "A list of the permissions the role is allowed to use" ],
        "Resource": [ "A list of the resources the role is allowed to access" ]
    }
}

  A trust policy for the role that allows the service to assume the role. For example, you could attach the following trust policy to the
  role with the UpdateAssumeRolePolicy action. This trust policy allows
  Amazon EC2 to use the role and the permissions attached to the role.

{
    "Version": "2012-10-17",
    "Statement": {
        "Sid": "TrustPolicyStatementThatAllowsEC2ServiceToAssumeTheAttachedRole",
        "Effect": "Allow",
        "Principal": { "Service": "ec2.amazonaws.com" },
       "Action": "sts:AssumeRole"
    }
}

  An IAM permissions policy attached to the IAM user that allows the user to pass only those policies that are approved. iam:PassRole
  usually is accompanied by iam:GetRole so that the user can get the
  details of the role to be passed. In this example, the user can pass
  only roles that exist in the specified account with names that begin
  with EC2-roles-for-XYZ-:

{
    "Version": "2012-10-17",
    "Statement": [{
        "Effect": "Allow",
        "Action": [
            "iam:GetRole",
            "iam:PassRole"
        ],
        "Resource": "arn:aws:iam::<account-id>:role/EC2-roles-for-XYZ-*"
    }]
}

  Now the user can start an Amazon EC2 instance with an assigned role.
  Applications running on the instance can access temporary credentials
  for the role through the instance profile metadata. The permission
  policies attached to the role determine what the instance can do.

The procedure(s) to attach a policy to a user/role (inlining it might work as well) are described in Attaching and Detaching IAM Policies:

  To attach a managed policy (console)

  Sign in to the AWS Management Console and open the IAM console at https://console.aws.amazon.com/iam/.
  In the navigation pane, choose Policies.
  In the list of policies, select the check box next to the name of the policy to attach. You can use the Filter menu and the search
  box to filter the list of policies.
  Choose Policy actions, and then choose Attach.
  Select one or more identities to attach the policy to. You can use the Filter menu and the search box to filter the list of principal
  entities. After selecting the identities, choose Attach policy.

...

  To embed an inline policy for a user or role (console)

  Sign in to the AWS Management Console and open the IAM console at https://console.aws.amazon.com/iam/.
  In the navigation pane, choose Users or Roles.
  In the list, choose the name of the user or role to embed a policy in.
  Choose the Permissions tab.
  Scroll to the bottom of the page and choose Add inline policy.

  Note

  You cannot embed an inline policy in a service-linked role in IAM. Because the linked service defines whether you can modify the
  permissions of the role, you might be able to add additional policies
  from the service console, API, or AWS CLI. To view the service-linked
  role documentation for a service, see AWS Services That Work with
  IAM and choose Yes in the Service-Linked Role column for
  your service.
  Choose from the following methods to view the steps required to create your policy:

  Import an Existing Managed Policy – You can import a managed policy within your account and then edit the policy to customize it to
  your specific requirements. A managed policy can be an AWS managed
  policy or a customer managed policy that you created previously.
  Create a Policy with the Visual Editor – You can construct a new policy from scratch in the visual editor. If you use the visual
  editor, you do not have to understand JSON syntax.
  Create a Policy on the JSON Tab – In the JSON tab, you can use JSON syntax to create a policy. You can type a new JSON policy document or paste an example policy.

  After you create an inline policy, it is automatically embedded in your user or role.


Azure DevOps - Conditionally Build and Deploy Say we have 6 microservices in a monorepo, each with their own CI build pipeline that is currently only building each project and running unit tests. Is there a way to identify which projects have had code changes since the previous release and only build and deploy those, leaving others untouched? I've not seen anything in the documentation examples that describe this as a possibility but I'm hoping that with all of the conditional and scripting abilities in YAML and possibly templates, never used them, that I could achieve this.

One way forward without this would be to manually comment out the services that I know had no changes in the YAML and commit that, then build from the updated definition. This is open to human error so I'd like to increase the level of automation.

Builds - Get Changes Between Builds
Something like this looks interesting but I'm not sure how I'd integrate it with YAML

The goal of all of this is to only version stamp and deploy what has changed. I may be missing some core DevOps concepts here and am happy to read more if you'll guide me.

You can create build pipeline triggers to trigger your builds based on branch along with path triggers. In your release pipeline, you can set triggers and filters to decide which artifacts to be deployed.


Jenkins pipeline fails with No such property error I have Jenkins container (version 2.138.1) with a pipeline which uses the plugins:

Job DSL version 1.7.0

Repository Connector version 1.1.3

P4 version 1.8.14

it fails with the below error:

11:23:17 ERROR: (int_dbp_ci.groovy, line 7) No such property: IntDBCIProperties for class: int_dbp_ci

what's strange, in the same machine, in other Jenkins container, for the same job with the same exact syntax and plugins version, it works. any idea what might cause the issue?

The issue was with one of the options present in global security configuration “ Enable script security for job DSL scripts”, which needs to be unchecked.


Is FaaS required to have an official "Cloud"? I'm in the process of refining my DevOps skills, but my "Cloud" is more like a lot of scripted virtual machines and monitoring the system.  For example, if I have Oracle VM Server, and I install Ansible, Docker, Kubernetes, etc., and I can do CI, provision VMs, containers, monitoring with Elastic...does that qualify as a Cloud?

What I might provision:

Developer needs CI.  The developer uses Team Foundation Server which has CI calling Jenkins or something else, deploying.

Developer's App starts getting hit, a lot, hey, we need x, I scale up with more powerful VM.  I scale out if the application is engineered to allow it.  Database scaling, sharding, caching more, indexes, the whole DBA part.

*To me this is more DevOps.  You understand both Dev and Ops world and know when Ops can't help anymore per se, and when an application needs engineering.

There are other things with the scripting and log monitoring.

But, I do not have true Cloud software like Azure, Oracle Cloud, and I don't have Faas.

I don't want to misrepresent myself as a Cloud Engineer or like title.

Google’s free computing course says:

  The US National Institute of Standards and Technology created a
  definition of cloud, although, there is nothing US specific about it.
  Here it is, cloud computing is a way of using I.T. that has these five
  equally important traits. First, you get computing resources on-demand
  and self-service. All you have to do is use a simple interface and you
  get the processing power, storage, and network you need, with no need
  for human intervention. Second, you access these resources over the
  net from anywhere you want. Third, the provider of those resources has
  a big pool of them and allocates them to customers out of that pool.
  That allows the provider to get economies of scale by buying in bulk
  and pass the savings on to the customers. Customers don't have to know
  or care about the exact physical location of those resources. Fourth,
  the resources are elastic. If you need more resources you can get
  more, rapidly. If you need less, you can scale back. And last, the
  customers pay only for what they use or reserve as they go. If they
  stop using resources, they stop paying.

This seems to be taken from the document at https://www.nist.gov/news-events/news/2011/10/final-version-nist-cloud-computing-definition-published

Using DevOps software with virtual machines, with or without containers, can happen without using cloud. In a similar manner, you can open a corporate account on a public cloud and not do DevOps at all. This happens more than we might expect when big companies are migrating legacy systems into the cloud where they simply have sysadmins or other Ops folks use the web interfaces to the cloud IaaS to manually commission VMs that they then manually configure. I have met teams that are convinced that they are doing DevOps when they are simply doing Ops with a few specialist Cloud Engineers who handle setting Virtual Private Cloud (VPS) meaning private connectivity into the cloud (e.g, VPN or a dedicated connection) and IAM (Identity and Access Management).

IMHO cloud and devops are most definitely not the same thing but are at their best when combined.


How to connect to a docker container? I have the following Dockerfile:

FROM ubuntu:latest
MAINTAINER Valter Silva
RUN apt-get update
RUN apt-get install -y apache2
ADD index.html /var/www/html/
CMD /usr/sbin/apache2ctl -D FOREGROUND
EXPOSE 22
EXPOSE 80

This is how I build my image:

docker build --tags mob:latest .

Next, this is how I bring my container up:

docker run -d --name website mob

Then I check my container's ip via the following command:

docker network inspect bridge

I would like to execute a curl on this ip to ensure that my website is up and running:

curl 172.17.0.2

But for some reason, my command keeps hanging. I'm not sure why this is happening. I'm running my dockeron a MacOS system.

You can get to know if your site is up and running in the following way,

Map both the ports of inside container to the host using -p option and try to curl localhost:port

Scenario:-

Run your container with the following command,

docker run -d --name website -p 80:80 -p 22:22 mob

Explanation:-  -p host_port:container's_port

Now, curl localhost:80 or localhost:22

If your service is running properly, you will have got the response

P.S:- Make sure nothing is running in 80 and 20 on host before running the docker run command


What are the expected effects of the acquisition of GitHub by Microsoft? When one navigates to github.com it is announced that Microsoft has acquired Github.

There are some rumors that multiple projects were migrated to other git providers like gitlab and bitbucket. What effect will the acquisition have from a DevOps perspective. To be more specific, the integration of Travis and GitHub works out of the box? Should one worry? E.g. will users have to buy licenses in order to use GitHub? Will the travisci integration be neglected?

Microsoft has heavily invested in their Git Virtual File System. It is unlikely then that such a key system central to Microsoft's internal development processes will be going anywhere any time soon, and GitHub is a smart dovetail with Microsoft's portfolio and strategic direction.

Microsoft has recently been moving towards more freemium services - which GitHub already is - and moving towards open source - the company was one of the top contributors to open source in 2017. Other notable examples are things like:

The Linux Subsystem for Windows
The open-sourcing of power shell
Contributions to the Linux Kernel

This also provides a strategic opportunity to integrate Visual Studio (which already has pretty good integration with Git) more tightly.

In short, Microsoft will probably do very little to harm GitHub - it will only work to make it solvent. This acquisition may have been very much about preserving GitHub to ensure that Microsoft will be able to continue to leverage its' features - the very same features you probably use. Nonetheless, guardianship of GitHub will be a difficult balancing act, as Microsoft must be careful not to alienate developers but they may have to end the gravy train as GitHub has been bleeding money for some time. Ultimately this acquisition is likely to be a good thing for the developer community as it will help to preserve and continue the public services that GitHub provides, but at worst things will look like the migration from SourceForge to whatever the new and shiny hot young thing is.


Does load balancer F5 support CasC (Configuration as Code)? Using CasC with F5 would allow for versioned and optionally dynamic network endpoint configuration, saving time and reducing risks.

Does this tool support this? Are Single Configuration Files (SCF, F5 term) the vehicle to do that?

Yes, F5's do support Configuration as Code. Historically, F5 has made an appliance to manage Configuration as Code called "Enterprise Manager" which pragmatically managed client endpoint F5s (LTMs, etc) using the iControl XML API.

They rather quickly discovered this management appliance was terrible and added a more robust REST API to the client appliances for managing devices (LTMs, etc.; also branded as iControl) which is much easier to use and more flexible and then began building a replacement for the Enterprise Manager branded BIG-IQ.

The point being, you can manage that same API using this REST interface. See their tutorial on DevCentral. Usually, you can find the exact REST syntax and calls on the DevCentral site on a per-version basis like this one for TMOS 12.1.0.

Generally speaking, it is NOT recommended that you use an SCF (Single Configuration File) pretty much ever for a couple of reasons. First, it lacks any supporting files, such as certificates and keys for your SSL profiles or scripted (so-called "external" health monitors), etc. Secondly, it plays poorly if  you use partitioning, as partitions are spread over several files in a foldered structure. These don't consolidate well into an SCF file. You would actually be better off to write TMOS scripts. One of the reasons F5 switched from the bigpipe command to the TMOS shell is that it could be scripted where bigpipe could not be easily scripted. But again, the REST API is preferred. SCFs are really a legacy of version 9 of TMOS and haven't aged well and work poorly in version 12. A significant reason for this is due to the changes in the HA peering between V10 and V11 when they changed to a clustered architecture. It really wreaked havoc on the usability of the SCFs.

Puppet actually has a module for managing F5s if you use this configuration management tool and salt has a runner for it - both using the REST API if you use either of these configuration management tools.


Relation of devops with release management How does DevOps deal with release management?

Is release management is the part of devops lifecycle?

Do devops take active part in release management or the organization has hire to another professional altogether.

Your use of the term "devops" suggests that you see it as a role.  I humbly suggest that developers that are part of a Scrum team aren't called "Agiles," so DevOps practitioners aren't "devops." :)

Release management is absolutely part of DevOps.  It's one of several specialty engineering roles.  Staffing for the skill of release management varies widely from one company to the next.  The size of the company, the industry, and the size of the IT department all influence how (and if) that skill is isolated to a person, a team, or distributed across IT.

Organizations of medium to large size tend to have dedicated staff that understand the practice and tooling of Continuous Integration.  This includes some systems admin skills, and increasingly includes development skills, at least scripting.  The growth of Infrastructure as Code as a practice is blurring the lines between release managers and developers.


Which problems does a container orchestration solve? Every company who has anything to do with containers these days, including public cloud providers, is creating their own solution for container orchestration.

These include:

Kubernetes
Docker Swarm
Amazon EC2 Container Service
Apache Mesos
Mesosphere DCOS
HashiCorp Nomad
other

What are the common problems with containers that must be solved by these orchestration solutions? What is the benefit of adding such a complex system in your production?

If you are only running a single container or two containers together you are correct in that an orchestrator may be unnecessary and add unneeded complexity.  However, these tools do solve several issues when you running several containers together (especially in production).

  What are the common problems with containers that must be solved by these orchestration solutions

Containers run in an isolated process (usually in it's own namespace).  This means that by default the container will not be aware of other containers.  Additionally, it will not be aware of the systems files, network interfaces, and processes.  While this can greatly help with portability of the software it does not solve several production issues such as microservices, container discovery, scalability, disaster recovery, or upgrades.

  What is the benefit of adding such a complex system in your production?

Adding a container orchestrator can greatly reduce the complexity in production as these tools are designed to resolve the issues outlined above.  For example, Kubernetes is built to allow containers to be linked together, deploy containers across an entire network, scale and load balance the network based on container resource consumption, and allow upgrades of individual containers with no downtime.


docker-compose.yml - make network name configurable via environment variable I have a docker compose file with env variables for stage specific configurations. As long as env variables are used in values in docker-compose.yml everything is fine, but the problem is with the network name, since it's specified in the tag.

networks:
  mynetwork.${STAGE_NAME}:
    driver: bridge
    ipam:
      driver: default
      config:
        - subnet: ${STAGE_NETWORK_PREFIX}.0/24

Any chance to get the network name mynetwork.${STAGE_NAME} configurable from outside?

If you have envsubst installed (part or gnu gettext) you can use something like that:

export STAGE_NAME="xyz"
export STAGE_NETWORK_PREFIX="1.2.3.4"
docker-compose -f <(envsubst docker-compose.yml) ...

If your shell does not support process substitution, something like that would do the trick:

envsubst docker-compose.yml | docker-compose -f - ...


Jenkins: mercurial scm: clean=false is not working as expected EDIT: It seems as if the MercurialSCM plugin hardcodes the update to use the --clean option. See this line. Anyone have any ideas what's the idea?

I have the following checkout step in my Jenkinsfile:

checkout([
        scm: [
            $class: "MercurialSCM",
            source: "ssh://hg@bitbucket.org/xxxxx/${repo}",
            credentialsId: "jenkins",
            revision: params[branch] ? params[branch] : "default",
            subdir: (repo == "f8_root") ? "" : "aarch64/${repo}",
            clean: false,
        ],
        poll: true,
])

The argument for clean is set to false. But, the plugin still seems to issue an update with the --clean option. From the logs:

[Pipeline] checkout
[f8_fw] $ hg showconfig paths.default
[f8_fw] $ hg pull --rev feature/test-framework
pulling from ssh://hg@bitbucket.org/doblesc/f8_fw
no changes found
[f8_fw] $ hg update --clean --rev feature/test-framework

How can I fix this?

Seems like --clean is hardcoded into the MercurialSCM Jenkins plugin.

See OP for the link to the culprit code.

I have opened up a ticket on Jenkins jira.


Is zipping a file supposed to happen in Jenkins or on Nexus? I am trying to figure out the correct way to do my pipeline.  Am I supposed to zip my code in Jenkins with maybe the File Operation plugin and then upload it to my private Nexus repository?  I don't send all kinds of files to Nexus and do something there do I?

If you are building these artifacts for a release/deploy, you will want them to be in as ready to release of a package as possible. If that is as a .zip, then yes you should create the .zip on the Jenkins server, then upload to Nexus. Your deploy system will then download, decompress, and manipulate the artifact as seen fit. In general, I would only use the node running Nexus to store artifacts and not execute any "build-like" processes. That is exactly what Jenkins was made for.


Jenkinsfile credential as paramter unable to use it I am trying to use the credentials parameter for a git clone.  But i am getting the error that the variables are not found
Param definition
  credentials (credentialType: 'Username with password', defaultValue: 'fcb2d7c3-4b35-4ef2-bdf0-24fc4ff1137c', description: 'Git credential', name: 'git_credential', required: true)

Usage in stage
withCredentials([usernamePassword(credentialsId: params.git_credential, passwordVariable: 'DOCKER_PASSWORD', usernameVariable: 'DOCKER_USERNAME')]) {
sh 'git reset --hard; git clone https://${DOCKER_USERNAME}:${DOCKER_PASSWORD}@repo_url/scm/${params.repository}.git --branch master'

Error: Wrong variable used

withCredentials publish environment variables, and in sh to access the environment variables, you have to do it like this: "${env.JOB_BASE_NAME}".
Try this:
withCredentials([usernamePassword(credentialsId: params.git_credential, passwordVariable: 'DOCKER_PASSWORD', usernameVariable: 'DOCKER_USERNAME')]) {
   sh 'git reset --hard; git clone https://${env.DOCKER_USERNAME}:${env.DOCKER_PASSWORD}@repo_url/scm/${params.repository}.git --branch master'
}


Skip stages in Jenkins pipeline I am using a scripted pipeline where I am required to skip some stages conditionally.Is there a way to show skipped stages on Jenkins UI other than importing org.jenkinsci.plugins.pipeline.modeldefinition.Utils in the jenkinsfile as the import is giving me "cannot resolve class" error while running build.gradle for some other script.

By default, there is no way to visualize skipped stages in the Jenkins UI.  See this issue report on the Jenkins JIRA about visualizing skipped stages.
There are a few workarounds that I'm aware of:
You can disable the Groovy sandbox to allow importing the aforementioned class.
You can also use the imperative-when plugin for scripted Pipelines as described in this blog post to add visualization (note that I haven't tested this personally).
Or you can skip the visualization altogether and wrap your build steps in an if clause to conditionally execute them.  You'll see in the Jenkins UI that a stage takes 0ms to execute if it has no steps, so there's at least some indication that it was skipped even if it doesn't explicitly say "skipped".


Memory being abruptly freed up every day at about the same time I have a few VMs on Windows Azure that run our ecommerce website, and lately we started using Telegraf, InfluxDb and Grafana to keep an eye on these machines. After a couple weeks of gathering data, I have noticed a weird pattern related to the Memory Available metric:

Everyday almost always at the same period of the day, I have noticed that there is an abrupt amount of memory being freed up which, due to my very very very limited DevOp skills, I cannot figure out what is causing this.

Here's a chart that shows this pattern:

My question is: What could lead to something like this? I feel tempted to suspect that a Memory Leak is to blame but...Free memory never drops below 70% and only happens in two of the VMs with the most traffic!

Should I be concerned when I see something like this?

P.S.: I have stared gathering metrics for Private e Virtual bytes for each of the windows services we have running and for the w3wp process...although I have read that this metrics are not very reliable to find out if you have a memory leak, but at least I'll try to get some sort of trend and see if it correlates with the pattern shown above.

I've seen this same "sawtooth" pattern in other systems, in particular a Java-based data tool.  Based on your description, I think you're looking at  .NET garbage collection (assuming this is a .NET app.)  Java and .NET are both memory-managed languages and frameworks that use garbage collection.

A memory leak is typically found in frameworks that lack memory management, or in a program on a memory-managed framework that is overriding or confusing the garbage collector.

The fact that these are your highest-traffic servers makes sense.  You're seeing the .NET framework allocate memory as needed, then the garbage collector kicks in on a regular cycle and reclaims unused memory using the garbage collection algorithms.  Unless you're tracking specific performance issues I don't think this memory usage pattern is a problem.


CI/CD free for hobby side projects. Jenkins on Heroku as alternative, with ephemeral dyno I have four side projects which are my hobby (NOT A JOB! NOT A WORK!).

I work on that at night, when I have time and when I want to.

All projects are javascript backend and frontend. Just experiments.

What I need is a Jenkins instance (or something like that for CI (continuous integration) and maybe also CD (continuous delivery)) but free. Maybe like free heroku dyno which is amazing: I don't work everyday on it, I work sometimes, when I have time and, like my free dyno on Heroku it's amazing the auto-sleep and go on just when I need it.

Is there something for Jenkins out there but free, with auto-sleep?

Jenkins on Heroku is still an option given the ephemeral state of dyno on Heroku? Is Jenkins still saving config on filesystem?

I can't afford to pay something for that because it's just an experiment.

Any idea?

You should try CI / CD like:

Codeship ( free tier, and totally free for open source projects )
travisci
circleci, ...

Openshift, as @JamesKnott mention, but this one is a full PaaS, so it is meant to host your app too.

Hope it helps


Linking VSTS build with jenkins artifacts So I have a local jenkins server building artifacts.  I trigger this build through VSTS.  When browsing through the logs when uploading the artifacts to VSTS from the local jenkins server I found this: Downloading content from Jenkins server:http://[ip]:[port]//job/[Folder Name]/job/[Build Definition Name]//api/json?tree=lastSuccessfulBuild[id,displayname] with strict SSL:true

This could pose an issue where the triggered build may get beat out by another build (theoretically).  While this is not a high likelihood, I would like to mitigate all possible artifact confusions as that is one of the main points my company is moving to build automation and continuous integration.  I did see something related to the VSTS Task in issue 4110 but not sure what is coming of this.

What I'm trying to accomplish is that when Jenkins finishes the build it uploads it to VSTS.  Because of the size of the build, when we go to releases, I would like to not download the artifacts back to our local premise, (using deployment groups) but instead copy it internally directly from the Jenkins build server.  Does anyone have any suggestions?

So I wrote this powershell script to be ran by the deployment group agents who have access to the server locally.

Param(
[string]$JenkinsPatArg
)
$isRunningLocal = $false
$jenkinsUrlBase = "[public ipaddress]"

if(Test-Connection -ComputerName "[private ipaddress/DNS name]" -Quiet)
{
    $isRunningLocal = $true
    $jenkinsUrlBase = "[private ipaddress/dns name]"
}

$buildDefinitionName = $env:RELEASE_ARTIFACTS_GAMEZIP_DEFINITIONNAME
$buildTeamProject = [Uri]::EscapeDataString($env:SYSTEM_TEAMPROJECT)
$targetBuildNumber = $env:RELEASE_ARTIFACTS_[Artifact Alias]_BUILDNUMBER

$Auth = "$($env:username):$($JenkinsPatArg)"

$Bytes = [System.Text.Encoding]::UTF8.GetBytes($Auth)
$Base64bytes = [System.Convert]::ToBase64String($Bytes)
$Headers = @{ "Authorization" = "Basic $Base64bytes"}

Write-Host "Searching for $($targetBuildNumber) in project $($buildTeamProject)->$($buildDefinitionName)"

$correctBuild

Write-Host "Querying: http://$($jenkinsUrlBase):[port number]/job/$($buildTeamProject)/job/$($buildDefinitionName)/api/json?tree=builds[id,url,actions[buildNumber]]"

ForEach($build in (Invoke-RestMethod -Uri "http://$($jenkinsUrlBase):[port number]/job/$($buildTeamProject)/job/$($buildDefinitionName)/api/json?tree=builds[id,url,actions[buildNumber]]" -ContentType json -Headers $Headers).builds)
{

    if($build.actions.buildNumber -eq $targetBuildNumber)
    {
        $correctBuild = $build
        Write-Host $build.id
        Write-Host $build.url
        Write-Host $build.actions.buildNumber
        break

    }
}

if($correctBuild -eq $null)
{
    Write-Error "Build not found.  Must rebuild source to obtain artifact"
    return;
}

if($isRunningLocal){$correctBuild.url = $correctBuild.url.Replace("[public ip address]", "[local ipaddress/DNS name]")}

$targetZip = "$Env:temp/$($buildDefinitionName)_$($correctBuild.id).zip"

(Invoke-RestMethod -Uri "$($correctBuild.url)/artifact/$($buildDefinitionName)/*zip*/$($buildDefinitionName)_$($correctBuild.id).zip" -ContentType Application/zip -Headers $Headers -OutFile $targetZip)

Add-Type -AssemblyName System.IO.Compression.FileSystem;[System.IO.Compression.ZipFile]::ExtractToDirectory($targetZip, $env:targetDirectory);
#Expand-Archive -Path $targetZip -DestinationPath $env:targetDirectory -Force
Remove-Item $targetZip;

passing in the a personal API key (created by Jenkins user manager to use by visual studio team services), this script first checks if it is can access the server using local connections, if so set the variables appropriately.

From there we create local handlers for environmental variables.  Our release artifact name defined in VSTS is the same as that in our jenkins server.  The build number used is created by VSTS and passed to jenkins and stored by the VSTS plugin(Jenkins TFS Plugin download).  We get this version number from the environment as the artifact build number.  We also ensure that our team project path url is windows URL encoded brief explanation.  Next we hold our authentication which is stored as a variable in our VSTS variables.  The $JenkinsPatArg is marked as secret. As such, we must pass it in as it isn't visible as an environment variable.  We then encode the authentication and add it to our request headers.

We then add a debug line to print what target we are looking for and what project we are looking in, just to make sure we are looking for the right thing.  We create a variable to hold the build information that we find and do one more debug write stating what are query url is so that we can manually run it if we run into any issues.

Next we then run this url query against the build server and iterate over all the builds returned.  The query filters out all the JSON to just the build ID, it's absolute url, and the VSTS build number.  We check to see if the build we are examining has the same build number as the artifact we are deploying, if so store it in the $correctBuild variable, print the was found for debugging, and exit the loop.

Coming out of the loop we check to see if we found a result.  If we did not find a build (due to it being cleaned up most likely) we inform the user that the artifact needs to be rebuild from source.  Otherwise we download the artifact zip to a temporary location from the jenkins server and extract it to the target directory, defined as a variable.  Lastly we delete the downloaded zip.

This is ran as a powershell script task and due to it's length, can not be an inline script.  So to deploy this, i have a devops git repo for my scripts and I just include this as an artifact that is downloaded and ran.  This task is preceded by other tasks which stop the current process from running, and delete the current files in place before running this task.  After this task is ran, more scripts are ran to run the downloaded artifacts.

I hope this helps someone else trying to coordinate and link VSTS builds with Jenkins artifacts.


How to get branch hash using git command I have JenkinsFile where I'm trying to get the hash for my current branch. I saw this post: https://stackoverflow.com/questions/9110478/how-to-find-the-hash-of-branch-in-git
but when I tried to run def BranchHash = sh "git rev-parse ${BRANCH-NAME} I got: fatal: ambiguous argument 'HEAD': unknown revision or path not in the working tree
I'm looking for the branch hash, and not the last commit

git show -s --format="%H" <HEAD or Branch Name>

-s supressess the diff output
--format="%H" specifies that only the hash should be shown.
See man git-show for more details.


AWS API Gateway as an http proxy to ALB I have a setup of API Gateway proxying all the requests to my EC2 instances behind load balancer (ALB). But I want api endpoint to have a nice domain my-api.mydomain.com. I'm trying to accomplish that via custom domains.

Then I register Target Domain Name generated in Route53 like this

and my http proxy setup is just a passthrough to the ALB which manages requests to underlying EC2 instances providing business logic.

This setup doesn't work, so I'm wondering what's wrong with this configuration and what critical conceptual thing I have missed. Doing curl -SL https://my-api.mydomain.com/v1/records/ (or any api endpoint) throws {"message": "Internal server error"}. CloudWatch logs shows

  Execution failed due to configuration error: Host name
  'my-api-lb-111222333.us-east-1.elb.amazonaws.com' does not match the
  certificate subject provided by the peer (CN=*.mydomain.com)

I'd be grateful for any suggestions.

Try directly accessing https://my-api-lb-111222333.us-east-1.elb.amazonaws.com from your web browser and you'll get a similar error.  The SSL certificate is not just used for encryption -- it also verifies that the host is not an impostor, by being consulted in a check of whether the hostname used to access the target matches the hostname of the certificate that's offered by the target

Because amazonaws.com is not your domain, you can't get a certificate for that.

You need to assign a domain name from your domain to the ALB in DNS, e.g. origin.example.com, and point API Gateway to that name, not to the system-assigned hostname of the ALB.


How to create a host on a DigitalOcean team with Terraform? Teams on Digital Ocean are very recent and there is no documentation about using them with Terraform.

All information I can found is how to work with them manually.

I was hoping that I could say something like:

resource "digitalocean_droplet" "host" {
    image = "${var.image}"
    name = "${var.host_name}"
    region = "${var.region}"
    size = "${var.size}"
    private_networking = "${var.private_networking}"
    ssh_keys = ["${var.ssh_fingerprint}"]
    team = "${var.team}"
}

Note the team = "${var.team}" at the end.

Since Terraform is just a wrapper on the top of the provider APIs I looked at the documentation trying to find some endpoint of field about this but found nothing.

How should I use Digital Ocean teams with Terraform? Or I am stuck with the main account if I want to automate things with Terraform?

According to the article you linked, Team Resources are shared. This means that all droplets (hosts) are shared across the team once you have a team in place.

source: https://www.digitalocean.com/community/tutorials/how-to-use-teams-on-digitalocean#team-resources

This means that each team member, using his own personal token, can manage all the resources shared with her. Terraform allows each user to use his own credentials when managing resources, and includes a state file (that must be shared as well). Managing shared tfstate and managing secrets is addressed in this DevOps.SE question.


Strange behaviour of Docker CE/EE client on Windows - timeout for certain image layers With a private registry, I cannot pull images because certains image layers fail to download. The pattern seems to be that this happens with larger, 50MB sized layers.
It shouldn't be a registry neither network problem as colleagues with other systems manage to get these images.

6bbedd9b76a4: Pulling fs layer
fc19d60a83f1: Download complete
de413bb911fd: Download complete
2879a7ad3144: Download complete
668604fde02e: Download complete
f257ac562635: Download complete
07486ea9391a: Download complete
c131a1091dea: Download complete
e4e87f3caf2a: Downloading
6ef05c7ca3e3: Retrying in 1 second
85b7b45e23a1: Download complete
077ff1474b65: Retrying in 1 second
f85147a0e9d7: Waiting

Other errors on new retries with same image:

2c1171c58a199d2ec9daee729d88146dfb91caba4788: local error: tls: bad record MAC

error pulling image configuration: tls: received record with version 5ea7 when expecting version 303

UPD. Strange enough, the very first image layer which always gets stuck before it downloads anything stucks rarely at "downloading... 987B".

UPD2. Same behaviour with Docker EE client (Docker-Client/17.03.0-ee-1)

UPD3. Filed this as a bug to the Moby project.

You may wish to try disabling and uninstalling any antivirus software you have installed and see if it resolves the issue (I of course don't recommend this in the long term). I have seen two different instances of antivirus interrupting SSL conections on Windows 10 recently. Chrome would report a similar error to "bad record MAC" and the issue was generalized to any SSL website or connection and any browser. Unstalling or changing A/V software resolved the issue.


Building Jenkins pipeline with GitLab info I am new to DevOps world and desperately looking for your help.
We have multiple API depositors in GitLab subgroup, I am trying to establish a CI/CD pipeline, which will connect to GitLab subgroup,

Any new API added into the subgroup, the pipeline will be triggered.
If existing API code is updated into any branch pipeline should be triggered.

In GIT repository, I have to provide the API URL, should I write #of pipelines as number API?

Gitlab Subgroups are essentially only additional path identifiers on the Gitlab server.

For example if you have:

Gitlab server on localhost
Group: test
Subgroup subgroup1
Repository myawesomerepo

Then the path to your repository will be:

http://localhost/test/subgroup1/myawesomerepo

And cloning the repo will be

git clone git@localhost/test/subgroup1/myawesomerepo.git

So when you connect any Jenkins pipelines to the repo you will need to connect them to that full path.

Any changes being done or API's added to the Git repository itself can easily be monitored automatically to do some action via Jenkins.

However, since a Gitlab Subgroup is not a Git feature rather a Gitlab feature, you need to try and write you own logic to query on the full URL and to see if there were any new Subgroups/ API endpoints added.


error validating data: couldn't find type I have following yaml file:

---
apiVersion: v1
kind: pod
metadata:
    name: Tesing_for_Image_pull.pod
    spec:
        containers:
        - name: mysql
          image: mysql
          imagePullPolicy: Always
          command: ["echo", "SUCCESS"]

When I use kubectl create -f Tesing_for_Image_pull.yaml , I get the following error:

error: error validating "Tesing_for_Image_pull.yaml": error validating data: couldn't find type: v1.pod; if you choose to ignore these errors, turn validation off with --validate=false

How can I solve my problem?

kubernetes is picky and case-sensitive:

kind: Pod

should do it.

Also, spec should not be in metadata:

apiVersion: v1
kind: Pod
metadata:
  name: Testing
spec:
  containers:
  - name: mysql
    image: mysql
    imagePullPolicy: Always
    command: ["echo", "SUCCESS"]


Docker MongoDB image - How to specify credentials other than in the compose file? I'm using this docker image https://hub.docker.com/_/mongo/. That image provides a way of initializing a mongodb root database user with env variables which I can specify in the compose file as follows:

environment:
  - MONGO_INITDB_DATABASE=db
  - MONGO_INITDB_ROOT_USERNAME=root
  - MONGO_INITDB_ROOT_PASSWORD=pass

It seems storing the password in config files is not very secure. How can I use more secure ways (i.e. docker secrets) to init the root user in this case? Same thing for initdb files, which I need to copy to the container and may as well contain more credentials.

I wonder if anyone here found a more secure way of handling this.

Secrets support for mongo is built in. You can see in that image's entrypoint script that if you add _FILE to end of username/password values it will pull from those secret files.

As for .js files, if you're storing decrypted passwords in those files, you'd need to make a custom entrypoint script or something that will take envvars you're passing in for which secrets to use and run a jq or sed on the file to replace I guess. You can store the .js files themselves in a secret as longas its smaller than 500Kb (max size of a secret).


How to filter vmware_vm_facts I'm trying to pull out the "uuid" from facts returned by ansible's vmware_vm_facts module. There is a solution to a similar question posted on Reddit, but when I try that, it fails with
'ansible.utils.unsafe_proxy.AnsibleUnsafeText object' has no attribute 'ip_address'

Here is my code:

  - name: Test vmware_vm_facts
    vmware_vm_facts:
      hostname: "{{ vcenter_hostname }}"
      username: "{{ vcenter_username }}"
      password: "{{ vcenter_password }}"
      validate_certs: no
    delegate_to: localhost
    register: rc

  - name: Debug...
    debug:
      msg: "IP of {{ item.key }} is {{ item.value.ip_address }}and power is {{ item.value.power_state }}"
    with_dict: "{{ rc.virtual_machines }}"

Anyone have any ideas?
Thanks!

Thanks for your response @Zeitounator. I'm running Ansible 2.8
Since posting this I have received an answer that works for me. I realized that these facts are presented as a list of dicts as noted in the doco, but I was still having trouble for some reason (confused myself badly.) The solution hinges on using with_items instead of with_dict and using an appropriate conditional like so:

  - name: Test vmware_vm_facts
    vmware_vm_facts:
      hostname: "{{ vcenter_hostname }}"
      username: "{{ vcenter_username }}"
      password: "{{ vcenter_password }}"
      validate_certs: no
    delegate_to: localhost
    register: rc

  - debug:
      msg: "IP of {{ item.guest_name }} ({{ item.uuid }}) is {{ item.ip_address }} and power is {{ item.power_state }}"
    with_items: "{{ rc.virtual_machines }}"
    when: item.guest_name is search(inventory_hostname)


Is Gradle used to automate deployment of EAR and WAR files to application servers? I'm getting a bit deeper into Gradle and want to further automate the process.  Generally speaking what are the approaches towards interacting with application servers?

I'd want to automate deploying an EAR.  Is there even a need, or case to be made, to step out of Gradle for that purpose?

I imagine it's much the same regardless whether it's EAR or WAR files and regardless of the specific application server -- but I'm using Glassfish.

If you want just to deploy faster to server(propably not only Glassfish) you can just write code in gradle to do that.
Example taken from here

/**
 *  ~/.gradle/gradle.properties:
 *  glassfishHome=/path/to/glassfish_home
 *
 *  or in Netbeans, right click project, Properties, Manage Build in Tasks, Run
 *  Add line to Arguments: -Dorg.gradle.project.glassfishHome=/path/to/glassfish_home
 *
 *  For more information about Exec tasks see
 *  http://www.gradle.org/docs/current/dsl/org.gradle.api.tasks.Exec.html
 */
task run(dependsOn: 'war', type:Exec) {
    workingDir "${glassfishHome}${File.separator}bin"

    if (System.properties['os.name'].toLowerCase().contains('windows')) {
        commandLine 'cmd', '/c', 'asadmin.bat'
    } else {
        commandLine "./asadmin"
    }

    args "deploy", "--force=true", "${war.archivePath}"
//from: https://www.schuermann.eu/2013/08/03/gradle-glassfish.html
}

In other words just use your asadmin


How do fail a Packer pipeline upon powershell provisioner pester test failure? I'm building a packer pipeline and having an issue in getting the failure of a powershell provisioner step to fail the packer job itself. This is important as without failing the Packer job, the AMI would be published, despite the test failure.

How I'm proceeding:

Run provisioning setup steps
Copy Pester Script to Remote machine
Invoke inline powershell statement to Invoke-Pester
Download the test results from the Pester remote run
Next provisioning step parses the file to count errors and throws error upon issue count > 0

However, even with issue count being > 0 it is not causing the pipeline to fail. Additionally some errors thrown in other provisioner steps also fail to cause the Packer job to stop. I've set $ErrorActionPreference = 'Stop' on one of them and it didn't seem to make any difference.

Here's part of the provisioning workflow for packer.

...     {
            "type": "file",
            "source": "{{ template_dir }}/tests/windows-server.tests.ps1",
            "destination": "{{ user `TEMP_DIRECTORY` }}/"
        },
        {
            "type": "powershell",
            "environment_vars": [
                "TEMP_DIRECTORY={{ user `TEMP_DIRECTORY` }}/"
            ],
            "inline": [
                "if(-not (Get-InstalledModule Pester -MinimumVersion 4.10.1)){ Install-Module Pester -Scope AllUsers -AllowClobber -Force -verbose} ",
                "Import-Module Pester -MinimumVersion 4.10.1 -Force -DisableNameChecking",
                "$null = New-Item -Path $ENV:TEMP_DIRECTORY -ItemType Directory -Force -ErrorAction SilentlyContinue",
                "$testresult = Invoke-Pester -Script (Join-Path $ENV:TEMP_DIRECTORY 'windows-server.tests.ps1') -OutputFile (Join-Path $ENV:TEMP_DIRECTORY 'TEST-pester-results.xml') -OutputFormat 'NUnitXML' -PassThru"
            ]
        },
        {
            "type": "file",
            "direction": "download",
            "source": "{{ user `TEMP_DIRECTORY` }}/TEST-pester-results.xml",
            "destination": "{{ template_dir }}/artifacts/TEST-pester-results.xml"
        }, ...

Once this part is done it's followed by this step which is some logic from this blog post:

  {
            "environment_vars": [
                "TEMP_DIRECTORY={{ user `TEMP_DIRECTORY` }}/"
            ],
            "script": "./scripts/Get-TestResults.ps1",
            "type": "powershell"
        },

The Get-TestResults.ps1 has the following function thanks to this article: Testing Packer Images with Pester written originally by Sam Cogan and modified slightly by me.

[cmdletbinding()]
param()
$TestFile = Join-Path $ENV:TEMP_DIRECTORY 'TEST-pester-results.xml'

if (Test-Path -Path $TestFile -PathType Leaf)
{
    [xml]$testResults = Get-Content -Path $TestFile -Raw
    $totalIssues=[int]$($testResults.'test-results'.errors) + [int]$($testResults.'test-results'.failures) + [int]$($testResults.'test-results'.invalid)
    exit $totalIssues
}
else
{
    Write-Error " Failed to find `$TestFile: $TestFile"
    Write-Warning "$(Get-ChildItem $ENV:TEMP_DIRECTORY -Recurse | Format-Table -autosize -wrap | Out-String)"
    Write-Error " Failed to find `$TestFile: $TestFile"
    exit 1  #not a good exit :-)

}

After resolving some syntax issues, I found the true cause of failing to throw the error.
In my prior usage I was doing this through Inspec provisioner, which handled the exit codes.

If you are using Pester and having to do the workaround I described above, then to ensure error's in a step are thrown you need to throw an exit code other than 0.

The two steps I took that seem to work fine (I like some emoji fire in my logs )

$ErrorActionPreference = 'Stop'

try
{
# Do code here
}
catch
{
   Write-Error " Failed to do something important `$TestFile: $TestFile"
  exit 1  # packer will recognize failure at this point
}

Note that you will want to make sure your Azure DevOps Pipeline has the following settings to publish test results regardless of failing:

# ℹ  Using Invoke-Build to call Packer. You can do this with a packer extension or any other way you prefer
- task: PowerShell@2
  displayName: Run Packer Configuration
  inputs:
      filePath: build/build.ps1
      arguments: '-Task PackerBuild -Configuration $(Configuration)'
      errorActionPreference: 'Continue'
      pwsh: true
  continueOnError: true #I want to publish the test results regardless

- task: PublishTestResults@2
  displayName: Publish Pester Tests
  inputs:
      testResultsFormat: 'NUnit'
      testResultsFiles: '**/TEST-*.xml'
  condition: succeededOrFailed()


How to avoid continuous integration-caused instabilities in test environments? Assume you're using continuous integration processes which frequently update some target environments, so that every time there are some changes "you" can test your changes right away. That's part of the goals of CI, no?

But, also assume that you have other people involved in your test cycle, e.g. managers or customers. Makes sense to get other people involved in trying to review (break?) your upcoming changes, no?

But if you continuously keep delivering changes in the environment in which those other people are, seriously, trying to test them, then multiple issues may arise, such as:

they might waste their time in reporting issues which, by the time they save the (in depth) report, they cannot even reproduce the issue themselves anymore (e.g because accidently you also ran into the same issue, and already fixed it in their environment).
you might not be able to reproduce issues they reported, since the environments in which they ran into some issue, is no longer identical (you (!!!) might have overlayed their environment).

So what can you do (how to configure things?) to avoid such (frustrating) situations?

I'll give my experience on this one, mostly because it showcases why some answers are not always applicable.

Some context to start:

We have 7 environments to host roughly 80 applications, most of them rely on each others through webservices or shared tables on db2-iSeries.
For good or bad, the iSeries are our DB system of reference.
This last point invalidates any idea of bringing the app with its dependencies in an isolated environment as bringing up an AS400 for each would cost too much and we wouldn't have the hardware to run it anyway.

What we are doing is not a complete automated Continuous Delivery, we have a schedule of releases to bring up a coherent lot of applications for the general operations. Aside from this each test teams can trigger a release in one of the Q/A environment for the application they are testing and can put a lock on some application version to avoid another team request breaking their tests.

Dependencies of applications are checked before release, so the system won't release something if other applications can't be updated or doesn't match its dependencies needed.
The main idea is to allow updates when it won't impact someone, if there are no tests planned, it should flow from previous environment (and we're aiming at removing the scheduled releases in the 5 firsts environments on mid term now we have validated this 'on demand' method system).

The short version is to have a 'semaphore' system around the applications in the environment, a team should be able to lock its target application with its dependencies (and transitive dependencies) for the time of manual tests.
The implementation of this semaphore is highly dependent on your automation system so I won't extend on that.

Of course the easy way is, as others mentioned, to create a fresh environment for an application with all its dependencies to avoid the semaphore described above.


What is Value Stream Mapping? I have heard a lot about Value Stream Mapping and how it can be used to analyse the value stream of manufacturing processes including the process of delivering software.  I have never seen it successfully delivered in the context of software development and operations teams.

What is Value Stream Mapping and how can it be applied to software delivery?

A value stream is a process that adds value to a collection of things that have less inherent value.  The canonical example there is an assembly line.  I have little interest in a disconnected collection of parts (steering wheel, shifter, gas pedal.)  On the other hand, I'll pay good money for a car.  Car manufacturers add value to the parts in the form of facilities, skilled work, supply chains, etc.  Parts go in one end of the stream, cars come out the other.

Enterprises use complex processes to deliver their products and/or services.  Value stream mapping is a process used by companies to figure out how and where they deliver value.  The purpose of value stream mapping is to develop a clear understanding of the steps, time, and resources required to support a business capability that delivers value.

Value stream mapping is useful in kaizen, or Continuous Improvement.  The "systems thinking" employed in DevOps and Lean encourages small, incremental improvements made throughout a complex process.  A value stream map can expose areas of waste, such as unnecessary or inefficient steps in a value stream.  A complete map tends to expose those areas of waste.  This helps prioritize improvements.

Value Stream Mapping can be applied at large scope to a business process, or to a smaller scope within software delivery.  The software delivery process itself mirrors an assembly line.  Code is written, reviewed, tested, integrated, and tested again.  It is delivered from one environment to another (in manufacturing this is called a work center) until it is finally delivered to Production.


How to test a Terraform Configuration? If you had a Terraform configuration that had a moderate degree of complexity, how would you write tests around the configuration that could be executed as part of a Continuous Integration / Continuous Delivery pipeline?

As an example, you may have a multi-cloud configuration that specifies the following desired state:

Azure Container Services to host Docker in Azure
Azure Blob Storage
SQL Azure
EC2 Container Service to host Docker in AWS
Amazon S3 Storage Service
Amazon RDS SQL Server database

Potentially terraform apply could create the above from scratch, or transition from a partially deployed state to the above desired state.

I am aware that Terraform splits its work into the execution plan stage and the application phase which actually makes changes to the target architecture.  Can this be used to write tests against the execution plan, if so are there frameworks to help write these?

There is currently no full solution to this integrated into Terraform, but there are some building blocks that could be useful to assist in writing tests in a separate programming language.

Terraform produces state files in JSON format that can, in principle, be used by external programs to extract certain data about what Terraform created. While this format is not yet considered officially stable, in practice it changes infrequently enough that people have successfully integrated with it, accepting that they might need to make adjustments as they upgrade Terraform.

What strategy is appropriate here will depend a lot on what exactly you want to test. For example:

In an environment that's spinning up virtual servers, tools like Serverspec can be used to run tests from the perspective of these servers. This can either be run separately from Terraform using some out-of-band process, or as part of the Terraform apply using the remote-exec provisioner. This allows verification of questions like "can the server reach the database?", but is not suitable for questions such as "is the instance's security group restrictive enough?", since robustly checking that requires accessing data from outside of the instance itself.
It's possible to write tests using an existing test framework (such as RSpec for Ruby, unittest for Python, etc) which gather relevant resource ids or addresses from the Terraform state file and then use the relevant platform's SDK to retrieve data about the resources and assert that they are set up as expected. This is a more general form of the previous idea, running the tests from the perspective of a host outside of the infrastructure under test, and can thus collect a broader set of data to make assertions on.
For more modest needs, one can choose to trust that the Terraform state is an accurate representation of reality (a valid assumption in many cases) and simply assert directly on that. This is most appropriate for simple "lint-like" cases, such as verifying that the correct resource tagging scheme is being followed for cost-allocation purposes.

There is some more discussion about this in a relevant Terraform Github issue.

In the latest versions of Terraform it is strongly recommended to use a remote backend for any non-toy application, but that means that the state data is not directly available on local disk. However, a snapshot of it can be retrieved from the remote backend using the terraform state pull command, which prints the JSON-formatted state data to stdout so it can be captured and parsed by a calling program.


What is the best way to share maven repository using Docker build (dynamic) image with maven? We have created a Docker build container to be run by Jenkins for Java based projects. We are using Maven for building our software. The problem we are facing is that, the builds are taking way to long to finish and the main reason for that is that maven is re-downloading all the dependencies for every single build, maven is run from inside the build container and not from the pipeline step.

We thought about using a shared folder but we have dynamic build agents so no shared folder. Is there anyone who is using the same approach and is there a better approach to this?

You might use docker builder pattern. Briefly you need to create Dockerfile.build which adds pom.xml and run mvn dependency:resolve:

FROM maven:latest
ADD ./pom.xml /src/pom.xml
WORKDIR /src
RUN mvn dependency:resolve

Rebuild that image every time prior build. Docker will use cached image if ./pom.xml has not changed.

Docker 17.05 introduced 'multi-stage builds' feature which implements this pattern. More information is available here: https://docs.docker.com/engine/userguide/eng-image/multistage-build/


How to handle services when the developers provide incorrectly in k8s I provided the script to provision a new feature environment inside k8s it will create all the configs required, namespaces, resources and whatever...
in my script, I described this param:
parameters {
text(name: 'SERVICE_LIST', defaultValue: '', description: 'Services to deploy, separated by line breaks.')}

when the developer is provisioning a new env, he will provide a list of services that he wants to enable on that environment
also in my script, I described the following piece of code which are enabling all services requested(its located inside the stage and step):
def services = replaced.split('\\n')
   services.each { SERVICE ->
    sh "echo turning on service ${SERVICE}"
    sh "yq w -i  ${DEPLOY_ENV_FOLDER}/services.yaml '${SERVICE}.enabled' true"}

this is an example of a services file:

so, when the developer provides the services that he wants, the script goes to this file, enable the services and apply to k8s
now my question is how to assure that the services provided by the developer are correct and that they exist in the file? otherwise, I need to abort the script. I tried to use try{} catch{} in the second piece of code above to define the exception for abort services which are provided by developers incorrectly, do you have any other solution?

@Mkash, don't you know in advance what services you have in your services.yaml file?
Maintain a list in groovy, and see whether the input is in that list or not.
Something like:
availableServices = ["ew_ws_gateway", "ingress"]

services = ["a", "b", "c", "ew_ws_gateway"]

services.each { svc ->
    if (svc in availableServices) {
        println("Service exist" + svc)
    }
}

The other option is to change the parameter from type of text to multiple-choice possibly using https://plugins.jenkins.io/extended-choice-parameter/


Use a volume or copy my artefacts into docker image I want to dockerize my php application and deploy it in production. Here is my application structure :
api
|
+-- src
+-- config
+-- vendor
+-- tests
+-- public

I need to embed api/* in my container, and I'm confused about the best approach.
Here is mine :

A light multi-stage docker image (around 85 mb but can be more optimized)
FROM php:7.4-fpm-alpine AS base

WORKDIR /var/www/html
ENV COMPOSER_ALLOW_SUPERUSER 1
RUN apk update && \
  docker-php-ext-install mysqli tokenizer pdo_mysql && \
  curl -sS https://getcomposer.org/installer | php -- --install-dir=/usr/local/bin --filename=composer

FROM base AS dev

COPY xdebug.ini /usr/local/etc/php/conf.d/xdebug-dev.ini
RUN apk add --no-cache --virtual .phpize-deps $PHPIZE_DEPS && \
  pecl install xdebug-2.9.1 && \
  docker-php-ext-enable xdebug && \
  touch /tmp/xdebug.log && \
  chmod -R 666 /tmp/xdebug.log && \
  apk del -f .phpize-deps

A mounted volume (docker-compose.yml)
api:
  volumes:
    - ./api:/var/www/html
  command: composer install && php-fpm;

I saw a lot of docker images using COPY .. instruction in Dockerfile instead of volumes. I believe COPY is great to copy small config files (or a small build for compiled languages), but it's not optimized to keep the images small (basic rule with docker).
Am I missing something?
For example, is it bad practice to handle volumes from outside the dev environment ?

To copy into the image or to mount volumes--my advice:

if the files will change infrequently, then copy them into the image, with the understanding that if they need to be modified, you will need to rebuild the image.
if you need to frequently change the files, or store some kind of state about the files (and re-use them with future container instances of the image), then mount them as volumes.


Control GKE CICD from a Jenkins in a lab with private network? For a test purpose I need to use my locally provisioned Jenkins with Vagrant in order to connect to GKE and use pods to build. Is that possible, because from what I read K8s will need access to Jenkins as well. How can I achieve that?
Looks to be possible, but I am stuck on access rights for now:
o.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: https://xxxxxx/api/v1/namespaces/cicd/pods?labelSelector=jenkins%3Dslave. Message: pods is forbidden: User "system:anonymous" cannot list resource "pods" in API group "" in the namespace "cicd". Received status: Status(apiVersion=v1, code=403, details=StatusDetails(causes=[], group=null, kind=pods, name=null, retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, message=pods is forbidden: User "system:anonymous" cannot list resource "pods" in API group "" in the namespace "cicd", metadata=ListMeta(_continue=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=Forbidden, status=Failure, additionalProperties={}).

Taking the url from cluster-info and creating a config file with jenkins and using at as secret did the trick to pass the connection.
I am still curious how to authenticate with a bearer token though. I have tried to use existing managed Jenkins secret, but it failed with it.
Update: The authentication happens, but after creating pods the job never runs.
Update: It looks like using jenkinsci docker image as a hardcoded podtemplate works, I try to work with Groovy provisioned yaml template that calls GCR and there I got the firstly mentioned error.


Placing an AWS Lambda in a public subnet Is it possible to place an aws lambda in a public subnet and thus avoid paying for NAT?

The docs state the following

  If your Lambda function needs Internet access, do not attach it to a public subnet or to a private subnet without Internet access. Instead, attach it only to private subnets with Internet access through a NAT instance or an Amazon VPC NAT gateway.

But do not explain why we can not attach the lambda to a public subnet.

Provided the Public Subnet where you're launching your Lambda is configured with Auto-assign public IPv4 address: Yes it should work in public a subnet without NAT.

Whether or not it's a good idea is another question. In general you shouldn't launch any backend services in the public subnets, the public subnets are for things like API Gateways, NAT Gateways, Load Balancers, etc. Not for launching the actual backend systems. That's the best security practice.

However that aside I'm pretty confident that with auto-assign IPv4 address enabled Lambdas will work even in the public subnet.

Unless you want the Lambda to connect to any VPC-internal resources (e.g. to your RDS) you may just as well launch it without VPC config (leave those VPC and Subnet settings empty). That will give it access to the internet without having to worry about NAT or public IPs.

Hope that helps


Is there a way to combine helm with some other tool for k8s configuration management without conflicts? Currently I have a k8s project with multiple services which are managed by a helm chart. Said helm chart is installed on my local machine and I want to keep it this way if it's possible. But now I need to provide a simple user-friendly way to change versions of some service's docker images for other people. I was thinking about creating a jenkins task which would allow changing "image" field for those services, but it would create conflicts with my helm release: next time I update something with helm, it would rollback jenkins "image" field changes to the values in my helm chart.
Is there a simple way to resolve this conflict, like saying helm not to change "image" field? If not, how can I continue using helm for project configuration while allowing "image" field changes from jenkins?

Helm isn't made to be used this way.  If you'd prefer to use a static deployment, you're better off using a plain Kubernetes manifest.  Then, people changing the image field won't break your release.
You can easily convert it to one simply by running helm get manifest command
Helm is used to manage release versions.  I.e. every new release (and a docker image change definitely qualifies as one), you would push a change to your helm chart.
The other alternative you have is defining docker image as a value.  Then you can pass a command-line argument to replace the image tag with a new one and re-run helm from Jenkins any time you have a new deploy.
Example:
in values.yaml:
image:
  repo: foo
  tag: bar

On command line when running helm:
helm upgrade --install my-app --set image.tag=newtag

Final, and least recommended way is to use a mutable tag (i.e. "master") and set your ImagePullPolicy to Always.  This way, when you have a deploy, you just need to bounce all the pods in your deployment/daemonset/statefulset object.