Creating Keycloak Realm via Ansible Currently there is no module to create a Keycloak realm with Ansible. There is a PR but it is stucked since half a year or so. So I though, there is a Keycloak REST API (https://www.keycloak.org/docs-api/9.0/rest-api/index.html#_realms_admin_resource) and I could call it with uri to create a Token and then send a JSON RealmRepresentation via REST. So I tried this:

- name: "Create Token for service Keycloak"
  uri:
    url: "https://keycloak-server/auth/realms/master/protocol/openid-connect/token"
    method: POST
    body_format: form-urlencoded
    body:
      username: "admin"
      password: "password"
      grant_type: "password"
      client_id: "admin-cli"
  register: keycloak_token

- name: "Create Realm for service Keycloak"
  uri:
    url: "https://keycloak-server/auth/"
    method: POST
    src: "realm.json"
    remote_src: "no"
    headers:
      Content-type: "application/json"
      Accept: "application/json"
      Authorization: "Bearer {{ keycloak_token.json.access_token }}"
  register: keycloak_realm_create

The token is available within keycloak_token.json.access_token and it is correct.

As realm.json I tested a minimal JSON (because all attributes in RealmRepresentation are marked as optional):

{
  "id": "myrealm",
  "realm": "myrealm",
  "displayName": "My Realm",
  "enabled": true,
  "sslRequired": "external",
  "registrationAllowed": false,
  "loginWithEmailAllowed": true,
  "duplicateEmailsAllowed": false,
  "resetPasswordAllowed": false,
  "editUsernameAllowed": false,
  "bruteForceProtected": true
}

or a complete JSON export from Keycloak via WebUI (export realm).

But in any case the following error happens:

fatal: [localhost]: FAILED! => {
  "changed": false,
  "connection": "close",
  "content": "{\"error\":\"RESTEASY003065: Cannot consume content type\"}",
  "content_length": "55",
  "content_type": "application/json",
  "elapsed": 0,
  "json": {"error": "RESTEASY003065: Cannot consume content type"},
  "msg": "Status code was 415 and not [200]: HTTP Error 415: Unsupported Media Type",
  "redirected": false,
  "status": 415,
  "url": "https://keycloak-server/auth/"
}

The documentation says it must be a POST to / and it consumes application/json. So, I'm not sure - is the URL correct? If I try https://keycloak-server/auth/realms/, I'm getting a 404. If I remove the Content-type - I'm getting a 406 with "Not acceptable". If I remove Accepts too, I'm getting a HTML page.

So - what is wrong?

As @JSapkota mentioned the URL was wrong. It must be https://keycloak-server/auth/admin/realms.

- name: "Create Realm for service Keycloak"
  uri:
    url: "https://keycloak-server/auth/admin/realms"
    method: POST
    src: "realm.json"
    remote_src: "no"
    status_code:
     - 201
    headers:
      Content-type: "application/json"
      Accept: "application/json"
      Authorization: "Bearer {{ keycloak_token.json.access_token }}"
  register: keycloak_realm_create

Take care about the return status code. It is "201 created" and the uri module must know, that this is fine.

This works one time. Because in the next run the REST API responses with 409 the uri module fails.
To be idempotent, you need to check first, if the realm exists. This can be done via GET /auth/admin/realms/{{ keycloak_realm_name }} and check the return code, if it is 404 or 200. If it is 404, then you make a "POST" otherwise a "PUT". So, this a complete example:

- name: "Set facts"
  set_fact:
    keycloak_admin_user: "admin"
    keycloak_admin_pass: "password"
    keycloak_base_url: "https://keycloak.server"
    keycloak_realm_name: "myrealm"
    keycloak_realm_data_file: "realm.json"

- name: "Create Token for service Keycloak"
  uri:
    url: "{{ keycloak_base_url }}/auth/realms/master/protocol/openid-connect/token"
    method: POST
    body_format: form-urlencoded
    body:
      username: "{{ keycloak_admin_user }}"
      password: "{{ keycloak_admin_pass }}"
      grant_type: "password"
      client_id: "admin-cli"
  register: keycloak_token

- name: "Find out, if Realm {{ keycloak_realm_name }} for service Keycloak exists"
  uri:
    url: "{{ keycloak_base_url }}/auth/admin/realms/{{ keycloak_realm_name }}"
    method: GET
    status_code:
     - 200
     - 404
    headers:
      Accept: "application/json"
      Authorization: "Bearer {{ keycloak_token.json.access_token }}"
  register: keycloak_realm_exists

- name: "Create Realm {{ keycloak_realm_name }} for service Keycloak"
  uri:
    url: "{{ keycloak_base_url }}/auth/admin/realms"
    method: POST
    src: "{{ keycloak_realm_data_file }}"
    remote_src: "no"
    status_code:
     - 201
    headers:
      Content-type: "application/json"
      Accept: "application/json"
      Authorization: "Bearer {{ keycloak_token.json.access_token }}"
  register: keycloak_realm_create
  when: "keycloak_realm_exists.status == 404"

- name: "Update Realm {{ keycloak_realm_name }} for service Keycloak"
  uri:
    url: "{{ keycloak_base_url }}/auth/admin/realms/{{ keycloak_realm_name }}"
    method: PUT
    src: "{{ keycloak_realm_data_file }}"
    remote_src: "no"
    status_code:
     - 204
    headers:
      Content-type: "application/json"
      Accept: "application/json"
      Authorization: "Bearer {{ keycloak_token.json.access_token }}"
  register: keycloak_realm_create
  when: "keycloak_realm_exists.status == 200"

Also I had the problem, that after every request the Keycloak server responses with a 403 when I don't refresh the token before the next call to Keycloak. But I think, this depends on Realm token settings of the admin realm.


Infrastructure Automation with GitLab I am looking to validate the use of devops for a team of seven people, each having a specific function.

1 Java Developer
1 System Administrator
1 Team Lead/Project Manager
1 Identity Management Admin
2 Access Management Admin
1 PKI Registration Agent

We have different workflows for each function. However, from my perspective, it's all the same. New ideas or requirements bring about new or changes to code, which initiates the workflow.  To me, application updates, OS patching, and innovation are all the same. Could CI/CD pipelines be created to standardize infrastructure security and configuration as well as enable a faster and more feature tuned application release cycle?  Any input would be most appreciated.

-Darryl

here are my 2 cents,

You can use GITLAB to have merge requests for all your security requests like adding users to the groups for every new user or existing users. the Mr can be approved by only your IAM admin. You can use GITLAB CI pipelines to go and perform the operation once the MR is approved. From the users side, the user has to  provide their keys and submit as an MR so that the IAM team can validate and approve it.
You can deploy your applications easily using Ansible and Gitlab CI. All you need is just source code of your application and some knowledge of YAML. I am sure, your folks already are equipped with that. Ansible can be used for infra provisioning as well so you can just run the show with them. With regards to os updates, I would say ansible can easily do it. Have the SDLC workflow in place for developing playbooks and once the playbook is modularized, it can used in production.

Let me know if I answered your query and if you have any other questions.


helm install causes nil pointer I'm really a beginner with kubernetes and really struggle with this simple tutorial from baeldung.com

The only thing I did different (intentionally)  was using a google cloud cluster instead of minikube.

Basically this is what I did so far:

Installed google cloud sdk.
Initialized gcloud with my google cloud cluster.
Installed helm.
Created helm charts like told in the tutorial.
Changed service.yaml, values.cyaml and deployment.yaml like told in the tutorial.

And when I now run

helm install hello-world .

I get this error message:

Error: template: hello-world/templates/serviceaccount.yaml:1:14: executing "hello-world/templates/serviceaccount.yaml" at <.Values.serviceAccount.create>: nil pointer evaluating interface {}.create

To be honest I'm not even sure the serviceaccount.yaml is here for and I'm grateful for any advise!

You can try below options to debug this further.

First verify values.yaml and check the values for serviceAccount (default value is true ) that means the helm chart when deployed will create a service account for the deployment. When you create a pod, if you do not specify a service account, it is automatically assigned the default service account in the given namespace.

serviceAccount:
  # Specifies whether a service account should be created
  create: true
  # Annotations to add to the service account
  annotations: {}
  # The name of the service account to use.
  # If not set and create is true, a name is generated using the fullname template
  name:

Try to deploy your helm chart with create: false and verify if it solves your issue.

Else remove the file hello-world/templates/serviceaccount.yaml and mark serviceaccount as create: false in value.yaml and pass name as default service account of your deployment try to deploy as below

Note I have removed the serviceaccount.yaml

~/hello$ tree
.
└── hello-world
    ├── charts
    ├── Chart.yaml
    ├── templates
    │   ├── deployment.yaml
    │   ├── _helpers.tpl
    │   ├── ingress.yaml
    │   ├── NOTES.txt
    │   ├── service.yaml
    │   └── tests
    │       └── test-connection.yaml
    └── values.yaml

4 directories, 8 files

set create field as false under values.yaml

$ cat hello-world/values.yaml | grep -i serviceaccount -A 10

serviceAccount:
  # Specifies whether a service account should be created
  create: false
  # Annotations to add to the service account
  annotations: {}
  # The name of the service account to use.
  # If not set and create is true, a name is generated using the fullname template
  name: default

You can fetch your list of serviceaccounts available as below

$ kubectl get serviceaccounts
NAME      SECRETS   AGE
default   1         2d4h

Once deployed helm list will show the namespace used to deploy your chart and serviceaccount from the name space is used for deploeyemnt

$ helm list
NAME                    NAMESPACE       REVISION        UPDATED                                 STATUS          CHART               APP VERSION
hello-world-1580665905  default         1               2020-02-02 17:51:45.631330487 +0000 UTC deployed        hello-world-0.1.0   1.16.0

Get chart details and values post deploeyemtn as below

$ helm show chart hello-world
apiVersion: v2
appVersion: 1.16.0
description: A Helm chart for Kubernetes
name: hello-world
type: application
version: 0.1.0

$ helm show values hello-world | grep -i serviceaccount -A 10
serviceAccount:
  # Specifies whether a service account should be created
  create: false
  # Annotations to add to the service account
  annotations: {}
  # The name of the service account to use.
  # If not set and create is true, a name is generated using the fullname template
  name: default

podSecurityContext: {}
  # fsGroup: 2000

Get deployed resources as below

$ kubectl get all -o wide
NAME                                          READY   STATUS    RESTARTS   AGE    IP               NODE         NOMINATED NODE   READINESS GATES
pod/hello-world-1580665905-84b9d4d469-xv5mx   1/1     Running   0          118s   192.168.58.215   k8s-node02   <none>           <none>

NAME                             TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE    SELECTOR
service/hello-world-1580665905   ClusterIP   10.108.73.21   <none>        80/TCP    118s   app.kubernetes.io/instance=hello-world-1580665905,app.kubernetes.io/name=hello-world
service/kubernetes               ClusterIP   10.96.0.1      <none>        443/TCP   2d4h   <none>

NAME                                     READY   UP-TO-DATE   AVAILABLE   AGE    CONTAINERS    IMAGES         SELECTOR
deployment.apps/hello-world-1580665905   1/1     1            1           118s   hello-world   nginx:1.16.0   app.kubernetes.io/instance=hello-world-1580665905,app.kubernetes.io/name=hello-world

NAME                                                DESIRED   CURRENT   READY   AGE    CONTAINERS    IMAGES         SELECTOR
replicaset.apps/hello-world-1580665905-84b9d4d469   1         1         1       118s   hello-world   nginx:1.16.0   app.kubernetes.io/instance=hello-world-1580665905,app.kubernetes.io/name=hello-world,pod-template-hash=84b9d4d469


Jenkins plugin recomendation My company wants to visualize the % of failed/successful deployments of jenkins jobs.

And the way jenkins is arranged complicates the task, each team or project is a folder name and inside the project folder the jobs are called same.

Folder1
   samejobname
   subfolder1
      jobname
folder2
   samejobname
   subfolder1
      jobname
.
.
.
folderN
   samejobname
   subfolder1
   .
   .
   .
   subfolderN

So I need a plugin (there must be one) which can give me this stats of each job so I can take it from API and pull them to prometheus somehow.

I will be very very happy to be pointed in the right direction so thanks in advance for all the help.

Have a look at the Summary Display plugin. Some more details about it (= quote from the linked page):

... allows an easy and fully customized build report display.
Features

Allow a rich summary report visible from both project and build page
Reports must be written in an XML format according to the syntax described in section Syntax Description
This plugin allow the parsing of several XML files
Reports displayed are ordered according to the XML file names
Element displayed are based on

In general, the answer to such questions is usually found by searching the endless Jenkins plugin index.


How to insert a minion Powershell variable into Salt Pillar or Mine? Say I have a Powershell script, named test.ps1, that has this:

$VariableForPillar = 5

I can call it from a Salt state with -

test powershell run:
 cmd.run:
   - name: 'C:\Windows\System32\WindowsPowershell\v1.0\powershell.exe  -ExecutionPolicy ByPass -File  d:/tmp/Powershell/test.ps1'

But what can I do to retrieve that value and insert it into a Pillar (or a Mine)?

You can only insert the return into the mine.

Just specify the cmd.run (or possibly cmd.script) in your mine_functions and you will get the whole return back.

If you want to have a script that sets up extra config on the minion, I would recommend writing a custom grain.

https://docs.saltstack.com/en/latest/topics/grains/#writing-grains


Can Cypress with cucumber replace php unit and regression tests in CI/Cd pipeline? Or there a language agnostic unit and regresion testing?

My team built an MVP with manual tests only and we got into the inverted icecream antipattern. Now to transform into production we need to fix the problem. The marketplace is fully going into Node and JS frameworks and convincing devs to invest more into php is hard considering the job marketplace. Right now we got into a problem that in order to fix 1 bug 2 more appears because there is no regression. Is there a way to fix the icecream antipattern using a php agnostic unit testing? or do we really need to get into phpunit / codeception?

Yes Cypress is capable to run in CI.

https://docs.cypress.io/guides/guides/continuous-integration.html#Record-tests

Most of the time it is just running cypress run. In our case we ran the docker cypress.


How to setup Azure DevOps Pipelines and Releases with multiple apps Wanting to deploy a collection of .NET Core API's (microservices) using Azure DevOps I run into some challenges:

Having 10+ small API's, how should I setup the Release Pipeline? One pipeline with one "stage" for each API/application? Or a separate Pipeline for each API?
The API's are deployed to IIS on Windows Server. I'm using the FTP Task now but it's really cumbersome as the AppPool needs to be stopped, then the code is copied over, then the AppPool is started again. I know (non-disruptive) deployment on Azure with App Service is way easier with deployment slots, etc. but what tools can I use on a plain VPS with IIS on Windows Server?
How can I transform or inject the correct appsettings.json values for each environment during Release using Azure DevOps? I could use Pipeline variables but they seem to be shared across the whole Pipeline, so I would need a separate Pipeline for each API then (see question 1). There's the task "Magic Chunks" but it doesn't handle arrays well (which is widely used for Serilog for example). Environment variables on the target machine would be an option, but the API's are deployed to IIS so how can I separate the environment variables per API (there could be the same keys with different values for multiple API's)?

I use separate pipeline for each app
you could use powershell task to restart IIS, but I dont think there is a way to do this without shutting down the website. You can also reconfigure website, so copy new code to location X, repoint website to location X, restart website. Then you can easily point it back if something fails and switch is almost instanteneous
You can use library variable sets (or groups, dont remember). those can be tied to the stage in the release. it might also be possible to scope regular variables, but I've never tested that.


How to monitor AWS S3 cross-region replication process? Amazon S3 has a cross-region replication which will handle copy of new/updated objects to additional region.

The problem is that solution does not provide visibility on state for replication process, for example at the moment there's no way to easily monitor missing objects on destination or any possible permission issues that can interfere with the process and can result with replication not doing it's job, as well missing feedback for replication lag.

We found several tools similar to CRR Monitor https://aws.amazon.com/answers/infrastructure-management/crr-monitor/
but we didn't have good experience with it as we manage tens of millions of objects which can cost several thousand of dollars per month for CloudWatch events, as well as additional management for services required by solution (DinamoDB, Kinesis etc.) another layer of "monitoring the monitor" required.

For now we are using daily comparison of S3 objects inventory between source and destination using AWS Athena query, which also feels like a hack and it's not optimal for problem detection as inventory generated with daily frequency.

I hope to see some feedback, suggestions, ideas or experience for alternatives. Ideal solution would be managed or "serverless" solution which will improve visibility on the process and will provide fast detection of errors.

Still no answers, and still no CRR monitoring in place on AWS, here's minimalistic solution, any feedback and suggestions are welcome. This monitor will only show which S3 objects are missing from destination, it will not provide visibility on versions of replicated objects.

Prerequisites

Enable inventory reports on src (Source) and dst
(Destination) buckets
Create Athena tables (src_inventory, dst_inventory) based on inventory reports for each bucket

Monitor

Run Athena query after both inventory reports generated e.g. on daily basis, can be executed from Lambda, Step Function, build server or any type of cron job.

Example of Athena query which should show number of missing objects on destination bucket at the time of generation of inventory reports:

FROM src_inventory src
LEFT JOIN dst_inventory dst
ON src.key = dst.key
WHERE dst.key is NULL
AND src.dt = '2018-11-11-08-00';

The result can be reported as CloudWatch metric and monitored with CloudWatch alarm.


Mysql with Kubernetes I am converting a webapp from monolithic to kubernetes based architecture.
But how can mysql be persistent with data when we have 2 mysql pods in each of 2 differnet nodes?
I think we can mount a volume but still if my server with the mounted volume fails, this will be a single point of failure.
How can I avoid this?
I want to acheive no single point of failure using kubernetes without cloud.
How can I acheive this?

I would suggest to only run stateless apps in k8s. One should run statefull apps like databases in dedicated database solutions like AWS RDS.


Best way to isolate programming projects? Not sure if this is the best place for the question but I have multiple development projects with different dependencies.. I want to know what the best practice is to isolate each project?

I have many unrelated projects that a designed and updated on my laptop but I typically push them to a rPi or VMs (esxi). I would like to isolate the various projects so its easier to track which dependencies belong to which project and to stop the OS from having libs and other deps that aren't needed after a project is 'completed'. I do code some for work but its largely a hobby helping neighbors and the local SPCA.

What I use:
I typically program on a Debian box or Windows 10 depending on what I need to accomplish.
I program in Java, Python, and the typical web stuff (jscript, html, css)..
Bash and Powershell if you consider either programming too.
Version control is done via Gitlab.

A more generic approach would be use a separate virtualized environment for each project, see virtualization. That could be either:

a container image - more lightweight, faster and definitely more popular these days, see containers
a virtual machine (VM) image, slower and a bit more heavier to maintain than containers

You'd have all the project dependencies (IDEs, libraries, tools, etc) installed inside the virtualized environment, not on your system itself.

You'd need to create/bring the project's corresponding image on your machine when you start working on it and you can delete or save it for later re-use on some external storage when done. A shared artifact respository would be recommended as storage if more than one developer can work on these projects.

A huge advantage of this approach is that such virtualized environments play really well with automated CI/CD pipelines, in true DevOps spirit, helping to minimize/eliminate the differences between the development and staging/production environments.

Depending on the programming language and/or the supporting development tools being used on a particular project you may have some more specific approaches/alternatives. For example Pipenv or Virtualenv/virtualenv can be used to obtain such isolation for Python projects. But these solutions are typically not usable in other contextes.


What's the relationship between ContainerD and Docker? I'm aware that Kubernetes uses containerd internally and that containerd is part of Docker.  Does that mean that Kubernetes only uses certain parts of Docker to provide orchestration of containers? Additionally, how does runc fit into this ecosystem, I'm aware Kubernetes uses it because CVEs affecting runc impact Kubernetes.

I've not been able to find out what the technical components of Kubernetes are; there are plenty of diagrams out there showing the logical elements (i.e. masters, nodes, kubelets, deployments, services, pods, etc.). However, I'm trying to get into the details a bit more and understand what parts make up Kubernetes.

Is it possible to describe how Kubernetes is constructed from its components such as runc, containerd and Docker?

Containerd is a daemon for running containers.  The Docker blog has a nice write up explaining how this fits into Docker.  While containerd is used in Docker, it can also be used as a daemon for other container solutions.  Containerd requires runc in order to function.

runc as defined by the project is a "CLI tool for spawning and running containers according to the OCI specification".

These tools are designed to follow the Open Container Initiative (OCI) specifications.  This has become a standard and defines how Docker and other tooling interacts with containerd and runc.

When you look at most Kubernetes diagrams they will not show containerd or runc as they are abstracted away into the container engine you use (e.g., Docker, rkt).  Because Kubernetes uses containerd, it will also use runc.  This is what allows multiple container solutions that follow OCI specifications to integrate with Kubernetes.

So yes, you could break down how part of Kubernetes is built by looking at runc, containerd, and the container engine.


what is difference between "sudo shutdown now" versus "aws ec2 stop-instances" When logged onto a AWS EC2 instance say some Ubuntu box and I issue
sudo shutdown now

how is that different from stopping the same instance using
aws ec2 stop-instances --instance-ids  $curr_instanceid --region $AWS_REGION

either way I see the instance gets into the state of stopped and it will transition into state running once this is run
aws ec2 start-instances --instance-ids   $curr_instanceid --region $AWS_REGION

Just to state the obvious for junior folks who might come across this: sudo shutdown now requires access to the operating system (via SSH keys, inbound security groups, etc), while aws ec2 stop-instances is an AWS CLI command which simply requires API authentication (via IAM credentials or STS tokens).
Aside from the obvious difference in invocation, the shutdown is the same.

When you stop or hibernate an instance, we shut it down. You can restart your instance at any time. Before stopping or hibernating an instance, make sure it is in a state from which it can be restarted. Stopping an instance does not preserve data stored in RAM, but hibernating an instance does preserve data stored in RAM. If an instance cannot hibernate successfully, a normal shutdown occurs.

Source: stop-instances documentation
The --hibernate flag defaults to false, so a normal shutdown is triggered immediately by your CLI command.
When --hibernate is true, the suspend-to-disk process differs from a regular shutdown (such as sudo shutdown now), and is further documented here.


Codeship cannot push image to Google Container Registry I created a Service Account in the Google Cloud console to let Codeship push Docker images to Google Container Registry, but when I test it with jet steps --push it says I don't have permissions.

{StepFinished=step_name:"push_app" type:STEP_FINISHED_TYPE_ERROR}
Image Push error for image gcr.io/my-project/my-app:1499432530, denied: Unable to access the repository; please check that you have permission to access it

The service account already has full owner permissions to my account. Is there anything I need to check in the Google console or my Codeship configuration?

The service account needs permissions to the storage segment where the containers will be pushed. To fix this

Log in to the Google Cloud Platform console
Go to the Storage section
Tick the corresponding storage segment and click Show information panel button in the top-right corner.
Add the storage object administration permission to your service account.

You can then retry locally with jet steps --push.


Consul: how to remove dead client nodes quickly Background: When running Consul client-mode in AWS, EC2s are constantly getting replaced, which results in a lot of dead client nodes when looking at the Consul UI or via consul members.

Already tried to enabled reconnect_timeout, leave_on_terminate (default to true on Client but tried to turn on anyway), and Autopilot cleanup_dead_servers to no avail.

The delay seems to be inconsistent as well, some got reaped after 4~7 days, some are never reaped, consul force-leave had to be used.

As a best practice, you should gracefully deregister the node gracefully. In this case, consul will know the the node has left and will be removed from the cluster. Otherwise, the consul cannot distinguish between a temporary failure, agent crash, network partition, etc.

There is a github issue as well related to this.

Quoting few important points from the issue.

  The nodes should automatically reap out after 72hours (not yet configurable, but soon). Otherwise, the best route is to issue a graceful leave before destroying the nodes (consul leave), so that they can be reaped immediately. They are kept around for that long since without a graceful leave, Consul cannot distinguish between a temporary failure, agent crash, network partition, etc.

.

  force-leave should push them into the "left" state. Nodes are not reaped until they are in failed for 72h or in the left state. "force-leave" just moves a node from the "failed" -> "left" state. They are not removed from the members list for 24 or 72h.


What is difference between release and deployment? What is the basic difference between software release and deploy? What is the relation between software release management and continuous deployment?

From Wikipedia:

  Release management is the process of managing, planning, scheduling
  and controlling a software build through different stages and
  environments; including testing and deploying software releases.

Depending on the context the term release is used to reference either of:

the version of the software being released (from a version control system prospective)
the artifacts actually being delivered corresponding to that version (which can include physical media, printed documentation, etc)
the process leading to making those artifacts available and, in the case where the release includes deployment - deploying them in the respective environment (which may or may not be the production one)

Continuous Deployment (together with Continuous Integration and Continuous Delivery) is the modern DevOps methodology aiming to releasing better quality software as fast as possible.

So you can see Continuous Deployment being part of the release management in some cases, but you can also see release management without it - for example in the waterfall development model.

Related: How does continuous integration relate to continuous delivery / deployment?


Bash Tokenization Errors on Jenkins JNLP Slave Why might the below bash commands give me an error on a Jenkins JNLP slave in Kubernetes cluster? Each work on a local machine.

Error: Syntax error: redirection unexpected

IFS='/' read -r -a branch <<< "${1}"

Error: Syntax error: "(" unexpected (expecting "}"):

IFS='/'
branch=($1)

Environment Details

JNLP Slave is using Dockerhub jenkins/jnlp-slave:latest
Slave being ran with kubernetes plugin
Jenkins v2.156 on master.
Command is from a sourced function in another script. (e.g. . "/script.bash")

I cannot say this is a the best answer so I do hope someone can provide a better clarification.

It seems the default shell for sh() in Jenkins is Bourne.  By updating the sh() with the correct shell this allows the below usage:

sh'''#!/bin/bash
    IFS='/'
    branch=($1)
    # Use it.
'''

This doesn't resolve the the redirection error.  Also I would think by updating the default shell in the image with chsh -s /bin/bash I wouldn't need to set it but I still do.


How to avoid code duplication in bitbucket-pipelines? The following bitbucket-pipelines file has been created:

options:
  docker: true

pipelines:
  default:
    - step:
        script:
          - echo "Login to the docker registry"
          - docker login some-docker-registry
    - step:
        script:
          - echo "Build some-image"
          - docker build some-docker-registry/some-image .
          - docker push some-docker-registry/some-image
    - step:
        script:
          - echo "Build some-image2"
          - docker build some-docker-registry/some-image2 .
          - docker push some-docker-registry/some-image2
        deployment: production

When it is run by bitbucket, it indicates that the authentication to the docker registry fails so now this step has to be added over and over again:

options:
  docker: true

pipelines:
  default:
    - step:
        script:
          - echo "Login to the docker registry"
          - docker login some-docker-registry
          - echo "Build some-image"
          - docker build some-docker-registry/some-image .
          - docker push some-docker-registry/some-image
    - step:
        script:
          - echo "Login to the docker registry"
          - docker login some-docker-registry
          - echo "Build some-image2"
          - docker build some-docker-registry/some-image2 .
          - docker push some-docker-registry/some-image2
        deployment: production

Is it possible to prevent such duplication? An attempt has been made (see first code snippet), but that does not seem to work, i.e. a docker login is required by every step.

Unlike the declarative pipelines in Jenkins, bitbucket does not seem to have such functionality. In order to prevent code duplication a script was created and the current bitbucket-pipelines looks as follows:

options:
  docker: true

pipelines:
  default:
    - step:
        script:
          - ./build-script.sh API true true
    - step:
        script:
          - ./build-script.sh Write true true
        deployment: production


Artifacts mentioned in secondary-artifacts section not getting uploaded to s3 bucket I have created a codePipeline project through which my application get build using maven and it's artifacts are uploaded to s3. I'm trying to upload multiple artifacts as a single zip file to s3 bucket through codebuild. Below is my sample buildspec.yml file

version: 0.2

phases:
  install:
    runtime-versions:
      java: openjdk8
    commands:
      - apt update -y && apt install maven -y
      - echo Install phase started on `date`
      - echo Install phase ended on `date`
  pre_build:
    commands:
      - echo Pre_build phase started on `date`
      - bash pipelines/aws/scripts/onCheckout.sh
      - echo Pre_build phase ended on `date`
  build:
    commands:
      - echo Build phase started on `date`
      - bash pipelines/aws/scripts/onBuild.sh
      - echo Build phase ended on `date`
  post_build:
    commands:
      - echo Post_build phase started on `date`
      - bash pipelines/aws/scripts/onPackage.sh
      - bash pipelines/aws/scripts/onPublish.sh
      - bash pipelines/aws/scripts/onPromote.sh
      - echo Post_build phase ended on `date`
      - echo $CODEBUILD_SRC_DIR
artifacts:
  files:
    - pipelines/**/*
    - target/spring-petclinic-2.0.0.jar
  base-directory: $CODEBUILD_SRC_DIR
secondary-artifacts:
  artifact1:
    files:
      - pipelines/aws/appspec.yml
    base-directory: $CODEBUILD_SRC_DIR
    discard-paths: yes

codebuild-output:

https://forums.aws.amazon.com/ 2019/05/24 12:00:27 Phase complete: POST_BUILD State: SUCCEEDED
https://forums.aws.amazon.com/ 2019/05/24 12:00:27 Phase context status code: Message:
https://forums.aws.amazon.com/ 2019/05/24 12:00:27 Expanding base directory path: $CODEBUILD_SRC_DIR
https://forums.aws.amazon.com/ 2019/05/24 12:00:27 Assembling file list
https://forums.aws.amazon.com/ 2019/05/24 12:00:27 Expanding /codebuild/output/src289881682/src
https://forums.aws.amazon.com/ 2019/05/24 12:00:27 Expanding artifact file paths for base directory /codebuild/output/src289881682/src
https://forums.aws.amazon.com/ 2019/05/24 12:00:27 Assembling file list
https://forums.aws.amazon.com/ 2019/05/24 12:00:27 Expanding pipelines/**/*
https://forums.aws.amazon.com/ 2019/05/24 12:00:27 Expanding target/spring-app-2.0.0.jar
https://forums.aws.amazon.com/ 2019/05/24 12:00:27 Found 52 file(s)
https://forums.aws.amazon.com/ 2019/05/24 12:00:28 Phase complete: UPLOAD_ARTIFACTS State: SUCCEEDED
https://forums.aws.amazon.com/ 2019/05/24 12:00:28 Phase context status code: Message:

Artifacts mentioned in secondary-artifacts are completely ignored due to unknown reasons, Am I missing any parameters?

In the CodeBuild "Artifacts" configuration, there is an "Add Artifact" button.

You need to click that and reference your secondary artifact to get it to upload to S3 as well.


Is there a system which automates everything (OS release, and above) to host websites on? This is not really a DevOps question but more of a System Automation question (which DevOps covers only a part of, by popular opinion, so I hope it's okay).

I feel like some 5 USD IaaS VPS with Ansible could give me the peace of mind of technical stability for say 15 years on a debian:stable system, but the debian:stable release itself will get old and vulnerable with the years.

Everything above the OS will generally not get old because by Ansible (given a correctly orchestrated playbook) will continuously update and upgrade it.

Yet the fact that the OS will indeed get old can't get out of my mind. Transferring websites from one IaaS to another even once per 4 or 8 or 12 or 16 years is not something I want to do in principle. One could say, well, you can't eat the cake and keep it whole.

I thought of using PaaS but PaaS solutions wouldn't necessarily bring me that peace of mind as well because my data might be "thrown" in some server environment with an old OS (could be old as 20 or 25 years) without me even knowing that and not all PaaS companies guarantee streamed OS release_upgrade.

So, I desire some full system automation solution. Is there even such a thing which is usually stable and countable in earth in 2018? I ask this totally seriously, maybe I need to be less stressed about this, but maybe such technical solution (which isn't SaaS of course) exist and might suit my needs?

I just want to build my websites (usually Drupal) on the most automated, most upgraded system (OS and above) possible, in the budget of 5-30 USDs.

Note: I never provided hosting services and doesn't provide them now either - usually my websites are personal websites that only I myself manage.

TL;DR Pick software that focuses on long-term support. Use containers so that you can separate the runtime you need for your app from the security patching of the underlying infrastructure.

Long Answer:

I used to run some old PHP sites running popular content management systems. The plugins would fall out of support such that I could not upgrade the content management system else the plug-ins would break. So then I would stick on the latest major version things worked on and hope that no new security holes would come. I then ran monitoring software and manually fought off any scripts that got injected through security holes. Then the operating systems that support that old version of PHP I needed would fall out of support. I then couldn't easily move the hoster to take better use of cheaper deals or high availability. So I would find ports of the legacy PHP I needed onto the oldest version of an OS that hosters would support. Total nightmare.

What this demonstrated to me was that some software isn't built to last more than three or four years. Millions of sites would have had the same issues as me but tens of millions more wouldn't have as they would have been shutdown else rebuilt from scratch on newer software. So the first lesson here is that if you think you need to build for the long term you need to figure out if your using software that cares about long-term support and has a published lifetime.

The world has moved on and the PHP7 world is taking a new approach to having a predictable software release cycle and lifetime (still quite short I might add, but more predictable). You should not just assume nor expect any distro or any community to be with you for the long haul unless that is exactly what they said they are trying to do. And if that is what they are trying to do then they won't do other things that are in conflict with that such as run hot new framework features that are trending on stackoverflow.

These days I happen to be helping a company that is running Node.js 8 and PHP 7.x on RHEL7 containers on kubernetes on AWS. They get long-term support of security updates with a published lifetime. I have automated the detection of new security patches to the PHP and Node.js containers. A weekly job checks the upstream container registry and if there is a new security patched version it is announce to a team chat room along with the command to patch the containers in the test environment. A commercial subscription to those patches is included in my managed kubernetes hosting costs. Yet as its all opensource I could fall back to using Centos7 if I wanted to cut some costs.

As I am running containers I don't care whatsoever what patches are being applied to the IaaS nor the version of kubernetes I am hosted on. I am on openshift.com and they have upgraded to 3.11 through security fixes without telling me. They just patch the worker VMs and bounce them and kubernetes starts my containers on other VMs. I am insulated from all the security patching of the cluster I am on because I am running containers that have inside them the minimal operating environment needed to run my app. That only changes when I choose to upgrade to a new container and with long-term security patch support; I don't need to do that for years. I understand that the Kubernetes hosts that run my containers themselves run a very stripped down distro to maximise their security and customer density. It doesn't matter to me. The idea of an IaaS stack of Linux that goes out of support seems a totally alien concept to me now even through I lived many years with that problem.

It also means that I am free to move to any cloud or hosting provider that can run my containers. It doesn't have to be kubernetes it could be Docker Swarm or whatever. All that I am betting on is that this better way of working with containers will be supported in the long term. I think that's a good bet. A container image is only a json file naming a load of tar files that are the layers in the image. Yes; its just tar files and json. As long as the container host can load my files from a tar and attach a TCP/IP stack my stuff will run. I am pretty confident that this approach will have a very, very long shelf life.

I don't think that anyone that is trying to run apps should start out on IaaS today. PaaS is one solution but tends to have very strong vendor lock-in. CaaS is the new sweet spot. Choice. Stability. Control. Portability. If it runs under docker on your laptop just push it to the container orchestrator you are renting space on. My containers only have one thing in them. The hosts that they run on only does secure container hosting.


Jenkins triggered by multiple git repos I've been looking around to see how to have ~10 different repos all on a git push trigger a single pipeline that does pretty much the same exact thing - build a docker image -except that the different repo names will be used for the docker image repo and the specific git commit will be the tag.

Is this possible or do I have to break it down into multiple pipelines? Ideally I would be able to have one that queues jobs when multiple pushes happen in a short period of time.

I would be inclined to use Pipeline Templating as this gives you the ability to define standard templates centrally but leave the SCM field to be defined by the individual job.

  Leveraging templates in this way lets you separate the business logic (what should happen when) of your pipeline from the technical implementation (what’s actually going to happen). The result of this is a CI/CD pipeline that’s proven to be significantly easier to manage when supporting multiple teams simultaneously.

I recommend that you do not have a single pipeline for all repositories as that is going to make it very hard to tell which repository is failing or slow via dashboards and alerts.


Can you inherit a template_cloudinit_config in another template_cloudinit_config object in Terraform? I recently discovered template_cloudinit_config within Terraform, and I'm hoping I can utilize it to specify something like the code below. Basically my goal is to declare a core template_cloudinit_config that contains a number of shell scripts that should get run on every machine.
Then I'd like to declare a 2nd template_cloudinit_config that, without duplicating code, lets me essentially inherit the core set of scripts that should get applied to every instance AND define an additional script to run.
That way from a compute perspective, I could easily create specialized instances that have custom scripts applied to them, but still inherit a number of basic configurations. I know I could make this work with remote-exec provisioners, but I am trying to avoid them since Terraform says they should be used as a last resort.
data "template_cloudinit_config" "core_config" {
  part {
    content_type = "text/x-shellscript"
    content      = file("scripts/core_script1.sh")
  }

  part {
    content_type = "text/x-shellscript"
    content      = file("scripts/core_script2.sh")
  }
}

data "template_cloudinit_config" "custom_config" {
  part {
    content     = "${data.template_cloudinit_config.core_config.rendered}"
  }

  part {
    content_type = "text/x-shellscript"
    content      = file("scripts/custom_script.sh")
  }
}

resource "aws_instance" "vm1" {
  instance_type = "t2.small"
  ami           = data.aws_ami.ubuntu_ami.image_id

  # Other variables that declare subnet, key, etc

  user_data = "${data.template_cloudinit_config.core_config.rendered}"
}

resource "aws_instance" "vm2" {
  instance_type = "t2.small"
  ami           = data.aws_ami.ubuntu_ami.image_id

  # Other variables that declare subnet, key, etc

  user_data = "${data.template_cloudinit_config.custom_config.rendered}"
}

The cloudinit provider only knows how to encode multipart MIME documents to pass to cloud init; there is no mechanism for parsing to modify and then re-encode.
However, a different way to frame your problem is of constructing a sequence of part objects in a local value and then passing them all together to the cloudinit_config data source, like this:
locals {
  core_config = tolist([
    {
      content_type = "text/x-shellscript"
      content      = file("${path.module}/scripts/core_script1.sh")
    },
    {
      content_type = "text/x-shellscript"
      content      = file("${path.module}/scripts/core_script2.sh")
    },
  ])
  custom_config = tolist([
    {
      content_type = "text/x-shellscript"
      content      = file("${path.module}/scripts/custom_script.sh")
    },
  ])
}

data "cloudinit_config" "core" {
  dynamic "part" {
    for_each = local.core_config
    content {
      content_type = part.value.content_type
      content      = part.value.content
    }
  }
}

data "cloudinit_config" "custom" {
  dynamic "part" {
    for_each = concat(
      local.core_config,
      local.custom_config,
    )
    content {
      content_type = part.value.content_type
      content      = part.value.content
    }
  }
}

resource "aws_instance" "vm1" {
  instance_type = "t2.small"
  ami           = data.aws_ami.ubuntu_ami.image_id

  # ...

  user_data = data.cloudinit_config.core.rendered
}

resource "aws_instance" "vm2" {
  instance_type = "t2.small"
  ami           = data.aws_ami.ubuntu_ami.image_id

  # ...

  user_data = data.cloudinit_config.custom.rendered
}

The above is using dynamic blocks to generate a dynamic number of part blocks based on the number of elements in a list, which means you can focus on making sure the lists have the necessary items, using whatever dynamic expressions you need to decide that, and have Terraform project that into zero or more part blocks automatically.
Note: the hashicorp/template provider has been deprecated and continues to be available only for backward-compatibility. For that reason, I wrote the above to use the hashicorp/cloudinit provider instead. Its cloudinit_config data source is based on what was formerly template_cloudinit_config.


Private docker registry that attempts a pull-through on images not found locally I have a docker registry image running on a local network with the command

docker run -d -p 5000:5000 --restart=unless-stopped --name registry -v /mnt/part_sdb/registry:/var/lib/registry registry:2

but enabling the caching of images stored in Docker Hub requires one to pull the image, and push it in the local registry

Is there a way to configure the docker registry to do automatic pull-through if the image requested is not available locally while preserving the ability to push local images?

from this article:

  The catch is that you will probably run the public Registry as a
  mirror or as a regular registry, so you will probably end up running
  two containers — one as a mirror, another as your local Registry (not
  a big deal). In artifactory you can set up as many registries as you
  want (local or remote), each with its own URL. Or you can “mix” them
  on a single URL with a virtual registry (if you think it is a good
  idea).

Yes, it is possible. The entire subject is discussed in Registry as a pull through cache, from where the below quotes are taken.

Basically you need to configure the cache as a proxy for Docker Hub:

  Configure the cache

  To configure a Registry to run as a pull through cache, the addition
  of a proxy section is required to the config file.

  In order to access private images on the Docker Hub, a username and
  password can be supplied.

proxy:
  remoteurl: https://registry-1.docker.io
  username: [username]
  password: [password]

Of course, you also need to configure your docker daemon to use your local cache (but I presume that may be already done from the context of the question):

  Configure the Docker daemon

  Either pass the --registry-mirror option when starting dockerd
  manually, or edit /etc/docker/daemon.json and add the
  registry-mirrors key and value, to make the change persistent.

{
  "registry-mirrors": ["https://<my-docker-mirror-host>"]
}


How can I clean/delete a Jenkins workspace after merge? I would like to keep a workspace in Jenkins until after the pull request is merged. Meaning, I don't want to clear the workspace after every build, but I also don't want to clear the workspace only when the branch gets deleted.

How can I use the github webhooks to trigger a workspace cleanup whenever a pull request gets resolved as merged?

Ideally, the solution would involve using my JenkinsFile/Pipelines.

edit: Background.

We have long-lived branches and short-lived branches. For brevity, I'll call the long-lived branches "team branches".  Our workflow is to take short-lived branches, such as feature branches and make a PR request to pull them into the team branches.

On Jenkins, we then build the feature branch and the PR request and the team branch.  Normally this works fine, but sometimes when multiple teams are merging to qa each team and feature branch needs to update(pull) and then get built on Jenkins again.  Currently, we are always cleaning the workspace which makes the update builds take longer than they need to. So I want to be able to preserve the workspace until all updates are done and the PR request actually gets merged, only then do I want to delete the workspace for the feature branch and the PR request and the team branch.

I am assuming you are using Jenkinsfile. You can do for example:

pipeline {
    agent any

    stages {
        stage('Build') {
            steps {
                echo 'Building..'
            }
        }
        stage('Test') {
            steps {
                echo 'Testing..'
            }
        }
        stage('Deploy') {
             when { branch 'master' } /* when in master branch */
             steps {
                echo 'Deploying....'
                deleteDir() /* clean up our workspace */
            }
        }
    }
}


Best practices to maintain forks of Git repository in an on-prem Azure DevOps Server We have an on-premises Azure DevOps Server and like to use the Git workflow "Fork-Clone-Push-PR".
In the moment of forking the system let allow us decide into which project and under which name the repository should get forked.
Forking it in the same project would mean tons of such forks after a while, which looks ugly.
So a dedicated project "Forks" might be the solution to host all these forks. But is this a nice option?
Are there any best practices or tips & tricks on how we should proceed?

I think that the best way to answer your question is to take a step back to focus on your goal.  Based on the fact that you're using an on-premise Azure DevOps Server, I am assuming that you're working on a closed-source project (I apologize if this assumption is incorrect).  I believe that your goal is to control quality via PRs in this environment.
As this Atlassian tutorial states, "The Forking Workflow is most often seen in open source projects". The reason for this is that the fork mechanism was created for a very specific problem, "How do you give complete strangers the ability to contribute to a project without giving the world write permissions to your repository?".  Note, I occasionally use the forking mechanism for other fairly advanced purposes as well, but this is its primary use.  There is a fantastic book called Git for Teams that describes git workflows and their intents very nicely.
My first experience with git was working in GitLab on a closed-source project.  We were a team of ten employees working on an enterprise software application.  We were required by the process to fork the repo and create a PR to get our changes back. The result was a big learning curve for the engineers who were new to git and a lot of confusion around setting up remotes and getting the latest code.
Many people erroneously think that the only way to have a pull-request mechanism is to fork the code, probably because their only exposure is through PRs in the open-source community.  To quote Microsoft, "Pull requests can come from either topic branches or from a branch in a fork of the original repository".  A topic branch is just a fancy term for a bug/feature/etc branch.  When I am working with a closed-source group that is new to git, I will often disallow forks in the Project/Repository settings and I will enforce a branching policy of a minimum number of reviewers for a pull-request to the master branch.
Regarding best practices for branching strategies in git, the book I mentioned above is a great first step.  There are many standard branching strategies, a couple examples are:

gitflow which is great for teams that are new to git and has some awesome tools and many tools incorporate the gitflow pattern, like sourcetree.
trunk-based development a very lean development process.

Regarding your original question, I suspect that creating a new Azure DevOps project to store all of those unsightly extra fork repos would just add another level of complexity to your work.  You would need to enforce security for both projects, manage users and permissions for the new project, etc.  In this case subtraction is likely better than addition, I suggest removing forking and using a good branching strategy for your project (if possible). It would simplify your life significantly.
Please let me know if this did not answer your question or if you have any further questions.


Change container size in a template from the Marketplace I am trying to use a template from the Azure Marketplace. Everything works, but it defaults to storage sizes that are by orders of magnitude too large for my usecase.

At the end of the process, I can "kind of" export the template files in text form - at  least parameters and variables, but there are no values for the storage here. I assume that those are hard coded inside whatever scripts they use internally to deploy that application. I see no way to have a look at or modify those scripts.

Is there something that can be done in such a case? Should the scripts behind marketplace apps be available somewhere (I just didn't see them)? Or are they usually closed off from the user?

According to https://azure.microsoft.com/de-de/blog/container-apps-now-available-in-the-azure-marketplace/ it seems to be possible to define the storage-size when a container-app is chosen.


what is better between Jenkins & AWS CodeDeploy for CI/CD I'm considering what is better and easier to use for CI/CD pipeline between Jenkins and AWS CodeDeploy.

e.g. Jenkins server request webhook to git repo(github), and deploy to EC2.

e.g. When push to git repo, deploy to EC2 go through codeDeploy.

Thank you for your help!!

Both are excellent tools, but it depends on your use cases for which is better. I would highly recommend using the entire AWS Ci/Cd suite (CodeBuild/CodePipeline/CodeDeploy) to gain the full benefit of using your resources in AWS. Here are pros for both:

Jenkins

Multitude of customizability plugins
Integration with all cloud providers (GCP, Azure, etc.)
Open sourced
Configuration as code through Jenkins Pipelines
Able to manage your own Jenkins Server (vs AWS which manages it for you)
Customizable GUI

AWS CI/CD

No plugins (no dependency hell)
Trivial integration with IAM (for both users and infrastructure)
Easy integration with AWS Secrets Manager, KMS, and other secret management tools
AWS managed infrastructure
Infrastructure as code through CloudFormation
Scalable out of the box

TLDR; Choose Jenkins if you want more control/configuration of your deployment infrastructure or you are integrating with many non-aws resources, otherwise, choose the AWS CI/CD suite for less overhead and better integration with AWS.


Connect two PCs to Azure vnet then RDP from one to the other I have an Azure vnet setup, to which two PCs are connected via a point-to-site VPN. I was hoping to be able to then RDP from one pc to the other using the IP assigned by the gateway, but this doesn't work, just can't find the other pc. I've dropped firewalls etc... to test this, but still nothing.

Can anyone advise if this is actually possible, or if there is something else I can do?

I have fixed this partially by updating the routes.txt file in the installation folder for the client app by adding an entry for the subnet used by the gateway to assign IP address to new logins.

While this now routes to the correct PC when RDPing, unfortunately a new IP is assigned to the PC every time it connects to the VPN, making it difficult to know the IP address when connecting.


kubectl create deployment not has the option containerPort When I execute kubectl create deployment nginx --image nginx --dry-run=client -o yaml it does not have options --port or --containerPort.  My yaml is:

apiVersion: apps/v1
kind: Deployment
metadata:
  creationTimestamp: null
  labels:
    app: nginx
  name: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  strategy: {}
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: nginx
    spec:
      containers:
      - image: nginx
        name: nginx
        resources: {}
status: {}

Is there a way to generate yaml with the containerPort option?

From the kubernetes docs I see this example:

$ echo 'apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: deployment-example
spec:
  replicas: 3
  revisionHistoryLimit: 10
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.11
        ports:
        - containerPort: 80
' | kubectl replace -f -

So under your containers entry you should be able to add:

        ports:
        - containerPort: 80

Is that what you had in mind or, at least, good enough?  You seem to want this as a command line option for the kubectl command and I don't see anything that indicates this is possible.


Re-using Python virtual environments on a build server Currently, every time we run a build through Jenkins+Ansible, we are re-creating a virtual environment and re-installing all the dependencies  listed inside the requirements.txt file.

This is very slow and does not scale well. How can we improve and speed up the process? Can we re-use virtual environments?

Our latest idea which we have not yet implemented was to build virtual environments outside of a Jenkins workspace, name virtual environments based on a project+branch and keep an MD5 sum of the requirements for every virtual environment. Then, before building an environment the next time, we calculate the MD5 sum of the current requirements in this branch and look up existing virtual environments by this MD5 sum. If an existing environment with this sum was found, just re-use it. We are not sure if this is the best way to solve the problem.

It's generally a best practice to make your builds idempotent.  Leaving artifacts behind only opens up opportunities for dependency management issues- and resolving those in a resilient way is exactly why you're using a build server.  I suggest you take a look at WHY your Python builds are running so slow- if you discover this is because of an overabundance of packages in each project, this might be optimizable.  If you discover this is because of server performance issues, those are easily fixable with more resources.  If you find you are blocked by network throughput, consider using an artifact repository to cache packages locally.  But building persistent virtualenvs seems like a recipe for problems that could be unpredictable when moving to another server, and extremely frustrating to troubleshoot.


How do I install local application dependencies in a Kubernetes manifest? In a container, there are RUN statements that you can add to a containerfile (such as a dockerfile) to add further dependencies using the package manager like APT or DNF within a container.

What is the proper equivalent to performing this in Kubernetes with a YAML manifest? For example, I'd like to install git in my Jenkins container so as to be able to use the git plugin in that application.

I tried to do this with "command" and "args" in the yaml, but it looks like it isn't executing as superuser? I get permission errors, and cannot install any packages that way.

You can't add run-time dependencies on an already build image using a yaml manifest.
You might attach/exec to an already running pod and install them, but it makes no sence, since you need to make sure that when the pod is re-created you do it again. The much better and right way is simply to describe it in your Dockerfile.

Kubernetes is not created for such kind of hacky moves. It simpy maintains state that you describe.  Essentially you need to add an instruction that adds git in your Jenkins dockerfile, build your new image and refer to it inside the manifest.

Commands and args are used if you want override the image's default entrypoint and its arguments.


Docker vs CMS - related or a different thing? I am new to DevOps and I am trying to catch the techniques and their names and meanings.

I looked wikipedia explanations on Puppet (a Configuration Management System example) and that of Docker.

Docker page says it is categorized as a virtualization platform. Docker manages many virtual containers on same Linux machine.

Is there some similar way as with Puppet and other CMS to tell what configuration is set up on container ad-hoc creation as there is in a CMS or is there anything similar on these two (a CMS and Docker and like)? Could they be used only separately or also together?

At least there exists a dockerfile to do some kind of similar sounding thing than system description language makes in CMS. That makes me think they perform similar tasks.

So in short I would like to how to categorize CMS vs Docker.

Something like puppet would help automate and configure the Docker Container.  CMS is configuration/provisioning and Docker is like you said a virtualization platform.  You're trying to compare two completely different things.  Here is an article on how to configure puppet with docker, sometimes an example helps.  https://puppet.com/blog/building-docker-images-puppet

For another way to classify, Docker is the VM you host your software.  CMS is automation on how to provision it.

As a side note, most people are moving to Kubernetes to orchestrate and provision containers on the cloud.


String to specify to KUBE_SERVICE_ADDRESSES (API Server configuration) What are the strings to set to KUBE_SERVICE_ADDRESSES and their meanings? Which documentation or GitHub repo to look into to understand?

Background: I have seen a few strings but not sure what these strings mean. Such as:

KUBE_SERVICE_ADDRESSES="--portal_net=10.254.0.0/16"
KUBE_SERVICE_ADDRESSES="--service-cluster-ip-range=10.254.0.0/16"

According to this Q&A and this code the --portal_net was renamed to --service-cluster-ip-range since version 0.18.

# --service-cluster-ip-range=<nil>: A CIDR notation IP range from which to assign service cluster IPs.
# This must not overlap with any IP ranges assigned to nodes for pods.
KUBE_SERVICE_ADDRESSES="--service-cluster-ip-range=${SERVICE_CLUSTER_IP_RANGE}"


Which Jenkins file is responsible for Enable cache setting I want to know how can I modify the below setting from Jenkins server and not UI:

http://XXXXX:YYYYY/configureSecurity/

Add the following lines in /var/lib/jenkins/config.xml under securityRealm and then restart jenkins service to enable cache. Remove the below 4 line if you don't want to enable it.

<cache>
  <size>500</size>
  <ttl>3600</ttl>
</cache>

Please refer the below image


Chef (well, Test Kitchen) hanging when using local cookbook I am new to Chef, trying to put together a POC for my company.
So I've got a project that has two cookbooks under the cookbook folder. One cookbook is a "base" cookbook called 'hydra-base', the other (called 'automator') is supposed to extend it. Right now all I have is a test in the extending cookbook, and a recipe invocation.

My problem is when I attempt to verify it with Kitchen, everything uploads ok until it gets to trying to apply the recipe, then I get the following line:
Recipe:  hydra-base::default and then kitchen just hangs. It has been idle for about 45 minutes now. If I break out of the tool (ctrl-c) then rerun kitchen converge, it gives me this error then hangs:

[2018-01-09T17:35:23+00:00] WARN: Chef client 1514 is running, will wait for it to finish and then run.

So it appears that chef still thinks its trying to update something. Here are the relevant files:

# automator/recipes/default.rb
include_recipe 'hydra-base::default'

# automator/Berksfile
source 'https://supermarket.chef.io'

cookbook 'hydra-base', path: '../hydra-base'
metadata

# automator/metadata.rb
depends 'hydra-base'

calling kitchen verify -l debug does not seem to provide any additional information after the line that hangs.

Can someone please give me a hint on how to fix this or how to get more information out of kitchen?
Thanks

It turns out that AWS was just excruciatingly slow and apt-get (applied in the base) was timing out. I would have known this had I gone back earlier and tried testing the base cookbook.


How to implement blue-green deployment with one master branch I was told that they easy way out is to have two production branches, master/blue and master/green, git merge action triggers CI to deploy to the matching stacks. Is that possible to do it with single master branch?

Some more details about our stacks.

CloudFront, origin set to S3 bucket which serves all the front-end SPA (single page app) static assets.
SPA app talks to AWS API Gateway, which connects AWS Lambda.
AWS Lambda is responsible for db CRUD operations with DynamoDB.
Front-end app and Lambda function each has their own Git repo, Git master branch merge action triggers 2 Circle CI jobs, one copy front-end assets to S3 bucket, the other deploys Lambda with serverless framework.

For blue-green deploy, I'm thinking to have 2 instances of S3, API gateway and Lambda function. To make the blue-green switch, we'll need to change origin from one S3 bucket to another, then create invalidations to clear the cache.

To make the single production git branch blue-green deployment to work, Circle CI needs to know is the current active stack is blue or green, copy & paste / deploy to the other stack, if Circle CI is not able to do that, the alternative that I could think of is to somehow put the checkout_current_active_stack action in the git precommit hook? To be specific:

(in the git precommit hook) send out a HTTP request to the current active stack API (AWS lambda function, which keep track of the current stack information)
deploy base on the return value ('blue' or 'green')

It is worth separating your Continuous Integration "Build" from your Continuous Deployment "Pipeline".  For the build portion, CircleCI seems to be your tool of choice, for the latter I recommend using something like AWS CodeDeploy which natively supports Blue-Green Deployments, for completeness you can use CircleCI to orchestrate AWS CodeDeploy:

Separating your pipeline phases has several benefits:

You only need to compile/test once from the master branch; then promote the packages through the Artefact Repository.
Your packages are stored forever in an Artefact Repository, for example Sonartype Nexus or JFrog Artifactory, this means you can reproduce an old build as you have the specific packages used for deployment.
CodeDeploy will handle the release patterns for you without having to think too much about how to achieve Blue-Green Deployments.

It is also worth stating that you can model the whole pipeline in Application Release Automation tools such as BuildMaster or Nolio.  Personally, I prefer to build my pipelines from multiple 3rd party SaaS solutions.


Docker memory storage Docker daemon allows, according to documentatio, using memory storage driver for its images.
Has anybody any daily business experiences with that?

According to the documentation, memory storage is intended for testing purposes only and should never be used in production.  The only practical purpose I can think of for this might be state testing, such as with Serverspec or Inspec, to validate configurations.  Using memory storage for this purpose might be a little faster and use less disk storage.  I'm not sure those benefits are worth the effort to implement.


Backup Jenkins log files to external cloud storage I'm looking to automatically back up Jenkins log files for all jobs running on our master and slaves once a week, moving them to aws' s3 and deleting them from their respective directories when done.

My question is, is there an existing plugin that has the capabilities I need, or will I need to do this manually?

I've found the Periodic Backup Plugin but it only seems to back up locally, and is focused on config files, not log files (although I could use negative pathing to ignore everything but log files...which would be a massive pain). Likewise, ThinBackup is also config-only. I'm hoping someone might have seen something googling isn't turning up.

Historically, I approached this a bit differently. I added a post-build step to each job, and had it copy the output to S3 at execution time via the S3 plugin.  This blog post covers the moving parts n a pretty granular detail.

You may want to reference

  wget -O console-output.log ${BUILD_URL}consoleText

as the output for the job in question in your build task.


How to solve `no live upstreams while connecting to upstream` when trying to rproxy jenkins using jwilder/nginx? https://github.com/jwilder/nginx-proxy

nginx.1    | 2018/03/04 22:24:54 [error] 102#102: *276 no live upstreams
while connecting to upstream, client: 83.85.143.226, server:
jenkins.releasesoftwaremoreoften.com, request: "GET / HTTP/2.0", upstream:
"http://jenkins.releasesoftwaremoreoften.com/", host:
"jenkins.releasesoftwaremoreoften.com"

The docker-compose file looks as follows:

version: '3'
services:
  jenkins:
    image: jenkins/jenkins:lts
    environment:
      - VIRTUAL_HOST=jenkins.releasesoftwaremoreoften.com
      - VIRTUAL_PORT=8083
    ports:
      - 127.0.0.1:8083:8080
      - 127.0.0.1:50000:50000
    volumes:
      - /usr/local/jenkins/jenkins_home:/var/jenkins_home
    networks:
      - rproxy
      - jenkins

networks:
  jenkins:
  rproxy:
    external:
      name: rproxy

Another attempt to make it work, was changing the VIRTUAL_HOST to 80, but that did not work as well.

Navigating to http://jenkins.releasesoftwaremoreoften.com/ returns a 502.

After a spending a lot of time I decided not to continue with this approach and decided to use Jenkins helm.


Auto-scaling, how to securely get code from private repo I want to setup an auto-scaling configuration but cannot think how I can securely get code from a private repo when I spin up new instances.

I obviously don't want to put passwords in startup scripts, or store private keys somewhere that can be downloaded without a password.

How can I achieve this?

Option 1:
You could place your private repo in the same VPC as the instance and then pull code from there. Just make sure the repo cannot be accessed from outside the VPC.

Option 2:
Ssh into the new instance and SCP the binary over.

Option 3:
Use docker and deploy a private container registry (e.g. Amazon ECR). Build the image on your machine / build server and then push it to the registry from where it is available to your instances to pull and execute.


Automatic Kubernetes deployments per Github pull-request I'm looking for a tool/service that will achieve a similar result as Heroku Review Apps using Google services.

When a new pull-request is created in Github:

Create new Cloud Build trigger for it with a given template configuration.
When done with build, create new deployment for it in Kubernetes Engine given template configuration.
Provide the public URL in the Github PR

When PR is closed:

Remove trigger, container and deployment
Remove public URL from Github pull-request.

I don’t know of a specific tool but I have setup a lot of webhook handlers on OKD kubernetes to do git driven updates to kubernetes. All the scripts are on GitHub as OCD. The tools I used are:

adnanh/webook a go binary that you configure to catch git webhooks and run scripts. Here is my hooks.json config to match GitHub release events and run a build script on openshift kubernetes. Looking at the docs there are PR events you can match on. So you should be able to setup Webhooks to fire on PRs and trigger s2i builds.
hub is GitHubs official command-line tool. I put it into my tools image that runs my webhooks. It has some pull request commands. I haven’t checked them against your stated requirements. Hub is just a wrapper to their API so worst case you would need to use curl to update PRs.
Helm is the installer/template engine I use for creating apps. Helmfile is an excellent extension. So here is a generic “deployer” helm chart that can spin up a deployment. So OCD responds to GitHub events of new Helmfile declarative configuration being pushed and deploys or updated any helm releases modified in git.

For builds I had to do quite a bit to have an OKD container build tag the final container image with the GitHub release tag which is a generic “ocd-release-build” helm chart. I think that is necessary as each PR will need by tagged with the PR number in the container registry to deploy from that specific tag. I haven’t ever tried to get OKD to build a refs/pull but I would be shocked it didn’t work as expected.

So while OCD doesn’t yet spin up builds and deploys based on PR events it appears is likely that it could be extended to do that. We have migrated our apps from our in-house scripts over to OCD. A “PR previews” feature could be a very cool feature to add.  Feel free to raise a feature request.


Howto run test on test computer instead of build computer, using gitlab I'm learning gitlab and am asking for advice / best practice here.
I want to install a program to a different machine than the build computer, and run some automatic tests on that machine. How is this best done?
I think this can be accomplished with the "multi project pipeline" feature. Is this the simplest/best way?
Here is my plan:
I could have one (shell) runner and project for my build machine, and a different runner and project for my test machine. The two would be connected using "multi project pipeline" feature. The artifacts from the build pipeline would be installed on the test machine and then system tests would run on the test machine.
Is this the best way to solve this? Or is there a simpler/better way?
Edit: The test machine has special hardware/enviornment necessary for the program to run. That's why program needs to be installed there.

Answering my own question here. "Multi project pipeline" is not necessary here. You simply have a single project and mark jobs with different "tags". You can then register runners for these different tags, on different machines.
(Artifacts are transferred from one job to the next the same, regardless if the runners are run on the same machine or on different ones)
https://docs.gitlab.com/ee/ci/yaml/README.html#tags


Configure Docker to use SSL for a private registry on Windows 10? Trying to use a customer registry from a Windows 10 system - a native Docker client in PowerShell.

PS > docker --version

Docker version 17.06.0-ce, build 02c1287
PS > docker pull REPO_URL/foo-bar:0.1

Error response from daemon: get https://REPO_URL/v2/: x509:
certificate signed by unknown authority.

Importing the authority certificate to local user store via Windows tooling works for itself but has no effect here.

The official Docker documentation says:

  Docker recognizes certs stored under Trust Root Certification
  Authorities or Intermediate Certification Authorities.

That is, you can proceed as the following:

Start > "Manage Computer Certificates" (also available in the control
panel)
Right-click on "Trusted Root Certification Authoritites" >
"All tasks" > "Import"
Browse to the crt file and then keep pressing "Next" to complete the wizard
Restart Docker for Windows

In my case, the catch was that I imported the certificate via the context menu, and therefore it went to another folder where Docker could not access it. The certificate store behaves however very much like the normal Windows Explorer tree folder, so I had just to copy the certificate over to the supported category, and after Docker restart it worked.


Deploy to GKE from CI server outside of cluster I'm setting up our CI (Jenkins pipeline) to deploy apps to a GKE cluster. Our Jenkins sever is running external to GKE or any Google Cloud infrastructure. I'm thinking of using a docker image with helm and kubectl to deploy, but need to set up credentials. On my local dev machine, I authenticate with:

gcloud container clusters get-credentials [CLUSTER_NAME]

What would be the best way of setting up credentials for a Jenkins job? Do I have to install and use the gcloud tools on Jenkins, or is there another approach?

The easiest approach does appear to be to use gcloud, as trying to get a kubeconfig without it proved tricky.

Fortunately, gcloud, kubectl and helm are available as a single docker image kiwigrid/gcloud-kubectl-helm. For example:

docker run -it --rm --volume ./gcp-key-file.json:/data/gcp-key-file.json:ro kiwigrid/gcloud-kubectl-helm:2.11.0-224.0.0 bash
gcloud auth activate-service-account --key-file=/data/gcp-key-file.json
gcloud container clusters get-credentials dev-cluster --project <project name> --zone <zone>


AWS Security Group Creation Using AWS CLI I have created a security group in AWS. I have added the ingress rule using the below command

aws ec2 authorize-security-group-ingress --group-id sg-07xxxx4d --ip-protocol tcp --from-port 22 --to-port 22 --cidr-ip 0.0.0.0/0

But instead of IP, if i want to give the source as another existing security group, how can i do that?

Ok I got the answer after exploring few things and looking at the error more attentively. Here is the way:

aws ec2 authorize-security-group-ingress --group-id sg-07xxxx24d  --protocol tcp --port 22 --source-group sg-d7dyyyy5


Terrafrom to declare both infra and application deployment in one state Are there any advantages of maintaining both infrastructure (Infra-as-code) and application/service (helm chart deployment using helm_release provider of terraform ) under one state file?
I don't like the idea of maintaining both infra and application under one state, because for some reason it could be possible that our intention was just to change application however as there is one state and it could detect some other infra related change, and it will deploy them, which could have unintended consequences on the infrastructure.
however I would like to be open and see suggestion, if there are any advantages which maintaining one state file provides.

I tend to prefer infra code living separately from application deployment code.  Multiple reasons for this:

You want your infra team to make infra changes.  You want your app team to make application changes (including application configuration).  Putting them separately lets you specify separate guard rails on them (for example, only the infra team is allowed to push changes to your infra repo).
You don't necessarily want to trigger a redeployment of the app if you make infra changes, and you rarely want to trigger a redeploy of your infrastructure if you trigger a new app build.
You don't want a bad build of your app preventing you from making any infrastructure changes until it's fixed, and vice-versa.

It actually makes sense to put your app deployment code (i.e. Helm chart) inside the app repo itself.
Then, you can actually package up application config changes along with the corresponding release of your application.  You also empower developers to do it themselves, without running the risk of breaking production more than they already can.
It will also allow your CI/CD tool to deploy both at once.
And since there's a strict mapping of app IaC and app itself in each GitHub branch, this actually lets you very easily create feature branch environments (i.e. a developer creates a new branch, your CI/CD picks it up, and automatically deploys it to a kube environment).


Coding conventions for docker-compose files? Is there some globally admitted coding conventions for docker-compose files?
Below is a sample docker-compose.yml from a small project we have here:

version: 3
networks:
  net:
    driver: 'bridge'

services:
  app:
    build: .
    networks:
    - net
    ports:
    - 3000:3000
  redis:
    image: redis:alpine
    networks:
    - net
  client:
    image: "app/test_clients"
    deploy:
      replicas: '3'

The quoting is especially inconsistent. Sometimes quotes are used around text and numerical literals. Sometimes they aren't. Sometimes single quotes are used, sometimes double quotes are used instead. The various example in the official docker-compose documentation seems relatively inconsistent in that area too.

This is valid YAML. And YAML is very permissive about quoting and, as far as I know, quotes could simply be omitted in most of the cases. I know this is probably a matter of internal policy but I wonder if there are some commonly accepted "best practices" in that matter.

One could use Composerize to create a consistent docker-compose.yml.

There seems to be two options. Options one is to enter some docker run command, e.g. docker run app/test_clients on their website, which will return:

version: '3.3'
services:
    test_clients:
        image: app/test_clients

Or use the command line: composerize docker run app/test_clients.


Change docker container’s log PATH Team:
All my container logs are under /var/log/containers..
Where in kubelet or docker config can I specify this path such that logs are instead created under new path?
Ex : /shared ? Instead of /var/log

I got a work around to this case. basically, My root volume is 100GB and I created a separate partition out of it as 70GB and I set it to a new device sda3 and mounted it to /var/lib.
now, all docker container logs are written in same path but in backend they are mounted to a different drive. with this setup, even if there is disk pressure, the node does not become unavailable.
so my drive now looks liek below
sda1
sda2 / 30GB
sda3 /var/lib 70GB.  << new device


yarn server with docker-compose So I am trying to start up an server that has:

a mongodb container
a backend container
a client container

Both server containers require me to run yarn start at the end. However, when I run this in the dockerfile the image never stops being built as the yarn start command technically never stops running.
The client requires the backend to be up already, and the backend requires the mongodb db instance to be up.
How do I get this server running in docker compose?
Here is my backend dockerfile
 FROM node:12-alpine

 WORKDIR /app
 COPY ./server /app

 WORKDIR /app/server
 RUN yarn
 RUN yarn start

It really does work until I get to yarn start (Which I should probably run in the docker-compose as it needs to mongodb container to be running. I just don't know how to run this in the docker-compose.yml, or force the client to wait for this command to be run before running its own yarn start).

yarn start should be the CMD of your docker image (I.e what is run when you start a container) and not a RUN command (which is run when creating the image used to satrt a container later)
From your exemple, something like this should be better:
 FROM node:12-alpine

 WORKDIR /app
 COPY ./server /app

 WORKDIR /app/server
 RUN yarn
 CMD yarn start


what environment does a build "RUN" run in? When docker build does a RUN command, what "environment" is it running in?

Explaining the question, does it run at build time, or when the container starts?  Is it using the operating system mentioned in the FROM?

In the Dockerfile, the RUN step is performed during the docker build process. The operating system is shared with the host (containers != VMs). The filesystem and rest of the configuration used to define the temporary container is based on the state of the image resulting from the previous step. Each step in the Dockerfile extends the image with additional metadata and filesystem layers.

The RUN step itself is performed inside a temporary container. The output of the RUN step is any filesystem changes that exist in the container at the time the command exits successfully. Changes to the shell (like a cd command or setting an environment variable inside the shell), spawned processes, and changes to files inside a Volume are all discarded. The changes to the container filesystem are captured as a filesystem layer that is included in the resulting image.


availability calculation of a azure service fabric stateful application I have looked at the question and answer in How do you calculate the compound Service Level Agreement (SLA) for cloud services?

I have a service fabric application which has a stateful service which stores the data in the disks of the VMs themselves.
Service fabric relies on underlying Virtual machine scale set's availability, which is 99.95 %
The azure storage VHD has a SLA of 99%

Going by the answers in the question above, is my applications availability 99.95 % * 99%  = 98.9505 % ?

This seems a little strange that i use a stateful service in azure service fabric which is supposed to be a distributed service mesh framework and the availability of the services built is so low.

I am sure I am missing something. Can anyone tell me if my calculation is correct ?
Do I need to consider the number of replicas in my calculations in which case I have as many VHDs as replicas ??

Virtual Machine Scale Sets (VMSSs) don't have a SLA of their own:

  Virtual Machine Scale Sets is a free service, therefore, it does not have a financially backed SLA itself. However, if the Virtual Machine Scale Sets includes Virtual Machines in at least 2 Fault Domains, the availability of the underlying Virtual Machines SLA for two or more instances applies. If the scale set contains a single Virtual Machine, the availability for a Single Instance Virtual Machine applies. See the Virtual Machines SLA for more details.

The above statement points to the fact that VMSSs are just an orchestration layer over the top of Virtual Machines and as you quite rightly pointed out results in an availability for a VMSS of 99.95%.

In terms of the storage calculation, there are several parameters here:

Whether you are using Hot or Cold storage, the availability for Hot storage is 10 times that of cold storage.
Whether your application is read or write heavy, what you need to consider here is that there are separate SLAs for reading and writing.

Let's make some assumptions:

Storage is, in general, cheep.  Therefore, only ever use the Hot Storage Tier.
Managed Disks are, in general, easier to use.  Therefore, you are limited to only using Locally Redundant Storage.

With the above two assumptions your storage SLA is 99.9%.

Multiplying these two figures together you get, 99.985%, this is still kind of low, but higher than your 98% figure above.  You can increase this number by hosting Service Fabric across multiple regions:

This would give you system availability close to four nines, i.e. 99.9898%


Multiple runners for all projects OR single runner per project What do we have:
We have a Gitlab EE with multiple projects running on a droplet, next to that we have a runner server for the CI runners. All projects are Docker projects and 4 runners can be busy at the same time.

What we need to know:
Now is my question, is it better to have like 4 runners on the runner server for all the projects OR is it better to have 1 runner per project?

I know when the CI gets stuck, the runner will be stuck too and it will mean that it is missing 1 runner till it gets reset. So at that point single runner per projects seems better, but it will make a large list of runners which perhaps use some idling resources from the server ?

See here nothing is running or doing work:

Anybody can tell me what is better in your opinion and why it is better in your opinion as both ways seem legit in my opinion and i want to know others experience.

After some research i found out that shared runners are good for the projects that are very much the same, and specific runners are better for the projects that are not like any project.

Shared runners can eat all resources available in the end with its queue's which makes specific runners more interesting.

My choice now is more like a little bit of both, so shared for all small and likewise projects, and specific for the large new projects as the documentation recommends.

From Shared vs specific Runners:

  ... The main differences between a shared and a specific Runner are:

  Shared Runners are useful for jobs that have similar requirements, between multiple projects. Rather than having multiple
  Runners idling for many projects, you can have a single or a small
  number of Runners that handle multiple projects. This makes it easier
  to maintain and update them. Shared Runners process jobs using a fair
  usage queue. In contrast to specific Runners that use a FIFO
  queue, this prevents cases where projects create hundreds of jobs
  which can lead to eating all available shared Runners resources.
  Specific Runners are useful for jobs that have special requirements or for projects with a specific demand. If a job has
  certain requirements, you can set up the specific Runner with this in
  mind, while not having to do this for all Runners. For example, if you
  want to deploy a certain project, you can setup a specific Runner to
  have the right credentials for this. The usage of tags may be
  useful in this case. Specific Runners process jobs using a FIFO
  queue.

  A Runner that is specific only runs for the specified project(s). A
  shared Runner can run jobs for every project that has enabled the
  option Allow shared Runners under Settings ➔ CI/CD.

  Projects with high demand of CI activity can also benefit from using
  specific Runners. By having dedicated Runners you are guaranteed that
  the Runner is not being held up by another project's jobs.

  You can set up a specific Runner to be used by multiple projects. The
  difference with a shared Runner is that you have to enable each
  project explicitly for the Runner to be able to run its jobs.

  Specific Runners do not get shared with forked projects automatically.
  A fork does copy the CI settings (jobs, allow shared, etc) of the
  cloned repository.

Still i would like to know everyone's mindset about this, as im still interested about your way of doing it!


How to report Code Coverage metrics across git repositories? We are using private git repositories in Azure DevOps and I have a requirement to create code coverage report across git repositories. Our build CI/CD pipelines are using both TeamCity & Azure DevOps.
I have looked at options and there is no straightforward solution to this problem. I am thinking about the following options:

using service like coveralls.io, there is an option to add private repositories; the drawback it is subscription based service
using TeamCity SonarQube plugin; It is not ideal and limited to the plugin features; I haven't tried it yet.
using custom script to query and aggregate existing code coverage metrics of last successful build in TeamCity/Azure DevOps across projects and group them by repository; Both Azure DevOps & TeamCity offer REST APIs that can be used to get the data.

I still need a solution for Azure DevOps builds, however I have to admit I have not researched this properly yet.

Has anyone done cross repository code coverage reporting and what approach/tools might be best?

If you use python and GitLab like I do, you can include a test coverage report in your CI/CD pipeline. You pip install coverage and then just run coverage in your pipeline. The following is an excerpt from an AWS lambda pipeline, but you should be able to find something similar for your environment:

 script:
    - pip install -r ./awslambda/requirements.txt
    - coverage run -m unittest discover -v -s "./awslambda" -p "*_test.py"
    - coverage report

The above generates a nice code coverage report and it is free of charge, for Python, AWS and GitLab.


ansible tower_project module, always getting "ansible-tower-cli required for this module" I have a control host that has the following:

ansible --version = 2.5.2
tower-cli --version == 3.3.0

tower-cli is able to connect to a remote Ansible Tower instance (note, Ansible tower itself is not installed on this control host).  Can run 'tower-cli user list' on control host and works as expected.

However, when I try to use a playbook with the tower_* modules, such as:

  tasks:
  - name: create the SCM project
    tower_project:
      name: "test project"
      state: present
      ...

Always getting the following:

  ansible-tower-cli required for this module

Any ideas, as I can use tower-cli just fine on this localhost that is running the playbook.

Found the origin of the issue, and unless someone else had run into it they wouldn't have known where to look --
check details of tower-cli --version and ansible --version, as well as which tower-cli which ansible;  in my case were installed in different python environments:

/usr/lib/python2.6/site-packages/ansible
/usr/lib/python3.4/site-packages/tower_cli

If you cannot see the python version using the above for tower-cli, you can run:
$ head -n1 `which tower-cli`
#!/usr/local/opt/python/bin/python3.7
$

The ansible modules couldn't see the tower_cli python modules as they were in a different python environment.
After-the-fact problem origin:  I had installed ansible via yum (package manager), and ansible-tower-cli via pip (python installer).  Being consistent in how you install both to make sure they go to the same environment resolves the issue.
Centos/RHEL: yum install ansible ansible-tower-cli
-OR-
Python:  pip install ansible
Python:  pip install ansible-tower-cli


External access to internet from Kubernetes Pods I'm having troubles with network access from pods. I had installed new Kubernetes cluster with flannel-network plugin,
Checking the health of server - good.

I tried to check network via busybox: kubectl run -it --rm --restart=Never busybox -n microservices --image=busybox sh, then ping 8.8.8.8 results in drop.

But ping 8.8.8.8 succeeded when executed from kubectl exec -it kube-flannel-ds-amd64-xxxx -n kube-system (checking network inside flannel pod).

What can it be?
I think it can be something with flannel or docker network configuration. But i did not find information about it.

I tried to change iptables configs, reinit cluster, create and changing docker network interfaces.

I have no more ideas, what can I do?

P.S. There is the same server installed before me and their settings and installation procedure are identical. And their connection is good.

I found the solution ...

Check your network interfaces (ifconfig -a cni0).
Good config is:

  cni0: flags=4163  mtu 1450
          inet 10.244.0.1  netmask 255.255.255.0  broadcast 0.0.0.0...

That subnet (10.244.0.0) configured in flannel-deploy.yaml

But, by default, a new server may have:

  cni0: flags=4163  mtu 1450
          inet 192.168.0.1  netmask 255.255.255.0  broadcast 0.0.0.0...

Change it to:

ifconfig eth0 10.244.0.1 netmask 255.255.255.0 broadcast 0.0.0.0


Ansible how to know what return value is provided by a specific module From Ansible's official doc

  Ansible modules normally return a data structure that can be registered into a variable, or seen directly when output by the ansible program

and

  Ansible modules normally return a data structure that can be registered into a variable, or seen directly when output by the ansible program

However, the document on aws_ec2 module

Doesn't document return values
Has this example, which seems like it actually has some specific return values

# Stripped version

# Launch instances, runs some tasks
# and then terminate them

- name: Create a sandbox instance
  ... # hosts, vars...
  tasks:
    - name: Launch instance
      ec2:
      ... # ec2 module specs
      register: ec2 #This seems to be one of the specific return structure by this module

    - name: Wait for SSH to come up
      ... # some module
      with_items: "{{ ec2.instances }}" # This is where the registered return value is used

Question: How to see all possible return values of a module, even without documentation

if you do something like:

- name: Launch instance
  ec2:
  register: data_struct

You can then output all the values returned by doing this below the above code:

- name: print all returned values
  debug:
    msg: "{{ data_struct }}"