Running the docker daemon without root Inspired by What are best and comprehensive practices to consider when running docker in production? , I stumbled over  Why we don't let non-root users run Docker...

They came up with  docker run -ti --privileged -v /:/host fedora chroot /host , which puts your unprivileged (but docker-grouped) user in a root shell on the host (not in the container).

No need to mention, that is rather bad news for production systems.

What can be done to avoid this or similar things? I assume the docker daemon cannot be run as a non-root user (or else that would likely be the default way to start it)? One solution that comes to mind is not putting unprivileged users in the dockergroup and only allowing specific docker command lines via sudoers. But is that enough?

Do we know how hard it is to break out of the container and gain a host root shell like in the example given, if the image is ran as usual (i.e. docker run --rm -it myimage)? I am not even talking about some attack via the service running in the container, but by, say, a specially prepared image.

EDIT: to clarify, I am specifically interested in this attack scenario:

Untrusted software developer delivers a ready-to-run image.
Trusted user runs that image on a production system in a normal fashion (i.e., docker --rm -it myimage), without --privileged.

Possibly with sub-scenarios of -u unprivileged_u or not.

Does the untrusted person that created the image have a reasonable chance to break out of the docker container runtime? I.e., more than with, say, classical chroot or classical permission elevation?

EDIT2: My research shows only a single real breakout incident which was fixed with Docker Engine 0.12, and then again in 1.0.1. Also, all things I found that are critical about this mention the erstwhile TCP/IP interface of the docker daemon, which was changed to the Unix socket for a long time now. Is there anything else, or would you say that there is no problem as long as the person typing/running the actual docker command is good?

Compared to just adding the user to the docker group the second links solution is not any more secure.

Note how that page still can launch with the --privileged flag, and that it is running unconfined.

unconfined_u:unconfined_r:unconfined_t

This means that the container can access all resources including the hosts disks and hardware.  It is security through obscurity, which is not security.  Anyone who knows how to use mknod or walk the /sys and /proc trees could easily compromise the host and all containers with zero logging.

The reality is that docker shifts both complexity and the security boundaries.  All users who can launch container or hit the API need to be restricted to trusted users in that security context.

--privileged disables all apparmor and selinux policies which is actually far less secure than a native package.

Namespaces are not a security function and depend on apparmor and selinux to enforce reasonable constraints.

Docker notes this on this page.

https://docs.docker.com/engine/security/security/#control-groups

'First of all, only trusted users should be allowed to control your Docker daemon.'

The security functions of docker/kube are not administrative boundaries like classical unix permissions, they are tools to prevent non-privileged containers from breaking out.

In a docker world the administrative boundary becomes the host, and the selection and segmentation required to isolate applications or users within that context needs to be applied at that boundary.

The benefits of this shift in complexity and responsibility generally outweigh the risks if implemented with those changes in mind.

TLDR

Docker API user == Sudo ALL user

Running a container with the --privlaged flag == running a web service as suid or as the root user.

Edited per the OP's request for additional information

The referenced issue with breakout int he OP's edit was an non uid0 privilege escalation.

Unfortunately, due to the need to perform root only actions Docker needs to enable some capabilities so that apt/dnf can install packages etc...

This need does pose a risk if production workloads are run in this default configuration and one should adopt the security principle of least privilege for production workloads.

--privileged disables apparmor/selinux and opens up capabilities

I am using ubuntu but it may be useful to work through the following steps. First start a default container with docker run -i --rm -t debian bash

From the parent host find the PID for bash using ps and note that the process is owned root.  If you look in /proc/$PID/status you will see the contexts it is running under.

# egrep '^Cap(Prm|Inh|Eff)' /proc/16026/status
 CapInh:    00000000a80425fb
 CapPrm:    00000000a80425fb
 CapEff:    00000000a80425fb

You will want to refer to man 7 man capabilities and /usr/include/linux/capability.h for better info but a summer is

CapInh = The Inherited capabilities (what docker provided)
CapPrm = The capabilities due to permissions (inside the container OS)
CapEff = The effective capabilities

You can decode these to human readable form by running:

$ capsh --decode=00000000a80425fb

Now do the same with $ docker run -i --rm --privileged -t debian bash and you will find that the effective capabilities are 0000003fffffffff

Also just do a dir /dev in both VM's and you will see just how much access a privileged container has.

By looking at /etc/apparmor.d/docker for apparmor systems or the lables in SElinux you will see the implications.

Going back to the principle of least privileges, I would ensure that my docker container is running process as a unprivileged user and with as few enabled capabilities as possible.

As an example, you can test this by running it first by dropping all caps.

$ docker run -i --rm -t --cap-drop=all -t debian bash

This can be validated through the /proc/$PID/status method above.

CapInh: 0000000000000000
CapPrm: 0000000000000000
CapEff: 0000000000000000

Note it is all zeros, from CapInh.  Then if I had to enable features for the application I would use --cap-add after the --cap-drop=all

Example, --cap-drop=all will break ping:

# ping -c 1 www.google.com
ping: Lacking privilege for raw socket.

So we can add the cap_net_raw cap. Docker expects the arguments with the cap_ prefix removed so the command is docker run -i --rm -t --cap-drop=all --cap-add=net_raw -t debian bash

# ping -c 1 www.google.com
PING www.google.com (216.58.216.164): 56 data bytes
64 bytes from 216.58.216.164: icmp_seq=0 ttl=54 time=4.779 ms

But be very careful adding them as an example.

cap_sys_module will allow a container to add or remove kernel modules from the parent host.

cap_sys_rawio will open memory and all block devices to attack

cap_sys_admin is super dangerous.

So in this case I would see if you can make things run in this context.

$ docker run -i --rm -t --cap-drop=all -u nobody:nogroup -t debian bash

Hopefully the robustness of apparmor and selinux profiles improves over time, but if that is not enough for your security needs you can look in there too.

Really, avoiding the --privileged flag and using principle of least privilege will make the most impact. Especially if you take advantage of the ephemeral nature of containers to keep packages up to date.

If you need more Red Hat covers some basic seccomp options here.

https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux_atomic_host/7/html/container_security_guide/linux_capabilities_and_seccomp


Zero Downtime Deployment with Database Backward Compatibility Question

Does anyone have experience with Database Backward Compatibility to achieve Zero Downtime Deployment? Are there good links or references that I can read?

Reason

I am working with a monolithic application where the code base is tightly coupled with the database. Because of this, when we want to do deployment (weekly), we have to work after business hours.

Goal

What I am trying to achieve is the ability to implement blue-green deployment where the old code is compatible with the new version of the database - and vice versa.

There is plenty of information about this. The basic idea goes like so -

Create a script that does the database migration, but make sure that it only adds tables and fields - never removes, renames, or changes the type of existing tables or fields.

The new version of the application will start using the new tables/fields when it is an existing field from the previous version that needed a change - the app must continue the previous version as it used to, and start using the new version as well.

In some cases, before the newly created field can be used it needs to be populated based on the previous field - the application can do this when it first starts.

If some functionality depends on the new field, and it is not yet ready, opening it to users can be hidden behind a feature flag that is toggled when ready to expose the new functionality.

When yet another version is released, the application can finally stop writing to the old field you were using and start using the new field exclusively.

The above mostly explains what happens when fields need to be renamed or type changed. For new tables or fields, it is much simpler since the existing version is not using these.


How do I create separate versions of lambda functions through terraform? I'm Running Terraform v0.11.14

resource "aws_lambda_function" "test_lambda" {
  filename         = "lambda_function.zip"
  function_name    = "test_lambda"
  role             = "${aws_iam_role.iam_for_lambda_tf.arn}"
  handler          = "index.handler"
  source_code_hash = "${data.archive_file.lambda_zip.output_base64sha256}"
  runtime          = "python2.7"
  version          = "1"
}

On plan I get the error
Error: aws_lambda_function.test_lambda: "version": this field cannot be set

I couldn't find an aws_lambda_function_version resorce so presume you create versions using the lambda fuction resorce and specifying the version.

I'm not sure what value I give to version to build versions of my lambda.

Im tring to get it to work for one version for now as a first step. I plan on adding count to create multiple versions as my next step if I can get it to create one version first?

I just needed to set

publish = true

I then enconter another issue for creating seprat versions but finding this answers my initial question


Dockerfile base image I have a Python application that I want to containerize with Docker. Should I use FROM python:3 or FROM python:3.7?

Does the python:3 tag automatically use the latest minor version of Python? Would there be a practical reason to pin to version 3.7 explicitly?

I could not find a clear answer to these questions in the official documentation.

Generally, when referencing 3rd party packages (Docker base images, Maven dependencies, Ruby Gems etc.) it is worthwhile to be as specific as possible. You want to avoid randomly upgrading dependencies at a later time, just because there's a newer version out there, just because you only specified a prefix of the actual 3rd party release number.

I personally would do whatever it takes to make sure that I get the same upstream version on every build, forever, and that upgrading always is an activity I have to d, that is never done for me. For example, in Ruby, the bundler keeps specific versions in the Gemfile.lock. While the Gemfile can specifiy vague versions (like "~> 3.1" giving you anything ">= 3.1 and < 4.0"), whenever a version is picked by the automatic resolving, the actual immutable version is fixed in the Gemfile.lock and will never change on its own.

The reasoning is very simple - we use tools like Docker specifically to get out of the upgrade/version hell we had in the past; to make sure that we have the same versions on all environments, and to make changes visible (in the form of git commits etc.). Introducing the same all over, by having "floating" upstream version numbers, would really be a shame.

There is really nothing elaborate like that Ruby Gemfile.lock, for Docker base images, so in this case you have to look at the images you are pulling; they should document which tags are considered permanent.  Then just specify those. And stay clear of latest, obviously.

For your particular example, to pick at random, a version like 3.6.6-alpine3.7 seems to be pretty fully specified. I'd expect not much more than bugfixes from upstream. But if I were especially paranoid on that day, I might specificy a @digest for the FROM (I'm not that paranoid normally...).


TeamCity - CI w/ Multi-project .NET solution using SOA build and delivery issues I have a multi-project .NET solution that I'm trying to set-up with TeamCity (TC) for CI and Octopus Deploy (OD) for CD.

I originally set-up TC to pull the latest from SVC, build the solution and deploy it as a .zip on the Dev server, but I've been having issues trying to deploy the solution to IIS with OP. I was informed that I should instead have TC take the solution and create multiple packages from it for OP to pick-up. I currently don't know how to do this.

I have the following build steps:

And I have this in my General Settings Publish Artifact settings:

Visual Studio solution set-up (Blue shows the projects that make up the entire application we want to deploy. Green boxed will be the separate packages, BLL, Services and Repository because of project dependency get rolled up into the API project.):

I have manually deployed the solution in IIS, below is how it needs to deploy in IIS:

The root site is a .NET project using MVC that communicates, hubs is a Signal-R project and API has the repository project and services project under it. So essentially The root website is a client, it communicates can only see API and hubs and any communication to the repository project would go through the API project first, it's supposed to be using a service-orientated architecture.

The goal is that TC should spit out 3 packages, one for the root website, one for the Signal-R project and one for the API which holds several projects but all projects are entirely in a single solution and this isn't something that I can get around for the time being. Once TC can do that I will need to set-up OD to deploy each of those packages to their required locations (which should hopefully be pretty simple).

Any information/assistance with this is greatly appreciated as I do not have a Dev-Ops background and this is the first time I've had to set-up this kind of a project with TeamCity.

Kind regards.

So you need to create packages for each of the three sites so that you can deploy each one individually.

The first step is to create a .nuspec file for each of the IIS websites you want to package:

Install OctoPack and read the entire article
Create three empty text files next to the .csproj, give it the same base filename as the .csproj, thus if it is MyApp.API.csproj you want to create an empty text file called MyApp.API.nuspec.
In each of these .nuspec files paste in the following template:

<?xml version="1.0"?>
<package xmlns="http://schemas.microsoft.com/packaging/2010/07/nuspec.xsd">
<metadata>
    <id>Sample.Web</id>
    <title>Your Web Application</title>
    <version>1.0.0</version>
    <authors>Your name</authors>
    <owners>Your name</owners>
    <licenseUrl>http://yourcompany.com</licenseUrl>
    <projectUrl>http://yourcompany.com</projectUrl>
    <requireLicenseAcceptance>false</requireLicenseAcceptance>
    <description>A sample project</description>
    <releaseNotes>This release contains the following changes...</releaseNotes>
</metadata>
</package>

Customize the template for each of your projects.
Update TeamCity to add /p:RunOctoPack=true to the msbuild command line
Publish your packages with the following msbuild arguments:

/p:OctoPackPublishPackageToHttp=http://your.octopusserver.com/nuget/packages
/p:OctoPackPublishApiKey=API-ABCDEFGMYAPIKEY

You may have to play around a bit, but essentially the above is taken from this article so treat that as your bible.


What package managers are supported on Microsoft hosted agents with CMake enabled? Here, I can read that it is possible to setup a CMake build on a Microsoft hosted vsts agent. What C++ package managers (e.g. Conan) are supported for these agents?

It would be great if at least one of these package managers is supported: https://github.com/nlohmann/json#package-managers

According to this page, the hosted agent supports NuGet, npm, and Maven.

It looks like it might also support pip.
You could install cget using pip and then use cget to install your package. A bit of a roundabout way, but it might do what you need.

"C:\Program Files\Python36\Scripts\pip" install cget
cget install nlohmann/json

I don't have a way to test this right now, but I will try some real examples and edit tomorrow if I have time.

Since it is a CMake/Git project, you could also just clone the repository (or add it as a submodule) and build it every time. Since I often use libraries that aren't managed, I tend to take this route and leverage the caching that many CI platforms offer to keep my build times low. Sadly, it looks like VSTS hasn't gotten to this yet.


AWS CloudWatch Logs Publisher plugin hangs on job completion We use the AWS CloudWatch Logs Publisher plugin on our Jenkins and most of the time the logs are sent within 2-5 seconds of the request being sent out. However, sometimes the step will hang for up to 5-10 minutes, with no errors in the console or in Jenkins' log. The job hangs with the following status message;

  AWS CloudWatch Logs Publisher is waiting for a checkpoint on [...]

It always completes successfully, even with the delay.

Has anyone seen this before? Is there anything I can do to prevent the step hanging?

Googling returns only this Q&A:

It looks like that it is not a reported issue. Also combined with your remark, i.e.:

  It always completes successfully, even with the delay.

I wonder whether it is really an issue as I have seen a lot of logs and also weird messages while everything worked. What I do in such situation is talk with the developer of the code. In this case I would create an issue in the Jenkins bug tracker and discuss whether this is an issue and whether the developers should change the code that triggers this message.


How do you fetch a Jenkins job's artifacts from a remote system? I'd like to download an artifact file from Jenkins CI (e.g. file from the latest successful job) from a remote system. How can I do it? Do I need a plugin to do this?

The public URLs of artifacts can be found in Workspace of the job. It is advised to use view version of URL, so it is more consistent.

For archived files, Archived Artifact Url Viewer can provide URL to a file inside a zip or jar archive within the artifact folder of a build


How can I achieve better load balancing of different systems on a Jenkins? We have four test machines spread over a Jenkins.

We would like to have a better distribution. Example: If machine one is running with a test job, the second job should detect that node one is running and directly take the next free instance to start another job there.

I have not found a suitable plugin, instructions on this topic, which represents exactly this sufficient.

The default behavior of Jenkins is to use the same node over and over unless it's unavailable. If you use labels you can target resources.

Take a look at plugins like Least Load and Scoring Load Balancer to help distribute the load. CloudBees also provides an Even Scheduler plugin as a commercial offering.

Kubernetes (e.g. Kubernetes Plugin) to schedule ephemeral agents could also satisfy your distribution needs.


Command returns 0.0.0.0 for IP but only for leader I have a 3 node docker swarm cluster and I am trying to get the IPs of each one via docker node inspect node-1 --format '{{ .Status.Addr  }}' but I am having an issue where when the node I am trying for (example node-1 from above), that I get back 0.0.0.0. This only happens on the leader node, the others work just fine.

What is happening in this case? Also, how can I use docker commands to get the IP of all the nodes when checking with their hostnames or docker node ls?

I ended up using {{.ManagerStatus.Addr}} and stripping the port number from the results with sed.

for NODE in $(docker  node ls -f 'role=manager' --format '{{.Hostname}}'); do
  docker node inspect --format '{{.ManagerStatus.Addr}}' ${NODE} | sed 's/:2377//'
done


How to add multiple jvm options in a saltstack jinja? I have a config.jinja file with below jvm options:

`-Dappdynamics.agent.appName=AP_{{ pillar['region'] }}`

However the AP_{{ pillar['region'] }} varies for different environments. For Dev/Prod it can be AP_{{ pillar['region'] }} and for QA it must be AP without the pillar value.
How can I define the appName option with multiple values to achieve above? Any suggestions?

if (pillar['region'] == QA)
   -Dappdynamics.agent.applicationName=ALIP
else
   -Dappdynamics.agent.applicationName=ALIP_{{ pillar['region'] }}
end-if

Perhaps a macro might be what you are looking for.

Understanding Jinja:Macros

In the lib.sls example in the link (pasted below), it sets a value contingent on the value of the if statement.  I think a similar approach could work for you.

{% macro pythonpkg(pkg) -%}
  {%- if grains['os'] == 'FreeBSD' -%}
    py27-{{ pkg }}
  {%- elif grains['os'] == 'Debian' -%}
    python-{{ pkg }}
  {%- endif -%}
{%- endmacro %}


How do I set a base domain for gitlab's auto devops? Docs seem to point to https://docs.gitlab.com/ee/user/project/clusters/index.html#base-domain
However it doesn't seem to specify how to set the base domain strangely enough.
Can't seem to find info no this via a web search, I've also looked at https://docs.gitlab.com/omnibus/settings/configuration.html with no luck.

https://docs.gitlab.com/ee/topics/autodevops/quick_start_guide.html#install-ingress

Go back to the cluster page on GitLab, and go to the Details tab. Add
your Base domain. For this guide, use the domain .nip.io.
Click Save changes.

I think this option only showed up AFTER I installed the ingress.


How to examine Jinja output in Saltstack? I have a templated SLS in Salt I'm trying to build, but it's emitting invalid syntax, which is resulting in errors such as:

my-minion-id:
    - State 'system' in SLS 'network' is not formed as a list

In principle, it should be possible to, somehow examine the output of the Jinja template before it attempts to parse the output as an SLS file. There exists a Python module for the Jinja renderersalt.renderers.jinja, but if I attempt to execute it on the CLI, I get an error:

# salt my-minion-id salt.renderers.jinja.render /srv/salt/network/init.sls
my-minion-id:
    'salt.renderers.jinja.render' is not available.
ERROR: Minions returned with non-zero exit code
# salt my-minion-id renderers.jinja.render /srv/salt/network/init.sls
my-minion-id:
    'renderers.jinja.render' is not available.
ERROR: Minions returned with non-zero exit code

How can I see the output of my template? It seems absurd it should be this difficult to debug.

Check out the slsutil.renderer module.

This should do what you want

salt my-minion-id slsutil.renderer /srv/salt/network/init.sls 'jinja'

This module just calls the compile_template function directly for you.

Edit: /srv/salt/network/init.sls is the path on the minion, if you are not targeting the master as your minion, you will probably need to do the following.

salt minion-id cp.cache_file salt://network/init.sls
salt minion-id slsutil.renderer /var/cache/salt/minion/files/base/network/init.sls

or point to whatever file that cache_file spits out.

If you are on 2018.3 or newer, you can just specify salt://network/init.sls


How to setup priorities between pipelines that might be triggered simultaneously on Azure DevOps pipelines I am using Azure devOps to build and deploy my application, and within that application I have a project that is being used by other related applications (let's call it project N). Thus I decided to deploy this project as a NuGet package, and consume it during the build.

Project N is likely to change often, thus I am having some trouble in defining a reliable setup that:

Creates the NuGet package only when there are changes in Project N and its dependencies
Guarantees that, if the NuGet package is to be created, as per 1), the pipeline does that before anything else

The way I have setup this so far, is to have two different pipelines, one that creates the NuGet package and deploys it (NuGet Pipeline), and another one that consumes that NuGet package and then builds and deploys the application (Main Pipeline).

The problem is, when changes happen both in files related with Project N, and in some of the remaining files of the solution, I don't know how to make a rule that says "Run NuGet Pipeline before the Main Pipeline", so for the moment I have to trust in luck for that to happen, and if not, I would have to manually trigger the Main Pipeline after the two build are finished. Because I want to be sure that the new changes introduced in the Project N do not break the Solution.

Is there an elegant way to do that? I could setup a trigger that, whenever the Pipeline NuGet finishes it would trigger the Main pipeline, but by doing it that way I would trigger the Main Pipeline twice whenever changes would be made to the Project N and to the rest of the solution.

It sounds like both of these pipelines are being triggered by the same source commit(s). If that's the case, you should be able to use CI trigger source path filters so that the builds only happen on commits to the relevant code.
If a developer does a single commit that is committing changes to both the main application and the package library (which would still trigger both pipelines regardless of the filter), you should consider promoting better code commit practices. Commits should be generally atomic to an appropriate context (i.e. don't commit to a stand-alone library and an app that consumes it together). Commit [and push if Git] the package changes first so they'll build and publish, then commit [and push] the main application. This will ensure that the trigger filters do what they should.
I don't know of a feature in ADO that allows you to serialize the pipeline execution order in the way you would like. As you suggested, the alternative of triggering one pipeline from another would kind of work but ultimately doesn't solve the core problem.
Alternatively, considering that "project N" changes a lot, and if it has that tight a relationship with the main project. Perhaps consuming it as a package is not appropriate. If the library is also to be consumed by other applications that have a more loose relationship, perhaps consume the dependency directly as code by the main project, but make its package publishing a part of the main application build instead of its own pipeline. (I've used this model for API applications that publish a data contracts library. The API application itself consumes the project directly as a project reference since the two are very tightly related, but others consume as a published package.)


Dockerfile parse error line 5: unknown instruction: -ESSENTIAL this is my dockerfile ;

FROM ubuntu:16.04
MAINTAINER James Turnbull "james@example.com"
ENV REFRESHED_AT 2016-06-01
RUN apt-get update -yqq; apt-get -yqq install ruby ruby-dev build
  -essential redis-tools

RUN gem install --no-rdoc --no-ri sinatra json redis
RUN mkdir -p /opt/webapp
EXPOSE 4567

CMD [ "/opt/webapp/bin/webapp" ]

and this is how i ran it :

  docker build -t jamtur01/sinatra .

however i get this error in regard to  -essential

  Dockerfile parse error line 5: unknown instruction: -ESSENTIAL

i am not clear what i have done wrong. is there an equivalent for -essential

I assume you have a copy-paste issue. The right package name is build-essential. And if you want to split a RUN statement over several lines, you will have to use backslash for line continuation.

All that leading to something along the lines of:

RUN apt-get update -yqq; apt-get -yqq install ruby ruby-dev \
          build-essential redis-tools

As a personal suggestion, when you have several commands to run, it is usually better to join them using && rather than ; since you want the build to fail if either command is failing.

RUN apt-get update -yqq && apt-get -yqq install ruby ruby-dev \
          build-essential redis-tools


Install apache on awa ec2 from locally installed ansible server on my laptop I would like to install apache2 remotely on aws ec2 instance from my locally installed ansible server on my laptop. I have also the. pem key of the instance. Now what Ip need to put in /etc/ansible/hosts file inorder to connect ec2 instance from my laptop?

You need to put public ip address or hostname assigned by aws ec2 service, if your aws ec2 instance is public.

If your aws ec2 instance is located in vpc, you won't be able to connect it directly. Either you need to setup routing through internet gateway or you have to setup ansible server in vpc.


How to pass the version to docker-compose? When I would like to update an app at the moment, docker-compose down, the version in the docker-compose.yaml is changed, subsequently the app is started and the latest version is running.

Now I would like to update it automatically. It it possible to update the version by passing a variable?

Workaround

At the moment I use sed to update the docker-compose file, but I wonder whether there is an easier approach.

Another way to execute the @golfNintyNine answer is putting inside docker-compose.yml the environment variable:

serviceName:
image: "imageName:${VERSION}"

And then executing simply:

$ VERSION=2 docker-compose up

Also you can put all together inside script file like this:

#!/bin/bash

function usage {
    cat << EOF >&2
Version argument is required, please use -v or --version.
EOF
    exit 1
}

OPTS=`getopt -o v: -l version: -- "$@"`

if [ $? != 0 ] ; then usage ; fi

eval set -- "$OPTS"

while true; do
    case "$1" in
        -v | --version ) export VERSION="$2"; shift 2;;
        -- ) shift; break ;;
    esac
done

if [ -z $VERSION ]
then
    usage
fi

docker-compose up


How to incorporate containers into a standardized dev environment? I'm currently working at a small company where we've had a lot of issues being on the same page with entity framework migrations and local databases. I want to start incorporating containers in hopes of standardizing everyone's development environment. Ideally, I'd like a disposable database that can be modeled using Entity Framework code first, seed data into it, then disposed of once the testing is done. What resources are there to help get me started and what other things should I know about standardizing the development environment.

Docker is an awesome tool for creating a database, seeding data into it, and disposing of it.

For an example check out Microsoft's SQL Server with Docker.  The following would pull and run a sql server 2017 instance: docker run -e "ACCEPT_EULA=Y" -e "SA_PASSWORD="
   -p 1433:1433 --name sql1
   -d mcr.microsoft.com/mssql/server:2017-latestAfter running this you now have a developer ready SQL instance that you can then connect to SQL Management Studio (or any other GUI management tool), or you can even connect via the command line.

If you are not using SQL you can run a container for the desired database type (check out the options on Dockerhub).

Once you have your database up and running, run the scripts to set it up, seed the data, and test without worrying about messing up your production database.


running a phpUnit test within a docker container i want to run a specific version of phpUnit WITHIN a docker container. This container will use a specific version of php. i.e

php:5.6-apache

Its a lareval application. i have install phpunit via composer on the hostfiles and then used the volume command to transfer this to the container.
my composer.json file has following entry:

  "require-dev": { "phpunit/phpunit": "^5.0" }

This is my docker run command to run the test on my testdev container:

docker run --rm -it -v ~/Users/mow/Documents/devFolder/testdev:/app testdev_php "php ./vendor/bin/phpunit"

This returns the error:

exec: fatal: unable to exec php ./vendor/bin/phpunit: No such file or directory

i am unclear why it says this because te vendor directory is at the root of my site directory.

this is my dockerfile

FROM php:5.6-apache

ENV S6_OVERLAY_VERSION 1.11.0.1

RUN apt-get update && apt-get install -y \
    libldap2-dev \
    git \
    --no-install-recommends \
    && rm -r /var/lib/apt/lists/* \
    && docker-php-ext-configure ldap --with-libdir=lib/x86_64-linux-gnu/ \
    && docker-php-ext-install ldap \
    && docker-php-ext-install mysqli pdo pdo_mysql

#install xdebug
RUN git clone https://github.com/xdebug/xdebug.git \
    && cd xdebug \
    && git checkout tags/XDEBUG_2_5_5 \
    && phpize \
    && ./configure --enable-xdebug \
    && make \
    && make install

RUN a2enmod rewrite

COPY ./docker/rootfs /
COPY . /app

WORKDIR /app

ENTRYPOINT ["/init"]

I guess the real question is this: what is the correct way to run a phpUnit test within a docker container so that its subject to the php version within that container.

Don't quote the command you're trying to run, just do:

docker run --rm -it -v ~/Users/mow/Documents/devFolder/testdev:/app testdev_php php ./vendor/bin/phpunit

Your image is looking for a file called "php ./vendor/bin/phpunit" instead of looking for "php" and passing everything else on the line to it.

This is the real reason docker requires all your parameters before the image, because the parameter after the image is the command and everything after it is parameters to that command - don't quote everything.

The Dockerfile equivalent is:

ENTRYPOINT ["/init","parameter1","parameter2","parameter3"]

If it still doesn't work, check the parent Dockerfiles for the location of "php"

docker run --rm -it -v ~/Users/mow/Documents/devFolder/testdev:/app testdev_php /usr/local/bin/php ./vendor/bin/phpunit


How does Hashicorp's Vault compare to Lastpass? In this section Hashicorp compares Vault with other solutions. How does Hashicorp's Vault compare to Lastpass?

Attempt to answer the question

It is possible to login to LastPass from a web browser, create secrets and share these with other users. There does not seem to be an audit log. Imagine that a password of one of the users is retrieved one could retrieve a password from a database without knowing it.

I don't have enough rep to comment but I'm going to expand on what RuBiCK said.

In the first case, if you have a password for JIRA that I'm going to use to log in to the website with and fiddle with some tasks, then this goes in LastPass. You can store it there and use the extension or the vault to help me remember a complex password. You can also use LastPass to send credentials to my team members and then they can use LastPass to log in with.

In the second case, if you have made a script which asks AWS for what instances you have running right now then that script is going to need some credentials to log in with. You could put them in LastPass if you wanted but every time you went to run the script it would mean you have to go and get them from LastPass, enter them into the script somehow and get my list of instances.

Vault however presents an API which your script can use to ask for credentials when it runs. The credentials are stored in a secure way by Vault and you can now store much more in there as well.

The above is also why Vault wouldn't be useful for most people to do what they want to do. You don't want to have to write something to talk to Vault and create browser extensions etc when LastPass is, kind of, affordable and good at what it does.

I hope this helps!


What are the correct permission settings when running Docker in a Jenkins Pipeline? I'm trying to get a new jenkins pipeline together to test new pull requests to our code. I'm using docker with the ubuntu:14.04 image to simulate our production environment.

Here is a minimum working example:

#jenkinsfile
stage('Checkout and provision'){
docker.image('ubuntu:14.04').withRun('-u root'){
    checkout scm
    sh 'chmod -R 770 ./'
    sh './init-script.sh'
    }
}

and

 #init-script.sh
 sudo add-apt-repository ppa:ondrej/php
 sudo apt-get update -y
 sudo apt-get dist-upgrade -y
 sudo apt-get install \
    apache2 \
    php \
    php-mysql \
    php-xml \
    libapache2-mod-auth-mysql \
    libapache2-mod-php \
    php5-curl \
    zip \
    htop \
    supervisor \
    mailutils \
    git \
    build-essential -y
 sudo apt-get autoremove -y

And the /etc/sudoers file for the container is as follow:

#
# This file MUST be edited with the 'visudo' command as root.
#
# Please consider adding local content in /etc/sudoers.d/ instead of
# directly modifying this file.
#
# See the man page for details on how to write a sudoers file.
#
Defaults        env_reset
Defaults        mail_badpass
Defaults        secure_path="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
Defaults:jenkins !requiretty

# User alias specification

# Cmnd alias specification

# User privilege specification
root    ALL=(ALL:ALL) ALL

# Members of the admin group may gain root privileges
%admin ALL=(ALL) ALL

# Allow members of group sudo to execute any command
%sudo   ALL=(ALL:ALL) ALL

# See sudoers(5) for more information on "#include" directives:

#includedir /etc/sudoers.d

jenkins ALL=(ALL:ALL) NOPASSWD:ALL

The problem I'm having is that docker is not running as root as I'm used to and instead retaining the user id of the jenkins user from the host machine. This makes it difficult to sudo.

I've tried adding the jenkins user to the containers /etc/passwd file and running chmod against that file but don't even have the permissions to do either of these from the jenkinsfile.

With this configuration I should be the root user. However, I either see Error: must run as root if I don't use sudo or sudo: no tty present and no askpass program specified if I do.

Is there a correct way to deal with this situation? Ideally from the jenkinsfile; I'd like to avoid creating an additional Dockerfile if possible as this will be a further differentiator between testing and production.

So after a debug session in chat what is needed is to allow the user running jenkins to be able to sudo docker passwordless on the docker host.

A typical sudoers file on ubuntu could be in /etc/sudoers.d/jenkins

jenkins_user ALL=(ALL) NOPASSWD:ALL

Be warned this allow jenkins_user to run as root without password any command, a better file should be:

jenkins_user ALL=(ALL) NOPASSWD:/full/path/to/docker
jenkins_user ALL=(ALL) NOPASSWD:<other needed command to be run as root>

This will allow to start the container as root and as such it gives all rights within the container itself by running as uid 0.

Useful resources used aside:

Jenkins docker plugin readme
How to setup passwordless sudo on Linux?


How to copy files recursively to a Kubernetes pod? I want to copy files recursively to a Kubernetes pod

I tried kubectl cp -r

I got:
error: unknown shorthand flag: 'r' in -r

What are the best ways to transfer whole directories recursively into a pod.

kubectl cp by default does recursive copies when given a directory, although it seems to be picky about trailing slashes.  If foo is the directory you'd like to copy, simply run

kubectl cp /path/to/foo <pod-id>:/path/in/container/


Where Azure DevOps Services Enviromnent come into play? I am trying to understand where you actually use the environments you setup in a project. I though I could use them to deploy on specific servers from deployment agent which are hosted on another on premise server.

The example given in the documentation shows that you can define the environment inside a deployment job.
But I can't find it inside the UI when defining a release pipeline

Where do you use environments if you're not using a yaml pipeline ?

- stage: deploy
  jobs:
  - deployment: DeployWeb
    displayName: deploy Web App
    pool:
      vmImage: 'Ubuntu-16.04'
    # creates an environment if it doesn't exist
    environment: 'smarthotel-dev'
    strategy:
      runOnce:
        deploy:
          steps:
          - script: echo Hello world

You need to enable the multi-stage pipelines preview feature. Here is a guide to enable preview features.


Where to put ansible-vault password We are planning to use ansible vault in our project to prevent leaking passwords or keys in git.

The idea is to put all our sensitive data into a plain file then encrypt this file with ansible-vault using a password before pushing to git.

To decrypt the file, we have to pass the vault password to ansible, I'm thinking about 3 possibilities:

Store it inside a server environnment variable
Pass it as an option to ansible-playbook command
Store it into a non versionned file.

Is there any other options, which is the best (and secure) way to store ansible-vault password , ansible best practices documentation is not saying anything on this.

The idea is to put all our sensitive data [...]

The meaning of "all" in this sentence should be analyzed very carefully before implementing the solution that you plan.

Ansible vault is a very useful tool, but it should be used only to store secrets that are:

Specifically needed for the ansible deployments
Easily made useless to owners that should become unaware of them, but that may illegitimately "remember" them (typically off-boarded employees)

The second point is critical.

Many people, and potentially the whole DevOps team, will have access to the ansible vault password and therefore all the secrets.

Therefore, for all the secrets stored in the vault, a condition should hold for which a person or machine with unauthorized access to them should be incapable of making any use of them if so is desired.

In concrete terms, if you use ansible to deploy a database and its users, you can store the passwords in the vault, but you will have to be very careful (and most likely consider another solution) if that service will be available from the Internet and without the need for any VPN authentication!

Users (DevOps) exposed to the secret, should be incapable of using "remembered" passwords if one security barrier is imposed on them (e.g., VPN access revoked). In addition to this, access to the source code repository (where the vault is stored) should be revoked as well before passwords are changed.

Under these conditions, ansible vault is a very useful tool.

Trying to store a secret that could be used by any person or machine on the Internet in the vault would be instead a mistake (e.g., VPN credentials of users).

  Is there any other options, which is the best (and secure) way to store ansible-vault password

Under the conditions from the previous paragraph, I think that a good practice would be:

Store the vault password in an external secure vault (something like Vault from HashiCorp or any SaaS for credentials management)
Allow access to the external vault item to DevOps (they will need the password for testing) and the CI/CD system or ansible controller
Keep a convention to use secrets! You will not be able to review changes to the secrets and you will not be able to grep for ansible variables in secrets files! So be thorough since the beginning. A good convention is to name all variables stored in the ansible vault with a secret_ prefix. When you will see something like:

postgres.yml:

postgres_password: "{{ secret_postgres_password }}"

you will know that the value is stored in the ansible vault.


Securely grab minion ID in pillar top file template? I found this SLS file in a pillar top.sls targeting example:

top.sls:

base:
  '*':
    - hostspecificsls

hostspecificsls:

# See https://groups.google.com/forum/#!topic/salt-users/UWfP6bMe6F8
{% include 'hosts/' + salt['grains.get']('fqdn') + '.sls' ignore missing %}

This strikes me as distinctly insecure, as it runs counter to one of the Salt Best Practices:

  Don't use grains for matching in your pillar top file for any sensitive pillars.

Why is this insecure? Well, grains are evaluated on the minion, so if a minion is sufficiently compromised, it can return whatever an attacker wishes for an arbitrary grain call. A compromised minion might claim to be an SSL (or VPN!) terminator to get at private key material, for example.

Fortunately, there's a (theoretical) solution; use the minion ID. Minion IDs are unique, and they're verified on the master cryptographically.

Except I can't find how to get at the minion ID without depending on grains. Remember, the grains are executed on the minion, so even if the Salt master knows at the transport layer what the minion ID is, if you query for the minion ID via grains, you're asking the minion for the ID, not the master.

So how can I get at the master's concept of the minion ID from Jinja on the master?

There is currently one safe way to do this.

You can use {{opts.id}} as an explicit way to grab the minion ID. opts is technically an implementation detail; the opts dictionary is a Salt internal structure that doesn't appear to be formally specified. Its candidacy for the purpose was pointed out to me in a bug comment.

In the next version of Salt (so, later than 2016.11.5) the id grain, will have been special-cased, making it possible to use {{grains.id}} safely.


How can a Windows server be a slave to Jenkins running on Linux when there in an error installing the slave-agent.jnlp file? I want to have a Windows Server 2016 slave node with a master Jenkins (version 2.89.2) running on Red Hat Enterprise Linux 7.x.  I am trying to install the slave-agent.jnlp file from the Jenkins web UI on the Windows server.  I run this command for PowerShell opened as administrator:

javaws -verbose C:\Users\Administrator\Downloads\slave-agent.jnlp

I receive this error:

Error: The field <jnlp><application-desc><argument> has an invalid value:

Launch File

<jnlp codebase="http://FQDNofJenkinsServer:8080/computer/FQDNofWindowsSlave/" spec="1.0+"><information><title>Agent for FQDNofWindowsSlave</title><vendor>Jenkins project</vendor><homepage href="https://jenkins-ci.org/"/></information><security><all-permissions/></security><resources><j2se version="1.8+"/><jar href="http://FQDNofJenkinsServer:8080/jnlpJars/remoting.jar"/></resources><application-desc main-class="hudson.remoting.jnlp.Main"><argument>32ba31ca5f150c2b3491802b0b60e56574d970a9e601cebad656a113fe325a85</argument><argument>FQDNofWindowsSlave</argument><argument>-workDir</argument><argument/><argument>-internalDir</argument><argument/><argument>-url</argument><argument>http://FQDNofJenkinsServer:8080/</argument></application-desc></jnlp>

Exception

BadFieldException[ The field <jnlp><application-desc><argument> has an invalid value: ,]
    at com.sun.javaws.jnl.XMLFormat$10.visitElement(Unknown Source)
    at com.sun.javaws.jnl.XMLUtils.visitElements(Unknown Source)
    at com.sun.javaws.jnl.XMLFormat.buildApplicationDesc(Unknown Source)
    at com.sun.javaws.jnl.XMLFormat.process(Unknown Source)
    at com.sun.javaws.jnl.XMLFormat.parse(Unknown Source)
    at com.sun.javaws.jnl.LaunchDescFactory.buildDescriptor(Unknown Source)
    at com.sun.javaws.jnl.LaunchDescFactory.buildDescriptor(Unknown Source)
    at com.sun.javaws.Main.launchApp(Unknown Source)
    at com.sun.javaws.Main.continueInSecureThread(Unknown Source)
    at com.sun.javaws.Main.access$000(Unknown Source)
    at com.sun.javaws.Main$1.run(Unknown Source)
    at java.lang.Thread.run(Unknown Source)

I expect the command to work.  I do not understand the error message.  What should I do?

You have two arguments that are empty:

<application-desc main-class="hudson.remoting.jnlp.Main">
  <argument>32ba31ca5f150c2b3491802b0b60e56574d970a9e601cebad656a113fe325a85</argument>
  <argument>FQDNofWindowsSlave</argument>
  <argument>-workDir</argument>
  <argument/>
  <argument>-internalDir</argument>
  <argument/>
  <argument>-url</argument>
  <argument>http://FQDNofJenkinsServer:8080/</argument>
</application-desc>

Your error message has a comma in it:

  BadFieldException[ The field  has an
  invalid value: ,]

Other bad argument error messages on the internet have comma separated lists of bad arguments:

  BadFieldException[ The field codebase has an invalid value:
  $$codebase,$$codebase]

So I'd guess those two empty arguments are the problem.


How to give value in runtime for chef I got stuck with a scenario where I need to provide runtime value while restarting/starting httpd service using Chef.

I installed SSL certs where it asks password for every reload or start. I tried giving:

service 'httpd' do
  supports restart: true, reload: false, status: true
  options "password"
end

But I don't have any luck.

Any help about how to pass this value while httpd restarts using Chef?

You shouldn't. Instead you should use proper secrets management system that would provide you unlocked SSL certificate.

By secret management I mean tooling like k8s secret management or Vault.

You can read Medium post about secrets.


Devops Jenkins nexus integration? I want to try to upload any set of artifacts from any type of project ( Java, node, PHP, .net, go ).

I know that once a developer pushes a code to git, Jenkins should take it and run the test case on it.

If it is success then push it to production? Am I right? I'm not sure about the flow.

I wonder how Nexus and Maven ( is it needed, if so why? ) will come into a picture of DevOps flow?

I know Nexus it's repository manager ( which store artifacts [ final executable file after code compilation ( which we can see in Java, go but not in PHP or node ) ] ) please, correct me if I wrong.

Do I need Maven?

No, you don't need maven.
Maven is a build automation tool as ant and gradle are, generally, but not limited to, jvm based projects.
Nexus is an artifact repository, but some others exists.
Artifact repositories software have different set of features (license, different storage options, different repositories type support like gems or npm ..., search support, web interface ...).

I know that once a developer pushes a code to git, Jenkins should take it and run the test case on it.
If it is success then push it to production? Am I right? I'm not sure about the flow.

Generally, in order to achieve continuous deployment, which is what you are asking for precisely here, you would:

setup a hook when you commit
have a continuous integration tool that receive the hook
the CI does whatever needs to be done for the project to be tested / build / runnable
the CI deploys the previously built software

Jenkins, for example, supports pipeline as code, it would allow you to put a Jenkinsfile in your repository, that would tell Jenkins how to do step 3 and 4 (travis and circleCI and others CI also supports this).
You can tell your CI in steps 3 to upload some artifacts to an artifact repository and on step 4, to retrieve them in order to deploy the software, but you don't need to.
My advice would be to design your CD system the simplest way you can, don't introduce not necessary parts like an artifact repository if you don't need it, add it when you face a problem that needs it.
As a first step, your CI, can during the deployment steps, simply copy your artifacts to the target server.


Crash course in Dev for Ops? I schooled in CompSci where we were primarily taught Java, but what I learned there is that my passion is systems, so I've always worked the ops side. I'm handy with scripting, so I'm not looking for a site to teach me Ruby, but something to explain more in-depth what you devs do all day. I want to understand the culture better and how you digest the sheer number of files in your projects -- the intangibles.

If I learned today I was being moved to a development team on Monday, what would I want to read this weekend?

Since you've tagged this question as "culture", I'm assuming you're not interested in a specific application, but in the broader questions of workflow and management.

I'd probably start with "The DevOps Handbook"; it's a good overview of different things to consider, without diving too deep.

"Continuous Delivery" by Jez Humble is also often referenced; I haven't read much of it yet, but it covers the concepts of source control & automation of builds.

If you're starting to get into applications at scale (this may be too much of an assumption), another good book is "The Practice of Cloud System Administration" by Limoncelli et al.


Diagnosing large delay in file sync to Docker for Mac Preface: I understand that Docker for Mac has file syncing performance issues, but everything I've read about the issues is related to slow read access. I don't really have a problem with slow read access. My problem is that updates I make to files on my host machine can sometimes take 10+ seconds to update on the running container!

Here's my docker-compose file:

version: '2'

services:
  ps-cms:
    image: [obfuscated]
    container_name: ps-cms
    mem_limit: 2048M
    ports:
      - "9080:80"
      - "9443:443"
    volumes:
      - .:/var/www/html/site:cached
      - ./xdebug.ini:/etc/php/conf.d/docker-php-ext-xdebug.ini
    environment:
      - MYSQL_HOST=ps-db
    networks:
      - connector
    extra_hosts:
     - "mysite.localhost:127.0.0.1"

  ps-db:
    image: mysql:5.6
    container_name: ps-db
    mem_limit: 512M
    environment:
      - MYSQL_ALLOW_EMPTY_PASSWORD=yes
    ports:
      - 33006:3306
    networks:
      - connector

networks:
  connector:

Software:

Docker Engine: 18.03.1-ce
Docker Compose: 1.21.1
Docker Machine: 0.14.0
PHP Storm: 2018.1

In the Docker for Mac preferences, under "file sharing", I removed everything there and just added my one project folder, since I heard that can affect performance.

Does anyone have advice on what could be causing this sync delay?

I figured this out, and it was related to the opcache settings for PHP in my docker image. The opcache re-validate frequency was set to 60 seconds, meaning that PHP caches files in memory for that long before referring to the filesystem again. Changing this to 2 seconds resolve the issue, but adds a significant performance penalty :(.

See more info: https://drupal.stackexchange.com/questions/253111/delay-in-code-changes-on-custom-pages-in-drupal-8-in-a-docker-environment


Where are dunder dictionaries stored in salt? I am new to Salt-Stack wanted to know where are all the dunder dictionaries stored(Ex:salt,opts,grains,beacons etc).

Thanks in Advance :)

There are currently 5 dunder dictionaries. They are available at runtime and generally not stored statically.

__opts__    - Master or Minion configuration options; stored in configuration files of master and minion, collected at startup

__salt__    - Execution module functions (i.e. __salt__['test.echo']('foo')); from built-in and custom execution modules stored in _modules/

__grains__  - Grains (i.e. __grains__['os']); collected from minions, dynamically generated, or stored in minion config or /etc/salt/grains

__pillar__  - Pillar data (i.e. __pillar__['foo']); collected from the pillar

__context__ - Context defined in state modules and execution modules

You generally don't need to use them, unless you are developing Salt or writing custom modules, runners, returners, or external pillars.


Every jenkins build shows error: ln builds/lastSuccessfulBuild /root/.jenkins/jobs/Build/lastSuccessful failed I migrate my jenkins to new machine succesfully,but after migrate all jenkins job run succesfully. but every build give following error:

ln builds/lastSuccessfulBuild /root/.jenkins/jobs/BeepnRide/jobs/43-Server/jobs/BR1.1.2QA-Build/lastSuccessful failed
java.nio.file.DirectoryNotEmptyException: /root/.jenkins/jobs/BeepnRide/jobs/43-Server/jobs/BR1.1.2QA-Build/lastSuccessful
    at sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:242)
    at sun.nio.fs.AbstractFileSystemProvider.deleteIfExists(AbstractFileSystemProvider.java:108)
    at java.nio.file.Files.deleteIfExists(Files.java:1165)
    at hudson.Util.createSymlink(Util.java:1193)
    at hudson.model.Run.createSymlink(Run.java:1957)
    at hudson.model.Run.updateSymlinks(Run.java:1938)
    at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:286)
    at hudson.model.ResourceController.execute(ResourceController.java:97)
    at hudson.model.Executor.run(Executor.java:429)

ln builds/lastStableBuild /root/.jenkins/jobs/BeepnRide/jobs/43-Server/jobs/BR1.1.2QA-Build/lastStable failed
java.nio.file.DirectoryNotEmptyException: /root/.jenkins/jobs/BeepnRide/jobs/43-Server/jobs/BR1.1.2QA-Build/lastStable
    at sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:242)
    at sun.nio.fs.AbstractFileSystemProvider.deleteIfExists(AbstractFileSystemProvider.java:108)
    at java.nio.file.Files.deleteIfExists(Files.java:1165)
    at hudson.Util.createSymlink(Util.java:1193)
    at hudson.model.Run.createSymlink(Run.java:1957)
    at hudson.model.Run.updateSymlinks(Run.java:1939)
    at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:286)
    at hudson.model.ResourceController.execute(ResourceController.java:97)
    at hudson.model.Executor.run(Executor.java:429)

I had a similar issue with Jenkins running under Kubernetes with Azure-storage-backed PersistentVolumes.

Azure storage doesn't support symlinks by default.

As per: https://docs.microsoft.com/en-us/azure/storage/files/storage-troubleshoot-linux-file-connection-problems#cannot-create-symbolic-links---ln-failed-to-create-symbolic-link-t-operation-not-supported

To resolve, I added the following to my PersistentVolume yaml:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: jenkins-pv
spec:
  storageClassName: azurefile
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteOnce
  azureFile:
    secretName: azure-secret
    shareName: jenkins
  mountOptions:
    - dir_mode=0777
    - file_mode=0777
    - mfsymlinks

The mfsymlinks allows symlinks to be created on Azure storage.


Build Docker images with DinD and BuildKit I use GitLab's pipelines to describe the deployment process. And, as part of it, I build Docker images. For that purpose, I use Docker-in-Docker to build those images (i.e. download the Docker-in-Docker image and do all the Docker-stuff, inside of it).

I also set to use Docker Buildkit features. And one of them is --cache-from option (which decreases build time). Which seems doesn't work with Docker-in-Docker (at least, for me).

Here's the pipeline's config:

stages:
  - build
  - purge
  - deploy

variables:
  DOCKER_HOST: tcp://localhost:2375
  DOCKER_TLS_CERTDIR: ""
  DOCKER_BUILDKIT: 1
  LATEST: ${CI_REGISTRY_IMAGE}/${CI_PROJECT_NAME}:${CI_COMMIT_REF_SLUG}
  IMAGE_COMMIT_TAG: ${CI_REGISTRY_IMAGE}/${CI_PROJECT_NAME}:${CI_COMMIT_SHORT_SHA}

services:
  - docker:dind

build:
    stage: build
    image: docker:stable
    script:
        - docker login -u gitlab-ci-token -p ${CI_BUILD_TOKEN} ${CI_REGISTRY}
        - docker build --cache-from ${LATEST} -t ${LATEST} -t ${IMAGE_COMMIT_TAG} .
        - docker push ${IMAGE_COMMIT_TAG}
        - docker push ${LATEST}

The log output shows the BuildKit is engaged, but there's nothing about using of 'CACHE'. And, the result of it -- 'build' stage takes much more time.

Please, your suggestions, ideas?

Caching of layers with buildkit in an external registry requires an extra step or two depending on how you want to cache your layers. The easy option is to include a build arg that enables the inline cache:

docker build --build-arg BUILDKIT_INLINE_CACHE=1 --cache-from ${LATEST} -t ${LATEST} -t ${IMAGE_COMMIT_TAG} .

Note that the inline cache only caches layers for the target stage that was pushed in the image, so other states in a multi-stage build would need to be built and cached separately, or rebuilt without caching, neither of which is ideal.

You can also cache to a local file, or push the cache to a different registry image rather than inline with the image you pushed. Unfortunately the standard docker build CLI doesn't have access to all the buildkit flags to enable this. Instead, you can install buildkit directly or use buildx which is a CLI plugin for managing buildkit. You can install buildx separately but in current releases it's available with experimental CLI options that can be enabled with export DOCKER_CLI_EXPERIMENTAL=enabled. I believe you'd need to create a container based builder for all of the buildkit options, which can be done with:

docker buildx create --use --driver docker-container --name local ${DOCKER_HOST:-unix:///var/run/docker.sock}

Then you should be able to run something like:

docker buildx build --cache-from ${IMG_CACHE} --cache-to ${IMG_CACHE} -t ${LATEST} -t ${IMAGE_COMMIT_TAG} .

More documentation on the buildkit caching is available at: https://github.com/moby/buildkit#export-cache

More details on buildx is available at: https://github.com/docker/buildx


Enabling Github webhooks for multibranch jobs? I'm setting up a CI scheme for a few projects, on Jenkins, using the multibranch plugin.
Everything is working pretty nicely, but the obsessive in me always wants a bit more.

Current situation:
The multibranch thingy is working: picks up and builds new branches and PRs. It does so by polling: Every N minutes it polls Github for new branches and branch updates.

I want:
I would very much like to build new pushes on Github Webhook push events and avoid polling. When not dealing with multibranch, it was as simple as checking the GitHub hook trigger for GITScm polling box.
In this context, I would have to add a new trigger to the Jenkinsfile. I assume something like:

triggers {
    githubWebhook()
}

Currently, my Jenkinsfile is something like this:

pipeline {
    agent {
        docker {
            image '${REDACTED}/builder:sbt-0.13-jdk-oracle-8'
            args '-v sbt-cache:/root/.sbt/ -v ivy-cache:/root/.ivy2/'
            customWorkspace '/src/'
        }
    }
    options {
        buildDiscarder(logRotator(numToKeepStr: '10'))
        timestamps()
        timeout(time: 1, unit: 'HOURS')
    }
    stages {
        stage('Testing') {
            steps {
                sh "sbt clean coverage test coverageReport coverageAggregate sonar"
            }
        }

        stage('Push snapshot'){
            when {
                not {
                    branch 'master'
                }
            }
            steps {
                echo "[DRYRUN] sbt publishSnapshot"
            }
        }

        stage('Push release') {
            when {
                branch 'master'
            }
            steps {
                echo "[DRYRUN] sbt publish"
            }
        }
    }
    post {
        changed {
            echo "[DRYRUN] changed build"
        }
    }
}

Has anyone dealt with something like this? Any other approach?

According to the official GitHub Branch Source Plugin documentation, the plugin can automatically configure webhooks for you if you have your GitHub API token configured in Jenkins global settings:

  You can change the build configuration to perform the folder
  computation at different times. The default setting will scan your
  GitHub repository for changes if it has not received any change
  notifications from GitHub within the past day. This scan will catch
  the repositories you’ve added or removed from the managed GitHub
  repository. You can always force the Folder Computation to run from
  the GitHub Organization page.

  While the build triggers are often enough, you can set up webhooks to
  automatically trigger builds when changes are pushed to your GitHub
  repositories. To do this you must have a GitHub login with a token.

  Go to the main configuration settings page, Manage Jenkins > Configure System
  In the GitHub Plugin Configuration section, add a server with your credentials
  If you need a token, generate one with the Additional Actions > Convert login to password and token

  You can also configure this manually through GitHub itself by
  registering the URL provided in the help section of the server config.


What's the relevance of periodicity in CI / CD processes? Part of what (I think) CI and CD is about, is processes that get repeated over and over again (= the "continuous" part of it). However continuous in this context is not exactly (I think) like a river where water keeps going (flowing) all the time. Instead it's rather something happening every X seconds, minutes, hours etc.

So assume some test-environment where every day, eg, from 5 pm to (max) 6 pm all required processes are executed to rebuild all executables that are impacted by the changed dependencies (sources, includes, etc), combined with appropriate CI/CD processes related to them. While before 5 pm and after 6 pm, none of these kinds of processes are allowed to be executed (eg to ensure the stability of the related environments).

Does such kind of periodicity (= between 5 pm and 6 pm, every day) fit with CI/CD or not? Or is it rather that periodicity has nothing to do with CI/CD?

I would separate the CI and CD contexts, as the periodicity in one of them is rather loosely coupled to the periodicity in the other one.

That's primarily because CI attempts to produce versions of software available for delivery/distribution, see How does continuous integration relate to continuous delivery / deployment?
But not all CI executions are successful. CD periodicity will, at best, be equal to the CI one, but typically it will be lower.

In CI the periodicity is tightly coupled to the desired development velocity (and maybe branch stability) and can be driven by different project-specific requirements. Here are just some examples of CI execution patterns, there could be others:

launched for every changeset commit - the recommended one, as it offers the shortest time to identify a changeset causing a regression, see Build on each commit - Continuous delivery. Costs vary with the commit rates.
launched every N (fixed number of) commits - trade culprit identification performance of #1 for slightly lower costs (overall fewer executions)
launched every T (fixed) time interval

fixed costs due to predictive amount of executions: no cost hikes for commit activity peaks, a day with twice the number of commits than another will still cost the same
culprit identification performance drops vs #1 if multiple changesets are committed within T
consecutive executions without any changeset committed in between offer measurements of the verification process reliability - different results in such executions could indicate unreliable verifications

launched whenever resources for execution are available - if that's where the bottleneck of the process lies. Any of the above categories can appear depending on the pattern of commits relative to the intervals between executions. This maximizes the verification resources ROI.
launched with one pattern during working hours and a different pattern overnight - to balance computing resources utilisation, for example.

Another possible reason for periodicity mismatch between CI and CD is the variability in the duration of the CI executions. Caused, for example, by that prohibited build schedule policy you mentioned (see also How to implement a frozen test environment?) Or by builds executed on classes of servers with significantly different performance, thus lasting different amounts of time.

Even for a successful CI execution in which the produced software version qualifies and is available for deployment it doesn't necessarily mean it will also be immediately deployed (or even deployed at all). There can be other criteria or policies preventing it (business reasons, for example).


Why can the certificate not be found by the Ingress Controller, while the certificate has been issued? kubectl logs nginx-ingress-controller-X -n nginx-ingress

W1222 06:52:04.575115       8 controller.go:1058] Error getting SSL
certificate "app/app-tls": local SSL certificate app/app-tls was not
found. Using default certificate
W1222 06:52:09.955475       8 backend_ssl.go:48] Error obtaining X.509
certificate: no object matching key "app/app-tls" in local store

kubectl describe certificates nexus-tls -n cert-manager

Message:               Certificate issued successfully
Reason:                CertIssued

Discussion

It is unclear to me where to deploy the certmanager. I have deployed in a separate namespace, the cert has been created successfully, but the certificate cannot be found.

https://github.com/jetstack/cert-manager/issues/482#issuecomment-449664935

  Once I followed this
  https://github.com/jetstack/cert-manager/blob/master/docs/tutorials/quick-start/index.rst#step-6---configure-lets-encrypt-issuer
  and deployed the issuers in the namespace of the app, it works


Using gitlab-ci to cache maven repository We have a server running gitlab and gitlab-runners on same machine. I've defined a runner as below

[abc@xyz bin]# gitlab-ci-multi-runner register
Running in system-mode.

Please enter the gitlab-ci coordinator URL (e.g. https://gitlab.com/):
http://abc.xyz.com
Please enter the gitlab-ci token for this runner:
9nev5wz3voabcdefgh
Please enter the gitlab-ci description for this runner:
[opera.ishisystems.com]: second-docker-remote-runner
Please enter the gitlab-ci tags for this runner (comma separated):
docker, remote
Whether to run untagged builds [true/false]:
[false]:
Whether to lock Runner to current project [true/false]:
[false]:
Registering runner... succeeded                     runner=9nev5wz3
Please enter the executor: docker, docker-ssh, ssh, docker+machine, docker-ssh+machine, parallels, shell, virtualbox, kubernetes:
docker
Please enter the default Docker image (e.g. ruby:2.1):
node:4.2.2
Runner registered successfully. Feel free to start it, but if it's running already the config should be automatically reloaded!

and my .gitlab-ci.yml is

image: maven:3-jdk-7

services:
  - wurstmeister/kafka:0.8.2.1

variables:
  MAVEN_REPO: "$CI_PROJECT_DIR/.m2"

cache:
    key: "$CI_BUILD_STAGE"
    paths:
      - .m2

build:
  script: "mvn -Dmaven.repo.local=$MAVEN_REPO test -B"
  tags:
  - docker

What I'm trying to do is setup a central maven repository on my gitlab-runner server so that every time a new job is fired instead of downloading all the maven dependencies it should use the cached repository.

To me the easiest route to this would be to have an instance of an artifactory running. There are several choices out there many of which have an open-source, community or free version that would support maven repositories.  Most options will support caching remote repositories (maven central for example) which can then be curated to enforce specific versions of the dependencies through your projects. It will also accept your own dependencies this way you can have a central unique access point for all maven dependencies in your team.

I've personally had success with Sonatype's Nexus (OSS version, or Pro version). There are other choices out there though check them out.

That said, if what you want is to prime the local repository for a new agent you want to provision, you could do so. But frankly, unless you have some real hefty dependencies (say in the GB size range), I would not bother doing so really and just let Maven naturally pull from the local binary repository instead. It makes for a simpler solution that will be more robust in the long run.


Can I keep a container started with --rm option? I started my docker container with --rm and -it options. I need to keep that container now because it has some important data. Can I cancel the --rm switch using the docker update command?

This answer from SO says it is possible, but I it doesn't say how exactly!

No, docker will delete the container when it stops, and there's no update option to change that behavior. You can see this in the help output:

$ docker container update --help

Usage:  docker container update [OPTIONS] CONTAINER [CONTAINER...]

Update configuration of one or more containers

Options:
      --blkio-weight uint16        Block IO (relative weight), between 10 and 1000, or 0 to disable (default 0)
      --cpu-period int             Limit CPU CFS (Completely Fair Scheduler) period
      --cpu-quota int              Limit CPU CFS (Completely Fair Scheduler) quota
      --cpu-rt-period int          Limit the CPU real-time period in microseconds
      --cpu-rt-runtime int         Limit the CPU real-time runtime in microseconds
  -c, --cpu-shares int             CPU shares (relative weight)
      --cpus decimal               Number of CPUs
      --cpuset-cpus string         CPUs in which to allow execution (0-3, 0,1)
      --cpuset-mems string         MEMs in which to allow execution (0-3, 0,1)
      --kernel-memory bytes        Kernel memory limit
  -m, --memory bytes               Memory limit
      --memory-reservation bytes   Memory soft limit
      --memory-swap bytes          Swap limit equal to memory plus swap: '-1' to enable unlimited swap
      --pids-limit int             Tune container pids limit (set -1 for unlimited)
      --restart string             Restart policy to apply when a container exits

Trying to change the restart policy will result in an error:

$ docker run --rm --name test-rm -d busybox tail -f /dev/null
14a7d9bfebc3356652e3c3f060e92d9cebb9f3eb4490873540c9eb1c272f6612

$ docker update --restart=unless-stopped test-rm
Error response from daemon: Cannot update container 14a7d9bfebc3356652e3c3f060e92d9cebb9f3eb4490873540c9eb1c272f6612: Restart policy cannot be updated because AutoRemove is enabled for the container

You can try to copy your data out of the container with:

docker container cp $container_name:/path/in/container /path/on/host

If you don't know which files to save, list which files have changed in the container with:

docker container diff $container_name


Continuous Integration infrastructure for multiple iOS projects As a iOS developer, I was looking to create a CI and CCQ (= Continuous Code Quality) infrastructure for the iOS projects we have in development so far. We already use Jenkins and SonarQube for almost all the Web and Android projects (using a VM foreach project, installing and configuring the CI and CCQ is automated) and it works great. But for the iOS projects, Jenkins needs to do the build on a computer running macOS, so I am not sure I have the perfect solution for that.

I was looking into a solution to virtualise the macOS. For each
project, to create a virtual macOS and install Jenkins as slave there
to handle the builds. The solution looked perfect, but it seems it is
not legal to have more than 2 VM running with macOS (and of course,
only on Mac computer)
http://images.apple.com/legal/sla/docs/macOS1012.pdf -> The point
2.B. So this isn't a solution for my case.
Another common solution I've read about is to have a Mac computer
(maybe a MacMini) that will handle all the builds for all the
projects. What do you think about this implementation? How many
projects may it handle? Developers may need to do some configurations
on their own project (especially in SonarQube), is it safe?
Can we have multiple Jenkins and SonarQube instances on the same machine, using different ports? Is this even a solution to be considered or I am talking non-sense?
Is there any other viable solution, maybe better than the ones above :) ?

Note: I am not sticking to Jenkins + SonarQube duo, if there are any other tools better suitable for iOS development, please share them with me.

Is there any other viable solution, maybe better than the ones above :) ?

For Jenkins probably no, due to the license issues you mentioned (you have to run macOS on Apple branded hardware). If you can work with that then it's definitely a solution.

If not, then I'd suggest you to check for a hosted iOS continuous integration system where you don't have to deal with the Apple hardware / licensing, the service will manage it for you, e.g. https://www.bitrise.io

There are also services where you can rent Apple hw and you can configure it any way you like, https://www.macstadium.com is one of them (probably the biggest one).


Does GitHub support VFS for Git? NOTES:

I really hope that this question hasn't been asked before and as far as I know, only Azure DevOps Service (Azure DevOps on cloud) and Azure DevOps Server 2019 (Azure DevOps on-premise) fully supports VFS for Git.
VFS in this context is Virtualized File System for Git. It is not VFS in Linux. The same virtualized, but this VFS is specific to ease managing large Git repository.

For those who don't know VFS for Git, VFS for Git is helpful to ease managing Git because it has virtualized file system that help managing large size repo, especially for those repo that has size > 2GB.

For more information, please visit the official landing page of VFS for Git: https://vfsforgit.org/

My question is, when will GitHub support VFS for Git? As far as I know, in early 2018, it says it will support but still no further update until now.

GitHub doesn't support VFS. There are news articles in 2017 saying GVFS and later VFS would be supported by GitHub. As of today, there is no support and no planned support for it.
My sources:

https://github.com/microsoft/VFSForGit/issues/1640
https://github.community/t/vfsforgit-setup/128184
https://github.community/t/microsofts-virtual-filesystem-for-git/2092
The Github public roadmap says nothing related to VFS or anything similar. https://github.com/github/roadmap


How to notate ipv6 addresses in inventory file? I have a machine, that is accessible via ipv6 only. It is setup for ssh with rsa key. Now I'd like to manage it with ansible. Unfortunately, this is not covered in the inventory documentation.

So I tried the following permutations:

docker_test ansible_host=fe80::88ea:e002:b409:4be0
docker_test ansible_host=[fe80::88ea:e002:b409:4be0]
docker_test ansible_host=\[fe80::88ea:e002:b409:4be0\]

None of this worked out.

For comparison I add my .ssh/config entry:

Host docker_host
    AddressFamily inet6
    HostName fe80::88ea:e002:b409:4be0
    BindAddress fe80::3e6b:335a:d2f7:3a75%enp3s0

I found two closed bug in the issue tracking system,which indicate the problem being at least partially solved: 3879, 3888

Any advise on this topic is highly appreciated.

This blog post covers all of the methods:

excerpt - Host names and patterns in Ansible 2

  Inventory hostnames

  Ansible 2 requires inventory hostnames to be valid IPv4/IPv6 addresses or hostnames (i.e., x.example.com or x, but not x..example.com or x--). As an extension, it accepts Unicode word characters in hostname labels. Any mistakes result in specific parsing errors, not mysterious failures during execution.

  Inventory hostnames may also use alphabetic or numeric ranges to define more than one host. For example, foo[1:3] defines foo1 through foo3, while foo[x:z:2] expands to fox and foz. Addresses may use numeric ranges: 192.0.2.[3:42].

  IPv6 addresses

  A number of problems with the parsing of IPv6 addresses have also been fixed, and their behaviour has been made consistent across the inventory (.ini files) and in playbooks (e.g., in hosts: lines and with add_host).

  All of the recommended IPv6 address notations (from spelling out all 128 bits to the various compressed forms) are supported. Addresses with port numbers must be written as [addr]:port. One can also use hexadecimal ranges to define multiple hosts in inventory files, e.g. 9876::[a:f]:2.

  A couple of small but necessary bugfixes go hand-in-hand with the parsing changes, and fix problems with passing IPv6 addresses to ssh and to rsync. Taken together, these changes make it possible to use IPv6 in practice with Ansible.

Therefore patterns such as these IPv6's would be allowed:

fully specified 128 bit address
[addr]:port
hexadecimal ranges - 9876::[a:f]:2
various forms of compressing ranges - FF01::[1:6]

Simplified examples:

# IPv6 Address                                # Simplified Notation
-----------------------------------------     ----------------------------
- FF01:0000:0000:0000:0000:0000:0000:0001     - FF01::1
- 2031:0000:130F:0000:0000:09C0:876A:130B     - 2031:0:130F::9C0:876A:130B
- 0000:0000:0000:0000:0000:0000:0000:0001     - ::1
- FE80:0000:0000:5EFE:0192.0168.0001.0123     - FE80::5EFE:192.168.1.123
- FE80:0000:0000:0000:1585:4868:495F:D521     - FE80::1585:4868:495F:D521

References

NETWORK BASICS: IPV6 ADDRESS SIMPLIFIED NOTATION


How do you know which secrets and credentials of your production services were used, and by whom/what? The context of the question is Secrets management.

As the title says, how do you know which secrets and credentials of your production services were used, and by whom/what?

The short answer: you would need to use a secret management system that provides 'audit logs'.

There are many options for handling secrets out there, but unless the tool you use has audit logging, you'll never truly know for sure what credential was used where.

I see you mentioned using Hashicorp's Vault. That's a really good one. They have audit logging built in; they call it 'Audit Devices', which can be enabled with a simple command.

Again, there are many perfectly valid options for handling secrets: Chef vault, 1password, environment variables injected by your build instances, home-grown solution using encrypted files stored on object storage, etc.... However, unless the system specifically says it provides audit logging, you'll never truly know what was used when and where.

You can design a secrets system with the best intentions, and make many rules that employees have to follow in regards to have the secrets are accessed, but for auditing purposes, you want to know where the rules have been broken. If you have a firm requirement to always know when and where secrets where used, I recommend you stick with something like Hashicorp Vault, turn on Audit Devices, and then ship the log somewhere for storage or processing.


Are git pre-receive/update hooks serialized? I'm investigating the possibility of implementing a "pre-commit" verification flow, enforced on the central SCM server side. By "pre-commit" in this paragraph I don't mean the git pre-commit hook, I really mean before the change becomes part of the integration branch, visible to all developers working on the branch.

For git in particular, which is a distributed SCM, the commit phase happens on the local server, which is not what I need. So I can't use the git pre-commit hook.

From what I gather the pre-receive hook would be what I'm after:

  This hook is invoked by 'git-receive-pack' on the remote repository,
  which happens when a 'git push' is done on a local repository. Just
  before starting to update refs on the remote repository, the
  pre-receive hook is invoked.  Its exit status determines the success
  or failure of the update.

  This hook executes once for the receive operation. [snip]

  If the hook exits with non-zero status, none of the refs will be
  updated. If the hook exits with zero, updating of individual refs can
  still be prevented by the <<update,'update'>> hook.

Or maybe the related update hook:

  This hook is invoked by 'git-receive-pack' on the remote repository,
  which happens when a 'git push' is done on a local repository. Just
  before updating the ref on the remote repository, the update hook is
  invoked.  Its exit status determines the success or failure of the ref
  update.

  The hook executes once for each ref to be updated, [snip]

The complication is that the verification itself can take a while.

My question: is the execution of these hooks serialized (by git)? Meaning will another hook invocation (due to a subsequent push operation performed before the previous hook execution completes):

fail,
block until the previous, still running execution finishes,

or

simply run in parallel with the previous hook execution?

In general git allows concurrent operations because one of them will eventually fail. Some specific database updates will have file locks (like updating the index files) but multiple receives can be in flight at once I think. In this case whichever verification completed first would be allowed in, and the second would fail due to having an invalid history (i.e. not being a fast-forward push).

In general this kind of operation is not well suited to git's server-side hooks since if something does fail, the person will then have to do some surgery on their branches to get things back in a valid state. These kinds of checks are generally run on the workstation side, though then you run into the "fun" that is distributing git hooks as they can't actually be checked in directly.

Another common solution is to have a CI service like Jenkins running verifications (tests) and updating a second branch or a utility tag showing which commits are acceptable.


Recipe for hiring a winning DevOps empowered team? The comparison of a software delivery team to a football team (all can kick the ball but only some will become an excellent goalkeeper or a forward) can be a nice input for further thought regarding division of labour and specialists vs. generalists.

So, it seems more to me than just "you should never try to hire a DevOps team" - because if you hire "a football team" you do not just want "somebody kick the ball in the FIFA World Cup", but you possibly want to have an idea what makes the team having chances to win.

So, winning teams can be a product themselves (can't find the reference for that but there are enough startup acquisions, I'm sure it is not always just about the product) - what is the recipe for success?
For sure there is a lot of soft skills/cultural related context, but this would be true for virtually every team.

What are mandatory DevOps specific requirements to skill sets for a winning team setup? Will be there a relationship between numbers of CI/CD experts and those who focus on development, or operations?

Note: this question is not a duplicate of  "why you shouldn't try to hire a DevOps Engineer" as a specific question is after required skills setup. E.g. would you have a CI-focused profile or expect developers to include pipelines as code to their projects? Clearly, just hiring somebody with the DevOps claim does not answer this question.

I believe DevOps is orthogonal to your question, i.e., it changes nothing compared to a "classical" approach (or to hiring any team at all, not only for software development). You identify what your key needs are (for example, an "architect" who is able to structure large software systems; some "hacker" who is able to fix a Kernel driver if needed; a "tester" who likes to test and such; and finally maybe a "DevOps engineer" who excels in creating good CI/CD tooling).

But all of them need, to stay with your image, to be able to kick the ball. I.e., they all need to work together in the context that the team is working in. They all need at least some basic understanding of what the other team members are doing; if you have a strict CI/CD pipeline, then they all need to be able to develop in that frame; and so on.


gitlab -ci script not being updated on runner I have a problem I can't get my head around in gitlab-ci: I have a monorepo where I store some infra code (IaC), in this repo I have a script which allow me to lint the saltstack states code, as :

#!/bin/bash

# Std var
_hostname=$(hostname -f)

for file in $( find . -iname "*.sls" ! \( -path '*/files/orchestration_states/*' -or -path '*/files/reactor_states/*' -or -name "top.sls" \) ); do
    state_path_name=$( echo ${file%.*}| tr -d "."| tr "/" ".")
    if ! sudo salt-call --file-root=. --pillar-root=/home/gitlab-runner/salt_pillars/testBed/ --local --retcode-passthrough state.show_sls ${state_path_name}  pillar='{"tgt_client_name": "dummy-dummy.domain}' ; then
        echo -e "> Error found while validating state file: please fix it!\
        \n
        \nDetails:\
        \n - File(s) name(s): ${state_path_name}\
        \n - Pipeline ID: ${CI_JOB_ID}\
        \n
        \n - Job link: ${CI_JOB_URL}"\
        |mail -s "${CI_PROJECT_NAME} | Pipeline #${CI_JOB_ID} has failed" -aFrom:${_hostname}@spie.com "mail@domain.com"
        exit 1
    fi
done

In this script I need to update the --pillar-root=/home/gitlab-runner/salt_pillars/testBed/ to replace the testBed string by something lese, let's say toto.
But when I update the string inside the repo the runner keeps using the old version of this script (which failed because the path is not correct).

I can see the correct version of the script inside the repo, but the runner does not update this script when fetching the branch to lint the states.

Any ideas about why the runner does not update the script ?

Thanks

OK, as usual the error was between the keyboard and the screen... I did use the "try again" button on the jobs list, which, of course was making use of the same commit, and therefore the changed made to the script were not taken into account ...


Docker Deployment strategy for microservices project i am new to the ops side of devops and was hoping for some clarification. Specifically, if in a microservice based project you have multiple jars and each jar gets deployed in a VM on the same hardware, how are you not limiting throughput? if a server only has a fixed amount of processing power and multiple services are competing for those resources, it seems like you are loosing performance. can someone explain how this works?

I'll ignore your usage of the word "VM" here, since you have "Docker" in the title and "container" in the tags.

One would indeed not deploy each microservice in their own VM, which would be an incredible waste of resources.

But Docker is different. It is no VM. There is no virtualization. There is no extra layer between the processes running inside the Docker container and the host kernel. There is no significant overhead in the form of RAM usage. It is more or less the same as just running the process without a container.

Docker is based on the concept of namespaces. Modern Linux kernels have namespaces for all their internal data structures (from user IDs to network interfaces to process tables). All aspects regarding accessing kernel ressources/hardware/networking etc. are exactly the same, no matter whether a process runs inside a Docker container or on the host itself. That's the beauty of it, that's also why a docker container spins up very quickly with almost no overhead, and why the actual runtime also has no appreciable overhead.

There is one single difference, which is the filesystem; and this is not due to virtualization of any kind, but due to the "onion" style of layering many filesystems on top of each other. There are options when building images to avoid this, i.e. pack many layers into one, which would alleviate such problems (and conversely, you can mount unionfs-style filesystems in pure Linux as well, and those will have exactly the same performance overhead).

But incidentally his overhead is mostly negligible because something like your standard microservice will load once, and then just run. If you use storage in a container, you usually don't write it into the container itself (though you can), but in a separate volume mounted when spinning up the container; and those are like normal Linux mounts, i.e. they have no performance penalty.

TL;DR: Docker has in the most cases no relevant performance penalty at all; namespacing instead of virtualization is the reason why.


fatal: ambiguous argument 'master': unknown revision or path not in the working tree when I modified one of my cookbook that I tried to add a cookbook dependency in berks file

cookbook name, '=0.1.0', git: 'ssh://url:xxx/cookbook', tag: '0.0.1'

and then tried to converge it which throws an error

'fatal: ambiguous argument 'master': unknown revision or path not in the working tree.use '--' to separate paths from revisions, like this:git <command> [<revision>...] -- [<file>...]'on test

I don't have much clue to explain even..but I can post my explanation as we go. I appreciate your help thank you

The error looks like you're using a non-existent reference (branch/tag). Try specifying the correct branch and tag using the branch/tag options. You can also use the commit hash directly using ref option.

Also ensure that your git URL is correct (your code does not contain the xxx.git part).


Jenkins downstream pipeline jobs not always triggered as expected I have three separate Multibranch pipeline jobs set up on Jenkins, which I want to be called sequentially as downstream-jobs. Please note that, all these 3 jobs have separate git repositories!
Each job has its own Jenkinsfile.
The first job is always triggered manually by me through the Jenkins UI. At the last stage of the 1st job's Jenkinsfile, the second job is being triggered, as shown below. (The second job triggers the third job, in the same way)
stage('Trigger second job')
{
   steps {
       build job: 'second_job_name', propagate: true, wait: false
   }
}

Now this step always seems to be executed successfully when the first job ends, no errors or warnings show up on the logs. This should mean that the second downstream job has been called, but unfortunately that's not always the case:

Sometimes the first job triggers the second job, but the second job never actually triggers the third one, although it seems to pass the corresponding Jenkinsfile stage.
Sometimes the first job completes but never triggers the second job at all.

The only time this setup works as expected, is when there are new commits to all 3 git repos. Then all 3 jobs are triggered sequentially without issues.
Does anyone know the reason behind this inconsistent behavior?
Is it maybe because the jobs are based on different git repos?
Is there anything I can do to fix this?
Thanks

I suspect this job name is the issue:
build job: 'second_job_name'

In a Multibranch pipeline, there are typically folders in the job name.  For instance, if you have a Multibranch project named "my_project", a repo in that project named "my_repo", and a branch within that repo that you wish to build, let's say "master", then your build command would look like:
build job: 'my_project/my_repo/master'

If you only specify up to the project or repo level in your job name, but don't include the branch, instead of running a build for the branch, Jenkins will scan for updates to all branches within that project/repo, and build any new commits which it hasn't yet built.  This results in exactly the symptoms you described:

The only time this setup works as expected, is when there are new commits to all 3 git repos. Then all 3 jobs are triggered sequentially without issues.


How to exclude a certain linenumber of an Ansible playbook when testing it using Molecule? The aim is to exclude a certain line that contains a checksum and cannot be smaller than 80 characters.

--> Scenario: 'default'
--> Action: 'lint'
--> Executing Yamllint on files found in /tmp/ansible-firefox/...
    /tmp/ansible-firefox/defaults/main.yml
      2:81      error    line too long (153 > 80 characters)  (line-length)

defaults/main.yml

firefox_checksum: sha512:49d776cfb5f42c6e5ea1a55a80d9f6bad223080b16baa0d39de63534c25e68340091b4e16be5355d565f81291cb94fb996f03ae7e3e4c7a28021b0a0929daf58

According to the yamllint documentation you need to add a special comment to the line -- <string-to-be-excluded>  # yamllint disable-line rule:line-length

---
#defaults/main
firefox_checksum: sha512:49d776cfb5f42c6e5ea1a55a80d9f6bad223080b16baa0d39de63534c25e68340091b4e16be5355d565f81291cb94fb996f03ae7e3e4c7a28021b0a0929daf58 #yamllint disable-line rule:line-length


Data sent from Logstash via elastic output plugin not showing in Kibana, but file output plugin works fine - what am I doing wrong? I have an "ELK stack" configuration and, at first, was doing the standard 'filebeat' syslog feeding from logstash with the elasticsearch output plugin. It worked just fine.

Now I have added a TCP input port (with assigned "type" for this data so as to do if [type] == "thistype" to differentiate in filters), its own grok filter, and output to both elasticsearch with its own unique index name and document_type name and file. When the data comes in over the TCP port it writes the properly formatted data to the output file as expected in the file output plugin but no data is showing up in Kibana when I choose the given index. Kibana recognizes the index from my output configuration and also lists all the fields/keys I assign in the grok filter; however, again, no data is searchable. But it sees the property names (and even adds the propertyName."keyword" property itself on them!) The data is definitely being grok'd properly as it is, as I mentioned, appearing in the file plugin output.

What am I doing wrong here? My configuration is as follows:

input {
  tcp {
    host => "10.1.1.10"
    port => 12345
    type => "odata"
    id => "odata"
    codec => line
  }
}

filter {
  if [type] == "odata" {
    grok {
      match => { "message" => "%{QUOTEDSTRING:oid},\"%{WORD:oword1}\",\"%{IPV4:oclientip}\",\"%{DATA:otimestamp}\",%{QUOTEDSTRING:opl},%{QUOTEDSTRING:oos},%{QUOTEDSTRING:oua}" }
      remove_field => "message"
    }

    date {
      match => ["otimestamp", "YYYY-MM-dd HH:mm:ss Z"]
    }

    mutate {
      remove_field => "otimestamp"
      remove_field => "host"
      remove_field => "@version"
    }
  }
}

output {
# the if .. is here because there are other types that are handled in this output since I centralize the input, filter, and output files to three distinct files.
  if [type] == "odata" {
    elasticsearch {
      hosts => ["10.1.1.1:9200", "10.1.1.2:9200"]
      sniffing => false
      index => "odataindex"
      document_type => "odatatype"
    }
    file {
      path => "/tmp/odata_output"
    }
  }
}

Again, the grok filter is fine; the data is ingested via tcp just fine; and the file output plugin outputs the interpreted/grokd fields fine. Kibana sees the "odataindex" index and the fields (such as oid, oclientip, oua, etc.) from the grok filter. It just doesn't return any data when I do a search.

Any ideas? I am new to elastic/kibana/logstash and would also appreciate any tips on debugging these things.

Thank you in advance!

Being unfamiliar with Kibana, I was not aware of the time constraint on data on the default search/display of just 15 minutes. The data coming in was timestamped (@timestamp key) via the 'date' plugin with the original open date, NOT the time of the, via TCP port to elastic, insertion event; thus, no data was showing and I had no idea that by default only the past 15 minutes of data based on @timestamp were displayed. If I had read the part about the time constraint I would have known. So I just adjusted the time to go back infinite years and saw my data.

So, if anyone else is having this problem, it's probably because you created a time-dependent index and have not clicked the 'time' button in the top right corner and changed the time frame.

This, my friends, is why you read the manual!