compiled opencv copy to another host Every time I spawn a new host I need to compile OpenCV on Ubuntu following official manual. The problem is that it takes about 20-30 minutes depends on the VM type. Is it somehow possible to store the compiled OpenCV i.e. at S3 as a package or any othey way to avoid compilation?

create a VM image doesn't fit my purpose

If the end result would be entirely contained in a directory somewhere you could simply archive that directory after the installation on one machine and restore it on the new machine. It does work for some apps.

But after going through the Ubuntu 18.04: How to install OpenCV procedure and peeking into the makefiles I believe this wouldn't be applicable to OpenCV: its artefacts are spread in various places on a Ubuntu system.

But I think it's possible to significantly reduce the installation time by skipping the actual build process, which appears to be the bulk of the process. From the above referenced procedure (which I'll also use in describing the proposed solution):

  This process may take 30 minutes or longer, so go for a nice walk if
  you are able.

I'll re-use the referenced procedure for describing the proposal, with a few tweaks (I'd like to think that the tweaks would easily be portable to future/similar versions of the procedure):

In the initial installation:

after the build completes but before the Installing and verifying OpenCV step insert:

Step 4a: save the pre-built official OpenCV source tree to S3

Follow Copying from s3 to EC2 instance instructions for S3 access setup, then run:

tar cvfz pre_built_opencv.tgz opencv*
s3cmd put pre_built_opencv.tgz s3://<bucket_name>

Optional: you can remove the local archive copy now:

rm pre_built_opencv.tgz

In the subsequent installations:

replace Step #2: Download the official OpenCV source with:

Step #2: Extract the pre-built official OpenCV source tree from S3

Follow Copying from s3 to EC2 instance instructions for S3 access setup, then run:

cd ~
s3cmd get s3://<bucket_name>/pre_built_opencv.tgz pre_built_opencv.tgz
tar xvzf pre_built_opencv.tgz

Optional: you can remove the local archive copy now:

rm pre_built_opencv.tgz

in Step #4: Configure and compile OpenCV for Ubuntu 18.04:

skip the Configure OpenCV with CMake section
replace the Compiling OpenCV on Ubuntu 18.04 section with:

Enter the pre-built OpenCV build directory

cd ~/opencv/build

I didn't run it myself, but I think it should work, at least for VMs of the same type.

I'd 1st try the same even for different VM types. If that doesn't work then I'd re-run the initial installation for them, using a different archive filename.


How can I eliminate the need for an Ansible control machine? Ansible is great for deploying infrastructure, but it requires you to have a control machine. This adds the burden of manually setting it up and securing it, and then pulling down your playbooks and making connections out to your servers.

I would like a way to run my playbooks 'in the cloud' rather than on my own system. I am aware the commercial offering Ansible Tower, however as far as I can see this is not suitable for personal use.

One option is to run them using a hosted CI tool such as Travis-CI or Gitlab CI, however this feels wrong from a security point of view, as somebody who gained access to my Travis/GitLab accounts would then have access to all of my servers.

Which solutions are out there that meet this requirement? Does my GitLab CI proposal have severe security problems or is it an accepted solution?

The solution is to automate setting up the control machine, i.e., the VM. You can use, for example, Vagrant, Terraform or similar tools (I'll stick with Vagrant in this answer for this example, it's about the principle). With that approach, the VM is wholly defined by a text file ("Vagrantfile"), and can easily and repeatably be created and re-created. The Vagrantfile species a base image, for example some particular Linux distribution; networking etc.; and necessary command lines to set it up, going from there.

You can set up your VM (i.e., install Ansible) by just providing all necessary shell commands. In this particular example, this is probably enough. If you need a more complex VM, you can get kind of recursive, and configure the VM itself by using Ansible (or one of its alternatives).

Obviously, you will want to treat the Vagrantfile the same as your other configurations; i.e., commit it to your SCM and keep it up to date.

This is just an example: You don't need to use Vagrant or a VM; you can also install your Ansible inside a Docker image or whatever your cloud provider of choice provides as containerization.


What exactly is a buildpack in DevOps? I am familiar with the term buildpack from a platform called Heroku. But I am not sure if it's a Heroku-specific term, or if it exists more broadly in DevOps nomenclature?
When I google, I see plenty of articles about buildpacks that don't mention Heroku, so I guess it's more general. If that's the case, what exactly is it? And how does it differ from something like a docker image? Or are they the same things with different names?

They are the same thing with different names. It's a general term.


Should I implement Devops on a small project. Is it too big for it? Just like the headline suggested, is it too big to implement devOps on a private application I'm solely developing. I mean creating the pipelines, continuous testing and continuous deployment.

And if I must do it for learning purpose which is the main intent. Is it fun to carry out? How's the learning curve and won't I waste too much of my development time? I'm curious enough to go down the rabbit hole.

The question you need to answer is "would I find it entertaining to set up a CI/CD environment if not to learn from it then just for the fun of it?".
You may not need it at this point in your project or it may benefit you. That I can't tell from the limited information. Linux started as a personal project, too. I set up something similar for my own pet project and I've gotta admit I feel a little bit proud every time I compile and deploy with a single click.

Should you implement it? That's for you to decide.


Exit a Makefile target called by parent without throwing an error Let's say that I have 2 targets:

test:
    # ...
    @$(MAKE) dosomething
    @echo test
    # ...

dosomething:
    # ...
    @if [ -z "$(SOMETHING)" ]; then exit 0; fi
    @echo dosomething
    # ...

I would like echo test to execute even if $(SOMETHING) is empty and echo dosomething only if $(SOMETHING) is not empty (which doesn't work with the example, it's still executed).

exit 0 of course doesn't work here since make ignores non-negative exit in subshells used to execute instructions.

Is there any other way to skip the rest of the target without breaking further execution of test target? I could make the rest a single long instruction of else but wondering if there's a different command I'm maybe missing.

Makefile

all: test dosomething

test:
    @echo "hello"

dosomething:
    @if [ "a" = "a" ]; then\
        echo "world";\
        exit 0;\
    fi

Running: make will return:

hello
world


What are possible backup status in velero? I am using velero for k8s cluster backup. I was looking for possible backup status in velero.
I couldn't find any proper documentation on this. Can someone please help

As I see in Velero code (version: v.1.5.3), the backups can have the following phases:

New
FailedValidation
InProgress
Completed
PartiallyFailed
Failed
Deleting

see: https://github.com/vmware-tanzu/velero/blob/v1.5.3/pkg/apis/velero/v1/backup.go#L195
Restores have these phases:

New
FailedValidation
InProgress
Completed
PartiallyFailed
Failed

see: https://github.com/vmware-tanzu/velero/blob/v1.5.3/pkg/apis/velero/v1/restore.go#L184


Microservices Deployment Manager I was wondering how information on deployed applications are kept in real, production environments (yeah, I am a novice) . For example, in a microservices based set up, if  5 microservices-based  applications were deployed in production environments. Let's say each of those had 4 - 6 instances each, are there special software for keeping record of these applications? I know the records can be easily written into a database but I can imagine some better solutions out there.

According to the Cloud Native Landscape, you are looking for a Coordination & Service Discovery:

https://etcd.io/

  A distributed, reliable key-value store for the most critical data of
  a distributed system

https://github.com/Netflix/eureka

  Eureka is a REST (Representational State Transfer) based service that
  is primarily used in the AWS cloud for locating services for the
  purpose of load balancing and failover of middle-tier servers.

https://github.com/alibaba/nacos

  Nacos (official site: http://nacos.io) is an easy-to-use platform
  designed for dynamic service discovery and configuration and service
  management. It helps you to build cloud native applications and
  microservices platform easily.


Docker image registries We are using AWS EC2 container services and we are using Amazon container registry but the problem with container registry is that it's only available in one region although it's have URI but is docker hub repository is good to go with EC2 container services or using Amazon ECR.
Thanks

It depends on several factors.

If your images can be public or must be private. If you can make a public image, docker hub could work.

With ECR you will have a fine grain control of your permissions, integrated with IAM

On the other hand, ECR it's only available on one region so if you pull an image from a different region you will be charged for that traffic and the latency could be a problem so if you need a multi-region registry yo should implement that mechanism by yourself.

In brief, I wouldn't use public registry for production applications. You'll have more control with a private registry like ECR or other 3rd party solutions.


Clear Sonarqube Current Analysis I want to clear Sonarqube project current code analysis and make the project blank. Is it possible? I know we can clear all the previous analysis, but I want to clear the current analysis of Sonarqube project.

If you mean removing a "snapshot" of an analysis according to the most recent SonarQube documentation you should be able to this.  Also if you have access to the database SonarQube is using then you could remove the analysis directly from there.  That option may introduce other issues though depending on how SonarQube handles it.


Installing ingress controller, services and pods on K8s via Ansible I've 3 Debian-10 VMs on a bare metal Proxmox host in my network and installed via Ansible the roles geerlingguy.docker and geerlingguy.kubernetes with complete default settings. The kubeadm on the master node shows me the three nodes. So, I think, the installation is fine.
And what now? I'm new to kubernetes, I know there are configs that need to be applied. But how can I "see" them or "change" them via Kubernetes/Ansible. I don't want go back to the command line here. I know about the 'k8s' module - but the examples are not very helpful. A kubernetes should have an ingress controller and some services. Is there a good example playbook for installing Traefik/Haproxy/Nginx/whatever and maybe a small example hello-world-example for Kubernetes? Is there a "map" of possible values for all the YAML options (I used to learn from examples)
Sorry for the very "open" question. I'm searching for some days now and either my questions to Google are wrong or Google sends me always to the same sites with either very "basic" informations that ends by installing the Kubernetes or very "heavy" information, where the author is far away from easy to read.
To make the question a litte bit more concrete.

The 3 VM's with IP 192.168.10.10-192.168.10.12 are running with Kubernetes 1.17.9 on Docker 19.03.12
I've installed Helm v3.2.4, some repos (Traefik, Nginx, Stable-Kubernetes-Charts)
I'm able to install with Helm some -don't know - are that pods or services?
I'd like to add a simple hello-world HTTP service, like that from gmolaire (thanks!)
That service is exposed to one of the three VMs on port 80

When I run that tutorial with step kubectl create -f ingress.yaml
I'm getting the error
Error from server (InternalError): error when creating "ingress.yaml":
Internal error occurred: failed calling webhook "validate.nginx.ingress.kubernetes.io": Post https://ingress-nginx-controller-admission.ingress-nginx.svc:443/extensions/v1beta1/ingresses?timeout=30s: dial tcp 10.105.231.128:443: connect: connection refused

This might be, because I cannot run kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/mandatory.yaml it ends with a 404.

Since your new in this platform, I would suggest you to explore in the bellow order (assuming you want to use nginx as the load balancer ingress controller):
1- This tutorial is the most complete to get started with k8s. I strongly suggest you to follow it through to understand the jargon and the working components before going to the next steps.
2- Take a look at this example where they go through creating an nginx application in a k8s cluster.
3- Look at this tutorial where the author describes the key concepts and working operation of an ingress controller.
4- Finally, look at this to start setting up your ingress controller from the official doc page. Other resources are available to allow you to setup as much as you'd like.
Extra: Here is the official k8s kubectl cheat sheet.


What's the difference between travis-ci.org and travis-ci.com? A few of the GitHub projects I contribute to use Travis CI for build testing. However, I've noticed that some projects use  travis-ci.org for build testing, while others use travis-ci.com. Both sites seem to function identically, even down to using the same UI.

What's the difference between travis-ci.org and travis-ci.com? Why do some projects use one over the other?

Travis CI originally created two separated platforms to differentiate between private repos / paid (travis-ci.com) and public repos / free (travis-ci.org).
However, as of May 2018, new users and projects (both private and public) should only use travis-ci.com (see this blog post). Note that travis-ci.org will be closed down completely on December 31st, 2020 (see this newsletter).
Although still in beta—which is a bit weird since travis-ci.org will be shut down soon—Travis CI provides a well-documented migration guide.


Need help getting documentation of Jenkins I need a little help over here:
In every site I seek info to get started using Jenkins (i.e.: devopscube) they say is very easy to understand, and theoretically is really simple once you have look into Jenkinks Architecture, but I'm having troubles to understand some concepts and I need a good site/pdf/free book (I'm from Cuba and my access to internet is a little expensive and no way to pay for courses and books :-( ) to read and fully understand, for example:

When creating the Jenkinsfile for a pipeline (using groovy) how is the code structure made? Where can I find some examples starting from the easiest to the complex ones?
What are the actual differences between stages and nodes on Jenkinks file?
Where can I learn how to make pipelines from scratch (but not just copy/paste from somewhere, understanding what I'm doing it), i.e.: I want to run a code on the server where I have my Jenkins master every time a fellow coder make a successfully git push

I downloaded the Jenkis.io's Jenkins User Handbook but is a bit hard to fully understand the questions I made right up.
Thanks

When creating the Jenkinsfile for a pipeline (using groovy) how is the code structure made? Where can I find some examples starting from the easiest to the complex ones?

The files typically ending in *.jenkinsfile can be made in any text editor or IDE. For examples the best place to look is the Pipeline Examples page. There is also a github repo with more examples.

What are the actual differences between stages and nodes on Jenkinks file?

Nodes - are physical machines or virtual machines that run Jenkins as worker. They can be different architectures.
Stages - These are best thought of as the recipe:

Mix the ingredients
Add water
Mix again
Bake

Each stage is an action that Jenkins needs to take.

Where can I learn how to make pipelines from scratch (but not just copy/paste from somewhere, understanding what I'm doing it), i.e.: I want to run a code on the server where I have my Jenkins master every time a fellow coder make a successfully git push

The first two links I gave are the best way of starting but it's purely subjective. I'd suggest starting simple and then building up from there. In the document you referenced there is also a section on Pipelines which might be the missing step for you.


JAVA_HOME error in Sonar Scanner for MSBuild I have set up a sonar scanner MSBuild in Jenkins, but suddenly I'm getting below error.

ERROR: JAVA_HOME exists but does not point to a valid Java home
       folder. No "\bin\java.exe" file can be found there.

Below is Environment variable for Java Jdk

JAVA_HOME=C:\Program Files\Java\jdk-12.0.2

How to solve this Issue?

I have added "C:\Program Files\Java\jdk-12.0.2" as value of JAVA_HOME environment Variable and its working. I don't know exactly this is the correct solution OR Not, but when I have checked sonar-scanner.bat file then it's taking %JAVA_HOME%/bin/java.exe location of Java.


How to patch AssemblyInfo in TeamCity for .net core 2.1 projects? In the .net core 2.1 project, there is a *.csproj file, that contains assembly information:

<PropertyGroup>
    <AssemblyVersion>2019.1.15341.0</AssemblyVersion>
    <FileVersion>2019.1.15341.0</FileVersion>
    <Version>2019.1.15341</Version>
</PropertyGroup>

In Standard .net projects there were Properties/AssemblyInfo.cs which contained the Assembly information, but in the .net core, there is no AssemblyInfo.cs, all Assembly information are in the csproj.

How can I change the Assembly Version, File Version in Assembly Info Patcher in TeamCity for the .net core projects?

You can use File Content Replacer build feature with predefined templates for .NET Core
https://www.jetbrains.com/help/teamcity/file-content-replacer.html#.NET+Core+csproj+templates


How do i print the list of jobs per plugin in Jenkins I'm trying to list the jenkins jobs per plugin is being utilized. I see there's a plugin in jenkins, https://github.com/jenkinsci/plugin-usage-plugin, which displays the plugins usage in the Jenkins UI. But how do i get the same output from the groovy script?

They quick and dirty answer is to "Display jobs group by the build steps they use" as updated below.
import hudson.model.*
import hudson.tasks.*

//All the projects on which we can apply the getBuilders method
def allProjects = Jenkins.instance.allItems.findAll{ it instanceof Project }

//All the registered build steps in the current Jenkins Instance
def allSteps = Builder.all()

//Group the projects by the build steps used
def projectsGroupBySteps = allSteps.inject([:]){
   map, step ->
   map[step.clazz.name] = allProjects.findAll{it.builders.any{
        it.class.name.contains(step.clazz.name)}}.collect{it.fullName}
   map
}

//Presentation
projectsGroupBySteps.sort().each{
    println "--- ($it.value.size) $it.key ---"
}
println ''

println 'Occcurences:'
projectsGroupBySteps.sort().each{
println "--- ($it.value.size) $it.key ---"
    it.value.each {
        println it }
    println ''
}
return

Note that a job has multiple sections: builders, publishers and buildWrappers.
Depending on what you are looking for, replace
[allSteps = Builder.all(), allProjects.findAll{it.builders.any ]
[allSteps = Publisher.all(), allProjects.findAll{it.publishers.any ]
[allSteps = BuildWrapper.all(), allProjects.findAll{it.buildWrappers.any ]

Plugins may also be involved in the SCM section and as properties (eg: hudson.plugins.buildblocker.BuildBlockerProperty, com.sonyericsson.jenkins.plugins.bfa.model.ScannerJobProperty)
Obviously, same limitations of plugin-usage apply: does not scan pipeline steps plugin usage or build steps nested within an org.jenkinsci.plugins.conditionalbuildstep


nginx rewrite with proxy_pass I want to proxy requests with nginx to different domain and also rewrite some part of the path. For example
http://origin.com/ - > http://target.com/_plugin/kibana
How to accomplish such automated addition of _plugin/kibana part to any route that I request and forward to my target server
server {
    listen 80;
    server_name _;
    location / {
        proxy_set_header X-Forwared-Proto $scheme;
        rewrite ^(.*)$ /_plugin/kibana/$1 break;
        proxy_pass https://my-es-custer-in4lf3ruqx6shv2ndhju.us-east-1.es.amazonaws.com/;

    }
}

Should I have different location blocks or should I have a rewrite directive on a server level for this to work properly?

You do not need to use rewrite for your case, since you want to append _plugin/kibana to any routes.
Assuming the nginx config is for http://origin.com/:
server {
    listen 80;
    server_name origin.com;
    location / {
        proxy_set_header X-Forwared-Proto $scheme;

        # http://origin.com/req/1/2 will go to http://target.com/_plugin/kibana/req/1/2
        proxy_pass http://target.com/_plugin/kibana;

    }
}


Which Release Management aspects help explain the difference between Waterfall and Agile? When explaining DevOps to somebody, it happens that a question comes up like:

  How does Release Management using the Agile methodology differ from Waterfall?

So what kind of criteria can you use to explain these differences to such audience?

IMO DevOps is culture, much like Agile (without choosing an agile methodology.) Therefore you don't "do" DevOps.
You "do" a release methodology called Continuous Delivery as part of a DevOps Culture. (full disclosure, I don't think I've ever referred to CD as a release methodology before, but in my jetlagged state I think it works)
If you'll buy that, then here's the definition of Continuous Delivery from one of the people who wrote the book by the same title, Jez Humble.

Continuous Delivery is the ability to get changes of all types—including new features, configuration changes, bug fixes and experiments—into production, or into the hands of users, safely and quickly in a sustainable way.
Our goal is to make deployments—whether of a large-scale distributed system, a complex production environment, an embedded system, or an app—predictable, routine affairs that can be performed on demand.
We achieve all this by ensuring our code is always in a deployable state, even in the face of teams of thousands of developers making changes on a daily basis. We thus completely eliminate the integration, testing and hardening phases that traditionally followed “dev complete”, as well as code freezes.

So then, you can work in an Agile methodology, having software you can demonstrate to the business, making sure you're doing proper automated testing, reacting well the change and all of the things that make it better than waterfall. All too often that doesn't mean you could actually deploy it to production.
You end up with something like this:

So the software will (probably) be better when you're done then if you didn't have some sort of iterative approach, but you don't really know because real users haven't ever seen it.
What you really want is something that looks more like this:

Every iteration, something gets deployed to production. So, the software is deployed. If you decide to create downloads, open up the web server, or however you get software into the hands of users it's released.
What the heck!? I asked about DevOps! Nobody asked about Continuous Delivery!!
So what does DevOps have to do with this?
It's very, very hard (approaching impossible) to truly have your software in a state where you could deploy it whenever you want unless that team is working in a DevOps culture. A culture where System Admins, DBA, SREs, Security People, Devs, QAs, etc., etc., are all part of a single team and not siloed part of an organization with handoffs.
Note:
About part of a comment posted to this answer, which was like so:

About your "... software in a state where you could deploy it whenever you want ...": that reminds me about "automatic pilot" software (in a plane) ... My favorite question about that: "Imagine an update is applied to such software ... How would you feel about doing so inflight ... While you're in onboard?".

I love that question (in bold, in the above quote)! The idea of "is it really ready?" is something I rant about all the time - blog. IMO it's vital that you're confident in security, performance and other too often "secondary" tests in order to practice CD. The features are done when they're done, but hackers are always there.


How to decrypt Jenkins passwords from credentials.xml? I've taken over the project where a lot of Jenkins credentials has passwords or passphrase strings which I need to know in order to progress with the project, unfortunately these weren't documented anywhere.

I've checked the credentials.xml file where these credentials are stored, but they're in not plain text, e.g.:

<passphrase>{AAAAAAAAAAAANzxft/rDzyt8mhxpn3O72dxvVqZksL5vBJ4jNKvAjAA=}</passphrase>

Note: I've changed it slightly for privacy reasons.

How can I decrypt its original password based on the string above?

Luckily there is a hudson.util.Secret.decrypt() function which can be used for this, so:

In Jenkins, go to: /script page.
Run the following command:

println(hudson.util.Secret.decrypt("{XXX=}"))

or:

println(hudson.util.Secret.fromString("{XXX=}").getPlainText())

where {XXX=} is your encrypted password. This will print the plain password.

To do opposite, run:

println(hudson.util.Secret.fromString("some_text").getEncryptedValue())

Source: gist at tuxfight3r/jenkins-decrypt.groovy.

Alternatively check the following scripts: tweksteen/jenkins-decrypt,
 menski/jenkins-decrypt.py.

For more details, check: Credentials storage in Jenkins.


Should we build our application inside a docker container, or copy it in? I am dockerizing our applications to have the environments recreatable. As a process of dockerizing, we will be required to do the below steps:

Build the application.
Prepare the environment as the container.
Deploy the application to the container.

Is it fine to have the application build, inside the docker, as part of docker building? In that way we need not install the build dependencies in local.

But the trade-off is, we have to build the complete docker for the even a small code change that we build. I understand this can be optimized to a level with multi-staged build.

I understand this can be optimized to a level with multi-staged build.

This is an understatement - you do not "optimize" that, but that's just the way you do it.

To answer your question: yes, by all means run the build inside a docker container. How else would you be sure that at build time every library or dependency is the same? There is no reason whatsoever not to do it.

Of course you need to pay a bit of attention when writing the Dockerfile. Ideally, whatever you change in your source code day-to-day, the only thing that happens when you re-run your build is the very last step of actually compiling your code and doing whatever it is you do with it afterwards (creating your deliverable archive...).

You need to use multi-staged builds afterwards to get rid of all the intermediate steps, and especially all the compile-time-only components (and it's pretty easy as well).

Note that there's usually some parts of your setup which are used by both the compile-time and run-time (unless you are just creating a .war file and sending that off to a webserver). I.e., if you have a ruby-on-rails application, the "ruby-on-rails" part could well be shared between compile-time and run-time. You want to make sure to put everything that is shared in this way very early in the process, and have a clean cut-off point that you can use as a FROM later on, copying the compiled files in as needed.


Continuous deployment strategies vs DevSecOps How do you implement security requirements in CI/CD while taking care of security requirements? What are best practices here?
Security requirements may differ, let's assume this simple level of security:

Direct administrative access to target system shouldn't be given to the CI/CD system
CI/CD system needs to be able to initiate deployment and show it's success/fail status (could be determined by a white box test routine i.e. without admin access rights placed on the CI/CD side)

As you stated, requirements will differ based on company and industry policy; for us, the adjustment from "separation of duties" to automated deployments has been rough. As interim steps, we've started:

Requiring multiple "sign-offs" before deployment; this is accomplished by both reviews for the PR in Azure DevOps and an audit log in Octopus.

Monthly review of deployments by higher-ups.  This is mostly a rubber-stamp activity, but it satisfies auditors.


Jenkinsfile for monorepo & monobranch I'm currently working on a Microservice architecture-based project and writing Jenkins Pipeline for it.

The project structure is as follows:

Web Service (master)

Jenkinsfile
Service 1/

Jenkinsfile

Service 2/

Jenkinsfile

Service 3/

Jenkinsfile

Within each Service, there will be a Jenkinsfile that contains information on building and testing for the specific service.

How do I use the Jenkinsfile on the base directory to build and test each individual Services before running integration test on a cross service level ?

I've been trying to look for ways to implement this, but the build seems to target project level, it is said to be able to process relative file path but I haven't had much success with it.

I have tried loading, but load seems to be made for groovy file, which doesn't seem to fit the requirement.

I have sometimes used the pattern where all pipeline things go in a .pipeline/ directory of the app, including the Jenkinsfile. I would, in your case, add several pipelines (ie, Jenkinsfiles) - one per microservice as you propose, and one for the web app.

Then, you define independent multibranch pipeline jobs, based on the location of the pipelines. In other words, you create 5 multibranch pipeline jobs, (one for the web app, one for each microservice), and attach each job to the relevant pipeline file in your repsitory. In this way, they will be tested independently.

The important thing is to properly specify the correct triggers. The microservices will be triggered on changes to code in their path. They may also have as post-build actions the triggering of the web app pipeline, for integration tests, etc.


Storing Code Signing Certificate I own a public and open-source Windows Forms/C# application in a Github repo that I want to start code signing with my certificate through Azure Pipelines CI.  I have set up my pipeline and it is building fine, the next step is to integrate automatic code signing, which until this point I have been doing manually. I have taken the following steps:

Certificate has been added to Secure File storage
Code Signing Extension Installed https://marketplace.visualstudio.com/items?itemName=stefankert.codesigning
Task variables have been created and marked as secret

After adding a task, the YML generating is:

- task: codesigning@2
  inputs:
    secureFileId: '$(certName)'
    signCertPassword: '$(certPass)'
    files: '**/*.exe'
    timeServer: 'http://timestamp.digicert.com'
    hashingAlgorithm: 'SHA256'

So, saving this, it is obviously going to update my azure-pipelines.yml in my repository. Is it possible to perform code signing in Azure Pipelines without exposing any information in the repository?  Is this considered best practice for code-signing in Azure Pipelines?

From what I can see you have done everything correctly, you have stored both the certificate material (i.e. the PFX) and the password for the PFX in a secure location that presumably no one else can access.  Assuming you don't give anyone else permissions on your Azure tenant and you don't blindly accept pull requests that change azure-pipelines.yml then you are in a good place.

If you are feeling paranoid, then you could create a second pipeline using a private repository that is triggered when the previous pipeline completes successfully.  However, IMHO, I would carefully consider the risk you are exposing yourself to against the complexity and fragility of trying to tie two pipelines together.


Is a decentralized Docker swarm possible? I want to run services in our office in containers. The problem I have is that we do not have a spare machine available to act as the dedicated host.

So I am considering creating a swarm using everyone's desktop workstations as the workers. But if the manager node (my computer) is shutdown then the entire swarm stops.

I envisioned a swarm where every worker can act as a manager if needed. When the manager dies, another worker assumes the role as manager and the service continues running uninterrupted.

Is it possible to create a decentralized swarm where any available node can assume the role as a manager as needed?

By Docker Swarm design it is dencentralized, so you can promote any node to a manager. Normally, 3 managers are enough, for landscapes with thousands of nodes 5 are enough.

https://docs.docker.com/engine/swarm/admin_guide/


Unable to get Helm to work with ECR I'm using Helm v3 with AWS ECR on a Kubernetes cluster that was creating using Kops. I have checked and the role kops assigned to the nodes does have ECR permissions. When I deploy my helm chart I just get imagepullbackoff when doing kubectl get pods in the correct namespace.

Below is some of my helm chart code, I hope this is everything that's needed to debug this issue. BTW I have looked around on SO and google, both come up short on ECR with Helm and nothing for Helm 3 - was released recently.

Inside my values.yaml

image:
  repository: <number>.dkr.ecr.us-east-1.amazonaws.com/apigw
  # tag: latest
  tag: ef86dfe0b703
  pullPolicy: IfNotPresent

Note that the tag there is a git commit ID and spinnaker will change that on each deployment so here being static is just to test.

I was following this guide and as a result changed some things that might have effected it.

https://aws.amazon.com/blogs/opensource/deployment-pipeline-spinnaker-kubernetes/

Main one I think that makes a difference is

 imagePullPolicy: "{{ .Values.image.pullPolicy }}:{{ .Values.image.tag}}"

to

imagePullPolicy: {{ .Values.image.pullPolicy }}

However... If I change it back I get this error when trying to apply the helm chart

Error: Deployment.apps "apigw" is invalid: spec.template.spec.containers[0].imagePullPolicy: Unsupported value: "IfNotPresent:ef86dfe0b703": supported values: "Always", "IfNotPresent", "Never"

Why are you adding the image tag to the pull policy? you dont need it there, just leave it like this: imagePullPolicy: {{ .Values.image.pullPolicy }}


What considerations should be taken into account for a good log file format? We have a practical question I struggle to answer :

  What is a good logfile schema?

Here are the caveats:

There are several workloads, each writing more than one log file
Applications are on more than one language
All applications are in the cloud, but on different cloud providers
All logs are sent to a central dashboard (e.g. Kibana) which can be queried.

The requirements which this schema should fulfill are:

must be queryable (structured)
should be able to determine failure patterns (correlations)
no proprietary file formats

For a particular case, we may want to set a trigger of some action based on a particular query.

Assuming, e.g. we write logs in JSON format, what would be a good schema to apply to the entries, to satisfy the caveats and requirements?

If you are dealing with application logs, here are the three broad categories of information that a good log file should cover:

basic context (timestamp, log level, application name, source file name, source module/function, source line number, log message)
server context (host name, data center name unless host name has it encoded, cluster/pod info, container info)
user context (request URI, username or a hash, request ID etc)

If we are using JSON for logging, care must be taken to make sure that the keys are short enough to reduce the redundant overhead and also make it easier to query.  Using all lowercase and underscores for keys could be a better idea because some of the log aggregators like Splunk are case sensitive.

In the structured format, stack traces become a little tricky to handle.  One solution is to use a marker line to indicate the end of a long line rather than just the newline.


Updating a TFS repo with Jenkins I want to checkout a TFS repo, make some changes to a file and send the changeset to TFS.
What are some ways to accomplish this with Jenkins?

I solved this using team-explorer-everywhere. We made a container that clones the code using it edits it an sends it back up. https://github.com/microsoft/team-explorer-everywhere


What is the difference between a jail environment in linux and docker container? I am curious about difference between a jail environment in linux and Docker.  What separates these two technologies?

While both Docker and jails have some similar functionality, they are built for different purposes.

The term "jail" comes from the FreeBSD distro.  A lot of people will also refer to a chroot as a jail.  At its core these jails virtually change the root directory for a process and its children.  This means that these processes are contained (jailed) within this new directory and should not have access to folder outside of it.  Jails are a great way to run a process that you want isolated from the rest of the system.

Docker Linux is an entire platform for running containers.  At its core it does this by using Linux namespaces and Linux Control Groups (source).  Docker is built around packaging these containers and allowing them to easily be shared with other systems.

Ultimately, both technologies are isolating processes.  However, Docker does this with the intent of making the isolated processes portable and repeatable.


Do you need to run it beside a VPN "OpenVPN service" ?

I think using below would solve the issue for clearing any unused networks by at least 1 container.

$ docker network prune


LAMP in docker must be multiple containers or one? I newbie in docker and I cant find answer what solution is more "right" for LAMP stack:

1) One "standalone" container that will consist from Apache, PHP, MySQL, etc

2) Special container for each component, build and run project via docker-composer.

Im tried to take a look on foreigh realisations of LAMP in docker, but found both variants. Maybe my question dont have right answer and for each project solution must be individual?

I would go with solution 2: use docker-compose. The idea is that with solution 2 you can easily update mysql for example without having to modify and rebuild your dockerfile. you just have to update the version of the mysql image in the docker compose file.

I found with a quick search on github this repository https://github.com/sprintcube/docker-compose-lamp

Note: for production, use terraform script rather than docker-compose


Sarbanes-Oxley (SOX) Versioning Strategy When dealing with publishing SOX application artifacts what is the best strategy for versioning SNAPSHOTS?

I'm currently following the semver pattern <major>.<minor>.<patch> and wondering if it is necessary to get granular with the version strategy and append something like a git hash to the end? e.g. <major>.<minor>.<patch>-SNAPSHOT<githash>

Or can I treat SNAPSHOTS as a SNAPSHOT should be and drop the hash all together?

My argument for the latter is that SNAPSHOTS are not cut as release candidates (rc) until they actually enter the release pipeline and therefore can be clobbered, especially when following a mature CI pattern where changes are continuously merged to master following early testing validation.

In addition to this, typically in an organization, a change request is only opened and tracked at the time you are performing a release.

(additional convo happening here)

I suggest to follow the principles that are defined in the Semantic Versioning 2.0.0 documentation.


Consumption Plan Function App to Firewalled Storage problem I am trying to connect a (python) function app to a storage resource. The storage has a public container that is accessed by several company internal HTML/CSS pages ($web).

Unfortunately, this container should contain sensitive information. So in order to prevent leaking data my choice was to configure the firewall to only allow the IP range that is used by the company. So far this seemed to work fine.

However, no I tried to automatically generate files (blobs) using a function app. Since only very few executions are necessary, there is no real justification of using anything but the "Consumption" plan of azure function apps. However, apparently this is not part of the trusted azure services, neither has a fixed IP nor can the consumption plan be set-up in a VNet, and thus it could not pass the storage containers firewall.

I tried adding the function app as a role with contributor rights. I tried adding the outbound IP addresses of the function app (to be found in the resource manager) to the firewall. But both things did not work.

Is there any workaround for this?

Consumption Plan Function Apps are not able to be joined to a VNET.

Premium Function Apps however, can be.

If the container has sensitive information then you should not make it public, IP restriction shouldn't be your only protection.

Use another service (such as a function app) to determine a user's identity: then grant them a SAS token for the container.


How to pass a /dev/disk device on macOS into Linux Docker? I've connected external HDD via USD on macOS, and I'm running Ubuntu's Docker container as:

docker run -it --device=/dev/disk3 --privileged ubuntu bash

but the device is not present in the container:

# ls /dev/disk3
ls: cannot access '/dev/disk3': No such file or directory
# fdisk -l
Disk /dev/sda: 59.6 GiB, 63999836160 bytes, 124999680 sectors
Device     Boot Start       End   Sectors  Size Id Type
/dev/sda1  *     2048 124999679 124997632 59.6G 83 Linux

Here is another attempt:

$ docker run -it --device=/dev/disk3:/dev/sdb --privileged ubuntu bash
# ls /dev/sdb*
ls: cannot access '/dev/sdb*': No such file or directory
# fdisk -l
Disk /dev/sda: 59.6 GiB, 63999836160 bytes, 124999680 sectors
Device     Boot Start       End   Sectors  Size Id Type
/dev/sda1  *     2048 124999679 124997632 59.6G 83 Linux

Here is the device which I'm trying to add to the container (as reported on macOS):

$ diskutil list /dev/disk3
/dev/disk3 (external, physical):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:     FDisk_partition_scheme                        *1.0 TB     disk3
   1:                      Linux                         1.0 TB     disk3s1

$ sudo fdisk /dev/disk3
Disk: /dev/disk3    geometry: 121601/255/63 [1953525167 sectors]
Signature: 0xAA55
         Starting       Ending
 #: id  cyl  hd sec -  cyl  hd sec [     start -       size]
------------------------------------------------------------------------
 1: 83 1023 254  63 - 1023 254  63 [         2 - 1953525165] Linux files*
 2: 00    0   0   0 -    0   0   0 [         0 -          0] unused
 3: 00    0   0   0 -    0   0   0 [         0 -          0] unused
 4: 00    0   0   0 -    0   0   0 [         0 -          0] unused

Similar issues are posted here:

How do I access a USB drive on an OSX host from inside a Docker container?
How to mount a device of host to host in a Docker container?

However, I cannot mount it on macOS host, as the partition is formatted to Linux file system (ext4) which is not supported. That's why I'm trying to pass through the device to the Linux Docker container, but the device isn't shown.

What am I missing?

Docker was built for linux, it runs on Mac through HyperkIT, which is a lightweight hypervisor. This means Docker only sees devices connected to the hyperkit hypervisor, rather than the devices connected to the Mac.

Unfortunately, Hyperkit has issues with USB device passthrough from your MacOS to the hypervisor (read more here) so the --device command won't be much use to you here. You have three options,

Option 1: Mount the ext4 drive to your Mac (not recommended)

Whilst your Mac doesn't natively support ext4, there are ways around this. I have installed ext4fuse on your Mac by following this tutorial. When you have mounted your device, add it to your ubuntu container with the -v flag,

docker run -v <Your_volume>:<dest_volume> -it ubuntu bash

Making sure that the directory of your volume has been added to the file sharing folders in Docker.

Option 2: create a docker-machine using the virtualbox driver and add the device to the virtualbox vm (still not recommended)

Follow this great tutorial here which goes through installing a docker machine with a virtualbox driver and mounting the USB stick.

Option 3: Run Docker on linux (recommended!)

I wouldn't recommend using Docker for Mac. It was built for linux and if you don't switch to linux you'll waste your time troubleshooting issues like these for no real reason.

I've got a mac, but run a few ubuntu cores in virtualbox and then install docker on top of them, that way I can play around with docker starm too (which you can't with docker Mac unless you use docker-machine!


Dev environment for legacy Visual Studio code? I work at a small company that has several legacy projects that all require different versions of visual studio to run. The company has servers on site and I have a small cloud budget. I want to set up the projects such that it doesn't matter which visual studio the project is opened it, it should still run the same. The projects are all asp.net, run on IIS, and use SQL Server. I'd like to know where to get started, such as should I try to containerize the applications? Should I set up some sort of VM Image with the projects on it? Any help would be appreciated!

... legacy projects that all require different versions of visual studio ...

Don't you just love Visual Studio?

  ... I want to set up the projects such that it doesn't matter which visual studio the project is opened ...

If your "legacy" goes back beyond .Net 2, then you're out of luck.

New, shiny versions of Visual Studio won't open .Net 1.* projects and will shrink away in horror from anything that needs anything created in 'Studio 6 (except, possibly, C++, the one language that's survived .Net-ification largely unscathed).

Most of the newer stuff will happily co-exist, but you'll probably need to think about  containerising the earlier stuff.


How do you handle the huge number of possible permutations of Dockerfiles / Teraform files? We have adopted Docker and Terraform as the foundation forour devOps.
We have a basic version of it up and running today.

Dockerfiles live in each projects git repository and Terraform files
are more centralized.

The number of projects we manage is increasing as do the number of environments, data seeding To make things more interesting we are also
moving toward splitting services up into microservices where that makes sense.

So for each project, we have options like:

What destination are you targetting (dev local, dev cloud, test (several variations), staging prod)
What version / tag of the source code should it build from.
What data should be seeded?
Who should be notified?
What infrastructure should it use? (We want to be able to allow "prod infrastructure) to be available for developers to spin up in dev cloud for debugging and testing) (along with several other combination)

There are a few other parameters we are considering and all in all the
number of possible permutations become very high.

There is no way we can cleanly keep order of that many Dockerfiles.

So I am looking for a tool that helps us keep track of all the different configurations and makes it easy to find them.

Ideally, inside my head, I am picturing a wizard-based web app, where you enter in the parameters and it executes the necessary scripts automagically.
This would also contain logs of what has been done, what environments are running and a few other things.

So far I havent had much luck.
I know a lot of companies must have solved it and I must be thinking about it
wrong, or there is a big opportunity here.

Is there such a tool? How do large companies solve this?

So let's imagine you have 20 microservices, each in a separate repository. Each microservice needs to be deployed into 5+ environments: dev local, dev cloud, test, staging, prod. Understandably all configuration will be different in each environment. However, you don't need to put it into the Dockerfile - in fact you don't put any configuration into the Dockerfile. Your app gets all required parameters from environment variables, or may be from a config file, if it gets too complex.

So you would have one Dockerfile per project, and if your developers need to spin it up locally, they will just use docker-compose, which has all environment variables set up, and maybe a config file mounted as volume. In stage envs the configuration will be totally different, but the Docker image will be the same. Your app will need to be able to read this configuration at start up time, and bail out early if something doesn't work.

Read https://12factor.net/config for some inspiration on how to configure your app.

Now, provisioning the infrastructure with Terraform would follow the same approach. You would have a generic module which provisions a set of some resources, and it would accept a bunch of parameters which would depend on environment. You can use workspaces to manage this quite easily.

As for the last question, large companies solve this with a CI/CD system. This can be Jenkins, or Spinnaker, or any of the other tools, whichever you prefer.


Where is Jenkins Pipeline script stored and can I control it? Several tutorials I'm referencing to learn about Jenkins Pipeline scripting guide one to create Jenkins Pipeline jobs as follows:

Click New Item in the left pane.

Click Pipeline and enter an item name.

Fill out the little Pipeline textbox with your script.

Question: where does this script physically reside? I don't see anywhere to save a filename -- am I creating foo.sh or /path/to/bar.py or /somewhere/../else/baz.pl? What is the filename and realpath of the script that I type out in that tiny little text box?
Question: is it possible to control the name/location of the Pipeline script that I type in that textbox? E.g. can this script be something version-controlled in some git repo? And specifically, when this job has no build triggers. I can't help but feel that anonymous script edited in that tiny little text box might be limiting for larger and more complicated scripts.
Sorry for the multiple questions; I've tried to limit them and keep their scope all related.

The good news, is that you absolutely don't have to use that little box to manage your pipeline script.
Jenkins also allows you to source your Jenkinsfile from an SCM repository like GitHub, and will even let you automagically import branches as separate jobs with separate Jenkinsfiles per branch using multibranch pipelines.
Pipeline scripts are in a language known as Groovy, not Perl or sh or similar. The definitive guides to Groovy are (of course) the Jenkins reference guide and the Groovy reference guide.


WorkItem Tracking error when performing clone operation in Azure Devops Today I received an error when trying to clone test plans in Azure DevOps web portal.
Steps:

In Visual Studio, I select the Test Case option
I find the Test Plan I want to clone, select it , and then select the "Clone test plan" button.
In the "Clone Test Plan" window, accept all defaults and click Clone.

Result: an error is received: "Clone operation encountered a WorkItem Tracking error while processing test case xxxx. Retry the operation. Error details: TF51535: Cannot find field 25."

It is extremely unfortunate that this issue happened today while I am trying to clone a large test plan. I will have to manually copy the hundreds of tests in order to have the new test plan ready for Monday morning.

This is an indication that someone removed a field from work item configuration that was represented by 25.  It's more of an issue of clean-up not being done when fields are removed.  See more about this error on Microsoft's Developer Community.

This other item in the community suggests a different problem when using a particular extension and a fix has been supplied.


Are there automated quality checking tools for Ops code In the dev world, we have several tools available to us for automated quality checks.  We have unit tests that run on each build, Sonar to check for code quality, metrics being sent to Grafana and Okta with automated alarms, etc.
I'm not very familiar with the ops world; are there similar automated tools available for our ops code?
For Terraform scripts, Dockerfiles, Helm/K8 scripts, Cloudbees configurations and other ops code, the concerns I would like to see automation for are:

Syntax checking
Code smells / anti patterns
Routing / Connection checking for accuracy
Security best practice checking (don't expose these ports or include these credentials,or grant these permissions kind-of-a-thing)
Appropriate provisioning (too many or too few resources allocated for the size of the task)
High cost warnings
Unit test-ish tools to ensure we aren't breaking the build in more subtle ways than "did it deploy successfully or not"

There's likely a lot more that I don't even know to look for given my lack of experience.  Are there support tools that will analyze our ops code that could provide automated support in this fashion?

It's difficult to provide a definitive answer given the scope of the question, but the  goal of testing infrastructure is both laudable and seldom done at all.
To my knowledge, you are looking for something akin to Open Policy Agent which provides a general policy language in which to express arbitrary policy as code. I would suggest investigating that at length.
At the risk of not providing a complete answer, I will give my best shot for some concrete examples as well.
Compliance as Code
In general, you need something which is often referred to as a "compliance framework", and in the DevOps world more often referred to as "Policy as Code". This allows you to define policies as code, which really means "machine-readable and executable". You specify what the policy is in the DSL of the framework, it is executed at some stage of your pipeline, and fails the change if it violates the policy.
To illustrate the point, consider someone submitting a pull request to a terraform module that inadvertently opens a port on a security group to the world which the policy says should be closed. This may be difficult to determine when you're writing the code, but the compliance test will catch it.
There are a few ways to do this, but I will mention two. A notable exception is Pulumi's Crossguard, but I don't have enough experience to speak to that.
Hashicorp Sentinel
In the Hashicorp world, there is the Sentinel Framework. This is an Enterprise product, so may not be relevant to your situation, but it is one of the best implementations I've seen. Importantly, it has a built-in testing framework  which aids in writing good policy code, which is also very expressive. I have used this as inspiration for how to write policy in Chef Inspec, which is the other example I will cite.
Chef Inspec
Chef Inspec is a framework for writing policy as code using rspec-style assertions. It uses a DSL based on ruby and supports major clouds (AWS, GCP, Azure) as well as OS-level resources. It is open source, so you can use it anywhere. You use it to make assertions in individual resources which are defined in the inspec DSL, e.g. "user on an OS" or "AWS EC2 instance". Writing the tests consists of assertions against the actual state of those resources, which makes it quite like a unit test. However, Inspec does not have a built-in testing framework for itself like Sentinel does - although you could use the ruby ecosystem for doing this.

Some notes:

Both of these tools allow you to express your policy as code, and to start extending the test coverage of that policy over your existing infrastructure.
Sentinel will catch violations before they are applied, but Inspec needs to be executed against an already-provisioned state.
Both will do what you required, but depend on your tech stack as to whether they will be appropriate for you
There is no magic bullet

for both of them you will need to write your policy as code. Usually this means taking some form of human-readable documentation and writing it in the DSL of the framework.
On the other hand, this provides an excellent opportunity to collaborate with the security team and do what folks are calling DevSecOps these days.


Limits of a singular Docker Swarm stack Normally, the example you start with Docker introduce environment to right more or less one app - frontend, backend, something else..

Would you put all containers of a larger app  (20-30 containers and above without replicas) still to the same stack or introduce a set of stacks representing system layers?

I would say It depend on you preference since redeploying a stack for only 1 or so image, will not restart all the containers present.

In the other hand network feature since docker 17.06 are reusable between stack with the attachable mode so you could easily separate the layers of your app while keeping them on the same sub-network (for usage like traefik or private data network).

Frankly for me this is the same problem as in dev when you separate the functions you are defining so it's easier for the others devs to understand a file of 50 lines compare to a file of 500 lines.

In the end it also depend on how this is integrated with your CD and if it's painful to update one document or multiple document when adding/removing services.

Personally I would go with multiples stack files.


Best Kubernetes storage option I know there are several options, I’ve heard everything from SSHFS, NFS, object storage all the way to storage OS, my issue is I need something fast and reliable to serve a media website built on Wordpress or similar cms. However I may want to do something with django also.
I have a home server I’ve been using over sshfs for some things but for a media site I don’t think that would be fast enough.
Any suggestions on the best method to store things? I also need to be able to access these items on an as needed basis through SFTP or an interface.
I’ve heard more than once that storageOS is the best option but after reviewing their site I don’t understand how it is persistent storage if it’s a container that is stored on an inaccessible node.
How can you back that up, manage space, etc?
I’m sure that a lot of you were looking at this question laughing because it seems so silly. Then there are probably some people who are just learning and saying OK yeah this is a legitimate question and then there are probably some people newer than myself you don’t even realize they need storage yet.

Your question is not silly at all, indeed managing WordPress at scale requires broad knowledges due to how WordPress is implemented.
It's good to know that WordPress consists of 1) core PHP code, 2) plugins/themes and 3) uploaded media assets.
To serve the core and plugins, you'll need to locate the code either in local disk or NFS. Local disk is obviously faster since it doesn't interact with network, but NFS is a good choice if you want to have multiple servers to share the same code.
The uploaded media assets are a bit different. The most common "assets" are image files, and you can completely offload the server traffic to object storage such as AWS S3 or GCP Cloud Storage. Unlike plugins and themes, the media assets  are static meaning it will never run server-side logics like PHP.
Moreover, WordPress is read-heavy application that most of the MySQL query is to render blog posts to your public audiences, means any-sized server may blow up when unexpected number of visitors rush to your website. This is where CDN shines, to cache rendered HTMLs in the edge servers at the first visit, then serve the cached files without reaching to origins. AWS CloudFront and GCP Cloud CDN are popular CDNs to turn cheap WordPress server to globally-scalable media site.
So, I recommend to have the following architecture:
Visitor --> CDN --> Web/PHP Server --> MySQL

When configured correctly, you can serve millions of traffics backed with $10 web server.


Docker container and their IPs on overlay (swarm) networks I have a Docker overlay network which includes 2 containers and the loadbalancer for this network:

    "Containers": {
        "0ece6b958aa6b451478b5004ba1b1681ec91b718b99a0c2c88b79d5e8e135a66": {
            "Name": "flvw_nginx.1.v9ycohipbvizlagm840o7jd0d",
            "EndpointID": "0fba09c56d5098efb3a89645a5dce575a12b5f3496d3839ce4e34bb29bc60091",
            "MacAddress": "02:42:0a:00:16:06",
            "IPv4Address": "10.0.22.6/24",
            "IPv6Address": ""
        },
        "25b2988afe5d0cf02f718831bf12858b3dccb875a3609804273da02bd4b422ac": {
            "Name": "flvw_pimcore.1.j3m0i52sksg0fmhdawzhwwhei",
            "EndpointID": "a2ea9ddcfba20b05017dda7a79af542ab09b0172a74f61be9fbb861fad39d008",
            "MacAddress": "02:42:0a:00:16:03",
            "IPv4Address": "10.0.22.3/24",
            "IPv6Address": ""
        },
        "lb-flvw_app": {
            "Name": "flvw_app-endpoint",
            "EndpointID": "c4cf9798cd3d2d8ee11057176c9f7395fd81546860f97c2623cd7f5043104294",
            "MacAddress": "02:42:0a:00:16:04",
            "IPv4Address": "10.0.22.4/24",
            "IPv6Address": ""
        }
    },

This states:

pimcore = 10.0.22.3
nginx = 10.0.22.6

So I exec into the containers and the following commands.

On the pimcore container:

[root@25b2988afe5d pimcore]# ip a | grep -i "10.0.22"
    inet 10.0.22.3/24 brd 10.0.22.255 scope global eth0

[root@25b2988afe5d pimcore]# ping nginx
PING nginx (10.0.22.5) 56(84) bytes of data.
64 bytes from 10.0.22.5 (10.0.22.5): icmp_seq=1 ttl=64 time=0.062 ms
64 bytes from 10.0.22.5 (10.0.22.5): icmp_seq=2 ttl=64 time=0.052 ms
^C

On the nginx container:

[root@0ece6b958aa6 /]# ip a | grep -i "10.0.22"
    inet 10.0.22.6/24 brd 10.0.22.255 scope global eth0
[root@0ece6b958aa6 /]# ping pimcore
PING pimcore (10.0.22.2) 56(84) bytes of data.
64 bytes from 10.0.22.2 (10.0.22.2): icmp_seq=1 ttl=64 time=0.036 ms
64 bytes from 10.0.22.2 (10.0.22.2): icmp_seq=2 ttl=64 time=0.065 ms

Result:

Pimcore says its own IP is 10.0.22.3 (which corresponds to the network inspect output) and that nginx' IP is 10.0.22.5 (which does not correspond).

Nginx says its own IP is 10.0.22.6 (which corresponds to network inspect) and that pimcore's IP is 10.0.22.2 (which does not correspond).

Containers seem to see other containers "1 number off", in a way of speaking. Why?

My TCP/IP knowledge is definitely not high level, but this feels weird. Maybe someone can englighten me!

When you ping nginx the docker swarm returns Virual IP of swarm service named "nginx" not the container ip.

This is is how internal load balancer works - you can have multiple replicas of "nginx" containers behind the "nginx" service.


How do I provision a Windows Docker container with Ansible? I want to provision a Windows Docker container with Ansible, but I am confused about how exactly it will work.

For instance if I use Packer with Linux, I know I can run ansible (which is ansible remote) and ansible-local. I have to build my Windows Docker containers from a Windows computer and the Windows computer cannot be an Ansible master at this point.

Does any tool, possibly Packer for Windows handle things automatically and spin up a Linux Ansible Master VM to run the playbooks automatically?

How can I easily provision Windows containers with Ansible as part of my image build process?

I am currently using Windows 10 Enterprise and Windows Server 2016 is on the horizon.

I've always had issues with using ansible to provision Windows machines with Packer due to the winrm connections breaking, example issue.

A workaround which you can do is to use the shell-local on your host machine which calls the Ansible playbook with the new host (although you would need to give it the IP address of your Windows Docker machine).

I've added the example which I did to get a test working. This uploads ip.cmd to the Windows virtual machine, which gets its IP address, and is then downloaded back to the destination. The ansible playbook is then called against that.

"provisioners": [
        {
            "type": "file",
            "source": "ip.cmd",
            "destination": "C:/tmp/ip.cmd"
        },
        {
            "type": "windows-shell",
            "inline": [
                "echo [{{ user `ansible_group` }}] > C:/tmp/hosts",
                "C:/tmp/ip.cmd"
            ]
        },
        {
            "type": "file",
            "direction": "download",
            "source": "C:/tmp/hosts",
            "destination": "./ansible/hosts"
        },
        {
            "type": "shell-local",
            "command": "ANSIBLE_CONFIG=./ansible.cfg ansible-playbook -v -i ./ansible/hosts -l \"{{ user `ansible_group` }}\" -e \"ansible_user={{ user `username` }} ansible_password={{ user `password` }} ansible_become_pass={{ user `password` }} ansible_port=5986 ansible_connection=winrm ansible_winrm_server_cert_validation=ignore \" ./ansible/site.yml"
        }

where ip.cmd gets the IP address of the Windows host:

@echo off
FOR /F "tokens=2,3" %%A IN ('ping %computername% -n 1 -4') DO IF "from"== "%%A" set "IP=%%~B"
echo %IP:~0,-1% >> C:/tmp/hosts

source


What are best practices for implementing SSL on Jenkins? One of the options is to configure SSL in NGinx and to put this in front of Jenkins. Are there any other options for securing Jenkins? I am looking for an answer that is focused on best practices that harden the layer 6.

Besides the transport layer you should first of all enable
Access Control

to protect jenkins from public access.


Cleanest way to prematurely exit a jenkins pipeline from inside a withEnv In a Jenkins scripted pipeline with the below structure, the return command will exit only the withEnv.

node{
    withEnv([...]){
        stage('1'){
        ...
        }

        stage('2'){
        .....
        }

        if( env.BRANCH_NAME != 'master' ) {
            currentBuild.result = 'SUCCESS'
            return
        }

        stage('3'){
        ...
        }
    }

    stage('4'){
        withEnv([...]){
            ...
        }
    }

    stage('5'){
        ...
    }

    stage('6'){
    ...
    }
}

How can I stop the entire pipeline after stage2 with SUCCESS?

Just the way you can exit any stage prematurely by using return keyword, you can use the return keyword (outside of the stage) to exit the entire pipeline prematurely. You can re-use the currentBuild.result value outside of Stage 4. See below.

node{
    withEnv([...]){
        stage('1'){
        ...
        }

        stage('2'){
        .....
        }

        if( env.BRANCH_NAME != 'master' ) {
            currentBuild.result = 'SUCCESS'
            return
        }

        stage('3'){
        ...
        }
    }

    stage('4'){
        withEnv([...]){
            ...
        }
    }

    if(currentBuild.result == 'SUCCESS') {
        return //this will exit the pipeline
    }

    stage('5'){
        ...
    }

    stage('6'){
    ...
    }
}


Automated Integration Testing of an API in Azure Functions using VSTS I am building a HTTP API as an Azure Function App in C# and I would like to run automated integration tests (calls to the API) as part of the CI/CD.

Is this possible to do using VSTS?

I have searched but only found how to do unit testing (not integration testing) as part of the deployment.

There is no difference on vsts for your test (unit, integrated or functional). You just set the right dll name with your tests:

or select Test plan if you have a test plan with associated test recordings


How are you enforcing git behavior, including locally (particularly on Windows)? I'm taking point on moving this .NET shop from svn to git, and have identified some ancillary issues I'd like to have a solution for before we flip the switch.

The one I'm asking about in particular in this question is line-ending enforcement. By default git for windows installs with the 'checkout crlf, commit lf', which won't work for a bunch of source that is (as far as I'm aware) exclusively comprised of crlf endings.

I don't know that I'd blindly trust any given dev to configure this correctly even given instruction, so I'm considering one (or both) of the following but was curious if anyone here had gone another route.

A pre-commit hook that checks for any lf line endings (or maybe all lf line endings), and rejects in that event.
An install script distributed to devs that populates the global config with the 'as-is, as-is'.

P.S. While writing this it occurred to me that the initial conversion from svn to git could commit the default way and as long as people stuck to the default that would be pretty seamless as well. Having been a dev using git in a .NET shop that installed git with the non-default 'as-is, as-is', I've created my own issues there too (they'd all rolled default prior to my arrival). So I'm still leaning towards some sort of enforcement mechanism.

To answer the question of how to enforce something locally, you can't without doing some very heavy lifting around managing and enforcing the state of every developers workstation, and I'm usually of the opinion that developers should probably be local admins on their development machine because if they aren't then they will just spend their time figuring our how to get those privileges anyway.

And that's probably because you shouldn't need to care about the state of local configuration when using distributed version control. You should only care about the state of your configuration. Assuming you are using git as a centralised version control system, because that's what basically everyone does anyway because its easiest, then let's just assume that "your" configuration is the copy of code saved on the central server.

If this is the case then you shouldn't accept merges that, as you mentioned, break crlf/lf line endings. So you enforce that when some other client tries to push changes you with server side logic that rejects the request to pollute your repo with their potentially breaking stylistic choices.


Anyone know how to use $CIRCLE_BUILD_NUM across different jobs in CircleCI? I use $CIRCLE_BUILD_NUM in different jobs in a workflow (but it seems like that get incremented in between jobs of the same workflow)?

I did some research and it looks like I can switch to CIRCLE_WORKFLOW_ID which is unique for the workflow but then the ID is some really long string, not like the build number which is a small integer, nice that I can use to append to build numbers and docker tags and such.

Any ideas?

This is the approach I came up with.  It's kinda ugly, but it does work:

- run:
    name: Create unique build identifier
    command: |
        GIT_SHA=$( git rev-parse --short HEAD )
        echo "export GIT_SHA=${GIT_SHA}" >> "${BASH_ENV}"

        # the always-increasing counter, based on CIRCLE_BUILD_NUM
        BUILD_COUNTER="${CIRCLE_BUILD_NUM}"
        echo "export BUILD_COUNTER=${BUILD_COUNTER}" >> "${BASH_ENV}"

        # the build identifier, which includes the short git sha
        BUILD_NUMBER="CIRC${BUILD_COUNTER}-${GIT_SHA}"
        echo "export BUILD_NUMBER=${BUILD_NUMBER}" >> "${BASH_ENV}"

        # output build id and counter
        echo -e "\nbuild counter: ${BUILD_COUNTER}; build id: ${BUILD_NUMBER}\n"

Then ${BUILD_NUMBER} is available as an environment variable in other steps.


How to create a preemptable docker container? Is there any way by which I can set a lifetime for a container after that much of time it should be stopped automatically and only awake when someone send a request. I wanted something like Preemptable Google Compute engine instance or Heroku free Dyno which sleeps after 1 hr of inactivity and awake again when a request is send to that instance.

Note: I have docker compose project so I want to set the life time for one service which depends on other services also. I have a traefik container runs separately beside the docker-compose project which handles the request and sends to containers. So is there something like request_post_hook which can start that stopped docker-compose container when receive any request ?

I'm not aware of any straightforward way to do that, without also running a different container to act as a middleware of sorts. You might want to look at one of the "Serverless" platforms. If your use case will fit that model, many of the docker/swarm/kubernetes serverless tools do what you're asking, but again, you're running other container(s) to manage it. I'd suggest starting with a glance at the OpenFAAS docs to see if it'll work for you.


Multiple Application in single SSL 443 server NGINX I have two application served in a single server app1.domain.com and app2.domain.com.

I want to proxy pass the request from app1 to serve static file and app2 to proxy_pass localhost:8000

I already have both SSL certificate using letsencrypt, how can I serve both apps using a single server?

I thought I can use different server_name. I read from here: the article use multiple NGINX server with the same port 80. But when I try for the port 443 the test fails "duplicate listen options for..". Here is my current config

server {
    server_name app1.domain.com;

    listen [::]:443 ssl ipv6only=on;
    listen 443 ssl;
    ssl_certificate /etc/letsencrypt/live/app1.domain.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/app1.domain.com/privkey.pem;

    index index.html;
    root /usr/share/nginx/html/;

    location ~* \.(css|js|png|jpg|jpeg)$ {
        try_files $uri $uri/ =404;
        gzip_static on;
    }

    location / {
        try_files $uri /index.html =404;
        gzip_static on;
    }
}

server {
    server_name app2.domain.com;

    listen [::]:443 ssl ipv6only=on;
    listen 443 ssl;
    ssl_certificate /etc/letsencrypt/live/app2.domain.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/app2.domain.com/privkey.pem;

    index index.html;
    root /usr/share/nginx/html/;

    location / {
        proxy_set_header  Host $host;
        proxy_set_header  X-Real-IP $remote_addr;
        proxy_set_header  X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header  X-Forwarded-Proto $scheme;

        # Fix the “It appears that your reverse proxy set up is broken" error.
        proxy_pass    http://localhost:8000/;
        proxy_read_timeout  90;
    }
}

server {
    if ($host ~ "app1.domain.com|app2.domain.com") {
        return 301 https://$host$request_uri;
    }

    listen 80 default_server;
    listen [::]:80 default_server;

    server_name domain.id;
    return 404;
}

And I thought maybe I could use if ($host = domain) inside the location but it fails too

Oops, turns out I just have to remove the line
listen [::]:443 ssl ipv6only=on;
and it works.

it's leftover from running NGINX certbot, I thought that was important


docker-compose run container before other container stop Can somehow with docker or docker-compose run container B before container A stop?

docker-compose support trigger event or something like that?

Cheers

If you mean make sure that container A does not execute and then exit then no.  Docker containers will stop if the container crashes or if the main process exits.

However, if you are attempting to run container B and it is dependent on container A, I think the depends_on flag may be what you are looking for.  This will allow you to ensure that service B will run after service A is started.  However, you may need to add something to service A that ensures that it is ready before attempting to connect to it (e.g. a database).


Is there a way to map ports two docker container Let's consider two docker conatiner named app and mysql.
I am looking for a way, where I can map app and mysql ports, so that if I try to connect to mysql from web, using localhost:3306 it can connect.

I know I can connect using dns and docker network. But I am curious if there is a way to do this.

On Linux, containers live in separate network namespaces, so localhost or 127.0.0.1 on the container is not localhost on the host. I'm not sure about other OSes. You could expose a port to the host like this:

docker run -p 3306:3306 mysql

Then you can access it from any container at the host IP address, which for the default network is 172.17.0.1:3306. You can add -p many times, read the docs here: https://docs.docker.com/

All that the -p does on Linux, is essentially:

sudo iptables -t nat -A DOCKER \! -i docker0 -p tcp --dport 3306 -j DNAT --to 172.17.0.2:3306

(Or whatever your mysql container IP address is.) Docker compose and similar utilities make containers able to see each other with --link eg.

docker run --link web --link mysql:localmysql app

The only thing this does is to put the correct IP address for the container in a file called /etc/hosts inside the container (but this might change in future.)

If you want to use --link, all containers need to be on the same network. You can connect them while running with:

docker network connect mynet web

The main reason for doing this is because the containers don't have fixed IP addresses. You can do:

docker run --network mynet --ip 172.18.0.10 -d app

To give your container a static IP - and then just use that instead of the name. The docker team keeps changing how things work, so you might need to run additional commands to allow different networks to connect to each other.

You can of course also do all of this manually, on Linux, using the firewall, on the host machine. (Start learning about it here: https://www.google.com/search?q=iptables (Also look at the Images tab!)

iptables -nvxL # shows the filter table
iptables -nvxL -t nat # shows the NAT table where port forwarding is done

To dynamically forward a port to a container, get its IP address, eg 172.18.0.10, then:

iptables -t nat -A POSTROUTING -p tcp --dport 1234 -j DNAT --to 172.18.0.10:4567

Be keenly aware that if you do this, and your container restarts, or gets another IP address, that this will not automatically update, and unless you use iptables-save and iptables-restore in your startup and cron scripts, these manually created rules will disappear. Also that the docker command and compose scripts won't know about any configuration that you manually add like this.

If you really want localhost in your container to point somewhere else, you can edit /etc/hosts inside the container, and change the IP address of localhost - be very keenly aware that other things might break, because localhost is supposed to point to the same host, not another one, so there will be unintended consequences at some point.


How to check return status of parallel branches in jenkins pipeline I am running a Jenkins job on multiple slaves. Following is the code structure:

def branches = [:]
def allNodes = Jenkins.getInstance().getNodes()
for (int i =0; i < allNodes.size(); i++) {
branches[allNodes[i].name.toString()] = {
    node(allNodes[i].name.toString()) {
    .
    .
    stuff
    }
}
parallel branches

Now, I want to get the status of the job parts running on nodes so that in case some job part fails on some node I can mark that node as offline. Is there any way to do this?

You may want to try the Join plugin. Some more details about this plugin (from the linked page):

  This plugin allows a job to be run after all the immediate downstream jobs have completed. In this way, the execution can branch out and perform many steps in parallel, and then run a final aggregation step just once after all the parallel work is finished.

  The plugin is useful for creating a 'diamond' shape project dependency.   This means there is a single parent job that starts several downstream jobs. Once those jobs are finished, a single aggregation job runs.

For more details on that, refer to my self answered question about "Why trigger something remotely (using Jenkins) and then just forget about it?".


dotnet not found on travis for an F# project I'm new to both travis and this website, so please bear with me ;).

I've been trying to set it up to work with my GH project. The repo is visible on travis side, but I get the following error while building:

./fake.sh: line 25: dotnet: command not found

Not sure why - most likely I've done something wrong with my .travis file. Here it is:

{
  "os": "linux",
  "dist": "trusty",
  "mono": "5.10.1",
  "sudo": "required",
  "group": "stable",
  "dotnet": "2.1.403",
  "script": [
  -  "./fake.sh build -t Nuget"
  ],
  "install": [
  -  "export FrameworkPathOverride=$(dirname $(which mono))/../lib/mono/4.5/"
  ],
  "language": "fsharp",
  "before_install": [
  - "chmod +x fake.sh"
  ]
}

The file "fake.sh" is my build script. It uses the F# FAKE library, but the target I invoke contains nothing, but invocation of "dotnet pack". It works on my local machine.

EDIT:
The fake.sh is the standard, unchanged F# FAKE file that is included in "dotnet new fake" template. Here's its content:

#!/usr/bin/env bash

set -eu
set -o pipefail

# liberated from https://stackoverflow.com/a/18443300/433393
realpath() {
  OURPWD=$PWD
  cd "$(dirname "$1")"
  LINK=$(readlink "$(basename "$1")")
  while [ "$LINK" ]; do
    cd "$(dirname "$LINK")"
    LINK=$(readlink "$(basename "$1")")
  done
  REALPATH="$PWD/$(basename "$1")"
  cd "$OURPWD"
  echo "$REALPATH"
}

TOOL_PATH=$(realpath .fake)
FAKE="$TOOL_PATH"/fake

if ! [ -e "$FAKE" ]
then
  dotnet tool install fake-cli --tool-path "$TOOL_PATH"
fi
"$FAKE" "$@"

All it does is it makes sure all the build requirements are there in place. Then it runs the "build.fsx" script:

open Fake.Core
open Fake.DotNet
open Fake.IO
open Fake.IO.FileSystemOperators
open Fake.IO.Globbing.Operators
open Fake.Core.TargetOperators

let nugetDir = "../nuget"

Target.create "Clean" (fun _ ->
    !! "src/**/bin"
    ++ "src/**/obj"
    ++ nugetDir
    |> Shell.cleanDirs
)

Target.create "Build" (fun _ ->
    !! "src/**/*.*proj"
    |> Seq.iter (DotNet.build id)
)

Target.create "Nuget" (fun _ ->
    !! "src/**/*.*proj"
    |> Seq.iter (DotNet.pack (fun o -> { o with OutputPath = Some nugetDir }))
)

"Clean"
  ==> "Build"

"Clean"
  ==> "Nuget"

Target.runOrDefault "Build"

OK - found the solution myself. It's a kind of tricky one. Even though my project is written in F#, I still have to set

"language": "csharp"

in my .travis.yml file. That's a bit counter-intuitive, but I must admit that it's written in the doc, so can't really blame anyone.


How to securely pass sensitive data to EC2 Spot instances? Right now, we only use EC2 on-demand instances, and we provision sensitive data (credentials, private keys, etc.) on the instances' EBS which are encrypted with a KMS key.

My understanding is that I have a few ways to pass those secret data to Spot instances:

Burn them in an encrypted AMI (but then, what about host-specific stuff, like SSH server private keys?)
Pass them as UserData when we call RunInstances (but then we would need to encrypt those data, and decrypt them with a secret burnt in the AMI)
Use Secrets Manager, but that could quickly cost a lot...
Use Parameter Store, and encrypt the payload with a KMS key (1 shared key for shared data + 1 key per node for node specific data?)

And I am not sure I could individually identify spot instances?

Am I missing something there? How do you solve those challenges?

When you're running ephemeral workloads like you would on a spot instance the best bet, if you don't want to bake an ami, is going to be a secret store of some kind.

The two most common methods I've seen for doing this are:

If your provisioning tool (terraform, cloudformation, salt cloud) allows it then you can create a kms key and use it to encrypt secrets in user data. Once the instance has started it can decrypt them using that key.
You can use a secret store like you mentioned. You said that AWS Secrets Manager would get pricey so you're probably going to want to use Vault. It's not that hard to set up and can use IAM permissions to allow your instances access to secrets. Using it also opens other avenues to doing secret storage right.


Pagerduty API integration and auto-resolution It is possible to implement auto-resolution with Pagerduty API integration. When a monitoring system detects an issue based on some metric and a threshold, it would create a Pagerduty incident through the API integration. Later, if the metric falls below the threshold, it could automatically resolve the incident, in case it is still open.
This feature could be very useful for several reasons:
Cases where a large number of incidents get triggered due to a common root cause
In this case, the on call person is spared from having to ack/resolve so many incidents and instead focus on troubleshooting and fixing the problem.
Auto-resolution would capture the incident duration more accurately
Humans are not good in doing things on a timely basis. If we are computing MTTR based on manual resolution of incidents, it may lead to inaccurate measurements.
Has anyone implemented auto-resolution with Pagerduty? If so, would you be able to talk about the pros and cons?

PD has a lot of native integrations that can do auto-resolve but for those that don't you can use the API and/or rulesets to do that.
Here is a link to the ruleset documentation: https://support.pagerduty.com/docs/rulesets
And here is documentation on the incident resolution API:
https://developer.pagerduty.com/api-reference/reference/REST/openapiv3.json/paths/~1incidents~1%7Bid%7D/put
In the API call you would pass the following json body to resolve it:
{
"incident": {
"type": "incident_reference",
"status": "resolved"
}
}


Accessing the committer's name as a Jenkinsfile environment variable Using the env.BRANCH_NAME variable accessible in the Jenkinsfile outside of the pipeline{} block, I have set up a variable used to conditionally set the node used to build the git branch. I would like to also access the name of the developer who committed the change, but I do not know how to access it.

I need a variable assigned before the pipeline block to assign to pipeline { agent { label selected_agent }}. If I attempt to use a method call instead of a variable there, the Java parser complains:

  java.lang.ClassCastException:
  org.jenkinsci.plugins.pipeline.modeldefinition.agent.impl.Label.label
  expects class java.lang.String but received class
  org.jenkinsci.plugins.pipeline.modeldefinition.ClosureModelTranslator

I cannot run code inside the pipeline block before the agent block to define the variable, and the agent block has to be the first part of the pipeline declaration.

If I could run bash commands, I could access the author's name, and in fact I do use it at the end of the Jenkinsfile to pull the git committer's name for the status email, but if I attempt to use sh() before the pipeline block I get this error:

Required context class hudson.FilePath is missing
Perhaps you forgot to surround the code with a step that provides this, such as: node

I am not sure what else to try. Does anyone have a suggestion?

I'm not entirely sure what you're trying to ask in your question, so I'm answering the question I believe you're trying to ask.

A build does not necessarily have a single commit author.  It has a list of committers, which can be empty or can contain many committers.  To create this list, you can use this snippet in a scripted Pipeline (note that you may need to disable the sandbox in order to use .collect()):

changeAuthors = currentBuild.changeSets.collect { set ->
  set.collect { entry -> entry.author.fullName }
}.flatten()

I use this with Git as my SCM; I'm not sure if it works with other SCMs.

See also the following questions on StackOverflow (some answers may be outdated):

How to get culprits or committers inside a Jenkins workflow with one or more SCMs
How do you pull git committer information for Jenkins pipeline


Ansible multiple commands to a file You can see bellow how it looks like my Ansible Script.
My question is what i need to change to get output from multiple commands to a file.The problem is that i get output only from the first command.
I have 200 vrf,do i really need to make multiple tasks fior multiple commands,or there is another solution for it.
I would highly appreciate any response.
---
- name: PING_TenGig_Ethernet_VRFs
  hosts: fr1.s05
  gather_facts: false
  connection: local

  tasks:
    - name: run multiple commands on remote nodes
      ios_command:
        commands:
          - show bgp vrf Mgmt-intf
          - show bgp vrf cha-00 all summary
          - show bgp vrf cha-001 all summary
          - show bgp vrf cha-002 all summary
          - show bgp vrf cha-00303 all summary

      register: print_output

    -  debug: var=print_output.stdout_lines

    - name: save output to a file
      copy: content="{{ print_output.stdout[0] }}" dest="./output/{{ inventory_hostname }}.txt"

Judging by the documentation, the result of your run multiple commands on remote nodes block is an array. You're writing only the first item of that array with {{ print_output.stdout[0] }}. The [0] here is the pointer to the first element of the array. Try removing it and see if the result fits your needs.
UPD: Use this to write several strings into a file:
- name: save output to a file
  lineinfile:
    create: yes
    line: "{{item}}"
    path: ./output/{{ inventory_hostname }}.txt
  with_items: "{{ print_output.stdout }}"


Is there a way to lint Terraform? I'm hoping there is something to pick up when I have misspelled variables and comments to save me some time when I'm checking my code and reduce the potential for any errors.
I normally use IntelliJ IDE for Terraform on a MAC but also have VS code set up that I can use.
Is there a plugin for either I could use or even a different tool?
I imaging variable might be difficult to lint due as I use underscores to separate words in my variable names so the tool would need to know to treat underscores as white space for linting purposes.

terraform fmt

The terraform fmt command is used to rewrite Terraform configuration files to a canonical format and style.


Running Automation on Google Cloud Virtual Instances using Ansible I'm trying to automate google cloud virtual instances remotely only using external ip addresses of virtual machines. I can ssh into the virtual machines using command line with user name shishir9159_gmail_com . But If I use any ansible commands like this:

ansible -i hosts -u shishir9159_gmail_com --private-key=~/.ssh/google_compute_engine -m ping all

and it results in this following error:

"msg": "Failed to connect to the host via ssh: shishir9159@35.202.219.6: Permission denied (publickey,gssapi-keyex,gssapi-with-mic)."

I've used some parameters in my ansible.cfg:

host_key_checking = False
ssh_args = -o ControlMaster=no

But I don't think they do much of a help according to this post:
https://serverfault.com/questions/929222/ansible-where-do-preferredauthentications-ssh-settings-come-from

And I tried many methods and recommendations. I have a service account but it doesn't seem to me necessary for this simple ping command.

You run the command with the user shishir9159_gmail_com but the error shows shishir9159@35.202.219.6: Permission denied ... seems like the _gmail_com portion of the username is truncated. Can you try setting the ansible_user config parameter in your inventory file or playbook rather than passing it on via the CLI?


GitLab Pages delivers random content I am experiencing weird behavior with the Pages feature of GitLab Omnibus package running on an Ubuntu 16.04 virtual machine. Some projects use Pages with Jekyll built by GitLab CI, which has been working as expected since it was first published with Gitlab CE.

For a couple of days now, visiting any of the homepages of those sites shows the content of just one of the projects. Each of them should of course show different content, but they all show the same. Even stranger: the content shown on each of the sites changes over time to one of the other projects, and I can not see whether this is deterministic.

Restarting the build processes of each of the projects did not fix this, neither did gitlab-ctl reconfigure, stop and start, nor rebooting the entire VM.

To investigate on that issue, I edited (which I assume is) the resulting file of the build process at /var/opt/gitlab/gitlab-rails/shared/pages/www/www.domain.org/public/index.html. Not in the first place, but later on during the already stated "rotating" content, the edits showed up on the webpage.

So what is going on there? Is this some caching issue? Is it malconfiguration? Is it a bug? Please help me find and fix the problem, as those are production websites.


What are best practices for running Pipeline tests into a CI framework? According to this github repo it is possible to test the Jenkinsfile, but this is run by Jenkins and if there are issues it will not run. Should one run such unit test in another CI? What are best practices to solve this issue?

Unit testing in an alternate CD/CI platform would be overkill in every situation I've built one.  The shared-pipeline code repo you connect to Jenkins will be triggered by a Jenkinsfile.  I suggest you define any and all testing in that particular Jenkinsfile - and keep this one simple enough that extensive testing of the Jenkinsfile defining tests for your Jenkinsfile does not also need a Jenkinsfile defining tests.


How to create ansible host file on mac? When I try to create ansible host file as follows:

sudo vi /usr/local/etc/ansible/hosts

When I try to save the file I get following error:

"/usr/local/etc/ansible/hosts" E212: Can't open file for writing

Run:

$ sudo touch /usr/local/etc/ansible/hosts
$ sudo vi /usr/local/etc/ansible/hosts

To create a file and then open it.


How do I configure Jenkins to trigger on a pull request based on the Git branch's name? I want pull requests to Git where the target is master and the source has a string "foo" to trigger a Jenkins build.

If this string "foo" appears in the source branch's name when there is a pull request, I want the Jenkins build to start.

The only way I can think of doing something this sophisticated is to put the logic into a Jenkins pipeline file.  Is there a plugin or supported way of making this work?  I am not using Github.

Is there a way to filter any branch that does not have "foo" in it?  How do I trigger a build based on a pull request?  I tried doing this by itself (without the search for the pattern "foo"), and even that failed.  There were no errors, but I know Jenkins was authenticating to Git with the credentials I configured in Jenkins.  The Jenkins job would not run when the conditions I configured were met.

Using a multi-branch pipeline you have several different ways of filtering branches, take for instance a regular expression for "foo". In order to use this job type, you must create a Jenkinsfile in your repo with build instructions using the scripted or declarative syntax. The term pull request is typically synonymous with services like Github and Bitbucket.

You can also filter within the pipeline itself using declarative syntax:

when {changeRequest branch: "foo.*", comparator: 'REGEXP' }

And also scripted syntax:

if (env.BRANCH_NAME ==~ /foo.*/){
    ...
}


Pulling windows base images fails I tried pulling the windows/nanoserver image.
After that failed I looked around and found this site
Every one of them fails with the same error message

#docker pull mcr.microsoft.com/windows/nanoserver:1909
ltsc2019: Pulling from windows/servercore
no matching manifest for unknown in the manifest list entries

Additional Info:
Docker version: 18.09.2, build 6247962
Win10 Enterprise 2016 LTSB: Version 10.0.14393 Build 14393

The problem is twofold:

The Win10 LTSB is on its newest version and Docker has stopped supporting this version.
This stops Docker from updating.
Microsoft has moved to its own container registry and in doing so abandoned support for older Docker versions. So while I was pulling from the right registry it could not find a fitting image.

Solution: get an up-to-date, non-LTSB version of Windows

Additional Info:
While the error message did not help in finding the cause the docker log did:

[WindowsDaemon  ][Info   ] debug: a Windows version 10.0.18363-based image is incompatible with a 10.0.14393 host