What functional differences are there between kubernetes, rancher and mesos? What are the main feature differences the popular container orchestration tools kubernetes, rancher, and mesos?

What would lead you to choose one over the other?

First of all, Rancher actually contains implementations of both Kubernetes and Mesos within itself. However, they did make their own system called Cattle which is heavily based on Docker's Swarm. I'll touch upon this in the section for Rancher.

Secondly, since each one offers similar base features (load balancing, resource isolation, etc) I'll try to go more into what makes them different rather than focusing on differences among those common features unless they're significant.

Kubernetes

Highly performance focused, also featuring cloud storage orchestration (a feature missing from Mesos, although there's probably a plugin for it). Has API options to allow for automated scaling of resource usage as needed by individual containers (and for the reverse if a container is not being hit hard while others are).

Something important about Kubernetes is that unlike other container orchestration software it doesn't provide a comprehensive  configuration or any kind of comprehensive self-healing. Instead, it focuses on continuous deployment of multiple apps with an easy rollback system at the app level (as a result you might want to look into micro-services when using it).

Each app is a small piece and can be deployed/configured individually. They make a point in their docs to say that Kubernetes is not a traditional PaaS (platform as a service) system since it lacks middleware for virtual hardware or databases and it doesn't build your app itself. It's designed to (as they say themselves) remove the need for manual container orchestration, instead automating the process by continuously pressing towards a target app state.

Mesos

Monolithic in comparison to Kubernetes. Focuses on the big picture moreso than the individual services, although it still allows for management of individual services. Provides built-in middleware for things like Spark, Hadoop, etc. Best usage of Mesos will involve many plugins as it's designed to be easily extended.

If you need fine-grained control over managing your application (insofar as there's a plugin available for what you want to do or you have a team member willing to build one if there isn't) you'll want to use Mesos.

Rancher (and Cattle)

Potentially the best option in that it is itself a superset of the previous two, having an implementation of both. This might also be seen as a downside as more complication in your management software is rarely a good thing and might lead to unforeseen issues.

Rancher features an application catalog that allows for one-click deployment, something Kubernetes doesn't have due to its design philosophy.  However, seeing as Rancher has an implementation of Kubernetes, you can use Rancher if you feel these features are missing from Kubernetes.

Cattle is based off of a stack system, where you group related services together. It also houses Rancher Compose, a feature similar to the Docker service of the same name. This is probably the most interesting part of Cattle, the rest of it being fairly standard (although the Secret management is still in beta). If you have experience with Docker Compose you should read up on it here (I don't, so I'm probably not the best person to write about it).

Resources: "What is Kubernetes?", "Overview of Rancher", "mesos.apache.org: 'What is Mesos?'"


Is it ok to have a Jenkins Server without slave node? In my team we have one Jenkins server with one master node & one slave node in the same server. We use Jenkins Pipelines and surround everything with node{}.

In a colleague's team I have seen that, for some reason, they have one master node and no slave node. All their builds run in master node.

Questions:

Is this a bad practice?
Are there any bad consequences for having only one master node?
Is my setup (one  master & one slave) a bad practice?

https://wiki.jenkins.io/display/JENKINS/Jenkins+Best+Practices

  In larger systems, don't build on the master.

  If you have a more complex security setup that allows some users to
  only configure jobs, but not administer Jenkins, you need to prevent
  them from running builds on the master node, otherwise they have
  unrestricted access into the JENKINS_HOME directory. You can do this
  by setting the executor count to zero. Instead, make sure all jobs run
  on slaves. This ensures that the jenkins master can scale to support
  many more jobs, and it also protects builds from modifying potentially
  sensitive data on $JENKINS_HOME accidentally/maliciously. If you need
  some jobs to run on the master (e.g. backups of Jenkins itself), use
  the Job Restrictions Plugin to limit which jobs can be executed there.

  One master node and no slave node. All their builds run in master
  node. Is this a bad practice?

Running the jobs on the master nodes means that the jobs have unrestricted access into the JENKINS_HOME directory

  Are there any bad consequences for having only one master node?

As the slaves have unrestricted access into the JENKINS_HOME directory this could be unsafe

  Is my setup (one master & one slave) a bad practice?

It is better than only one master as long as all the jobs run on the slaves

In summary, from a security perspective it is a bad practice to run the jobs on the master.


s3fs to mount S3 bucket with iamrole on non-aws machine I am trying to mount S3 bucket on my on-premises CentOS-6 server using s3fs as below:

[roots@myhost ~]# s3fs bucket-name /mnt/s3mnt -o  uid=1001,gid=1001,allow_other,umask=022,iam_role=iam-role,use_cache=/tmp

This gives error :

s3fs: unable to access MOUNTPOINT /mnt/s3mnt: Transport endpoint is not connected
debug :
    set_moutpoint_attribute(3293): PROC(uid=0, gid=0) - MountPoint(uid=0, gid=0, mode=40777)
s3fs_init(2597): init
s3fs_check_service(2896): check services.
RequestPerform(1524): ### CURLE_OPERATION_TIMEDOUT
    RequestPerform(1617): ### retrying...
    RemakeHandle(1288): Retry request. [type=-1][url=http://169.254.169.254/latest/meta-data/iam/security-credentials/iam-role][path=]
    RequestPerform(1620): Failed to reset handle and internal data for retrying.
s3fs: Failed to check IAM role name(iam-role).

Please let me know if somebody did it successfully.
Tried with endpoint parameter and URL parameter but it says:

fuse: unknown option `endpoint=ap-northeast-1'

Any suggestion is highly appreciated. Thank you.

Have you got AWS credentials configured on that machine?

~ $ aws configure

And then verify with:

~ $ aws sts get-caller-identity

Hope that helps :)


Bridged network adapter for Packer + VirtualBox? So,
I load an .ova image with packer of a minimal ubuntu linux installation.
I realized after everything went through (Provisioners etc) and it successfully exported the output file. However, when loading the output image with VirtualBox I can't communicate with the box, because the box doesn't have a valid network adapter that allows me to ping it etc.

So I tried adding a bridged adapter one during the build process:

"vboxmanage": [[ "modifyvm", "{{.Name}}", "--nic2", "bridged", "--bridgeadapter2","enp0s31f6" ]]

However the machine is still not reachable.
This is how it looks on the VM:

How can I ensure that my box is always reachable from the outside?
Is adding a network bridge adapter even the right way? What am I doing wrong?
I think I need the VM to have an IP like 192.168.x.x

With a bridged adapter, your VM would get its IP from the host machine's network DHCP server. This would allow it to be pinged from your internal network.

I'd also specify in your configuration that your cable is connected as so:

VBoxManage modifyvm "VM" --cableconnected1 on

Port forwarding in the VM's configuration would also work. It would allow access from the host machine's IP by just specifying the forwarded port.

To forward port 22 to 2222 you would do as so:

VBoxManage modifyvm "VM" --nic1 nat
VBoxManage modifyvm "VM" --natpf1 "guestssh,tcp,,2222,,22"
VBoxManage modifyvm "VM" --cableconnected1 on

If you want it to be accessible by the internet, you would need do port forwarding on your router to route access from your public ip:port to your internal ip:port.


Trade offs of putting an international master database in us-west-2 I'm hoping this isn't so open ended that it veers towards a "discussion" more than an "answerable" question.

I need to place an RDS Postgres instance in an AWS region that will act as the master DB for writes from all other regions. Those regions will include Australian and European regions, as well as North American.

My concern is latency on writes. I will use replicas for reads, so that is not so much a concern.

I will be using interregion VPC peering for all regions to be able to communicate with the RDS instance.

I'm thinking that placing the RDS server in us-west-2 as it seems to be in the most "central" spot for all regions of concern.

I understand that the AWS "backbone" is very fast, and perhaps fast enough not for this to be a concern.

Has anyone dealt with this problem and have any metrics or other ways to quantify a decision on where to place this RDS instance?

It's been a while since posting the question, and a comment I made above seems to be helpful despite the fact it's using a quantitative assessment, so I'll post it here as an answer.

As per comment above:

@030 I can offer my experiences since asking the question, albeit it's a subjective view as I have no quantitative data. I can offer that a year and a half later, I haven't regretted my decision putting the master into us-west-2. Our users in any region don't seem to be impacted by having the master in us-west-2, at least from a UX perspective. I suppose at some point it could be worth measuring write times -- but to date I haven't had to. There's no business impact from any latencies due to the location of the master, that we can detect at least, so we're happy here


Build / Deployment process for .Net/Mulesoft/Oracle stack I have the following challenge to streamline and setup a development process.

The program runs simultaneously, and each one has its own timeline for release, as follows:

Program 1
Program 2
Program 3
Program 4
Program 5

Under each of the programmes, there are multiple projects running in parallel. Each project has a different technology, like Web layer (.NET), middle-ware (Mulesoft) or database (Oracle).

The challenge is that I have to find a configuration management tool, which considers branching, automated build process, deployment and integration tracking with JIRA.

Can some one suggest the best tool to seamlessly integrate and orchestrate automated build and deployments?

Technologies involved are:

Oracle (Basically SQL scripts)
Middleware (Mulesoft - maven script)
.NET web layer (UI part)

Key points to be considered is the build and deployment process should be technology agnostic, as I have an environment with java, .NET and oracle world.

I am thinking of ( Github - Bitbucket/Bitbucket pipeline) actually but any suggestion would be highly appreciated.

For something that requires being agnostic to the tooling, and having the flexibility you need, your best bet is Jenkins.  https://jenkins.io/.  They have a ton of plugins that would meet any use case.  Be aware that Jenkins can be heavy for small teams, but your other options lack the flexibility and support for different communities.

You can check out other tools:

Team city
CircleCI/Travis CI
Semaphore
ETC


nginx for dummies? How to enable Jenkins over https I'm running Jenkins on an Ubuntu machine bare metal, and as I'm only experimenting and learning how the ops of it all works, I now want to learn how to enable https (with a nominally valid certificate obviously). I want to talk to a git instance and other tools and keep the automation tools secure. I believe the easiest is to use a reverse proxy, NGINX. Which I assume just acts like a forwarder; this is where I get stuck as a newbie.

I am unable to set up or follow the nginx initial steps to obtain a certificate, because the key generation tool (keytool) just does nothing when I give it a URL. It keeps saying my computer's FQDN is not valid.

I guess that's because my ubuntu machine is behind a NAT. I am guessing it's not realistic to create a trusted server on a private LAN which only has egress, but no ingress? Do I really need a real domain? I have one, but since this is work related, I'm loathe to create any kind of link-up between work and play. This feels a little workaround hacky-like, so keen for alternatives to what looks like an overkill configuration. For an internal experimental Jenkins setup, is there a 3 step process that a beginner who knows very little about keys can follow?

Your question is very broad, but I'll try to answer it.

You really need to read on what TLS/SSL is and how it works before you continue. You can find a bunch of questions in security.stackexchange.com that ask about this and many of them even link to RFCs where you can read the full detailed explanations of how it all works.
To configure nginx to work with TLS/SSL certificate you can use https://www.digitalocean.com/community/tools/nginx to generate the template. Just follow the guide and it should create a relatively secured vhost. If you would like to enhance the security even further - you can follow cipherli.st's guide. They update it relatively often so the proposed settings are usually up-to-date.
What makes a certificate "valid" or "invalid"? It's the Certificate Authority. It is possible to create a private CA certificate and use it to sign a Certificate Signing Request (or CSR). The CSR can then be signed by the CA and returned to whoever needs it (in this case the Nginx server). There are multiple tutorials for this available online and you can use easy-rsa to do the heavy lifting. However, the down side of doing it this way is that you need to go over all machines in the network and install the CA in browsers and all necessary keystores. Otherwise those machine will treat the site's certificates as "untrusted" because it will be coming from an unknown Certificate Authority. In some cases you can also create a Certificate Bundle for nginx.

If you had an ingress available you could have used LetsEncrypt to automatically issue certificates and skip complicating your setup by having CA and issuing certificates on your own.

Note: Image has been borrowed from ssl.com.


Kubernetes - Internal only API Endpoints Hypothetical question.

Say we have a Java application running in a Kubernetes pod (single docker container) that has 'public' endpoints and 'internal' endpoints.

The /public endpoints should be accessible from public internet and exposed via the Ingress.

The /internal endpoints should only be accessible to other pods in the Kubernetes cluster.

Is there any way we can limit access to these /internal endpoints in 'vanilla' Kubernetes? One way I guess would to be declare the public endpoints in the ingress but them seems messy.

I also believe tools like Istio allow policy based access?

I'm not sure I understand why listing the public endpoints in the ingress is messy. The common solution is to list the ingress endpoints in the ingress controller.

As you mentioned, there are alternatives such as Istio, Consul and other network policy solutions. There's also the Open Policy Agent that can help with this, if you don't want to write YAML.


M2 Repo contents is not available in artifactory server We have recently installed artifactory OSS 6.2.0 and then created the following repositories (with Package Type - Maven).

Local
 - wellows
Remote
 - wellows-remote
Virtual
 - wellows-virtual

And in Virtual Repositories (wellows-virtual), I have included the remote repository which is 'wellows-remote'. I have a maven project and in the distribution tag, I have the following added and as part of the build now the artifacts gets published to the artifactory server. However, I couldn't see my .m2 repository contents anywhere in the Artifactory server. Why so? Am I missing out any configuration here?

<distributionManagement>
    <repository>
        <id>central</id>
        <name>wellow.jrp.com</name>
        <url>http://wellow.jrp.com:8081/artifactory/wellows/</url>
    </repository>
</distributionManagement>

Welcome, Vivek

This is not how it works

Your local cache (~/.m2) is not transferred to Artifactory if anything the direction is the other way around (in some point of time you'll see files from Artifactory appear in your local cache).

When you run a Maven build, the configuration you provided in Maven (check the Set Me Up button in Artifactory UI) will make Maven searching for dependencies in the following order:

Local cache (~./m2)
Artifactory (first local, then remote caches)
If Artifactory won't find the artifact, whenever willows-remote is pointing to.

After the artifact is found, it will be saved in Artifactory cache and in a local cache on your machine.

The <distributionManagement> tag you used in your pom.xml is for deploying the artifacts you created when you run a mvn deploy command. It will take the artifact produced (one file, not your entire ~/.m2 cache) and upload it to the repo specified in <distributionManagement>.


Jenkins and docker login issue I am trying to create a Jenkins pipeline that will pull a public image from docker hub and build a Vue.js application inside that container. In my Jenkins server I already have other pipelines that use AWS ECR to push images too. It appears that my build that is trying to access a public image is trying to log into AWS ECR  as well. I'd be okay with putting that public image on ECR if that would solve my issue.

Here is the output of the failed Jenkins build

[Pipeline] withDockerRegistry
$ docker login -u AWS -p ******** https://index.docker.io/v1/
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
Error response from daemon: Get https://registry-1.docker.io/v2/: unauthorized: incorrect username or password
[Pipeline] // withDockerRegistry

The error shows that you are using AWS ECR credentials to login to dockerhub which will not work. Dockerhub is a different registry(both public and private), however ECR is private registry only.

For pulling public images from dockerhub there is no need to login to dockerhub. You can simply use docker pull command and it will pull an image from dockerhub registry.

Example: docker pull mongo

However, when you want to pull an image from ECR, you need to first login to the AWS ECR and then only you can pull an image from ECR.


Decoupling data of devops tools eg jenkins nexus Building devops pipe with tools such as Jenkins, Nexus, Hashicorp etc.. Deploying via terraform ansible & packer. Intending to use load balancers in front of each tool, want to deploy tools listed above with their data decoupled enabling easier tear down for updates etc. (eg load balance x2, nexus x2, data ?) Does anyone have any advice or suggested examples of how this is achieved. Not quite ready for containers, but trying to decouple data now in readiness for such a change. Cloud provider not defined yet AWS, Open Stack etc etc ?? Thoughts welcomed.

You might want to explore immutable architecture.

Steps

CI with static analysis(Sonarqube, because free!), pushing version artefacts to Nexus.
Base Image using Packer. These must be revisited from time to time. Base Image has security hardening, log shippers, monitoring tools etc.
Environment/Application specific images from Base image.
healthCheck and version API for each application(healthCheck API in deployment and version for visibility of code in different VMs).
Deploy App using Terraform and Ansible.

Can use Jenkins 2.0 (pipeline as code) to achieve points 1,2,3. It can be part of each application itself.

Let me know if this answers your question.


Helm Repo: can it host dockerfiles and other image-building artifacts I have a service that I want to port to Helm. It's a django application that requires memcached and postgresql and a forward proxy like nginx.
I would ideally like to have one repository with all that stuff (dockerfiles/buildah scripts/charts). I think I can that easily with Helm repository if the images were already created. But they're not.
Is there any convention to include Dockerfile's or buildah scripts inside the Helm repository? It seems weird to have to create another repository to build the images, and then to have a helm repository to tie them together. Is this the way it's done currently?

Well that's a legit question IMHO, but there are multiple way of doing things (and no one is worth than other).
1- Everything from dev to prod at the same place
If you want to grab everything from dev to production in a single project, it's possible, but you cannot build the docker image from hell command.
Helm is only here to organize, use template in yaml and sync releases following kubernetes file structure.
You can follows helm repository structure and add other scripts, but you have to execute steps in the right order: build, push, deploy
2- The "regular" way to do
I more often saw a separation of production configuration files and application files.
What I do is:

One repository which carry my application source code + everything for the CI (test, build, docker file)
One repository which carry my helm.charts for all my infrastructure.

Choose what suits the best for you, but keep in mind that helm is a templating system for kubernetes yaml files, so it will not build the docker image.


Teaching an old dog new tricks: Finally adopting the micro-service architecture Is it too late to teach an old dog a new trick?

I've always been the guy with a monolithic application/codebase.

For more than 10 years now, I've gotten good at caching, writing good SQL queries, using Redis to store frequently called read data, using background jobs to perform updates, edits, long-running tasks, e.t.c

Then again, I haven't had to deal with more than 100 million rows in a db before.

Not until now, I've used up all my tricks and alas no significant improvement.

Now I'm coming for advice from you smart devops people.

For my current app, just recently I had a rails app, my DB, and some script in the /lib, Redis, Sidekiq, all on one, you guessed it, server.

I read up on Docker and now I have Sidekiq, Redis, my app, and my db all on different containers.

Now, I'd like to split some functions on my app itself. This is where I need your help.

For example:

Script A: I have this ruby script(a ruby class) that crawls the web for backlink data. Script A only gathers links and removes links that have been parsed already.

Script B: Another script that gets the links script A collected and retrieves the information I want from that link.

Let's stop at that for now, so it doesn't get confusing.

I'd like to run these scripts on their own servers, and they would connect to my app via api's.

Both scripts are less than 1000 lines of code and are written in Ruby.

Requirements

Both scripts require Selenium, Watir, and headless chrome to run.
The server must be able to autoscale as needed.
If the server is rate-limited or stuck in a link blackhole, I should be able to spin up a new server(host), with a different wan IP, with all the content of the script in as little time as possible.
The server should be affordable, as it's these are long running tasks. The script does not require a database as it only sends back important data back to my app via API.
In short, both scripts should be on their own server (host) and it should be easy to spin up a new one in no time.

How do you suggest I go about it?

In your approach I see a few scalability problems - I'm judging from a Google App Engine (GAE) context, where scalability is achieved via breakdown of the work in small tasks/work items and strict limitation of the response time for a task execution:

you're duplicating some piece of functionality: both scripts need to parse the same page, leading to your requirement #1
the task execution time for script A is fundamentally unbound, it can vary widely depending on the number of links discovered which need to be checked if already crawled

I'd approach this a bit differently to avoid these issues. I don't even see a need for separate services for this particular job:

the service pulls individual page URLs as separate tasks/work items
it checks if the URL was already crawled and, if so, does nothing, otherwise continues
gets the page content and parses it to extract 2 kinds of info:

the list of other URLs to crawl, creating and enqueueing new work items for each one of them (it doesn't check here if they were already crawled, which may take a long time depending on the number of URLs)
the other info needed from the page

In this approach the duration of each task execution is reduced to the minimum necessary: one page parsing and only one check if a URL has already been crawled. And each page is only parsed once.

The number of pending work items would be the driver for scaling - if it goes above a level new instances can be started, if it drops below a level and multiple instances are running some may be stopped. If implemented as a GAE app the GAE infra can take care of that for you automatically.

This solution can be implemented using a noSQL database (like Google Datastore, in a GAE context), which can scale better than a SQL one.


Debug Docker caching issues I am trying to improve CI/CD performance and part of that is improving the caching behaviour of Docker builds. I find myself trying to debug what exactly invalidates the cache. Are there any ways to query the Docker build process to get more information than "cached" and "not cached" ? For example, at some point, I add a large directory, which has not changed as far as I can tell, but busts the cache. I would like to find out what is causing that, exactly.

The docker cache needs the following:

Build has been run on the same docker host previously (or you explicitly added flags to trust a pulled image) and
The previous image needs to still be on the build host (not pruned) and
The same previous layer and either
The same command being run (including the same environment/args) or
The same hash on the files being copied

If you see the cache being busted on a COPY or ADD command, and not the previous step, then you need to look at the hash being generated. All the files need to be identical, same file names, cases, file permissions, file owners, and the contents need to be bit for bit identical. You can look at the resulting checksum for this hash by running a docker image history command, e.g.:

$ docker image history golang:alpine
IMAGE               CREATED             CREATED BY                                      SIZE                COMMENT
6b21b4c6e7a3        3 weeks ago         /bin/sh -c #(nop) WORKDIR /go                   0B
<missing>           3 weeks ago         /bin/sh -c mkdir -p "$GOPATH/src" "$GOPATH/b…   0B
<missing>           3 weeks ago         /bin/sh -c #(nop)  ENV PATH=/go/bin:/usr/loc…   0B
<missing>           3 weeks ago         /bin/sh -c #(nop)  ENV GOPATH=/go               0B
<missing>           3 weeks ago         /bin/sh -c set -eux;  apk add --no-cache --v…   344MB
<missing>           3 weeks ago         /bin/sh -c #(nop)  ENV GOLANG_VERSION=1.12.7    0B
<missing>           3 weeks ago         /bin/sh -c [ ! -e /etc/nsswitch.conf ] && ec…   17B
<missing>           3 weeks ago         /bin/sh -c apk add --no-cache   ca-certifica…   551kB
<missing>           3 weeks ago         /bin/sh -c #(nop)  CMD ["/bin/sh"]              0B
<missing>           3 weeks ago         /bin/sh -c #(nop) ADD file:0eb5ea35741d23fe3…   5.58MB

You can include the --no-trunc option to get a full line. The key piece of data relevant to your question from the above image is in the ADD step where you see file:0eb5ea35741d23fe3…. If that checksum changes between two image builds, the cache will bust and you'll start a new set of layers.


Does QA in DevOps roles means CI/CD or more? One of views on DevOps is to say that it combines fileds of Development, Operations and Quality Assurance.

Now many DevOps roles have focus either

on development with the left shift in terms of operations awareness (up to a No-Ops full stack, you build it - you run it (c) Besos, Amazon's CTO),
or a sysadmin who is expected to write CaSC (Configurarion as Code), IaC (infrastructure as code) and automation codes to reduce work required in operations.

In both cases you see lots of tools typical for these roles with and without DevOps context.

What does QA DevOps roles do today as a formal commitment? Test automation always has been there, for example. Is QA just meant here to support CI/CD infrastructure and teams working on it, or something else?

First, DevOps is a journey, not a destination.  Not every organization approaches similar problems with the same solution; while automation is a component of a DevOps platform, it's not the single defining characteristic.

The role and responsibilities of QA professionals haven't really changed; they're still responsible for validating that code is ready for release.  What HAS changed is the focus on both automation and smaller releases.  Much like operational roles, QA staff should now have the opportunity to focus on adding additional automatic quality checks into the development life cycle.

In a shop with a mature CI/CD pipeline, QA becomes more about designing great tests, and letting the software do the actual heavy lifting. As Tensibai commented above, "some QA tests can't be automated" (yet), so there will still need to be some engineering responsibilities, but the goal should be to utilize automation in every aspect of delivery in order to facilitate adding additional quality.


How Do I Log Task Output To A File? One of my ansible tasks imports an Oracle database using impdp.

This generates a lot of output to the console so I have set no_log: True.

However, when this fails I want to see the log!

How can I make this particular task log to a file and not to the console?

[Converting my comment to an answer]

One way to do it would be to write the logs to some external file, and then having a task after it which makes use of failed_when condition, and remove the log file, if the previous task was successful.

Something like this should help you.

 - name: Run Py script
      command: <>.py  > <>.log
      become: yes
      register: PyScript
      ignore_errors: True

    - name: PyScript on success
      command: rm <>.log
      when: PyScript|succeeded

Note: This might not be the best way to handle your problem. But, this was a hack which helped me do my logging and monitoring.


Is there any latest technology which will replace Docker, Kubernetes in future? What's the future of engineers starting career in tools like Docker, Kubernetes?

Perhaps serverless could substitute it, but it will be hard to predict. I would suggest to follow industry standards and run things using these tools in production to stay up to date. The most changes in all branches occur in IT. Having said this, every engineer has to learn new things everyday to stay up to date. Whether k8s and docker will be replaced? Probably yes, but on the other hand there are still VMs and on-premise hardware. Perhaps k8s and docker will be used less in the future, but when I do not know. I am not a fortune teller with a crystal ball.


Playbook without relative path to vars I had already bunch of playbooks, so I started to organize them in playbooks dir. It cause there must be relative paths to vars files.

Actual playbook:

---
- hosts:  aws_instance.jenkins-agents
  user:   ec2-user
  vars_files:
    - ../../../vars/main.yaml
    - ../../../vars/vault.yaml
  ...

Is there any way to keep the dir structure, but avoid the lochness family (../../../)?

I know for roles You can specify path to them, but I did not find anything similar for vars_files

Project conditions:

We need to run the playbooks from the root of the project
Both vars and playbooks dir are in root of the project
This project doesn't have static inventory/hosts, but it is generated on the fly when You run some playbook

You'll notice that the official recommendations for directory layout put all of the playbooks at the root level.  This is intentional, as Ansible doesn't handle other schemes well.

You can put playbooks in a subdirectory, as you've started to do, but that will require (as you've already found) the use of relative paths from the playbooks to any other resources they use; Ansible uses the playbook path to start its search, that's just how it's written.  For the most part, that's just what you get to deal with when you make the choice to move where playbooks live.

However, you seem to have more nesting than you should:

---
- hosts:  aws_instance.jenkins-agents
  user:   ec2-user
  vars_files:
    - ../../../vars/main.yaml
    - ../../../vars/vault.yaml
  ...

Why are you navigating up three directories?  I'd expect a layout that's like the official one, but with one subfolder for playbooks; then you'd end up with just

---
- hosts:  aws_instance.jenkins-agents
  user:   ec2-user
  vars_files:
    - ../vars/main.yaml
    - ../vars/vault.yaml
  ...

which is a much more minor change.

Secondly, it's pretty rare that you actually want to use vars_files.  Most variables used in Ansible either vary based on the host (and thus should go in group_vars/host_vars in the project root) or are role-specific (and thus should go in vars/defaults in the role directory).  A few documentation links:

https://docs.ansible.com/ansible/intro_inventory.html#splitting-out-host-and-group-specific-data
https://docs.ansible.com/ansible/playbooks_roles.html#roles
https://docs.ansible.com/ansible/playbooks_roles.html#role-default-variables

Personally, I think it's best to have as little as possible in your playbooks, but delegate everything out to roles.  Here's an example entire playbook in your setup:

- hosts: aws_instance.jenkins-agents
  roles:
    - ../roles/jenkins_agent

This gives you far more flexibility for re-use.

And if you find yourself not wanting to use relative paths for roles, you can override roles_path in an ansible.cfg in the root directory where you run your Ansible commands from.


Testing R package on Travis fails because of TeXLive incompatible versions I am writing an R package and using Travis for Continuous Integration. The tests on macOS fail because the system has a newer version of TeXLive and is unable to install the inconsolata font to build vignettes.

The Travis logs show:

sudo tlmgr install inconsolata upquote courier courier-scaled helvetic
tlmgr: The TeX Live versions supported by the repository
http://mirrors.concertpass.com/tex-archive/systems/texlive/tlnet
  (2016--2018)
do not include the version of the local installation
  (2019).

and further on, R is unable to build the vignettes:

* checking PDF version of manual ... WARNING
LaTeX errors when creating PDF version.
This typically indicates Rd problems.
LaTeX errors found:
! LaTeX Error: File `inconsolata.sty' not found.
Type X to quit or <RETURN> to proceed,
or enter new name. (Default extension: sty)
! Emergency stop.
<read *>

l.276 ^^M

!  ==> Fatal error occurred, no output PDF file produced!
* checking PDF version of manual without hyperrefs or index ... ERROR
* DONE
Status: 1 ERROR, 1 WARNING

This failure throws a warning for R, which is upgraded to an error because I have warnings_are_errors: true in .travis.yml.

I tried this StackOverflow answer of reinstalling inconsolata in .travis.yml:

before_install:
  - if [[ "$TRAVIS_OS_NAME" == "osx" ]]; then
      brew cask install basictex;
      sudo tlmgr update --self;
      sudo tlmgr update --all;
      sudo tlmgr install titling framed inconsolata;
      sudo tlmgr install collection-fontsrecommended;
    fi

and this TexStackExchange thread of changing the TeX repository and running the same commands as Travis does when installing the system:

before_install:
  - if [[ "$TRAVIS_OS_NAME" == "osx" ]]; then
      sudo tlmgr option repository http://mirror.ctan.org/systems/texlive/tlnet;
      sudo tlmgr update --self
      sudo tlmgr install inconsolata upquote courier courier-scaled helvetic;
    fi

and neither worked.

How can I install the inconsolata font?

This issue is due to the CTAN mirrors having yet to update. The issue appears in the Travis R community forum:

  might be related to the recent switch to TexLive-2019 as the LaTeX package manager seems to fail finding inconsolata in the repository.

  ...

  It looks to me like the mirror chosen was has not updated to 2019, so I think this will resolve itself naturally when the mirrors catch up. If not I can look into it.

In the meantime, I solved it by forcing a previous version of TeXLive with a downgrade to OSX 10.12 in .travis.yml:

# macOS 10.12 version
osx_image: xcode9.2

See this Travis page for the correspondence between XCode and OSX versions.


Logstash multiple pipelines going into same index I'm trying to set up a centralized syslog for multiple log sources.

So I have a logstash that has two separate inputs and two separate outputs

However for some reason the data from one of the inputs ends up in both indexes

What am i doing wrong?

Below are both pipelines' configs

input {
  tcp {
    port => 5052
    codec => "json_lines"
  }
}
output {
  elasticsearch {
    hosts => "10.50.6.116"
    index => "remote"
  }
  file {
    path => "/var/log/logstash/remote-tcp.log"
  }
  stdout { codec => rubydebug }
}

input {
  file {
    path => "/data/vmlist/*.csv"
    start_position => "beginning"
    sincedb_path => "/tmp/sincedb"
  }
}
filter {
  csv {
    separator => ","
    columns => ["VM Name","Creation Date","Owner","Type","Message"]
  }
}
output {
  elasticsearch {
    hosts => "http://10.50.6.116:9200"
    index => "vms"
    document_type => "csv"
  }
  stdout{ codec=> rubydebug}
}

There are two ways to accomplish this, though one of them was only available recently.

The old-school version, the one you can do as far back as Logstash 1.5, is to pay attention to tags and use conditionals to separate your inputs. Roughly...

input {
  tcp {
    port => 1525
    codec => json_lines
    tags => [ 'tcp' ]
  }
}

input {
  file {
    path => '/var/log/app.log'
    codec => 'json'
    tags => [ 'file' ]
  }
}

output {
  if 'file' in [tags] {
    elasticsearch {
      host => 'logstash-es'
      index => 'files'
    }
  }
  if 'tcp' in [tags] {
    elasticsearch {
      host => 'logstash-es'
      index => 'tcp'
    }
  }
}

This results in two inputs that output to two separate outputs. This is all one file, though. Elastic figured out people were muxing pipelines this way, and came up with a way to do multiple pipelines in separate files.

- pipeline.id: tcp-inputs
  path.config: '/etc/logstash/pipelines/tcp.cfg'
  pipeline.workers: 3
- pipeline.id: file-inputs
  path.config: '/etc/logstash/pipelines/files.cfg'
  pipeline.workers: 2

This approach is somewhat more maintainable since the pipelines are in separate files, and humans don't have to reason out how the flows work when presented in a single big file. Pipelines are available in Logstash 6.0 and newer.


Why do so many circleCI configs use `version: 2` for their workflows? I keep seeing this pattern around, and I haven't been able to figure out why people do it. I figure there's some dusty tome somewhere explaining why, or maybe just my google skills are off today, but can anyone tell me why so many people have workflows at version 2?
e.g. the example here, excerpt below
version: 2.1

executors:
  my-executor:
    docker:
      - image: buildpack-deps:jessie
    working_directory: /tmp

jobs:
  flow:
    executor: my-executor
    steps:
      - ...

  downstream:
    executor: my-executor
    steps:
      - ...

workflows:
  version: 2

  btd:
    jobs:
      - flow
      - downstream:
          requires:
            - flow

I figure I should probably be cargo-culting this in until I know why people do it, but I dislike doing that without any understanding :)
Thanks!

The Workflows version field was used to issue warnings for deprecation or breaking changes during v2 Beta, and it is deprecated as of CircleCI v2.1.
2 is the only valid number besides 1


Why does my Terraform config require variables in multiple places? My folder hierarchy looks like this:

/
  -> live/
    : main.tf
    : variables.tf
  -> modules/
    -> logs/
      : logs.tf
      : variables.tf

In my main.tf I have:

module "logs" {
  source = "../modules/logs"

  env = var.env
  ...
}

In my logs.tf I have:

resource "datadog_monitor" "logs-fatal" {
  env = var.env
  ...
}

In both variables.tf files I have:

variable "env" {default = "test"}

If I remove this from either of them, I'm given an Unsupported argument error - why? It's unclear to me in this scenario which variable will be taking precedence and why both sets are needed.

Each module has its own set of input variables.

The input variables for the root module are set when you run Terraform, e.g. on the command like with the -var option.

The input variables for any child module, like your module "logs" here, are set inside the module block.

Each variable declaration is valid only for the module it appears in. You happen to have given both of these variables the same name, which is fine and sensible as it helps a human reader understand that they represent the same concept, but Terraform considers these two variables entirely separate and requires you to declare and set both of them.

As an analogy, consider a module as being somewhat equivalent to a function in a programming language. Each function defines its own arguments, and two functions might have an argument of the same name but they are still distinct by virtue of belonging to different functions:

function main(env: string) {
  logs(env)
}

function logs(env: string) {
  // "env" in here is "test" because it was passed through
  // the arguments of both of these functions, not because
  // the arguments happened to have the same name in both
  // functions.
}

// Calling main here is, for the purposes of this analogy,
// corresponding to running: terraform apply -var="env=test"
main("test")


Packer and compressed ISO images I am not using packer yet, just looking through the documentation. Some VMs it supports can be built from iso images. The examples cover use cases where the ISO is available online through the iso_url key in the JSON description file. But, can Packer handle the cases where the iso file is compressed (e.g. https://example.com/images/image.iso.xz)?

I tried the dumbest approach, i.e. just to plug the iso.xz file into iso_url and see what happens. Of course, this failed (even after manipulating the iso extension key).

What I ended up doing --- knowing a lot about Bash scripting, and not much about Packer, Go and the like --- is to wrap packer calls in a script that downloads and decompresses the image, then calls Packer on the decompressed image.

I would be curious to know if there are more elegant approaches.


Running parallel builds from jenkins pipeline job triggers and waits for same job on other project I have this test pipeline code, it runs 2 build on different nodes on build_servers as expected.

pipeline {
    agent { label 'myserver' }
    stages {
        stage('1') {
            steps {
                script {
                    def tests = [:]
                    for (f in [ 'x', 'y' ]) {
                        tests["${f}"] = {
                            node('build_servers') {
                                stage('Clean Workspace') {
                                     echo "cleaning up Ws ${f}"
                                     step([$class: 'WsCleanup'])
                                }
                                stage('create Wspace txt file') {
                                        script {
                                            props="Wspace=${workspace}"
                                            dir ( 'wspace') {
                                                    writeFile file: 'params.txt', text: props
                                            }
                                        }
                                }
                                stage('Run other job') {
                                    script {
                                        println('calling other project')
                                        props = readProperties file: "wspace/params.txt"
                                           file_params = props.collect {
                                                string(name: it.key, value: it.value)
                                        }
                                        build job: "other_job2", parameters: file_params, wait: true, propagate: true
                                    }
                                }
                            }
                        }
                    }
                    parallel tests
                }
            }
        }
    }
}

I expect it to trigger 2 different builds on other_job2, but it submits just one job and both tests['x'] and tests['y'] waits for the same job to finish, why is that? should it not create 2 different  jobs in other_job2 project, one for x and one for y?

in the console output, I see build #5 for both [x] and [y] exactly as shown below:

[x] Starting building: TEST » other_job2 #5
[y] Starting building: TEST » other_job2 #5

Also, I am printing ${f} in echo "cleaning up WS... line, I expected to see both x and y, but I see only cleaning up Ws y printed for both [x] and [y] as below

[x] cleaning up Ws y
    [Pipeline] cleanWs
[x] [WS-CLEANUP] Deleting project workspace...
    [WS-CLEANUP] Deferred wipeout is used...
    [WS-CLEANUP] done
    [Pipeline] echo
[y] cleaning up Ws y
    [Pipeline] cleanWs
[y] [WS-CLEANUP] Deleting project workspace...
    [WS-CLEANUP] Deferred wipeout is used...
    [WS-CLEANUP] done

Something needs to be different in other_job2 for tests['x'] and tests['y'].

Example(untested),

for (f in [ 'x', 'y' ]) { {
    {some code}
    build job: "other_job2", parameters: [file_params, string(name:'dummy', value: "${f}")], wait: true, propagate: true
    // the dummy parameter is for preventing mutation
}

More info:
https://www.jenkins.io/doc/pipeline/examples/#jobs-in-parallel


jenkins - gitparam jobdsl - branchFilter I want to fill branchFilter automatically on Jenkins using JobDSL plugin and language.

This is the url of Jenkins Job DSL plugin:

https://jenkinsci.github.io/job-dsl-plugin/#path/javaposse.jobdsl.dsl.jobs.FreeStyleJob.parameters-gitParam

I want to select all branches but master. To do that I have written the following lines:

  //This Build is parametrized
  parameters {
      gitParam('branch') { // https://jenkinsci.github.io/job-dsl-plugin/#method/javaposse.jobdsl.dsl.helpers.BuildParametersContext.gitParam
          description('Git Branch')
          type('BRANCH')
          branchFilter("^(?!.*master).*$")
      }
  }

But when compiling it I get error that such method is not available.

  Processing provided DSL script ERROR: (script, line 28) No signature
  of method: javaposse.jobdsl.dsl.helpers.GitParamContext.branchFilter()
  is applicable for argument types: (java.lang.String) values: [*]
  Possible solutions: tagFilter(java.lang.String)

Is there any workaround to do this automatically on the script or we have to fulfill that field after job has been created?

This is how I did:
gitParameter {
            // The name of the parameter.
            name('SELECTED_BRANCH')
            // The type of the list of parameters: Tag - list of all commit tags in repository - returns Tag Name Branch - list of all branch in repository - returns Branch Name Revision - list of all revision sha1 in repository followed by its author and date - returns Tag SHA1
            type('PT_BRANCH')
            // In my case this is a variable pointing to the gitlab hook variable
            defaultValue('$gitlabAfter')
            // A description that will be shown to the user later.
            description('Select a git the configuration you want to deploy')
            // Name of branch to look in.
            branch('')
            // Regex used to filter displayed branches.
            // Whatever is captured in the group will be the name used as the value of the SELECTED_BRANCH
            branchFilter('origin/(config\\/.*)')
            // This parameter is used to get tag from git.
            tagFilter('*')
            // Select how to sort the downloaded parameters.
            sortMode('NONE')
            // Which value is selected, after loaded parameters.
            selectedValue('NONE')
            // If in the task are defined multiple repositories, this option specifies which the repository is taken into account on getting data.
            useRepository("<my git url>.*")
            // When this option is enabled will show a text field.
            quickFilterEnabled(false)
            // Specify the number of items the list will display.
            // listSize(String value)
        }

As strange as it seems, some parameters shouldn't really be necessary, but the plugin requires them anyway, so that's why the config is verbose.
Also, pay attention that this one is gitParameter and not gitParam, yes, both exist, and they take different parameters.
For more information about the properties you can go here:
https://plugins.jenkins.io/git-parameter/


Jenkins 2.164.3 - programmable plugins installations for specific versions Which tool should I be using in order to be able to control Jenkins plugins from a cli/script in such way that when I have a new installation (e.g. 2.164.3)?

For example, I want to type blueocean - 1.1.15 (older version then current for example) and it will download the dependencies for the given version without messing everything else?

As I see I shouldn't be using jenkins-cli, but https://github.com/jenkinsci/plugin-installation-manager-tool? If so can anyone provide a simple example for the scenario I have given as the documentation is short of simple/clear examples.

The already existing topics are not answering the question in 2020 and I can't solve my scenario with them.

Why not use Plugin installation manager tool?

Your alternative would be the install-plugins.sh referenced in the tool readme.md. You can explicitly list all plugins and versions or let it auto-resolve dependencies.

See also answer in S/O re:Jenkins plugins for old versions.


Using Amazon VPC provided DNS trough AWS Site-to-Site VPN Currently I have an AWS VPC connected to our corporate network using AWS Site-to-Site VPN.

I can reach my EC2 instances trough the tunnel by their private IP Address.

Inside the VPC each EC2 instance can resolve private DNS hostnames ip-private-ipv4-address.region.compute.internal using the provided Amazon DNS server (reachable at IP address VPN-CIDR +2)

Is it possible to query Amazon DNS server from our corporate network trough the VPN tunnel?

The goal I want to achieve is the possibility to reach EC2 instances through VPN using a DNS name without create each time a record in our DNS.

Is it possible to query Amazon DNS server from our corporate network trough the VPN tunnel?

Directly? No. The recursive resolver available by default when you create a VPC can only be accessed by an IP address in that VPC. This is a restriction that Amazon imposes.

What you can do is create a forwarder using the Route 53 Resolver service or software like Unbound. You can then (conditionally) forward your onprem DNS for certain domains to it.

Some useful links:

https://docs.aws.amazon.com/vpc/latest/userguide/VPC_DHCP_Options.html#AmazonDNS

https://docs.aws.amazon.com/vpc/latest/userguide/vpc-dns.html#vpc-dns-support

https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/resolver.html


How can I pipe jq output I recently learned about an outstanding little tool, jq that allows you to pipe unformatted JSON output into it and then have that output reformatted and output to the screen in a very nicely formatted color-coded json layout. Eg:

echo '{"value": "New", "onclick": "CreateNewDoc()"}' | jq

I have found however, that I am unable to pipe the formatted results (from jq) into additional tools in my standard toolset, such as grep, sed, awk, etc. How can I make make this output pipe-able? Eg, this doesn't work:

'{"value": "New", "onclick": "CreateNewDoc()"}' | jq | grep value

Ok tried to understand your problem exactly... This is what is happening:

$ echo '{"value": "New", "onclick": "CreateNewDoc()"}' | jq | grep value
jq - commandline JSON processor [version 1.5-1-a5b5cbe]
Usage: jq [options] <jq filter> [file...]
[rest of output is omitted for brevity]

The important thing to notice here is the usage line (emphasis is mine):

  Usage: jq [options] <jq filter> [file...]

The filter is not optional, if you don't give one, jq tries to parse the rest of the command line as its filter and throws an error. A workaround for your case here is:

$ echo '{"value": "New", "onclick": "CreateNewDoc()"}' | jq '.' | grep value
  "value": "New",

which indeed greps the "value" line as the filter passed is '.' - this just pretty prints the json content, but that's far from the best use of jq. If you wish to get only this line it would be better do:

$ echo '{"value": "New", "onclick": "CreateNewDoc()"}' | jq '.value'
"New"

If you wish to get the output without quotes then you can add add -r option to jq like so:

$ echo '{"value": "New", "onclick": "CreateNewDoc()"}' | jq -r '.value'
New

from jq --help:

  -r     output raw strings, not JSON texts;

that's a little introduction to jq, that's probably not enough to solve your problem at all but as you didn't specify your problem I can't help more than that.


How to run Ansible command to update imunify360 settings How can I use Ansible to run the following CloudLinux Imunify360 commands properly on CloudLinux 7 (similar to CentOS7)?
I tried different ways to "escape" the colons in the commands and it doesn't run properly, even when a YAML syntax checker showed it's correct.

imunify360-agent config update '{"MALWARE_SCANNING": {"rapid_scan": true}}'
imunify360-agent config update '{"MALWARE_SCANNING": {"default_action": "cleanup"}, "MALWARE_SCAN_SCHEDULE":{"interval": "week"}}'
imunify360-agent config update '{"MALWARE_SCANNING": {"enable_scan_inotify": true}}'

from:
https://blog.imunify360.com/new-default-settings-for-imunify360

Note: Imunify360 has a free trial.

It turns out single quote works where double quote and backslash didn't.

- name: Enable rapid scan
  command: imunify360-agent config update '{"MALWARE_SCANNING"':' {"rapid_scan"':' true}}'

I learned that folding block scalar (>) is an easy alternative and a good way to avoid escaping.


Is it possible to use a list with 'file.exists' in Salt? I asked a question recently regarding the most efficient and scalable way to check which files/services exist in multiple environments. Thanks to an answer I have successfully set up roles for all of my salt minions.

I am now in the process of writing my first salt state which will check that all expected files/services are present for all minions with the apiserver role.

I am using the file.exists feature, however as you can expect there are 100's of files/services I wish to check, I'm sure there must be a way of listing these files/services in a list or string? I can't find anything in the documentation under file.exists but it is possible in other file.'s

Here is the current state of my init.sls file:

# This state will check all minions identified as 'apiserver' for files and/or services which should exist

# This section checks the 'checkservices.sh' is present in /opt (used for Jenkins check jobs)

checkservices.sh:
  file.exists:
    - name: /opt/checkservices.sh

# This section checks for expected files/scripts within /opt/apiv2/

apiv2.properties:
  file.exists:
    - name: /opt/apiv2/apiv2.properties

jvm.options:
  file.exists:
    - name: /opt/apiv2/jvm.options

restart.sh:
  file.exists:
    - name: /opt/apiv2/restart.sh

status.sh:
  file.exists:
    - name: /opt/apiv2/status.sh

shutdown.sh:
  file.exists:
    - name: /opt/apiv2/shutdown.sh

start.sh:
  file.exists:
    - name: /opt/apiv2/start.sh

# This section checks for 'bootstrap.jar' in /opt/apiv2/repo/

bootstrap.jar:
  file.exists:
    - name: /opt/apiv2/repo/bootstrap.jar

My second question is; is my current method (above) considered bad practice or is it just an ugly/inefficient way of doing it?

Yes - there are at least two options available for managing files in the manner you describe. The first such way is to manage the entire directory using file.directory:

/opt/apiv2:
  file.directory:
    - user: root
    - group: root
    - dir_mode: 755
    - file_mode: 644
    - recurse:
      - user
      - group
      - mode

The second way can manage a manifest of several files at many paths is to use the source_hash feature of file.managed:

This allows you to provide a source tar file and a file containing md5sum hashes:

apiv2-0.7.3.tar.gz:
  file.managed:
    - name: /tmp/apiv2-0.7.3.tar.gz
    - source: salt:///apiv2/distrib/apiv2-0.7.3.tar.gz
    - source_hash: salt:///apiv2/distrib/manifest-0.7.3.hash

You would then create a text file on your salt server /srv/salt/apiv2/distrib/manifest-0.7.3.hash with contents similar to:

37b51d194a7513e45b56f6524f2d51f2    /opt/apiv2/apiv2.properties
acbd18db4cc2f85cedef654fccc4a4d8    /opt/apiv2/repo/bootstrap.jar
73feffa4b7f6bb68e44cf984c85f6e88    /opt/apiv1/apiv1.properties

Your might even be able to use templating,

apiv2-0.7.3.tar.gz:
  file.managed:
    - name: /tmp/apiv2-0.7.3.tar.gz
    - source: salt:///apiv2/distrib/apiv2-0.7.3.tar.gz
    - source_hash: salt:///apiv2/distrib/manifest-0.7.3.hash
    - template: jinja

however Salt Stack might want to push down the file each time because after the templates are rendered, the hash will have changed, so the creators might just have elected to throw an error if you try these two features together. In short, Your Mileage May VaryTM


Automated patching of docker images Docker and other container images have become important building blocks especially for software production pipelines. Unfortunately, several researchers (e.g. BanyanOps) have shown how vulnerable docker images can be , including those from the official distributors. I was researching into the possibilities for automatic patching of such vulnerable images when discovered. According to this link, container patching is very sensitive and error-prone. Infact, there seems to be no solution as all the papers/reports about container image vulnerabilities do not proffer patching approaches. I will be grateful if someone has some opinions, options or even a possible solution.

Note: I am interested in how patching is conducted with or without update notifications.

As this question is quite high-level, I give a high level answer.

Technically, if there is an incident, you would normally enforce a shutdown of a running container and a replacement with a newer image with an implementation of the change request resulting from the incident.

How much automation will be possible to get there, I think it really dependends on single cases or at best their classes.

Best is therefore I think to assess different scenarios, group them into classes and derive solution workflows. Once a problem catalogue and a required set of organization policies are there you will be able to sort out which tools can help to realize them and which automations grades are possible.

For example:

discovered system environment level vulnerabilities are normally patchable through operation system security upgrades. You could have batch job delivering daily updated base images.
application environment level changes like "let's migrate from Java 8 to Java 9" is an organizational decision which requires an organizational process
application code level vulnerabitites requiring updating to patched versions requires some ways of dependency management governance in your organization, tools for quality gates are available but without an organizational decision nothing will happen here.
default credentials in production - again where do they come from? This is very special case and possibly a design failure; not sure how solve this in an automated way.

So I have to conclude that this question is more about culture than automation.


GIT Monorepo with enterprise Jenkins Need bit of help regarding Jenkins CI jobs. I am pretty new to Jenkins world

As per the current set up in our project we have different repositories in GIT and they have web hooks setup with Jenkins to trigger CI jobs in the event of any push. Please note there are separate CI job for different repos.

My requirements :

Build a mono repo which will consist all existing repos. ( for test purpose I have created 2 repos and merged them in mono repo ).
Set up Jenkins job which will trigger task only for repo where PUSH has occurred.
e.g if my monorepo consist 2 merged repos repo1 and repo2 , so if any changes in repo1 should trigger only related task for repo1 and likewise for repo2.

I’m not sure how could I achieve point 2. Any help?

My personal opinion is that each deliverable should reside in its own repository. However if one would like to keep the monorepo one could use changeset in conjunction with when:

pipeline {
    agent any
    stages {
        stage('build matchengine') {
            when {
                changeset "**/matchengine/*.*"
            }
            steps {
                echo 'building match engine'
            }
        }
        stage('build posttrade') {
            when {
                changeset "**/posttrade/*.*"
            }
            steps {
                echo 'building post trade'
            }
        }
    }
}

or change the directory in several build stage and build the code inside this directory:

stage('Build') {
  dir('web_app') {
    sh 'ls'
    sh 'git pull'
  }
}

One could also decide to use the parallel option to ensure that the build times will be shortened:

/* .. snip .. */
stage('run-parallel-branches') {
  steps {
    parallel(
      a: {
        echo "This is branch a"
      },
      b: {
        echo "This is branch b"
      }
    )
  }
}
/* .. snip .. */

Regarding the question:

  Set up Jenkins job which will trigger task only for repo where PUSH
  has occurred. e.g if my monorepo consist 2 merged repos repo1 and
  repo2 , so if any changes in repo1 should trigger only related task
  for repo1 and likewise for repo2.

I think that changeset would be most useful: a change will trigger the build, but as a change only occurred in a certain path, only that build will be started.


Vagrant inside a VM to control the VM host Background

I have an application which comes with a Vagrant setup that is designed for this usage scenario:

Laptop (Windows, Virtualbox host) => edit source code
+----> Vagrant VM (Linux) => build and run

Plain, default, simple, works. The sources are on a shared/synced directory, i.e., they are usually edited on the Windows host and then compiled and ran inside the VM.

Unfortunately, I prefer to use Linux development tools myself. Right now, I have this setup:

Laptop (Windows)
+----> Vagrant VM
+----> My personal VM (Linux, non-vagrant)

So my editor etc. run in the personal VM, and I have a common directory shared through Virtualbox. Again, everything works just fine.

Here is the problem:

To control the vagrant stuff, I need to enter the vagrant commands in the Windows shell, which is slowing me down. For example, to run the build inside the vagrant VM, I would prefer to use a hotkey in my editor to start vagrant ssh build.sh instead of switching to the CMD.EXE and entering that command. Unneccessary context switch.

Question

I don't wish to open the can of worms that is "virtualization inside virtualization" (i.e., I won't run Linux Virtualbox inside a Windows-hosted VM); so this keeps me with these two alternatives:

Is there a way to control the "Windows-Vagrant" that is running on the Windows host from inside a Linux VM (which itself is not under Vagrant control), i.e., to use vagrant up or vagrant ssh?

Alternatively, is it possible to install "Linux-Vagrant" in my existing VM right away (i.e. the Debian version) and have it create its VM in the Windows host?

Or is there a completely different way to solve this problem in an elegant fashion?

Is the answer as simple as setting up a dual-booted machine? Linux partitions for development tasks and as the VM host, Windows partitions for normal computing use.

Laptop

Windows partition
Linux Partition

Vagrant VM


Task to delete users without mount points in /home (using register and when) I have a list of users that I want to delete (del_users), but I only want to delete those that haven't mountpoint on your /home directory. My currently code is:

- name: Verify if exist mount point on /home
  shell: "mount -l | grep bind | grep -ci {{ item }}"
  ignore_errors: yes
  check_mode: False
  loop: "{{ del_users }}"
  register: del_users_reg

- name: Delete user if doesn't exist mount point on /home
  user:
    name: "{{ item }}"
    state: absent
    remove: yes
  when: item.stdout_lines == 0
  loop: del_users_reg

When I execute this task appears the next error:

TASK [osbasic : Verify if exist mount point on /home] *******************************************************************************************************************************************************************************************
 [WARNING]: Consider using the mount module rather than running 'mount'.  If you need to use command because mount is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid
of this message.

failed: [LGAUSUX10] (item=user1) => {"changed": true, "cmd": "mount -l | grep bind | grep -ci usuario1", "delta": "0:00:00.025089", "end": "2019-06-11 13:33:40.978563", "item": "user1", "msg": "non-zero return code", "rc": 1, "start": "2019-06-11 13:33:40.953474", "stderr": "", "stderr_lines": [], "stdout": "0", "stdout_lines": ["0"]}

TASK [osbasic : Verify if exist mount point on /home] *******************************************************************************************************************************************************************************************
...ignoring

TASK [osbasic : Delete user if doesn't exist mount point on /home] **********************************************************************************************************************************************************************************

TASK [osbasic : Delete user if doesn't exist mount point on /home] **********************************************************************************************************************************************************************************
fatal: [LGAUSUX10]: FAILED! => {"msg": "Invalid data passed to 'loop', it requires a list, got this instead: del_users_reg. Hint: If you passed a list/dict of just one element, try adding wantlist=True to your lookup invocation or use q/query instead of lookup."}
        to retry, use: --limit @/etc/ansible/osbasic.retry

PLAY RECAP **********************************************************************************************************************************************************************************************************************************
LGAUSUX10                  : ok=9    changed=1    unreachable=0    failed=1

How could put a condition for delete users only if the result of first task is 0?

Several problems in your playbook.

loop is basically expecting a list, not a string containing a variable name. You need to expand your variable with a jinja2 expression => loop: "{{ some_list_variable }}"
It will be hard to give you the correct syntax without seeing the exact data structure of your result var but... Typically, the register from a loop contains a results list of hashes, each containing info about the ran command for each loop iteration. This is your base loop for the second task => loop: "{{ del_users_reg.results }}"

If you still have trouble fixing your playbook, update your question with a debug of the registered var and I will update my answer accordingly.


Cloudfront for ELB, automatic redirection I have a question regarding a redirect (302) that happens when accessing the /admin panel of my Craft CMS application.

I've setup CloudFront for my app that runs on Craft 3. DNS points to CF (CNAME), CF has Elastic Load Balancer (ELB) as a default origin. The ELB is a subdomain on my main domain (for certificates reasons). There is a CNAME set for ELB on the main domain DNS.

Now, when I'm visiting any page on my application, there is no redirection happening (eg. when I go to www.myapp.com/product). Only when I'm visiting the /admin panel, I get redirected to the ELB domain. Eg. I go to www.myapp.com/admin and I get redirected to www.elb-domain.awsloadbalancerdomain.com (ELB's CNAME).

Any of you has any ideas about why this might be happening and how to prevent it?

Thank you in advance!

As the issue is happening only with /admin URL, this might be related with cookies which are being passed to the backend application's /admin URL.

You may need to check with CraftCMS team (or some other forums) and get the list of cookies which are being used by the /admin URL. You can whitelist those cookies in cloudfront specifically for /admin URL.


Apply resilience-testing in a company (to promote DevOps)? If true, why, how and what effect does it have on the Engineers? If false, why not? Back in the day this Q&A was created.

ChaosMonkey is a tool that is able to stop random services on a platform in a certain time period. The goal of this is to get a stable platform that is fault tolerant. If this happens during the day when most of the Engineers are working, they get proactive and the infrastructure will be hardened.

At the moment, more and more services are created in AWS, GCP and Azure. The services on these platforms are getting more and more critical. Now it is time to onboard more and more engineers. Especially most of the developers are still silo minded, i.e. some of them only want to develop, but this is shifting drop by drop.

According to Wikipedia, ChaosMonkey was invented at Netflix and additional resilience tools were created as well:

| Purpose                                        | Tool              |
|:----------------------------------------------:|:-----------------:|
| Stops random services in production randomly   | Chaos Monkey      |
| Drop an availability zone, e.g. in AWS         | Chaos Gorilla     |
| Communication delays                           | Latency Monkey    |
| Fix unhealthy services                         | Doctor Monkey     |
| Dispose unused resources                       | Janitor Monkey    |
| Detect nonconforming instance and notify owner | Conformity Monkey |
| Disable instances that are vulnerable          | Security Monkey   |
| Detect problems between 10-18                  | 10-18 Monkey      |

Based on this table one could also achieve also the same purpose by start to write some shell scripts that stop random services for example.

Based on the information that was read, the definition of resilience-testing is basically testing if an app and underlying infrastructure is error prone in production.

This is also confirmed by this blog post that refers to how resilience testing is done at AWS, but also describes IBMs view on this type of testing. According to IBM there are two parts that have to be taken into account, i.e. problem impact and service level. In an ideal world the customer will not notice any downtime at all, but if a machine dies (minutes) or a complete data center (hours) then the customer could experience downtime. The most important thing to be transparently as possible to the customer and to minimize impact.

Currently, resilience-testing is not used in the company, but the more is read about it, it becomes interesting. Especially as it is also aligned with DevOps and the company tries to make Engineers DevOps minded. It looks like that resilience-testing promotes a DevOps mindset.

Resilience testing is DevOps related

Some people could argue that testing is not part of DevOps, but according to Wikipedia, tools like ChaosMonkey are aligned with the DevOps toolchain.

I think that chaos testing will promote DevOps unless people have been informed in advance and proper alert management has been enabled. If people do not get triggered then this will not work.


Restrict user logon in Jenkins We have been using groovy script to read the build queue state:

def builder = new groovy.json.JsonBuilder()

builder {
  queueItems(Jenkins.instance.queue.items.collect { obj ->
      [
        id: obj.id,
        name: obj.task.name,
        isStuck: obj.stuck,
        isBlocked: obj.blocked,
        isDisabled: obj.task.disabled,
        inQueueSince: obj.inQueueSince,
        why: obj.why,
        causesDescription: obj.causesDescription,
        queueItemUrl: obj.url,
        taskUrl: obj.task.url
      ]
   })
}

println builder.toPrettyString()

Then we add Azure AD plugin. It requires an account with the hudson.model.Hudson.Administer permission to run a script.

We use a tech account, but the problem is that everyone can logon interactively using this login and became Jenkins admin.

Is it possible to have this account enabled, but restrict interactive logon to Jenkins?

If you're running this code today via script console, you're probably running into a RejectedAccessException from the sandbox which is why you require admin access to run this coupled with and or needing administrator access to use the script console.

If you put this code into a shared library, the code becomes trusted, and you no longer require administrator access to run the code and you no longer need script console. You just need a job that can call the shared library, if needed you can restrict access to the folder or job.


Is it possible to run every testing.xml file with different jenkins job? I've around 100 test method in my testing.xml file. and that testing.xml configured in pom.xml file so I'm able to run my all those testcase as maven test and I'm getting the proper reports and all.

I have created one Jenkins job and configured maven project in that and mentioned pom.xml file path of local. I'm able to run successfully.

I'm planning to separate some testcase category wise and manage then in another testing.xml file. lets say 2 xml file having 50 50 test methods.

I'm stuck here. Is it possible to configure every testing.xml file in separate Jenkins job ? if yes then how do i configure ? (Note: I have to run maven test for getting the reports)

I got the answer on StackOverflow :

You can pass the xml file name as parameter to the maven test command. First need to change the pom file as follows.

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-surefire-plugin</artifactId>
    <version>2.19.1</version>
        <configuration>
            <suiteXmlFiles>
                <suiteXmlFile>${SuiteXmlFile}</suiteXmlFile>
            </suiteXmlFiles>
        </configuration>
</plugin>

Then, you pass the different file name for each execution in goals and options like :

test -DSuiteXmlFile=mytestng1.xml

For second xml file :

test -DSuiteXmlFile=mytestng1.xml

You can configure each Jenkins job with different parameter.


Health monitoring of web pages without using GET check I am looking for a solution to monitor the health (DOWN or UP) of my web pages.
Let's say I have three services,

site.com/data
site.com/user
site.com/id

Now, what I can do is that I can create a GET request which will be served every second and check if response status code is 200 OK. If it is anything else than 200 OK, then it will trigger an alert. But, I don't want to hit the server continuously, as it will create unnecessary logs. I am using a mongodb logger. What is the recommended way to achieve this?

I see two way to solve this problem:

Check your logs often and ensure there was an access with code 200 in the last N seconds/minutes and no code 5xx meaning there's a server side error. (the often should be coherent with how long you are ok with no entries)
Keep an active check but:

use a HEAD request (so there's less data to return by the server)
use a custom user agent for the check
configure your log exporter to ignore this user agent.
take a larger check interval, 1 second is probably over obsessive

Depending on how often your sites are called you may wish to mix both methods to have a proper check even if there's no traffic from real clients.


Is DevSecOps indeed a new practice worth a professional term? Or rather marketing slang?
Currently, I am not always sure how to build a consense which words deliver informational value and which are thought to attract customers.
So, DevOps would include security testing if required without calling it DevSecOps? Otherwise we will have DevPerfOps,  DevDevOpsOps etc

In my experience, the purpose of the "DevSecOps" (or whichever order you prefer) has mostly to do with including Security as a first-order concern of software delivery.  Security is commonly treated as a "bolt-on" (often meaning optional,) non-functional requirement.  DevSecOps attempts to describe a process where security concerns are folded into the delivery process at every step of the value stream.


How to run a single one-time command on a kubernetes instance/pod Currently I have a setup on AWS with 4 EC2 machines. They provide a web interface that among other things also has a database search. My recent project was to move the database search to AWS Elastic Search. That took a while but in the end it was fine, because to create the initial index I simply had to SSH into one of the machines, run the command to create the index and I was done.
I could not have done it without SSH'ing into it because the production environment (meaning DB and ES instances) is not directly accessible from dev PCs (which makes sense to me).
So, being quite new to Kubernetes my question is: if we were to migrate these EC2 instances and switch to a Kubernetes kind of setup, how would I run the initial command to create the ES index?
I know I can connect to a pod when it's a single container but these would most likely have containers at least for nginx, redis and django (the one I'd want to connect).

A little more information is needed about specific operations you are doing, but generally your options are:

For one-time operations: use kubectl exec to get into containers and do commands (works similar to docker exec).

Also for one-time operations, spawn separate pods with shell access to interact with existing pods, something along the lines:

kubectl run -it --rm --restart=Never --image busybox tempbusybox -- sh

For reproducible (not one-time) operations, you may use kubernetes jobs.

In some cases you may use init containers for setup tasks.


What are the security risks of CD in an open source project? In the processes of setting up continuous deployment for an open source project with Travis I came across a predicament, can the encrypted private key be easily exfilitrated? For context, the deployment workflow is:

In a Travis build job, build and test code
If it passes, deploy the code
Decrypt the encrypted private SSH key and start the SSH client
Push to the Git remote on the production server

Using travis encrypt-file deploy_rsa --add, the private key is encrypted and only decryptable within the Travis build job. Am I correct in the assumption that someone forking the repo and creating a Travis build job will not be able to decrypt the key? In addition, the only attack vector would be for someone with push rights to modify the .travis-ci script to decrypt the key and send it to themselves?

You essentially have three threats here:

Someone could modify .travis-ci and use that to exfiltrate the unencrypted key material.
You could accidentally check-in or publish the decrypted key material as part of your release process.
Someone could attempt to attack the encrypted key-material directly; this is very unlikely given modern practice and current computing power.

The challenge of secret management is common in open source projects; one approach I have taken in the past is to have a second "deploy" stage in a private repository and pipeline.  A private repository and pipeline, then ring-fences the sensitive data from malicious actors entirely, protecting yourself from 1 and 3.  You can safeguard against two by whitelisting files that can be checked in to decrease the probability of making a mistake.

Finally, an approach that can be used to increase the detectability of an attack is to store the key in AWS CloudHSM, Google Cloud HSM or Azure KeyVault with the goal of marking the private key as non-exportable and then handing off any encrypt, wrap or sign requests to the cloud.  This means even if the credentials for the HSM/KeyVault do get exfiltrated you know every time they are used to sign something and can effectively revoke access to the key entirely.  Unfortunately, there doesn't seem to be any way of doing this with SSH keys at the moment, however, it works will with code-signing certificates.


Is DevOps restricted to companies with SaaS products? The practices describing DevOps, such as continuous delivery, automation, etc. are relevant to products that provide continuous service, such as SaaS products.

For example, a software development company that mostly does projects for other clients might never be maintained these after the project is over. And client projects are not shared with other clients, because irrelevant.

Does DevOps even apply to companies who develop multiple projects that are one-offs? What DevOps practices apply in this case, if at all?

Absolutely not!

DevOps is all about breaking down the traditional silos (departments) in order to be more efficient.

Better communication between teams, improved visibility and reliable and automated process are ways to achieve a better product.

I used to work for a big media company where we would support an internal tool and develop public-facing websites.

The benefits of DevOps in our case were the following:

Through continuous building, we let know the development team earlier rather than later if there are integration or build problems with their code. They can fix issues while their mind is still on the piece of code they just committed.
Through continuous testing and delivery (into QA), we enabled the QA team to find problems earlier and report them earlier. This reduced the time it took to find and correct bugs as well as reduce the complexity of these investigation.
With out log collection & aggregation tools, we gave to the developers access to something they wouldn't usually look at (they were very keen on the debuggers :) - understanding how logs are seen and used by other teams improved the overall quality of logs
We often shared information and created documentation to share knowledge between teams, trying to break down walls. By understanding the Ops' needs, we create a few guides for what should always be kept in mind when bootstrapping an application (where/how to manage properties, etc.). By understanding the Dev's reality (code more features, faster, gogogogo!) we were able to have the ops create servers and clusters that were better suited to the dev's needs.
The overall quality of deployments was greatly improved too. Deployments were handled by our team, so we had perfect visibility on both Ops and Dev. This eliminated many issues related to the "code hand-off" where the dev would hand over a package and one-page document to the ops saying "Install this!".

Overall, I would say that regardless if you are updating your production environment once per day or once per month and regardless of how many customers you have or your business model, every enterprise can use better communication, better tools, better visibility, faster feedback, etc.


backup dockerise mysql database from host volume I use a VPS (my host) where a LAMP server was installed.
I already have a database associated to host mariaDB server Ver 15.1 Distrib 10.1.47 and automysqlbackup backups it. This part works well.
For a "new project", we use now docker servers and one with a mysql 5.7 and a volume to save this new project database in a host folder, for exemple: /path/on/host/dockerise/db/
I want to backup too this new database.
What is the right way?

Is there a way with automysqlbackup? I think no because it uses host mariaDb server and mysqldump, and you shouldn't have two servers working on the same database (https://dev.mysql.com/doc/refman/5.7/en/multiple-data-directories.html)
Directly with mysqldump ?
Or just create a bash script and CRON it?

I have a little script for my MariaDb database inside a docker container.
The setup is comparable. I have a database container with a mounted volume. Furthermore I have a second volume for backups mounted under /backup.
The script is run on the host via cron and uses the database instance inside the container, so only one database accesses the data. That is the docker exec part.
I'm sure, you can adopt it to mysql. You only have to configure the CONTAINER_NAME and change the command inside the container.
#!/bin/bash
DATE_TIME=`date +"%Y_%m_%d_%H_%M"`
CONTAINER_NAME="mariadb"
CONTAINER=`docker ps -q -f name=${CONTAINER_NAME}`
PASSWORD=`docker exec ${CONTAINER} sh -c 'echo $MYSQL_ROOT_PASSWORD'`
BACKUPFILE=${DATE_TIME}_${CONTAINER_NAME}.backup.gz

echo "Backup of MariaDb"
echo ""
echo "Old backps will be deleted."
echo "Datenbank Container Name: $CONTAINER_NAME"
echo "Datenbank Container:  $CONTAINER"

echo "Writing backup to ${BACKUPFILE}"
echo ""
docker exec ${CONTAINER} sh -c "find /backup/ -mindepth 1 -mtime +8 -delete; /usr/bin/mariabackup --backup --user root --password=${PASSWORD} --stream=xbstream | gzip > /backup/${BACKUPFILE}"
echo ""
echo "Backup finished"


Running Ansible no-op's faster with a dynamic inventory and cached facts files I've got an ever expanding playbook that patches a few hundred machines to the latest version, I just add the same role with different parameters.

Obviously, the playbook is taking longer and longer to run on each machine, even though it's not really doing anything on them.

What I'm doing is in the pre-tasks, I'm getting a changelog file, checking that and running each of the roles when an identifier that a successful run puts in that changelog file does not exist.   But, even so, ansible still takes a fairly onerous amount of time to do nothing.

When I strace the main ansible process it looks like its trying to hit a bunch of non-existent cached facts files and the inventory script.

I think I've created the dynamic inventory correctly, it accepts both the "--host" and "--list" parameters (although I'm not certain that --host is being called).

I've got about 100 devices that I haven't gathered facts for in the inventory file and I've got:

fact_caching_connection = /tmp/factscache
fact_caching = jsonfile

set up.

If I take that out, it doesn't try to hit up the missing facts files, but it there is a stat in the strace against the inventory generator file over and over again. I know it can't be generating the entire inventory because that take 5 seconds.

I've tried generating regular inventory files and using those, I've tried generating inventory files with both _meta and the regular host_vars (currently using _meta to house the host vars because I heard that was faster).

(running ansible 2.0)

Instead of storing the Ansible code centrally, one could extract the code and distribute it to every project. Another benefit of this approach is that config and code will be seen together and that developers will change config and Ops could fix e.g. bugs instead of working in silos. Running the code per project will prevent that the executions will take longer, longer and longer.


How can I manage changes in desired capacity due to scaling policies in AutoScaling Group state? How can I manage changes in desired capacity due to scaling policies in AutoScaling Group state in terraform?

Specifically, suppose I provision an aws_autoscaling_group resource with terraform with a desired_capacity of 4 and a scaleup policy on high CPU utilization. Later, the autoscaling group has updated via scaling policies to a desired capacity of 6 but this state isn't captured in my terraform .tfstate.

If I want to make a subsequent change to my autoscaling group state via terraform, how can I make the change without resetting the desired_capacity (since it has not changed in the .tf)? Can I automate updating the desired_capacity to match current group size somehow, or should I just not set desired_capacity at all?

The desired_capacity in Terraform is marked in the documentation as optional. So with a proper min_size value, Terraform can wait until the minimum capacity is reached before continuing.

The above, with the addition of scale policy can effectively manage capacity without being specific about desired_capacity in your Terraform code. This will prevent you from having Terraform override the capacity in place at the moment of execution.


Context of DevOps in ISO 27001 security audits Correct me if I am wrong, ISO 27001 addresses mostly organizational and operational aspects. This sounds a little bit abstract and regulatory.

On the other hand, in DevOps, if you wish DevSecOps, there is proactive and very specific demand for processes and tools to ensure that no vulnerabilities are delivered to customer.

How do these two domains fit together?

Do security auditors ask actually in a security audit questions like,

"Do you have a vulnerability scan as part of your component dependency management?
Have you checked your API for OWASP breaches?
Do you scan your Docker images, being aware of risks connected to base images?"

In my experience auditors tend to follow a Top-down Risk Assessment process whereby they review the inherent risk to business and IT for a company or system, then look at the controls that company has in place to mitigate the inherent risk and the residual risk that remains.

A top-down risk assessment opposes the way most technical people work, in that we look at specific vulnerabilities, defects and design flaws and then work up from there in a more bottom-up approach.

There is nothing inherently wrong with either approach they both have a place in any given company, so in answer to your questions:

Auditors do not tend to ask domain-specific questions such as Have you checked your API for OWASP breaches; they do tend to ask questions such as What controls are in place to mitigate the risk of data loss due to software vulnerabilities?
The thing to realise about ISO 27001, and most standards, is that they are not prescriptive - they are designed to provide a standard business language to enable communication between Business/IT and Auditors.  More often than not "implementing" any standard is about identifying the existing processes that meet the requirements of the standard and documenting them, then looking for any gaps or flaws and plugging the holes.

The same logic goes for ISO 9000, ITIL, COBIT, COSO, etc. it's very rarely about enforcing some pre-defined process. Instead, it is a case of creating a mapping between what the auditor is looking for and what the business is doing.


What is the advantage of using Prometheus over ingesting metrics using logs? I want to setup monitoring on our environment. I am wondering on what are the advantages of using something like Prometheus for making metrics vs. just using something like the ELK stack for creating similar metrics using logs?

Efficiency

If you don't need the file it saves space and trouble to leave it out.  Logs require rotation and retention policies which are nice to skip.

Reliability

Skipping the file avoids issues with the file that could caused lost metrics like a full disk.

Latency

Skipping the file means that you remove latency from disk I/O.  For most metrics you don't care if they show up a few seconds late, but if you're trying to build things to respond to some metric quickly.  You might want to put it on Kafka also.


Why Run Concourse CI in Kubernetes I've just started exploring Kubernetes and Concourse CI. Learning still, so I'm still understanding all the concepts. Forgive me if this question sounds naive.

Why would we deploy Concourse CI in Kubernetes. Is it just for the ease of deployment? I mean concourse has its own way for scaling workers which itself if I've understood is through containers.
So why run something that can already scale and handle containers(I mean concourse) inside Kubernetes?

In our case yes, it is just ease of deployment/operations. We have existing skills with k8s and that's what we're most comfortable operating.

I guess you're right that its somewhat odd to run Concourse's scheduling inside of k8s scheduling, and it would be great to see Concourse being able to leverage Kubernetes directly at some point.


Running two containers in a kubernetes job each with their own initContainers In this SO article I understand that I can run a sequence of initContainers before running the containers defined in spec.template.spec.containers.

https://stackoverflow.com/questions/40713573/how-to-run-containers-sequentially-as-a-kubernetes-job

I have two independent containers, each of which requires a sequence of initContainers to run in series prior to the container running.

Is it possible to run both sequences of initContainers at the same time? Something like this:

apiVersion: batch/v1
kind: Job
metadata:
  name: my-test-job
spec:
  backoffLimit: 0
  template:
    spec:
      initContainers:
      - name: init1
        image: busybox
        command: ["echo", "Init 1 Ran!"]
      containers:
      - name: task1
        image: busybox
        imagePullPolicy: Always
        command: ["echo", "Task 1 Ran!"]
      restartPolicy: Never

    spec:
      initContainers:
      - name: init2
        image: busybox
        command: ["echo", "Init 2 Ran!"]
      containers:
      - name: task2
        image: busybox
        imagePullPolicy: Always
        command: ["echo", "Task 2 Ran!"]
      restartPolicy: Never

It's not valid to duplicate spec as I did in that example of what I'm attempting to achieve. But it demonstrates the sequence of events I want. Initially init1 and init2 kick off, and once each finishes, task1 and task2 start following their respective init containers.

What you are describing is best achieved by either using two jobs or let task 1 also be an init Container as they run in sequence.
Alternatively you could use Argo Workflows.
https://github.com/argoproj/argo


Jenkins Console Output character issue, but plain text show correct Jenkins working with newman command (postman) newman run https://www.getpostman.com/collections/631643-f695cab7-6878-eb55-7943-ad88e1ccfd65-JsLv, and the Console Output shows:

But when I go to View as plain text, the character shows correct:

Here more details:

Jenkins Version: 2.7.0
Job Type: free style
Slave OS: Ubuntu 16.04 LTS
Jenkins Installation way: apt install jenkins
JAVA_ARG: JAVA_ARGS=...  -Dfile.encoding=UTF-8 -Dsun.jnu.encoding=utf-8

that is formatting issue
you may try to configure jenkins job to use formatted output as well
check it out https://wiki.jenkins.io/display/JENKINS/AnsiColor+Plugin
try all available schemes one by one, some of then should works well


Monitoring K8s Pods in case of outage K8s has self-healing builtin features, if a pod crash then it will be replaced with the new pod but its log is gone. So what is the best way to monitor K8s pods in case of outage?

For health monitoring, I would recommend the prometheus-operator. If you are looking for application logs, you should consider a logging stack such as elasticsearch-fluentd-kibana. I personally would recommend the kube-fluentd-operator. Additionally, if you would like to see the kubernetes events I find eventrouter quite useful which can also be wired into your logging infrastructure.


How do I trigger a build pipeline on a TFVC branch commit to map and get that branch? I have project in DevOps TFVC not Git. I use branches, they live and die in the order of months. I want to create a build pipeline that is triggered when a commit is done on any of the branches and the agent job must map and get the relevant branch that triggered the pipeline. I have no idea how to do this. The initial setup of the pipeline seems to look like I can only map/cloak and get a static set of folders.

You've discovered one of the great pitfalls with TFVC. I have also not found an answer to this issue.

The only possible option I've seen around getting code from TFVC out of different branches is to base your pipeline off the highest point of the source tree possible (i.e. the highest common parent folder of all branches) and always build using source labels. You have to do this otherwise a label based build request will checkout nothing if the labeled path(s) do not fall under the pipeline's source root path.

A side effect of this is that in the build agent working directory the checked out code is going to land in a physical folder path that includes all the parent folders going up to the root specified by the pipeline's source root. If your build tasks are generic enough to tolerate this it might work. However, in my experience, the build tasks need to collect artifacts which will essentially be in an unknown location (based on the arbitrary branch) so you'll need something a bit smarter to get the path from which to collect said artifacts. It gets messy quickly.

Another side effect of this is that you can't do continuous integration builds. In order to support getting code from labeled locations, the pipeline's source location is the base path of all possible branches. A CI triggered build then has no context for which instance of a project to build (i.e. which branch). While you could probably extract the branch of a commit from the commit data, you'd need more complicated build scripts to do so.

I've found the pain of this to be far greater than just migrating to Git which allows for considerably more flexibility for building from arbitrary branches since the pipeline is based on the standard repo root from which all branches are based. Since Microsoft has done all but outright state that TFVC is going away, your time may be better invested in migrating to Git to take advantage of this and the other things it provides.