How does Vagrant differ from VirtualBox and similar software, if at all? Both Vagrant and VirtualBox (or their alternatives) seem to me to virtualize an OS on top of another OS,

How do they differ from one another essentially, if at all?

Vagrant is a tool to create and customize virtual environments (containers and VMs) in a reproducible fashion (across platforms) using code and a virtualization provider of your choice. VirtualBox is only one of the virtualization providers it supports. Vagrant can also create VMs using Hyper-V, VMware and AWS.

While VirtualBox's VBoxManage does let you create and configure VirtualBox VMs, Vagrant, by contrast, will allow you to create and configure VMs with the hypervisor provider of choice, which may or may not be VirtualBox. Vagrant is agnostic as to virtualization provider plug-in you use with it.

To sum it all up, Vagrant does not provide virtualization services itself.

Further reading

The links below provide helpful insights.

An itemized comparison of Vagrant to Hypervisors can be viewed on this HasiCorp github issue.

These links provide some additional overview of Vagrant that may help bring the difference into focus.

https://www.vagrantup.com/intro/index.html
https://en.wikipedia.org/wiki/Vagrant_(software)

Also, if you sign-up for a free account at LinuxAcademy you can view and work through this simple example that includes these tools.


Jenkins Job vs Jenkins Pipeline BACKGROUND : We have one Selenium automation project which is shared in BitBucket repository. Project is maven project. we have 10 testng.xml files for all modules

OVERVIEW: Now we want to trigger the automation build once any changes committed in AUT application code.
Also want to trigger the build for failed testcase in Post Step as well, So we will be sure with actual test failure.
Once this execution done then it should send the email with report attachment.

PROBLEM: We are unable to decide What would be feasible Jenkins Job or Jenkins Pipeline ?

TRIALS : I have idea regarding Jenkins Job like setting up maven job and configuring post build trigger, email configuration but not clear enough as a layman.

Jenkins Job and Jenkins Pipeline are basically the same. In a pipeline you define the steps of your job as groovy code (actually it is CPS https://github.com/jenkinsci/workflow-cps-plugin, but that should in general just be a custom groovy interpreter).

The point, that is making pipelines "better", form my perspective, is, that you can add those in so-called Jenkinsfiles alongside your code. So you have your build job versionized alongside your application code.

So I would recommend: Take the time to learn how these Jenkins pipelines work and use those. But if you do not want or if you do not have the time, then doing it with plain-old Jenkins jobs is fine as well.


Is there a Jenkins plugin for downloading source code? Although our own sources are checked out from a revision-control systems, the third-party software we depend on needs to be downloaded from their respective sites.

The plugin I'm looking for would offer a "Source Management" (or "Pre-Build") step, where I could register one (or more) sources for the same file. For example:

https://www.sqlite.org/2019/sqlite-autoconf-3290000.tar.gz
http://www2.sqlite.org/2019/sqlite-autoconf-3290000.tar.gz
http://www3.sqlite.org/2019/sqlite-autoconf-3290000.tar.gz

and its digest (such as SHA256).

At build- (or at a prebuild-) time, it would check its cache and, if the file is not present there, download and verify it, and make it available to the subsequent build-steps.

I'm sure something like this, or very similar, exists already, but I cannot find it. Sonatype's Nexus seems to offer this, but only for files already found in one's Nexus repository, not for arbitrary URL's.

Any help?

Perhaps you are looking for the Job Cacher plugin?
To quote the plugin page:

Summary
Enables caching of files on transient executors for use on subsequent builds to improve performance
Features

Item storage extension point supporting on master storage and AWS S3
Cache Wrapper for free style jobs that manages the cache
Arbitrary File Cache where user specifies paths to be cached
UI on Job page to review the job's caches
Supports Pipeline jobs with a cache directive
Cache Extension Point for other plugins to provide opinionated caching capability such as Gradle caches
Incremental storing of cache into storage to minimize data movement

However, these are for files created by the jobs, not needed by them. In order to use it in your case, you might need a few dummy jobs specifically for downloading the content from the arbitrary URLs, and then caching them in your Jenkins cache. I guess you'd have to write the script for that.
To be frank, I've never used this plugin, and what I describe above doesn't seem to be an elegant solution. Your question specifically ask for a Jenkins plugin, however I believe the general problem is how to cache job dependencies in Jenkins. If you agree to this wider interpretation, there are some other options.
I have often solved the problem you mention using a combination of managed scripts and persistent storage which Jenkins and workers have access to, in Jenkins.  If you need really arbitrary dependency management, a tool like Spack or EasyBuild may be useful. Although these have evolved in the HPC domain, they do quite well at caching dependencies for workloads.
However, since you mention just a few dependencies (specific sqlite versions e.g.), I think it may be worth your time to write a script which does the caching for you and, together with globally-available persistent storage, do the caching and retrieval that way.


chef database cookbook deprecated I frequently use chef database cookbook and it is being deprecated doesn't receive any updates. They even disabled issues on the github so it is not possible to check if any issues I encounter have been discussed and possible solutions were proposed. The deprecation notice says that functionality of this cookbook is supposed to be moved to individual database cookbooks but this still isn't implemented, at least for MySQL. Is there any alternatives to this cookbook or I'm just stuck with creating another fork alongside hundreds of others?

If what you are looking for is a chef cookbook to manage MySQL, I would recommend checking this It is an officially maintained cookbook. Keep in mind that:

  This cookbook is concerned with the "MySQL Community Server", particularly those shipped with F/OSS Unix and Linux distributions. It does not address forks or value-added repackaged MySQL distributions like MariaDB or Percona.

I hope this could help.


Is Serf an added value when a container orchestration platform like kubernetes is used? When the Hashicorp's website was checked it seems that there is another tool, i.e. Serf. Back in the day there was another tool called otto, but that was deprecated at a certain moment.

After reading the use cases I wonder whether it is an added value to use serf when a container orchestration platform like kubernetes is used.

Serf is gossip protocol implementation and has absolutely nothing to do with orchestration tools like k8s. Even if such protocol is used internally, then in most cases it isn't available for you applications running in given orchestrator.

So if your app or workflow depends on gossip protocol, then you will have use out of Serf, otherwise there is no point.

Gossip protocol is way of messaging to all nodes without:

Centralized point of knowledge
Single point of failure
Using whole network capability of any of the nodes

In simple words it works like "message chains" ("If you do not send this to N people you will never find your love" or something like that).

One node send notification to N out of M of its fellows
If you receive message that you have already received, then do nothing.
Otherwise go to step 1.

Given such simple protocol you can calculate probability when all your nodes will receive message with 99% chance.


Alternating VC branches to build deployment artifacts for different environments We use Jenkins for CI. The app is written in Java, built with Maven (invoked manually in local dev environments and by Jenkins), deployed on various Tomcat servers. Nothing unusual about the application stack. Version control is SVN for the time being (we're looking at Git and I hope that comes soon).

We have a multi-phase Jenkins job called something like build.all which does svn update using the Maven plugin and then runs mvn clean install from the root dir where the code is checked out by svn. Then, another set of jobs deploys the artifacts made by build.all using deploy plugin by sending them over to the target servers using a Tomcat manager web service. The deploy jobs specify the latest successful build of build.all as the source of artifacts to deploy.

That works great when we are in a single branch mode. The hard setting of build.all as the source of artifacts to deploy is what is giving me headaches while trying to support transition periods during which some environments receive deployments from branch X and some from branch Y.

Without belaboring you with the minutiae of random low level tactics I'm speculating might make this possible -- can you recommend a high level strategy to support parallel artifact builds from different versioning branches? One goal is to not have to change the job source of artifacts to deploy because there is a rather large number of jobs that do deployments and changing them all by hand would be prohibitively labor intensive. For example, is there a way to maintain a key lookup mapping environments with source branches (or references thereof) so if a deployment job starts with STAGE or UAT, Jenkins looks up which build job (specific to a branch) is mapped to that environment?

You can create a release process to manage deployments in your environments.
A plugin that can help you is the maven-release-plugin, where you 'freeze' the package and can promote them between environments (DEV -> UAT -> PROD). This ensures that the artifact (package) is exactly the same as the Production environment.


Multi-container server of PHP apps with reverse proxy? I am going to deploy my app to production on VPS server, using Docker. My goal is stable version from latest tag at example.com, and dev version from dev tag at dev.example.com.

Because this is an PHP application, container has own nginx+php set. I don't know how - if it's even possible (without Swoole in PHP etc.) to avoid this.

Problem is in resources - I see that each app container has own nginx instance, and server (VPS) must have global nginx, working as reverse proxy for containers.

Is there are any better way to archieve my goals?
Thanks!

Problem is in resources - I see that each app container has own nginx instance, and server (VPS) must have global nginx, working as reverse proxy for containers.

Generally speaking yes, this is how this sort of thing goes.  It's more of an issue for PHP than other languages because PHP doesn't really have a standalone application server, but instead is commonly deployed using php-fpm with nginx.  Since php-fpm supports communication via TCP sockets, theoretically you could run only fpm in the containers and have nginx on the host talk to fpm across the Docker networking layer, but I haven't researched this option enough to be convinced it would work.

Docker makes the most sense when used within the context of an orchestration layer like Kubernetes.  Rather than running several Docker containers manually or semi-manually on a particular host, you would schedule them with kubernetes and it would distribute them across its cluster of nodes - the specific machine that runs the container becomes irrelevant.  The kubernetes service then provides you with an IP via kube-proxy that load-balances across the pods of that service.


Laptop Internet connection monitoring I have a need to monitor laptops both Linux and Windows that are deployed to remote locations which we don't have physical access to all the time.

The laptops are used for different services but need Internet access, what we would like to have is a solution to alert us when a Internet has lost Internet connection

For web servers we use nagios and pingdom but wondering if someone has come across a solution that is lightweight and for laptops

Ideally having a dashboard to view what laptops are connected would be the best

I've started abusing SyncThing for this purpose (among others), configuring it as a system service on laptops, then locking down it's UI to prevent it from being able to be used by a local connection originator to manipulate or access files. I get connectivity monitoring and robust rsync-like backup of field data.

It's fantastic at punching through difficult network conditions, so once you peer a node with a central node (say, a backup landing pad itself backed up via more conventional means), you can use Syncthing's API on your landing pad node to track connectivity.

If you're looking for end-to-end IP connectivity, and not just "can the remote host reach the Internet at all" knowledge, you can disable NAT traversal and relay use, at which point there will need to be a clean, unfirewalled path between the two nodes.

The downside is that SyncThing is very much a rapidly-developed project, with the issues that carries with it.


How to enforce policies for manifests in Kubernetes? I have built a self-service platform based on Kubernetes, where we create namespaces for each team and allow them to 'do whatever they want within the namespace' (we set resource limits so no one can kill the whole cluster).

However, now I want to implement some kind of standard across the organization. For example, I want every PodSpec to define its own resource limits, and I want every resource to have a label that specifies what application it belongs to.

Is there a mechanism that will allow the API server to check the manifests being applied against a set of rules, and if it fails the check the manifest is rejected.

For example, the following manifest would be rejected because it has neither a label nor are resource limits set.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  selector:
    matchLabels:
      app: nginx
  replicas: 2
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.7.9
        ports:
        - containerPort: 80

But the following manifest would succeed because it satisfies all the rules:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: foobar
spec:
  selector:
    matchLabels:
      app: nginx
  replicas: 2
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.7.9
        ports:
        - containerPort: 80
        resources:
          limits:
            cpu: "1"
          requests:
            cpu: "0.5"

I would be inclined to implement the Open Policy Agent as this will give you the ability to define a policy similar to this:

apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sContainerLimits
metadata:
  name: container-must-have-limits
spec:
  match:
    kinds:
      - apiGroups: [""]
        kinds: ["Pod"]
  parameters:
    cpu: "200m"
    memory: "1Gi"

OPA will enforce any policy you defined via an Admission Controller, beware though it's very powerful and I have bricked clusters by not thinking through the options carefully.


Process auditing in containers I've read numerous articles on this going back to 2013, but what I wasn't able to ascertain is whether a reliable method has yet been devised for auditing processes inside a container. This question applies primarily to Linux, as I'm not very familiar with containerization on other platforms.

The problem is essentially this: How to track operations such as Linux system calls back to the user who created the container?

With Docker, that information is lost because the container is started by a daemon, and Docker doesn't have an authentication mechanism in place for identifying the user.

Podman takes the approach of creating a container as a process fork, rather than using a client/daemon, which preserves the loginuid, but what about across a cluster.

Kubernetes has auditing functionality, but I haven't been able to determine from the documentation whether that applies to processes running in containers, or only to K8s API calls.

It's seems like it would be possible to do this by:

1 - Require a user to authenticate when starting a container.

2 - Send the user name when starting the container on each node.

3 - Preserve the user ID in a process attribute.

For 2, to work, the user would have to be known to each host, and not just the container daemon.

A trivial case of 3, would be for the user in the container to have the same UID, and the authorization stage could enforce that. But that's pretty limiting.

Has anyone managed to address this successfully, or is it still an open issue? Either a global solution, or for a particular containerization technology/orchestrator.

Here are some links regarding proposals for implementing this in the kernel:

https://lwn.net/Articles/740621/
https://lwn.net/Articles/699819/
https://www.redhat.com/archives/linux-audit/2018-January/msg00014.html

I think there is a way to do this on docker:

  As of Docker 1.10 User Namespaces are supported directly by the docker daemon. This feature allows for the root user in a container to be mapped to a non uid-0 user outside the container, which can help to mitigate the risks of container breakout. This facility is available but not enabled by default.

You can map the root user inside docker container to specific user from the host machine (uid).


Can Ansible use a local repository? I have multiple hosts and my playbook has apt, pip modules to download necessary packages from repositories. When any (apt,pip) download module attempt to download packages, it does it for each host one by one. So I have to wait too long. Can ansible download these deb or pip packages to its local (or anywhere) and install to each host to decrease time.
What is the best practice when there are lots of hosts and lots of packages?

You can set up a proxy. A reverse proxy can be a good choice for this kind of setup. You would configure each machine to use the proxy as its apt/pip repository. The first time a package is requested the proxy downloads it from the internet and stores it, the next machine that requires it will get it from the proxy directly.

One popular choice is Squid. Another one is Nginx, set up in a reverse proxy configuration.

The advantage over mirroring entire repositories is that you'll only get the packages you need. The advantage over having Ansible copy the files is that it acts like the real proxy, dependencies and upgrades and all. But you'll  have to change apt and pip repository configurations. Or "hijack" those repository names through your local DNS.


Is there a conceptual limitation to have a clone stage function in Bamboo? In Bamboo, you can:

clone a job
clone a plan

Why is it not possible to clone a stage, I mean, could there be a certain reason to not have it?

Stages group jobs inside plans. So technically they are cloned when the respective plans are cloned.

Which means that missing the ability to clone a specific stage should be just a UX/usability/functionality issue, not a conceptual one.

There might be a non-trivial implementation details when cloning stages - they may carry inside stage sequencing information, as in "pointers" to the stage objects preceeding and/or following them inside the plan "pipeline", which might be a bit more difficult to handle when cloning just one stage in the sequence. These would be easier to handle when cloning entire plans as the entire sequence of stages is cloned, there will always be a 1:1 relationship between the stages.


Prometheus Alertamanger - how to silence all alerts for a given period during a maintenance? A work scenario that I can't cover currently is that I want to set a maintenance mode, meaning all alerts received from Prometheus to be ignored, I want to be able to set it through the UI for a given period until maintenance finish. One way to do it is manually to start covering each alert which in Prometheus-operator scenario is not sane, or I would lose a couple of hours. I can change config from secret to route all to null, but that is not practical as it requires multiple actions and is not good for on time or other user usage.
So I tried to use regex for key and value. Both set to .* .* regex. On preview I get all alerts covered, but when I click Create I get "Bad Request"? I couldn't find examples in internet and on official page which surprised me quite a lot.

What about Name = "prometheus", Value = ".*". That is something that worked for me.
An alternative can be Name = "severity", Value = ".*" also.


How to connect to a kubernetes cluster that is inside of a docker container? For CICD purposes, I need to run a docker container and then create a kubernetes cluster inside of it using kind. Here is my Dockerfile:
FROM ubuntu:18.04

RUN apt-get update
RUN apt-get install curl -y

RUN apt-get install -y gnupg2

RUN curl -fsSL https://apt.releases.hashicorp.com/gpg | apt-key add -
RUN apt-get install -y lsb-release
RUN apt-get install -y software-properties-common
RUN apt-add-repository "deb [arch=amd64] https://apt.releases.hashicorp.com $(lsb_release -cs) main"
RUN apt-get update && apt-get install terraform -y

RUN apt-get install -y wget
RUN wget https://golang.org/dl/go1.15.6.linux-amd64.tar.gz
RUN tar -C /usr/local -xzf go1.15.6.linux-amd64.tar.gz
RUN GO111MODULE="on" /usr/local/go/bin/go get sigs.k8s.io/kind@v0.9.0

RUN curl -fsSL https://get.docker.com -o get-docker.sh
RUN sh get-docker.sh

It's basically an ubuntu image with kind installed.
I run this image using docker run commad:
docker run -v /var/run/docker.sock:/var/run/docker.sock -it ef33f1a0ac5e bash

And then I create a kind cluster inside of it:
kind create cluster --name kind

Then I install kubectl on it.
When I try to connect to my kind cluster using kubectl (kubectl get all, for example), I get the following error:
The connection to the server 127.0.0.1:37583 was refused - did you specify the right host or port?

Here is my kubeconfig (kind get kubeconfig):
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: CERTIFICATE-AUTHORITY-DATA
    server: https://127.0.0.1:37583
  name: kind-kind
contexts:
- context:
    cluster: kind-kind
    user: kind-kind
  name: kind-kind
current-context: kind-kind
kind: Config
preferences: {}
users:
- name: kind-kind
  user:
    client-certificate-data: CLIENT-CERTIFICATE-DATA
    client-key-data: CLIENT-KEY-DATA

While you are running a container linked to your PC/Server Docker, the Kind Kubernetes port is available on your PC/Server not on the container.
So if you run kubectl commands from your machine it will work (including the kubeconfig).
Solution: Using --network="host" in your docker command, the container will act as your localhost (eg. docker run --network="host" -v /var/run/docker.sock:/var/run/docker.sock -it ef33f1a0ac5e bash


DevOps vs SRE vs Production Support Engineers DevOps primarily focuses on Delivery Speed and SRE focuses on Reliability in production but where does Production Support Engineers fit who also focuses on production monitoring, alerting, performance, user experience, incident management, RCA and work on code defects and understand business functionality?

In a SRE world, would production support engineers be more aligned to SRE or merged with SRE?

The best answer to DevOps vs SRE is here What's the Difference Between DevOps and SRE ? From the above series, you will understand Class SRE implements DevOps and both are working on a similar ground and it's pretty align goal for both of them.

As a production support, you will be doing the task which you have mentioned in the question, apart, you need deep visibility, alert and notifications, latency, metrics, tracking, security, etc. lots of things to take care. And, it basically providing the service to the end users who is utilizing your application or system. Also, I see production support team have written the dockerfile and bash scripts to automate many tasks. So, whenever user faced any issue that first comes to Production Support team and it might went to other IT teams if it's not easily resolve. But, if you look at SRE, their sole purpose is to keep system in reliable state every time, maintain resiliency, etc. In some organization, the DevOps does this part too.

From my perspective, all the three terms are same as their goal align and just differ in functionality. These functionality differs from organization to organizations. But, DevOps are intended to bring culture and shift in paradigm so that everyone will work collaboratively. Hope this helps.


Git Repository Manager conditionally deny pull request I am looking to migrate a handful of Subversion repositories to Git. Each of these repositories has a development and production branch. When ready to release, a developer will submit a pull request to production. There is a version file that contains the version that has to be incremented in some way. Is there a Git repository manager (Gitlab, Bitbucket, etc.) that allows you to automatically deny pull requests based on a condition? My condition in this example would be the version in a version file is not incremented. I am also willing to consider other methods, such as tagging the repositories instead of a version file, if auto-denying in that way is possible.

Most repository managers have some way of implementing this.

GitHub (and GitHub Enterprise) has the idea of statuses on a commit; you'd set up a webhook that pings your tool's webserver when a commit is pushed, and then it would use the status API to set whether or not it passes the version check.

Bitbucket, when self-hosted, supports plugins to integrate nicely with it, including ScriptRunner, which can be used to do all sorts of things similar to what you want.

I'm not familiar with GitLab, but it probably has some similar method.

You will need to do some programming to implement the checking logic, but these pieces should allow you to kick that off when appropriate and use the results to block the merge.


kubernetes quota vs limit. What are they? How are they different? When should I use one or the other? I have been reading through the docs but it still is not clear as to why I should use one or the other. They have great examples of how to configure them but don't answer the why for quotes or limits very well.
I want to understand
kind: ResourceQuota

and
kind: LimitRange

I am not so interested in the yaml this is very clearly given in the examples.
I want to understand the use-cases better.

ResourceQuota is

the total amount of memory and CPU that can be used by all Containers running in a namespace

LimitRangeis

if ... the Container does not specify its own CPU limit, then the Container is assigned the default CPU limit

Example of ResourceQuota:

Imagine you have 2 namespaces in the same cluster, the namespace "backup" and namespace "test". "backup" run critical pods and you want to guarantee that will always have enough resources (CPU and memory) to run it. So you can configure a ResourceQuota in the namespace "test" to use only a part of the total amount of CPU and memory in your cluster.
Is like you say to k8s: "The sum of all memory and CPU of all pods in the namespace "test" can't be higher than what is specified in the ResourceQuota, 20G for memory and 12 vCPU's"

Example of LimitRange:

You can specify in the yaml the minimum and maximum amount of CPU and memory a container can consume but instead of configuring in every yaml file you can set a default value with LimitRange.
Is like you say to k8s: "If there is no resource configured in the yaml file apply this LimitRange.


How to deploy software every second? Companies such as Amazon, Etsy, Google, Facebook and others are publicly saying they deploy software thousands time per day.

What are they doing that is so different from other companies?
How can other software companies deploy software thousands times per day, what is stopping them from doing so today?

The "how" of extremely high-frequency deployment is all about very mature engineering practices and tooling.  The "unicorns" (Gene Kim's term for the type of company you list) have some of the best talent in the industry, and invest in the best technology.

One of several key practical differences between these unicorns and most other orgs is the level of automation they employ.  It is very difficult to automate the quality checks needed to maintain a constant flow of work from DEV, through testing environments, to Production.  Development and Operations must maintain high service levels internally.  Ops must provide self-service resources so developers aren't waiting on tickets to get new infrastructure.


Which server to set up for distribution of windows based application updates, patches and upgrades? We are looking forward to set up some server to distribute application patches, updates, and upgrades. Our applications would be checking for their updates by querying this server. Presently, developers here are considering Chocolatey, and just rejected Squirrel for windows due to its narrow scope.

How is this being done in the world generally with best practices?

I recommend Chocolatey. There are more and more companies that are adopting this. For example, if one would like to install Gradle and navigates to the installation page:

  Chocolatey is “the package manager for Windows”.

$ choco install gradle


How to change ownership of folder inside the container after build? I have this service in my docker-compose.yml
php-fpm:
container_name: php_app
restart: always
build:
context: .
dockerfile: ./docker/php-fpm/Dockerfile
volumes:
- ./src:/var/www/html
that take everything of the host folder src and add to /var/www/html:
FROM php:8.0.2-fpm-alpine
WORKDIR /var/www/html

RUN docker-php-ext-install pdo_mysql
RUN docker-php-ext-install mysqli
RUN apk add icu-dev

COPY docker/php-fpm/config/php.ini /usr/local/etc/php/

RUN apk add --update nodejs nodejs-npm
RUN npm install gulp-cli -g
RUN npm install

COPY src src/

CMD ["php-fpm"]

EXPOSE 9000

accessing the container using sudo docker exec -it php_app sh  and executing ls -la I have:
drwxrwxr-x    6 1001     1001          4096 May 12 15:54 writable
which must be owned by www-data, how can I set the ownership automatically when the builds complete?

you need to create the www-data user in the dockerfile eg
RUN addgroup -g 1000 www-data && adduser -G www-data -g www-data -s /bin/sh -D www-data

RUN chown -R www-data:www-data /home/www-data && \
    chown -R www-data:www-data /var/www


Pometheus-operator - modify Grafana mixin dashboard Whenever I try to edit the provisioned dashboards I get an error stating that the provisioned dashboard can't be edited. I saw that there is an option to allow changes from the UI, but my questions is if there is a way to modify it in an automatic/code way. I want to be able to add additions to the mixin without interacting with that repo but a clone of mine.

You should use https://github.com/helm/charts/tree/master/stable/prometheus-operator helm chart , also you can write Grafana dashboard script that would create dashboard dynamically , can be update as well later on.


How to list security groups for the specific instance via CLI? I'm looking for syntax like:

aws ec2 describe-security-groups --instance-id i-0xyz

however, the above sub-command doesn't support --instance-id parameter.

I've checked and there are --filters and --query parameters, but I'm not sure about the syntax.

What would be the easiest way to display the description of the security group of the instance?

You can use aws ec2 describe-instances instead, you can specify the id --instance-ids <value> and it will output the security group.


Recommended SVN branching strategy for DevOps We are using SVN as our version control system and we are using the below standard layout for our projects.

Tags
Branches
Trunk

Trunk:
    -> Trunk will always hold the latest code changes.

Branches:
    -> When a release is planned a new branch will be created from the trunk and worked upon. Once the same is released it will be tagged.

Tags:
    -> Now after the release if we notice any issues in the delivered application then we create a new branch from the tags and then it is worked upon. Finally, the changes made will be merged into the trunk. Or sometimes we make the fix on trunk first and then backport it.

Is this the standard approach? or is there any recommended best practice?

We use Git for our VCS. We follow Trunk based development strategy.
Trunk-Based Development (TBD) is where all developers (for a particular deployable unit) commit to one shared branch under source-control. That branch is going to be colloquially known as trunk, perhaps even named “trunk”.
 lie of omission

This trunk is always stable, but it can have unfinished features covered by feature toggles. This type of development works

When you are just starting up
When you are iterating quickly

In short, feature branches will give you better isolation, but require you to deal with the pain of deferred integration, and merges. Toggles give you continuous integration, but require you to design/implement your code in such a way that supports toggles, and introduce the risk that unfinished feature code could negatively affect production.


Kubernetes External Access Of A Single Pod I am running a kubernetes cluster in AWS. I have 2 nodes. I have one pod that should be accessible externally. The problem is, that pod can be on either node.

The solution according to the docs:

Expose the deployment as a NodePort service.
get pods to find the name of the node running the pod.
get nodes to find the IP address of the named node.
Open the NodePort port through EC2 Security Group.

The problem here is that this pod could be restarted/switch nodes, I think? At that point the application will be unavailable by the previous node IP address.

How can I make sure an application is always available, even if this specific pod switches nodes during a restart/etc?

Note: I definitely only want one of this specific pod. I do not want replicas.

The NodePort will resolve correctly, no matter the node where the pods is (even if it gets rescheduled on a different node).

Any <Node IP>:<NodePort> will resolve to a Service type NodePort, no matter what node the pods are running.

Every node will forward traffic to that port to the Service.

You need to be on the same network (or create a SSH tunnel) or open that port in one of your nodes to the world.


Scheduling a build using Kubernetes The doc https://docs.openshift.com/container-platform/3.9/dev_guide/cron_jobs.html provides details of creating a cron job.
To start a scheduled task that executes a build every 10 mins I use the command:
oc run run-build 161/my-app --image=myimage --restart=OnFailure --schedule='*/10 * * * *'

Which returns:
cronjob.batch/run-build created

But the job fails to start:
The log of pod displays:
Error: unknown command "161/my-app" for "openshift-deploy"
Run 'openshift-deploy --help' for usage.

Have I configured the command ( oc run run-build 161/my-app --image=myimage --restart=OnFailure --schedule='*/10 * * * *' ) to start the cron job incorrectly ?

According the documentation you're giving, you need to remove run-build from the command :
$ oc run NAME --image=<image> \
    [--generator=<resource>] \
    [--port=<port>] \
    [--replicas=<replicas>] \
    [--dry-run=<bool>] \
    [--overrides=<inline_json>] \
    [options]


How do I return just the full path of a file found with the find module in Ansible? This has been reworded, expanded upon and clarified via an update late on 11/9/18.  I request it be upvoted now that it has enough code.

I want to find a softlink (to another file) in the $PATH environment variable on Linux managed nodes.

I have an Ansible playbook that uses a find module.  I have a string of potential directories to look in.  This string is the value assigned to the variable "result_from_prev".  My playbook looks like this:

 - debug:
     msg: "{{ item }}: {{ lookup('env', 'PATH') }}"
   with_dict: {a, 1}
   register: prevar
 - set_fact:
     midvar: "{{ prevar | dictsort | join('') }}"
 - set_fact:
     result_from_prev: "{{ midvar.split(':') }}"
 - find:
     paths: "{{ item }}"
     patterns: "goodfile"
     file_type: link
   register: files_found
   loop: "{{ result_from_prev }}"
 - set_fact:
     jsonvar: "{{ files_found | to_json }}"
 - set_fact:
     var: "{{ files_found | list }}"
 - debug:
     msg: "{{ item }}"
   loop: "{{ files_found | json_query('files[*].path' }}"

I want to extract the path of the file name for the file that is found.  How do I do this?

The above gives me an error about invalid data being passed to loop.  A list must be passed. I think that files_found is a dictionary.  How would I use wantlist=True here?  I tried, but I had no luck.

I tried using jsonvar instead of "files_found" in the loop.  But I got the same error about invalid data being passed to 'loop.'

I tried using listvar instead of files_found.  But I got the same error about invalid data being passed to 'loop.'

I tried using "with_items" instead of "loop".  But this prints out nothing for the msg.

In the JSON that the jsonvar variable can print out, there is a "matched" key.  The value is "0" for each directory path that the file was not found and the value is "1" if the file was found.  But I do not know how to extract just the full path of the file found.  I'd like it to be clean like this:

  /tmp/subdirectory/goodfile

The potential locations of where the file will be will change.  I have a framework that works: I have the potential files in a list in a variable named "result_from_prev".

I have tried using the "when" conditional and the item.matched notation to analyze the value of specific keys.  These efforts do not work for me.  I have tried putting the JSON into a file or using the "join" feature to create a long string.  But I have not had luck parsing this long string.  I have also tried using the "lookup" keyword with set_fact to assign a variable to that directory path that was found with the find module.  I tried to start over completely with the stat module instead of the find module.  But I found this to be difficult to use a list of potential paths whereas the searching by the file's name is supported with the find module.  I want to avoid using the "shell" module.  Nothing I have tried works.

I expect the find module to return only successfully found files with a supported syntax that makes extracting the directory path relatively easy.  This is not what I have experienced.  How do I return just the full path of the file named "goodfile" that is matched with the find module above?

Answers to some parts of your question are:

"I want to find a softlink (to another file) in the $PATH environment variable on Linux managed nodes"

Use "shell" module instead of lookup.

  Lookup occur on the local computer, not on the remote computer.

"I want to extract the path of the file name for the file that is found. How do I do this?"

Use "files_found.files" to see the complete path. Checkout the return values section in find module documentation.
As far as i understood, you want to do something like this

Try out the following:

- name: finding files matching "goodfile" pattern
  find:
     paths: "{{ some_path }}"
     patterns: "goodfile"
     file_type: link
  register: files_found

- name : get the complete path
  set_fact:
    files_found_path: "{{ files_found.files }}"

 - debug:
     var: files_found_path


Docker managers are losing quorum on AWS I have deployed Docker EE on AWS by choosing the package directly from the AWS Marketplace: Docker EE for AWS (Standard/Advanced) - BYOL. The cluster has been successfully launched and deployed on AWS.

After deploying several stacks on the UCP something really strange is happening, the managers are not able to see each other, they are losing the quorum inside the Swarm and the docker info command says that they are out of Swarm.

The AutoScalling group from AWS starts launching new managers inside the Swarm and the new launched managers are terminated in a couple of minutes because they are not Swarm's participants as well. Actually the AutoScalling group goes in a infinity loop by launching new managers and terminating the old ones.

The worst thing is that I am loosing the data from the Docker Services and there is no way of recovering.

Any ideas?

It seems like the AutoScaling Group from AWS are not that synchronized with the Docker Swarm configuration and the default timeouts in the AutoScaling are not accepted in the Docker Swarm managers.

Actually the Health Check is not configured good by default in the EE CloudFormation template, even if the managers are running and stable the Health Check fails and the AutoScalling Group tries to launch new managers. New launched managers are not able to become cluster participants, right after their running state.


Passing parameters from Jenkins to Ansible playbook How can I pass parameters from Jenkins to my Ansible playbook? Eg. I want to take IP address from users and pass it to the playbook so that it will execute only on those machines.

You can use -e switch for that purpose:

ansible-playbook -e 'foo=${FOO}'

where the FOO can be defined in the Jenkins envirionment variables. If it is defined within the parameters block of your Jenkinsfile, then you need to use ${params.foo}. Then, the value can be accessed by the Ansible playbook with {{ foo }}.


Can Ansible alone be a complete CI-CD tool? I am new to Ansible and was wondering about the capabilities of Ansible. Can Ansible alone be a complete CI-CD tool?
Scenario that needs to be implemented:
As soon as a developer perform a git commit, the build should be generated and placed some central server or maybe Amazon S3. This is a JAR file assuming a Java application. One can then download the build and deploy that build to the testing environment and after deployment invoke the automated test cases. Upon successful implementation of the test cases,one should be able to deploy this build onto the production environment in a rolling fashion. Note that one must not deploy this build every time, but it should be a discretionary power. I.e. even if during a day let us say, 5 builds were created and 3 of them passed the test cases, It should not go and deploy the builds on production. I should decide when to deploy the build, this could be a flag that I set to "Go" and then the deployment to production should take place.
Points that I am apprehensive about Ansible's capability are:

Will ansible be able to detect the git commit event. Is there a way my playbook to constantly poll the GIT server on a regular basis to detect any commits?

Is it possible to create Build using just Ansible. (Can I use command module to just execute the build creation process(assuming there is some command to create the build and I was thinking of firing those command using the command module, maybe maven or ant command).
Kindly forgive me for my ignorance if there is no such thing. I have never done development and hence am complete beginner.

How can I invoke automated test cases on the testing environment where the build was deployed (I am assuming a successful deploy on the stage environment).

Can I capture the success or failure result? (It maybe be a suite of test cases and won't be an objective o/p as pass or fail.)

Is there a way to achieve the discretion based deployment to production systems using Ansible?

Note that I don't want to use Jenkins or any such tools for creation of builds or deployment to test environments .
This may sound like a beginner question on DevOps, but I just wanted to assess the scope of Ansible as a CI tool. Is Jenkins (or any other variant) inevitable for achieving this? Or can people do away without using Jenkins.
Also if someone can comment on what is the recommended way to achieve this pipeline.

This is totally possible.  Of course you can achieve it without jenkins or similar tools being required.  We're always free to reinvent the wheel.  The question becomes is it worth the effort?  When it helps you avoid jenkins I'd be inclined to put the effort in.  (I'd also suggest looking at Concourse before doing jenkins again, but that's not what this question is about...)

For your case you need two things:

a playbook that automates your build using ansible.
a git hook to invoke your ansible playbook, probably post-commit.

If you are using gitlab or github you should look at their specific documentation for creating hooks, but otherwise the generic help I linked to should do it.

To answer your specific questions:

Yes, use the git hook.
Sure.  You probably want ansible to do a fresh git checkout first.  Eventually you will want to add any prerequisites to be verified too.
You don't have to assume anything, just add these steps to the end of your playbook.  If the build fails, it will stop before wasting effort on testing anything.  You could split build, deploy, and validate into their own roles within one larger playbook.  This will help make each of the sets of tasks easier to manage and you could call them manually if needed for debugging or emergencies.
You could do this in a myriad of ways.  I'd create a text file with a date/time stamp and the results inside.  You could write it to a database or K/V store.
I'm not sure what you mean.


How to bring a docker data container to Kubernetes? I have an application that relies on 4 docker data containers for the data it needs to run some tests. At the moment I have their docker-compose file and their images but I don't have access to the data containers Dockerfile or actual data.

How this application works in docker-compose is as below: The application is called rtsp_server and it loads different volumes from the data containers such as st-andries4(I just wrote one of them).

This is their docker-compose:

# The application
rtsp_server:
  image: rtsp_server_image
  volumes_from:
    - rtsp_data_st_andries4:ro
  ports:
    - "8554:8554"
  command: python ./rtsp-server.py /rtsp_data/st-andries-4 st_andries_4

# Data container
rtsp_data_st_andries4:
  image: st_andries_4:1
  volumes:
    - /rtsp_data/st-andries-4
  command: /bin/true
  mem_limit: 10000000000

I already brought the application, rtsp_server, to Kubernetes and it works fine but I need to also bring the data containers to k8s but I don't know how to do it.

You'll want to look through the Kubernetes Volume documentation to see what options are available.

In particular look into the PersistentVolumeClaim.  This allows you to setup a shared data volume for your pods.  Here's an example snippet from a k8s deployment YAML using a volume:

spec:
      containers:
      - image: **yourdbcontainer**
        name: mysql-db
        env:
        - name: example
          value: example_value
        ports:
        - containerPort: 3306
          name: mysql-db
        volumeMounts:
        - name: mysql-persistent-storage
          mountPath: /var/lib/mysql
      volumes:
      - name: mysql-persistent-storage
        persistentVolumeClaim:
          claimName: mysql-pv-claim


Terraform - gitlab provider does not start I am using terraform docker image hashicorp/terraform:latest and I define the provider in main.tf as follows:

terraform {
  required_providers {
    gitlab = {
      source = "gitlabhq/gitlab"
      version = "3.3.0"
    }
  }
}

provider "gitlab" {
    token = var.gitlab_token
}

When I run terraform apply, I get the following error:
Initializing provider plugins...
- Finding gitlabhq/gitlab versions matching "3.3.0"...
- Finding latest version of hashicorp/aws...
- Finding latest version of hashicorp/gitlab...
- Installing gitlabhq/gitlab v3.3.0...
- Installed gitlabhq/gitlab v3.3.0 (signed by a HashiCorp partner, key ID BC097C3333027B14)
- Installing hashicorp/aws v3.22.0...
- Installed hashicorp/aws v3.22.0 (signed by HashiCorp)
Partner and community providers are signed by their developers.
If you'd like to know more about provider signing, you can read about it here:
https://www.terraform.io/docs/plugins/signing.html
Error: Failed to query available provider packages
Could not retrieve the list of available versions for provider
hashicorp/gitlab: provider registry registry.terraform.io does not have a
provider named registry.terraform.io/hashicorp/gitlab
If you have just upgraded directly from Terraform v0.12 to Terraform v0.14
then please upgrade to Terraform v0.13 first and follow the upgrade guide for
that release, which might help you address this problem.
Did you intend to use gitlabhq/gitlab? If so, you must specify that source
address in each module which requires that provider. To see which modules are
currently depending on hashicorp/gitlab, run the following command:
    terraform providers

I can't find the solution to this issue from the documentation. Any suggestions?
terraform --version
Terraform v0.14.3

GitHub is one of the providers not yet included in the Terraform registry - see this diff to the documentation. This appears to be a temporary situation:

The following providers will be published on the Terraform Registry soon, but aren't quite ready. Until they're published, their documentation is available at the links below:

I can't tell from the release notes whether this is only in 0.14.x, but I can confirm that the "old" (i.e. current at the time of writing) github provider works with 0.12.x


Issue pushing to ECR using Jenkins I'm trying to push an image to ECR via a Jenkins pipeline. Currently I have the ECR plugin installed, and a role assigned to my EC2 instance that allows ECR access.

I am using correct credentials that are valid for the region I am trying to push to.

However I am getting this error message

Pipeline] withDockerRegistry
Using the existing docker config file.Removing blacklisted property: credsStore$ docker login -u AWS -p ******** https:/123456790.dkr.ecr.us-east-1.amazonaws.com
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
Error response from daemon: Get https://https/v2/: dial tcp: lookup https: Temporary failure in name resolution

Jenkins pipeline script

docker.withRegistry('https:/1234567890.dkr.ecr.us-east-1.amazonaws.com', 'ecr:us-east-1:aws-type-creds') {
    app.push(env.GIT_COMMIT)
 }

EDIT **
Now getting this error with small change to pipeline

+ docker push 1234567890.dkr.ecr.us-east-1.amazonaws.com/sample:66e4bf8547da1b2fd8f852e093fd152a6b1ee598
The push refers to repository [1234567890.dkr.ecr.us-east-1.amazonaws.com/sample]
1f15c22274e7: Preparing
70da67205593: Preparing
f0726b634cb9: Preparing
ab91817addbe: Preparing
1be049a99262: Preparing
cc82fb177433: Preparing
97a48aebb8e5: Preparing
1f7e5026a4f7: Preparing
9e864b659f2b: Preparing
f1b5933fe4b5: Preparing
cc82fb177433: Waiting
97a48aebb8e5: Waiting
1f7e5026a4f7: Waiting
9e864b659f2b: Waiting
f1b5933fe4b5: Waiting
no basic auth credentials

Line changed was line 2 of the pipeline code I showed before. It is now docker.image('alerts').push(env.GIT_COMMIT)

The docker login failed.

Error response from daemon: Get https://https/v2/: dial tcp: lookup https: Temporary failure in name resolution

The url is invalid.

change from

docker login -u AWS -p ******** https:/123456790.dkr.ecr.us-east-1.amazonaws.com

to

docker login -u AWS -p ******** 123456790.dkr.ecr.us-east-1.amazonaws.com


Kanban to catch a quick-moving situation We have a brownfield project which was recently struck by disaster (many people leaving for unrelated reasons, too many new customers, much too large backlog for much to few people, etc.). Management support is still fine. Thank bob the product owner (decades of knowledge) is still here. The devs have seen some glimpses of DevOps benefits from a very nicely-received DevOps colleague who is doing practical work as well as evangelisation, so there is no resistance on that front, either.

At the moment, we are bringing in new people from all over the place (i.e., split across different cities and even countries, though all speak the same language), and getting the know-how under control etc. We are also focusing a lot on automation before starting actual development again, because everybody saw what happens when we don't. Also, we have bundled everybody together - devs, ops, testers, to make use of every last bit of knowledge and avoid future silos. So it is a very dynamic situation.

My main objectives are:

Get people to work instead of panicking, i.e., give the individual devs focus.
Shield devs and ops from panicked management and stakeholders.
Make it more transparent for stakeholders to see what is happening with their tasks/epics, to reduce panicked zero-warning status calls.

Most people, as well as me, are used to Scrum (or more likely Zombie-Scrum or Scrum-But ;) ), but I believe it is too early to do proper Scrum here. So I will likely implement Kanban (in Jira).

What are your best practices here?

One column per person/pair? Or rather one column per stage for the tasks (i.e., "in development", "in testing" etc.)?  I think I'd prefer the first, because it very clearly shows whether a person/pair has their hands full - we could use a full column really well for telling a manager to go away. Also, we have quite diverse roles in the team (i.e., a DevOps expert who will be focusing on the CI/CD pipeline and such; or the testers).
In the usual Scrum process, the P.O. would not be part of the actual sprint board, but do his good work in the backlog. Would you propose to give a column to the P.O. so he can focus on fleshing out a few task at a time? This would put the P.O. well inside the team, which I would like. Is it a good idea? (Yes, I am aware that Kanban does not actually label those roles, but I hope you see what I mean...).
Alternatively to the previous: I believe The IT Sceptic had an article about having two boards - one focused on the stakeholder epics, the other on the actual short-term tasks. Do you have experience with that? Would that help in this situation, in your experience?

These may seem like three distinct questions, but are, in my mind one: how to structure the Kanban board - around people, or around tasks?

We are a product development shop - and in fact we build a Kanban tool (SwiftKanban), in addition to others.  We have a lot of experience with Kanban and DevOps and separate boards for epic/ story detailing vs. dev/ ps tracking. Our DevOps setup is still being fully automated and being gradually fleshed out. We've been on this journey since 2010-11 - and we have achieved significant business benefits exactly of the kind you are looking for.

We started with a single Kanban board, but with 2 separate swim lanes for the product manager and the dev team - see below -

Planning Lane for the Product Owner

Dev Lane for the main dev activity

Since we had a single team doing development as well as maintenance/ customer related work, as also refactoring work, we also had a few other lanes to help them visualize their overall workload -

Over the last 6 years, we have evolved, as have our dev/ engineering practices.  While we never used Scrum - and don't have sprints, we do production releases every 4-6 weeks, we deploy to an internal staging server (which we use as our internal production server, where we use our product for dev as well as other functions such as marketing, sales, support, implementation services, etc.) every week or less, and we have a single workflow between Dev and Ops as you will see below.

From that separate planning lane, we have evolved to a full upstream Kanban board - which is our roadmap planning board, where we track Themes, Epics and Stories - and lay out stories in up to 4 releases out. We did this as we gradually figured out just how much work needs to get done to identify and detail the work to be done as well as to get stakeholder alignment so everyone agrees that we are working on the right stuff.

Roadmap Planning Board

Stories from this board get pulled to the Dev board.  Internal and customer reported defects are directly added to this board's Ready queue.

Dev Board

As you can tell, the Product Owner (Product Manager) is responsible for the Roadmap board, while the Sr. Dev Manager is responsible for the Dev Board.  While Dev team members deply to Staging server, the Ops team ensures they have the right CI/ CD environment to do that.  After Dev/ Validation, it is the Ops Team's job to  deploy to production once sufficient valuable work has been completed (typically 4-6 weeks).

Both boards are accessible to all senior stakeholders as well as CEO, Sales and Marketing so everyone can track what is going on and what is prioritized for upcoming releases.

At the same time, we ensure we follow some of the recommended Kanban cadences, including the Daily Standup Meeting, the Weekly Replenishment Meeting (where fine-tuning of the story/ defect prioritization takes place) as well as Release Retrospective and a Quarterly Strategy Review Meeting.

The Kanban boards and cadences ensure that everyone is aware and aligned; and if there are conflicts, they are resolved during one of these meetings (other than the Daily Standup).

Besides key Kanban practices - including a reasonably strict adherence to WIP Limits - we have also, of course, invested heavily in engineering practices to enable a true continuous delivery/ deployment environment - and that journey still continues.  We are a Test Driven Dev shop and developers are responsible for JUnits, Functional Automation, and Peer Review.  Our code repository is Subversion and integration/ deployment is orchestrated using Jenkins. Subversion is integrated with our Kanban tool so all work at each stage is also updated on the card as well (via comments on the card).

Overall, to answer some of your questions more specifically,

Yes, it makes a lot of sense to provide a separate board to the product owner and another to the dev/ ops teams and link cards across them as needed.
And it makes sense to organize the board by tasks/ workflow stages rather than people as that gives you greater flexibility with using cross-functional folks in different stages of the workflow.
Most importantly, it lets you easily visualize and measure which workflow stages might be bottlenecks (rather than which people - which would typically be inaccurate) and what tweaks you might need to make to improve your flow and throughput and reduce lead time.

All the best with your transition to Kanban! If you need any more help with respect to Kanban, I strongly recommend reading David Anderson's book on the Kanban Method as well as refer to our comprehensive Kanban guide.

Hope this helps.


docker "Cannot mkdir: Permission denied" I'm trying to run wordpress inside docker.

I've been following this tutorial : https://www.sitepoint.com/how-to-use-the-official-docker-wordpress-image/

But when I run this command :

docker run -e WORDPRESS_DB_PASSWORD=xxxxxx -d --name wordpress --link wordpressdb:mysql -p 127.0.0.1:8080:80 -v "$PWD/":/var/www/html wordpress

The container crashes, and in the logs I can see things like :

tar: ./wp-admin: Cannot mkdir: Permission denied

Try to run such command as root or grant user and group rights to the folder with chown.

That's only what comes to my mind for now.


Open source private cloud VM provisioning for container workload There is a new generation of tooling developed that tries to integrate different IaaS providers (Azure/AWS/GCE/DigitalOcean) into one "infrastructure codebase". It's dubbed "Cloud Native" and it's supposed to result in "one tool to rule them all". There's Terraform from HashiCorp, InfraKit from Docker, Juju from Canonical, independent Foreman, Pulumi and others.

It's easy to get lost. Which of these tools is best suited to IaaS on premises? Let's suppose you have a bunch of metal servers and you want to provision some VMs there. Let's suppose you don't want VirtualBox/VMWare/ESX/Xen, you want KVM. And on top of that an obvious choice is libvirt. To manage multiple hypervisors we need to connect to the libvirt daemon remotely using connection URI. For KVM we can use qemu+ssh. Let's suppose we set those up using PXE boot. Now we have a bunch of qemu+ssh:// links along with ssh access.

Now we need a central component on top of that to manage resources and some tool that will provision virtual hosts according to some declarative specification.

On top of those VMs I want to run containers, so automatic setup of container orchestration cluster is desired. In addition, some way of attaching storage is desired. At least directly attached block devices or fs mounts.

Any suggestions?

Note: Found a similar question on SO, but for Xen.

Clarification: "best suited" needs metrics. So these are the metrics I'm looking to optimize:

Zero closed source LOC
Number of tools if not "All-in-one"
UI actions to setup 3 master + 5 workers k8s/swarm cluster

number of commands (for CLI tool)
number of clicks (for Web based/GUI tool)
length/size of the config file

UI actions to install the tool
UI actions to maintain configuration (preferably none, self-healing infrastructure)

"best suited" is the question. How do you measure this? what are the metrics and why do you choose them? I think you are looking for some kind of "tools combination", earlier tested on production by someone else's if preferred. What you want to do is try some opensource/free private cloud on-premises and run IaC on this private environment. You can test tools forever. How do you compare those tools? I believe all of them works fine as long as there is not any inconsistency problem. I think there is a missing/main part in your search when it comes to IaC. Because of few options for that missing angle as well. I think it is better to create a chart for all of us as you are going this process in order to benefit people here. Do you have a working environment? If so could you please detail all the tools you used to create this environment?

Some vendors run open source projects. How about them?

The Cloud Native Computing Foundation seeks to drive adoption of this paradigm by fostering and sustaining an ecosystem of open source, vendor-neutral projects.CNCF Cloud Native Definition v1.0

I started to wonder if this is cloud-native then what is it that is not cloud-native?

I think this is what you are looking for. The distinction between provider and provisioner is blurred. A workflow automation tool should be doing what cloud orchestrator with GUI does. Terraform makes a distinction between provider and provisioner but Cloudify removes them as I see it. For an architect designing the whole workflow with GUI might be useful to reach the desired state. Writing desired state for which you do DRY tests already beforehand or showing with graphs. There is something you know already where it goes since you already tested, seeing it flowing in graphics workflow makes sense? Then go for it. To me it is painting the history I do not see the benefit.

There are many ways to break down the types of automation, whether it’s imperative versus declarative, or orchestration versus desired configuration state.

Most orchestrators are fully CLI based, or offer very limited GUI operations.
A distinct advantage here is that Cloudify enables you to one-click your way through architectural relationship diagrams. See: Terraform vs. Cloudify.

Providers

A provider is responsible for understanding API interactions and exposing resources. Providers generally are an IaaS (e.g. AWS, GCP, Microsoft Azure, OpenStack), PaaS (e.g. Heroku), or SaaS services (e.g. Terraform Enterprise, DNSimple, CloudFlare).
See: Providers.

Provisioners

Provisioners are used to execute scripts on a local or remote machine as part of resource creation or destruction. Provisioners can be used to bootstrap a resource, cleanup before destroy, run configuration management, etc.
See: Provisioners.


Pass parameters to jenkins job with the help of userContent I am trying to create a so-called proxy page which will accept user parameters in Jenkins and then pass them to url for running pipeline with specified parameters.

I am trying to achieve it with the help of userContent Jenkins feature:

   <form action="http://user:token@jenkinshost:port/job/dev/job/dev-pipeline/buildWithParameters?n1" method="post">
       Number1:<input type="text" id="n1" name="n1">

       <input type="submit" value="Submit">
  </form>

But nothing works.

I was even trying to set up the next policy through Jenkins console:

System.setProperty("hudson.model.DirectoryBrowserSupport.CSP", "sandbox allow-same-origin allow-scripts; default-src 'self'; script-src * 'unsafe-eval'; img-src *; style-src * 'unsafe-inline'; font-src *");

It also didn't work.

Am I doing something wrong by missing something?

Looks like I read documentation not attentively.

userContent supports only static content but not dynamic.

Thus, PHP/JS or something similar can not be used there: https://wiki.jenkins.io/display/JENKINS/User+Content


How to skip reviewer for project administrator pull request? Is it possible to configure the repository to skip "At least one reviewer must approve the most recent iteration" in case the person who performs the Pull request is the project administrator?
I have the below branch Policy:

I made the review approbation for my own commit.

But when I'm going to complete the pull request, I get the below message:

At least one reviewer must approve the most recent iteration.

It's possible to skip review requirement only for project administrators?

As a workaround, you can add "bypass policies" for project admins.

The documentation: Security groups - Git repository (object-level)


Apache container cant access php files mounted in /var/www/ - 403 error I want to run my php website in a container with apache and php7
I am using this docker image : https://hub.docker.com/r/nimmis/apache-php7/

I run Fedora so I enter the example command given in docker hub in my terminal.

docker run -d -p 8080:80 -v {my_php_app_path}:/var/www/html nimmis/apache-php7

The container pull and run fine. But once i try to access the asset (on localhost:8080) I mounted to it, I receive a 403 error. So now i try to connect to the container to see what happens.

I use the command adviced by the author:

docker exec -ti <container id> /bin/bash

Once in it, I cd to /var/www/html all the file are presents but I cant read or ls in the folder. the root of the container cant do anythig with it. Here is what I get :

>>> root@a1f13dc4c29d:/var/www/html# ls
>>> ls: cannot open directory '.': Permission denied

Why is it that way ? Am I missing a step ?

Any help would be loved :D

SELinux prevent the docker filesystem to work in the intended way, to fix this issue just add the :Z flag at the end of the volume declaration. It goes something like this:

docker run -d -p 8080:80 -v {my_php_app_path}:/var/www/html:Z nimmis/apache-php7


How to integrate a platform as a service with configuration management tools? Is a configuration management tool (Chef/Puppet/Ansible) necessary for managing an application on a PAAS system (Azure app service, Heroku, Elastic Beanstalk)? How would it be utilized? While a tool such as Chef does not inherently work on a PAAS system, there are other tools (such as Habitat) that can support this functionality. What kind of steps would it take to integrate these technologies?

A configuration management tool is not required to manage a PAAS. For example, when one has created a VM on Azure then it is possible to export the configuration snippet as powershell or as yaml and use yaml to deploy and manage systems.

In my opinion one should restrict configuration management tools like puppet or ansible to manage configuration files and installing additional packages and yaml to manage the PAAS as I prefer KISS.

If one would like to manage different PASS then one could consider to use terraform.


how to stop and start kubernetes cluster I have launched kubernetes cluster using kubeadm with 1 master and 2 worker node. Is there any safe way to stop and start the entire cluster without disturbing deployments.

If I restart the master VM directly, It shows most of system pods containers in a stopped state.

You can maintain your data while stopping the master if you have persistent volumes attached, this is especially the case when using GKE google Kubernetes engine. But if you wanted to keep your services running while completely stopping a master that would require High Availability, Allowing one master node to take over the responsibility of another master which is clustering.


Is it possible to create a robot account in bitbucket without creating a new email account? Introduction

According to this article a user account could be given to a robot for Continuous Integration purposes.

Method

The IT department created a new email account, this was used to login to the git repository, a ssh key was created, the public key added to bitbucket and finally the jenkins user was able to create commits in bitbucket.

Problem

As the email account is only used by the robot and the robot itself is not reading emails it feels that the email account is misused.

Question

The question is whether it is possible to create a robot account in bitbucket without creating a new email account.

Discussion

If there is no separate email account how could bitbucket deviate between the robot account and an existing real user? A unique email account should be used right?

After doing some research and comparing various documents it seems that a bot account is basically a real user account and needs a separate email account in order to create such an account as using an email account to register a new account results in, e.g.: This email account is already in use.

Combined with quote retrieved from the link that is defined in the question:

  User accounts are intended for humans, but you can give one to a
  robot, such as a continuous integration bot, if necessary.

It seems to be necessary to create a new email account if a bot account has to be created as:

git repositories, like github, bitbucket and gitlab check whether an email account already exists if a new user is created
to clarify that the user is a bot and not a human


How to get 'diff' on Windows TravisCI machines? Attempt was choco install git -y, but it does not seem to be available:

The command "diff -u <(echo -n) <(gofmt -d ./)" exited with 2.
diff: /dev/fd/62: No such file or directory

Windows Travis Build

When git is installed it usually doesn't add the diff executable to the path because it would conflict with other versions of diff native to Windows, for example, the Compare-Object alias in PowerShell.

You can usually get around this by adding C:\Program Files\Git\usr\ to the path which contains all of the UNIX ports of functionality such diff, grep, awk, sed.  You do have to be careful as some commands are used more commonly on Windows than others, the most common one is find.
A second option would be to fully qualify the call to diff by appending C:\Program Files\Git\usr\bin.


Docker on Development Workstations I recently had an interesting discussion with a colleague.  We have different perspectives on what is required to develop Docker containers.  I won't say who believes what, to keep from prejudicing the responses.  Here are two competing assertions:

Assertion 1:
It is not necessary to install Docker on development workstations.  Container build is a CI task that can be performed by a CI tool, then containers can be troubleshot/debugged in a DEV environent.

Assertion 2
It is critical for developers to run Docker locally so they can smoke test and execute containerized processes before they are committed to SCM or CI.  Containerization is a core developer skill, which is specialized by development platform.

Is it critical, nice-to-have, or not necessary for Docker to be installed on developer workstations where containerized workloads are developed?

Someone said that to each question, there are always three answers. So here's your third answer.

Install docker on the dev machine, and build images with it.

But:

The images should not contain your source code or anything that changes during the usual edit-build-test-run cycle. The images only need to contain the basic stuff that does the build/test/run parts.

That is, if you were to develop a Maven application, your image/container would contain the mvn command with all its depdendencies. A second image/container would contain your wildfly  or whatever you are using as an application server.

If you were to develop in ruby on rails, you would have one image/container which contains everything to be able to run bundle exec rake cucumber, another image/container which contains everything to be able to run rails s or rails c.

The images will be recreated seldomly, for example if you want to upgrade the application server, and they will certainly not be docker build't on each run by the dev. Both images would not contain anything that belongs to your application. No source code, no .war file, no precompiled assets or application-specific gems for the ruby application. These would be mounted as volume when starting the containers, and point directly to whatever local directory the dev is currently working with.

This way, it is guaranteed that your dev uses the exact same base environment as your CI or production. New devs can start building/testing/running instantly and never need to install anything to their host machine. They can use the editor of their choice (if they wish to use something like Eclipse which has lots of magic and wants to compile the stuff itself, then they are free to do so of course - as long as they do at least a cursory check whether everything still works in the docker variant, before pushing).

And if you wish to deploy your application as a fully self-contained docker image, finally, then you are free to do so. Your CI server can build those images. It should use the smaller images used by your devs as base image, obviously.

This gives you a nice medium ground which should cover everything.


Should I run several applications in Docker in a single server? I have a server,32Gb ram, i7 processor and 2TB hard disk, with Docker engine running in it. I have like 100 development application to be migrated to this docker server.

So is it a good practice to run all your application in Docker in a single server?

Yes, sure. As long as your resources are enough (i.e., you're not running out of RAM, HDD, CPU, networking), and as long as you are fine with one gigantic single point of failure, there is no reason whatsoever to not run many docker containers on one machine.

Keep in mind that the overhead of docker is really small - it's all just namespacing on the host kernel, it's not like you're spawning copious amounts of VMs or anything like that.


How to build a complex parallel Jenkins pipeline? I've been interested in converting our bespoke Jenkins integrations into a pipeline. However, I can't seem to figure out how to do it.

Can anyone help me with the Jenkins script that could do the following?

1---2---3-----------9---10
    |           |
    |---4-------|
    |           |
    |---5---6---|
        |       |
        |---7---|

1: Start pipeline
10: End pipeline
5: Build some files
   * needed by 6, 7,
   * needed as artifacts at the end
2, 3, 4, 6, 7: Have jUnit result files, should be available at end of
   test (somewhere), even if one failed

Is this even possible? Or should I just join after 3, 4, 5? Like this:

1---2---3-------6-------9---10
    |       |   |   |
    |---4---|   7---|
    |       |
    |---5---|

Based on the comments to my question, and some basic testing the following seems to work:

Use nested parallel statements. (Thanks @lawnmowerlatte)
Although today (May 15th 2017) Jenkins Blue ocean does not visualise this properly. Possible solutions discussed here


How to run WSGI Python server without Apache or NGINX Is there a way to run a Python server that uses WSGI bindings directly? I am trying to go with the one process per container philosophy which will naturally get rid of Apache/NGINX from the container and just run a python process.
Is there a way to run python server directly with WSGI bindings without using Apache or NGINX?

https://www.quora.com/unanswered/Is-there-a-way-to-run-a-Python-WSGI-server-directly-without-Apache-or-NGINX

This question doesn't really make sense - WSGI is the mechanism that a web server such as Apache would use to invoke an application or script, such as something written in Python. If you have no web server, then what would be using WSGI in the first place?

Is the question really, how do I take a pre-existing WSGI application and turn it into a standalone? Flask is one such framework that can run as WSGI or using its own internal web server.

But, taking a step back, I still think this is the wrong question. There is no need to blindly apply the "one process per container philosophy" - which is really just some random person's opinion, at the end of the day - if it doesn't make sense in your use case. Think about it for a second. A container containing one process is just... a process. We already had those. Containers work best for bundling things together that logically belong together and administering them as a single "thing". What's inside them should be completely opaque to who- or whatever is managing them. You will never see developers jumping through hoops to follow a "one shared object per executable philosophy" because that doesn't make sense either!

If you have something that already works well with Apache/Nginx fronting your application, then containerize it that way, it will be fine.


Application monitoring in multiple EC2 instances I am running 10 AWS EC2 instances. Each of them running different applications. I am looking for application level monitoring solution. Suppose, if Apache fails in any instance, it should notify me with logs. I know about "Nagios" but it is mainly for infrastructure monitoring. There are few tools for application monitoring (like, "Monit") but I have to install them manually in each server, which is not possible when you are having multiple number of EC2 instances (A global set up would be better, because number of instances will keep increasing).

It looks like your problem is two fold:

You want to monitor the status of critical applications provisioned on virtual machines
You want to be able to install these checks rather intuitively.

I'll answer these in the reverse order.

The easiest answer for #2 is that you can use configuration management like Chef, Ansible, or Puppet to provision machines. That way you could write a cookbook that as a software package is detected on the system a simple agent based check is dropped in as well. Codifying these things makes them much more manageable in the long term.

In terms of #1 you're probably looking for an agent based monitoring tool. Some examples of this would be Prometheus or Sensu but there's an endless list of them out there. These allow you to describe checks on the server and the master just collects whatever information is given to it.


Is it possible to load an image into S3 via an URL? I'm going through some steps documented in How Do I Upload Files and Folders to an S3 Bucket? to get my ova into S3. Seems like a waste of time for me to download it and then upload it, is there a mechanism to import an S3 object from a web resource?

I have a UCS Director job that builds OVAs for me, uploads them to a Cassandra big data cluster and serves out obfuscated links for me and customers to download them - I'd like to provide said obfuscated link to AWS instead of downloading it on my computer and uploading it to S3.  In the end, I think I could just make my images go straight from UCS Director to AWS, but for the time being, I'd like to save a step.

Nope, credit goes to this SO answer, still valid today: How to upload files directly to Amazon S3 from a remote server?.

But if you have a server in the cloud, ideally in AWS, it'll probably go faster to execute the copy operations on that server - higher bandwidth and lower latency to/from such server when compared to your computer.


Time series data stores in CI context - meaning? From reading this job ad:

  You believe CI servers, push button deploys, time series data stores,
  metrics dashboards, and centralized logging are not just “nice to
  haves” they are critical pieces of infrastructure that rapidly pay for
  themselves.

What does using a time series data store mean for CI?

Since time series datastores are indexed by timestamps they can be very useful in a CI/CD environment, depending on what kind of information they store, for things like:

incident analysis and management
service operation and SLAs monitoring
overall SDLC pipeline performance/efficiency analysis and improvements
resource capacity planning
budget and ROI computations


Why is the K8s UI not available when rancher is used? K8s has been created using rancher, but when one would like to access the UI it shows:

Service unavailable

Methods

Method 1

rancher was deployed locally using vagrant
k8s was deployed
click the ui button

Method 2

perhaps the local firewall was causing the issue?

rancher was deployed on digital ocean
k8s was deployed
service unavailable again

I've seen this happen, especially on the newest (as of now) version. Try going into the "Infrastructure stack" and look for stopped containers. Often it helps to just refresh the entire deployment (hit the button "up to date" next to the "kubernetes" stack and hit save (If I remember correctly) in the bottom to force the "refresh").

You could also try manually just restarting the kubernetes-dashboard containers.

Lastly you could use the kubectl cli to delete the dashboard pods and force them to get recreated.

Let me know how it goes! I know I had to do this fix a few times.


Start only one pod at a time in kubernetes Is it possible to tell kubernetes, that when performing a deployment of multiple pods that it should start only one at a time and only create the next pod if the previous has completely started up?
It would be useful in our usecase:
Our application automatically migrates/upgrades the database when starting up. But when there are multiple pods starting up at the same time, it could happen that multiple pods try to upgrade the database at the same time which could corrupt the database.
But if kubernetes would wait with starting up the second pod until the first pod has fully started up, it would mean that the database has been correctly upgraded by then.
If kubernetes does not have such functionality, how would you handle such a case?

Use a combination of Readiness Probes and a proper deployment strategy.

Configure your Deployment spec like so:
spec:
  replicas: 4    # or as many as you need to have
  strategy:
    rollingUpdate:
  maxUnavailable: 0
  maxSurge: 1

maxUnavailable = 0 means that there should always be a number of pods available equal to the replicas count. maxSurge = 1 means that kubernetes will create the new pods one at a time, and destroy the old ones one-at-a-time as well.
Implement a readinessProbe which should return a successful response when the database migration has completed. You'd have to write the logic for that yourself, but if you do it right, kubernetes will create a pod with the new Deployment and wait for it to complete and only after that, evict 1 of the old ones.

PS: It's worth noting that this approach will cause your node to temporarily contain more pods than your replicas config implies, namely replicas + 1 for a short period of time while deploying. You need to make sure that your node can handle the little extra load untill the rollout is complete.


When exactly and what for should VOLUME in a Dockerfile be used? When writing a Dockerfile, I do have the option of specifying volumes, such as:

VOLUME /var/www

But even without this declaration I am able to mount volumes to this directory and other places, e.g. by using -v /foo:/var/www or -v ./httpd/conf.d:/usr/local/apache2/conf/conf.d for bind mounts when running docker.

So:

Is the specification purely informational?
What are the (dis)advantages of (not) specifying volumes?
Why and in which cases should I use VOLUME and when shouldn't I?
What about files that may optionally be mounted? Do they count as volumes in that case, too?

Short version

VOLUME instruction and -v dest are used to create unnamed/anonymous volumes, -v scr:dest option is for mapped volume.

Long Version

Dockerfile's VOLUME does not allow you to specify a host path.
On the host-side, the volumes are created with a very long ID-like name, these volumes are often referred to as unnamed/anonymous volumes.

Given this Dockerfile:

FROM php7:latest
VOLUME /var/www

Build it:

docker build -t myTest

Run it :

docker run --rm -it myTest

Inside the container, run ls and you'll notice the directory exists; /var/www

Running the container also creates a directory on the host-side.

While having the container running, execute docker volume ls on the host machine and you'll see something like this

DRIVER    VOLUME NAME
local     c984..e4fc

Back in the container, execute touch /var/www/myFile

This file is now available on the host machine, in one of the unnamed volumes.

ls /var/lib/docker/volumes/c984..e4fc/_data

Similarly, you can try to delete this file on the host and it will be deleted in the container as well.

Now run a new container, but specify a volume using -v:

docker run --rm -it -v /myVolume myTest

This adds a second volume and the whole system ends up having two unnamed volumes.
On the host-side, the new second volume is anonymous and resides together with the other volume in /var/lib/docker/volumes/.

It was stated earlier that the Dockerfile can not map to a host path which sort of pose a problem for us when trying to bring files in from the host to the container during runtime. A different -v syntax solves this problem.

Imagine I have a subfolder in my project directory ./src that I wish to sync to /src inside the container. This command does the trick:

docker run -it -v $(pwd)/src:/src myTest

Both sides of the : character expects an absolute path. Left side being an absolute path on the host machine, right side being an absolute path inside the container.

We run this command:

docker run -v $(pwd)/src:/src myTest

Is the specification purely informational?

No, it does mount a volume, it's similar to -v /var/www you just don't specify a mount point in the host machine, so docker will take care of it.

What are the (dis)advantages of (not) specifying volumes?
Why and in which cases should I use VOLUME and when shouldn't I?

It's probably a best practice to never use VOLUME.
The first reason we have already identified: We can not specify the host path.
The second reason is people might forget to use the --rm option when running the container, you will end up with a couple of unused volumes and it might be a daunting task to figure out which of all anonymous volumes are safe to remove

What about files that may optionally be mounted? Do they count as volumes in that case, too?

In this case you don't need volumes; mapping a single file does not make sense, I would use ADD or COPY inside the dockerfile.


Jenkins Testing PR's - allowing broken builds I have a very large codebase that currently takes ~30 minutes to build via Jenkins.  It's connected to our Source control/review system (Let's just say GitHub for now) so that every PR automatically gets tested and must build before we allow it to be merged.

We're seeing a behavior where 2 PR's (We'll call them PR#a and PR#b) can exist and each one is tested successfully (code builds, unit tests run, etc).  However, they're each tested against the tip of develop.  That means once PR#a is merged, PR#b shows that's been tested and valid.. However merging it exposes an API break merged as part of PR#b, and now the build is broken pending a fix.

What's the typical way to prevent this? If not in GitHub, what about other systems? (GitLab, Gerrit, or Stash/Crucible ?)

On GitHub - you can't. Basically, their PRs are temporary branches, and there's no check if base commit is HEAD of branch to merge in.
This problem is solved with gating systems like Zuul, which puts all PRs into queue to run tests.

You are talking about systems for managing and storing code, while the issue is in CI scope (i.e. Gitlab flow has this issue described in "Testing before merging" paragraph), Zuul is made exactly to fix this problem, because the amount of merges to OpenStack codebase per day is large (year ago they had 6 zuul instances to run tests for merges).


What is the way to update images in a configuration git repository when doing gitops? i'm trying to do gitops

I have a Configuration repo containing k8 manifests and different app repos.

1) App repo's CI build docker images and pushes into registry.

2) App repo's CI commits into Configuration repo with newly built image tags.

A gitops CD like ArgoCD is listening to configuration repo and deploys(manual/automatic) it to kubernetes.

Is this the right way to do it ?

2) App repo's CI commits into Configuration repo with newly built image tags.

Yes, this makes sense. Instead of using image tags you can use image digest, since the digest is generated from content, it is immutable, while tags may be updated (intentionally or unintentionally).

This manifest update can be done with multiple tools:

Using yq

yq write --inplace  deployment.yaml 'spec.template.spec.containers(name==myappname).image' gcr.io/my-image-repository/myappname:labelOrDigest

Or using a newer tool; kpt with the ktp set command.


A systematic way to test RESTful APIs with curl? I have noticed during integration testing that I actually work on recurring use cases with restful APIs (or in general HTTP interfaces) I check here and there with bash+cURL.

It starts looking quite messy and gets harder to maintain. Why deliver mess?

Typical use cases are:

Check that an URL returns http response code e.g. 200
Check that content type matches some MIME you need in that case
Check that returned content matches some pattern, or passes an abstract validation procedure

What I have found so far and consider a workable option without reinventing the wheel, is:

have a go with PyCurl - in a hope it fully implements all cURL options esp. proxying but also other switches I might need
use Python's built in unit testng

Then I could have, for example, one unit test per service I'd like to check:

import unittest, pycurl

class TestService (unittest.TestCase):

    def test_1(self):
        self.assertEqual(pycurl.returncode("some_url"), 200)

    def test_2(self):
        self.assertTrue(pycurl.response("some_url").matches ("xxx") )

    def test_3(self):
        self.assertTrue (pycurl.ContentType("some_url").equal("xxx"))

if __name__ == '__main__':
    unittest.main()

Does this make sense or is there a more high-level (but not too complex to pick up and integrate) tooling?

You could look at tools such as Postman which focuses on testing REST APIs with JavaScript - it has some nice features but you lose the use of Python.

Instead, I'd suggest looking at REST-related plugins for pytest, a Python test framework that simplifies your test code, while still running tests written using unittest.

For example, writing parameterised tests avoids boring and error-prone duplicate test code.

Pytest has a huge set of plugins that simplify various tasks, including:

Tavern, which specialises in testing REST APIs and seems highly relevant here - sort of "Postman for Python unit tests".
pytest-curl-report - when  testing with the requests library, will print a curl command you can use to reproduce the error from shell.
Testinfra - focuses on server testing (e.g. state of OS packages, files, processes, etc, usually tested on remote servers) - strongly recommended if you also need this type  e.g. to test Ansible code.

For those who use Puppet or Chef, Testinfra is similar to Beaker (with RSpec), ServerSpec or InSpec.

If you don't like Tavern, you can of course use pycurl with pytest, which makes it easier to diagnose exactly what failed. This example from the pytest-curl-report site uses only generic pytest features:

$ py.test test.py
============================= test session starts ==============================
platform darwin -- Python 2.7.9 -- py-1.4.27 -- pytest-2.6.4
plugins: curl-report, httpbin, cache, capturelog, cov, flakes, pep8
collected 1 items

test.py F

=================================== FAILURES ===================================
______________________________ test_requests_get _______________________________

    def test_requests_get():
        r = requests.get('http://httpbin.org/get')
>       assert False
E       assert False

test.py:7: AssertionError

Pytest lets you write all tests with a plain assert, and optionally include a helpful message as part of the output.  For example, one of your tests could be written:

def test_2():
    assert pycurl.response("some_url").matches ("xxx"), "xxx not found in response"


How do I get the hosts from a specific section of an Ansible `hosts` file in a Jinja2 Template I have a hosts file including the following contents

[elasticsearch_nodes]
host1 os=linux
host2 os=linux
host3 os=linux

In a .j2 template I want to get the the list of [elasticsearch_nodes]

{{ some_ansible_magic_here }}

# should output "host1","host2","host3"

How do I do this?

I do have this as well to help with the quotes and the commas when I have the list.

{% for host in some_sort_of_ansible_magic_to_get_me_that_list_i_need %}
 "{{ host }}"{% if not loop.last %},{% endif %}
{% endfor %}

The magic you need is magic variables, in particular the groups variable, which is

  a dictionary/map with all the groups in inventory and each group has the list of hosts that belong to it.

Then you can use template filters to format the list: use map with regex_replace to add the quotes to all list elements and then join them into the final result.

{{ groups['elasticsearch_nodes']|map('regex_replace', '(.*)', '"\\1"')|join(',') }}


On-Prem and Cloud Deployments If I have a mix of cloud and "on-prem" (across different networks) systems to deploy to is there a common tool, technique, or software available?  I've used Kubernetes for cloud deployments, but am unsure of what approach to take when On-Prem installs/upgrades are mixed in.

Is is considered best practice to attempt to standardize the deployment or work with the ops team to develop two separate deployment strategies?

If you can keep one common deployment strategy across both that would be simpler and less effort from the ops side of things.  Kubernetes seems to be a great choice for the cloud orchestration layer at this point in time.  Does Kubernetes make sense for the on-prem side though?  If you're not planning on each on-prem install including 5-10 servers to spread Kubernetes out over then it probably doesn't make sense to use it there.  All hope is not lost.  :)  You could take the same Docker containers that you ran under Kubernetes and run them yourself for on-prem.  You're going to be stuck dealing with service discovery, but if you're only installing on one server that can be solved in some old school way like picking and assigning ports to specific services.


How do I launch Lambda in my own VPC? The VPC configurations option in Lambda tells this:

  All AWS Lambda functions run securely inside a default system-managed
  VPC. However, you can optionally configure Lambda to access resources,
  such as databases, within your custom VPC.

So, can't I launch the Lambda function in my own VPC?

I have also asked the AWS Support folks for help too, as Lambda's security was crucial for us [HIPAA compliancy].

This was their response:

  Lambda has two options, you can have your lambda functions without a
  VPC or you could place your lambda function within a VPC. If you want
  to put your lambda function within VPC, then the most important thing
  to note is lambda function must be placed in private subnet. If you
  want to access internet resources from Lambda or make any AWS SDK API
  Calls, then you need to have the internet access for which you need to
  have a NAT Gateway or NAT Instance setup for the VPC, where your
  lambda function resides and have the private subnet point to the NAT
  Gateway. When lambda functions are placed within public subnet ( with
  an IGW route), then the lambda function won't work.

So, a lambda function can be put into a private subnet of a VPC, but not inside a public subnet. And for being able to create instances and ssh'ing into them, from this Lambda, add the private subnet to the security group of the public subnet [or another hacky way, is to add the VPC's IP range to the security group].


Serverless framework: how to specify a c++ lambda function with provided deployment package I have followed the guide in https://aws.amazon.com/blogs/compute/introducing-the-c-lambda-runtime/ to package my c++ lambda function into a zip file. The zip file has the contents:
function.zip
  bin
     function
  lib
     many .so files, etc.
  bootstrap

In the guide the lambda is deployed using the aws lambda create-function with the handler specified to be the same as the zip file name.

I would like to configure this lambda function in my serverless.yml and deploy it with the rest of my lambdas. I am not sure what the handler field below should be and using the zip file name didn't work out.

functions:
  function:
    runtime: provided
    package:
      include:
        - function.zip
    handler: function
    events:
      - http:
          path: function
          method: get
          cors: true
          authorizer: aws_iam

When I run serverless deploy -v I get warnings like:

  Serverless:  WARNING: Entry for function@function could not be
  retrieved. Please check your service config if you want to use
  lib.entries.

I don't see any documentation from serverless.com that they support C and C++ (while AWS Lambda might do), I have asked about it now so that we can know:

https://forum.serverless.com/t/c-and-c-support/8045

("Supports Node.js, Python, Java, Go, C#, Ruby, Swift, Kotlin, PHP, Scala, & F#") but if you want to try then I think that you should try some other name than "function" for your function. I did not try this at home but all examples use some other name than "function" because that word might be reserved.

https://serverless.com/framework/docs/providers/aws/guide/resources/

If that still won't work, check your path to your code and maybe you also want to set up everything up with a CI/CD pipeline with GitLab for example. That's how I do it.