Mastering CI infrastructure changes with GitLab With Bamboo, a best practice for infrastructure management of agents seems to be using Docker: any time you need a new version of compiler or test tool, the path to deploy a modified version of an agent is very fast.

How could this work with GitLab?

Gitlab CI supports running all of your builds inside Docker containers so you have a clean build environment every time. I'd recommend starting with the basic Docker executor but there is also an opportunity to autoscale your runners using Docker Machine or to execute the runners on a Kubernetes pod.

If you need to use Docker as part of your build (such as building, tagging and pushing images) then you can either use Docker in Docker or, preferably, bind mount the Docker socket of the host into your runner allowing it to run Docker containers as a sibling rather than child process. Note that the original author of DIND recommends using bind mounting instead of DIND, especially for CI.

To bind mount the Docker socket register your runner with the following command:

sudo gitlab-runner register -n \
  --url https://gitlab.com/ \
  --registration-token REGISTRATION_TOKEN \
  --executor docker \
  --description "My Docker Runner" \
  --docker-image "docker:latest" \
  --docker-volumes /var/run/docker.sock:/var/run/docker.sock

And then in your .gitlab-ci.yml file you can define your job as so:

image: docker:latest

before_script:
- docker info

build:
  stage: build
  script:
  - docker build -t my-docker-image .
  - docker run my-docker-image /script/to/run/tests

Using just the basic docker image rather than the docker:dind image


tools to gather variant information from thousands of servers and update into database Recently, I use ansible to gather information periodically from thousands of servers(which already installed collectd to gather CPU, memory, disks, netowrking info) in our company.

For example: I need to get those information daily

kernel version
installed packages versions(JDK, docker, etc)
running process info
some commands output (lldp, docker ps, etc)
some files content (cat /etc/some/config | grep word)

I use ansible shell module to execute those variant commands and parse the output into our inventory  NOSQL.

By using ansible which use SSH/WINRM to access thousands servers seems inefficient.(which fine tune with more forks, enabled ControlMaster pipeline, etc).

I are trying to find a better way to replace ansible for those information gathering and parsing.

Below are some solutions which I am surveying.

facebook/osquery is flexible and powerful, but maybe not easy(comparing to bash script) to run a customized commands and parsing the output by writing an extension.

collectd with exec plugin to run customized scripts seems straightforward. in this way I need to deploy new scripts to all servers from day to day.

Are there some tools which is more suitable for such variant commands output gathering? and easy to deploy, reporting, or may with some dashboard and API for operation.

Doing something with collectd feeding into Prometheus or datadog is definitely a good idea for gathering metrics, but it doesn't help you with things like the kernel version.  Handling process monitoring might be better handled with nagios or datadog as well.  But for the bulk of what you're doing I'd say that it makes the most sense to stick with ansible.
What you're doing sounds very similar to the ansible-cmdb project:

Ansible-cmdb takes the output of Ansible's fact gathering and converts it into a static HTML overview page (and other things) containing system configuration information.
It supports multiple types of output (html, csv, sql, etc) and extending information gathered by Ansible with custom data. For each host it also shows the groups, host variables, custom variables and machine-local facts.

I've been using this for the last couple of years and it continues to evolve and improve.  I hope you'll find it useful as well.


What tool is able to analyze images by connecting to clair? This clair documentation indicates that a number of tools should be able to scan images:

clair-scanner
reg
klar
clairctl

analyze-local-images

All of them have been tried, but none of them seems to work. All kind of issues occurred, including this:

2017-09-19 11:11:23.022625 I | Analyzing faild: Could not analyze layer:
Got response 400 with message {"Error":{"Message":"could not find layer"}}

Another issue that occurred using clairctl:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x518490]

  does this happen with all images including official like ubuntu,
  alpine, tomcat?

Using clairctl it was possible to scan postgres an other images that are available on dockerhub, but scanning local images failed

Clair-scanner is able to scan the images locally.

https://github.com/arminc/clair-scanner/issues/13

docker run --net=host -d --name db arminc/clair-db:2017-09-18
docker run --net=host --add-host postgres:127.0.0.1 -d --name clair --net=host arminc/clair-local-scan:v2.0.1
./clair-scanner nginx:1.11.6-alpine example-nginx.yaml http://127.0.0.1:6060 127.0.0.1


Google Cloud SQL: Can I change machine type with zero downtime on my database clients? I want to update the machine type of my Google Cloud instance, but this takes several minutes to update (second generation instance). The instance will be unavailable until the instance has restarted. Because of this downtime, we have to update the machine type at night time, so our visitors are the least troubled by this update.

Is there a workflow how we can minimise this downtime to zero or maybe a few seconds? I already thought about possible solutions like adding a temporary failover or maybe make use of read replica.

I contacted the support of Google Cloud about this question and they told me that Cloud SQL isn't build to perform this change without downtime. If I want to be able to make these changes, I should look at Cloud Spanner which is a horizontal scalable SQL solution provided by Google.


Change owner of files created inside a Docker container without changing the Dockerfile When I create a file inside a container with docker-compose run web touch test the file owns to root:root. I want to change it to be $USER without to use chown.

I'm aware of the --user option but it requires to have the user created on the Dockerfile. I prefer to not change the Dockerfile because it is committed on a git repo and I don't want to commit this because it is kind of personal config.

I tried to mount the file /etc/passwd but the user is not present when we run a command. I can use su $USER -c "..." but there is some issues with this.

So, what's the best way to set create new files with the owner set as the user of the host machine?

No need to change the Dockerfile. Just define a certain user, e.g. jenkins and a certain uid, e.g. 1000 and ensure the same uid is used when a folder is mounted.

ARG user=jenkins
ARG uid=1000

# Jenkins is run with user `jenkins`, uid = 1000
# If you bind mount a volume from the host or a data container,
# ensure you use the same uid
RUN mkdir -p $JENKINS_HOME \
    && chown ${uid}:${gid} $JENKINS_HOME \
    && addgroup -g ${gid} ${group} \
&& adduser -h "$JENKINS_HOME" -u ${uid} -G ${group} -s /bin/bash -D ${user}

RUN chown -R ${user} "$JENKINS_HOME" /usr/share/jenkins/ref

https://github.com/jenkinsci/docker/blob/master/Dockerfile-alpine


Jenkins Credentials Manager + Active Choices I want to access Jenkins Credentials to authenticate with Git before the pipeline script - to fill Active Choices parameter.
In the pipeline I simply do it with withCredentials. But during pre-pipeline stage (i.e. Active Choices parameter) Hudson extensions such as withCredentials or sshAgent are not accessible. Currently I execute groovy git ls-remote -h git@gitrepo.example.com:repo.git with unencrypted key in jenkins_home directory (which is not a good idea).
I know that there is some possibility to import and construct Java objects by reading Jenkin's and its plugins' source code - but this seems endless.
So: is it possible to somehow easily import and apply Hudson extensions to current Groovy script or does anyone have some ready solution for this (apparently nobody published anything about such case)?

It sounds to me like you're already aware of this, but there are a couple easy ways to do this, and a couple hard ways to do this.
The easy ways:

Make your repo available for checkout without SSH keys or authentication credentials.
Manually add an SSH keypair for repo checkout to the Jenkins user on your Jenkins master node (since Active Choice Groovy scripts run on the master node) (i.e. don't use the Credentials store).

The hard ways:

Use the Jenkins Java API in your Active Choices code to extract credentials from the credentials store.  API docs are available for the credentials store and the ssh credentials type.  There is no documentation on how to use the Java classes available in these plugins, so it's up to you to figure it out yourself.
Create a secondary "meta" job to update the parameters in the original Jenkins job (including Active Choices scripts) - I describe this a bit below.

If you can't do any of the easy solutions for security reasons, there isn't an easy, straightforward way to update Active Choice parameters based on the contents of a git repo.  This is because:

Like you said, "during pre-pipeline stage (i.e. Active Choices parameter) Hudson extensions such as withCredentials or sshAgent are not accessible".
The configuration for parameters, including Active Choice parameters, is taken from the previous run of that Pipeline, not the current one.  Meaning that, to update your parameters from within your Pipeline code, you'll need to run your Pipeline, which defeats the point of what you want to do.

There is a solution for 1., which is to use Scripted Pipelines instead of Declarative, but that won't get around the problems of 2.
The solution that I came up with when I ran into this issue is to create a second Jenkins job (I call it a "meta" job) which updates the parameters of the original job, including Active Choices scripts.  This second meta job is triggered by a git hook on the repository that I use to fill the options in my Active Choices parameters.  This has the advantage that you do not need to run the original job's Pipeline in order to update its parameters, solving the issues in 2.  Also, because it's still a Pipeline job, you can use normal Pipeline steps, thus allowing you to access the credentials store.
My job looks (roughly) like this - I simplified it a lot to make it fit here, so you'll need to do a bunch of tweaking to get it working the way you want.  There are probably a ton of errors and bugs in this because I ripped apart an existing script to make this.  A few things to note:

This makes heavy use of Jenkins APIs, so you'll need to either disable the script sandbox or whitelist a ton of functions in the Jenkins security manager in order to get this to run
This is still technically a Scripted Pipeline even though it looks much more like a normal Groovy script than a Pipeline file
Because this is a Pipeline, you can use steps like withCredentials to check out repositories using the credentials store.

import groovy.json.JsonOutput

import org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SecureGroovyScript

import org.biouno.unochoice.*
import org.biouno.unochoice.model.*

def metadataUrl = 'ssh://git@**************/metadata.git'
def metadataBranch = 'master'

def metadata = [:]

node('master') {
  git(
    url: metadataUrl,
    branch: metadataBranch,
    credentialsId: 'jenkins',
    changelog: false,
  )

  def dataDir = "${pwd()}/myJsonFiles"

  // can't use File.eachFileMatch() due to:
  // https://wiki.jenkins.io/display/JENKINS/Pipeline+CPS+method+mismatches
  def files = new File(dataDir).list().findAll() { f->
    FilenameUtils.getExtension(f) == 'json'
  }

  files.each() { f->
    // TODO: nice error messages if files are misformatted
    def name = FilenameUtils.removeExtension(f)
    def data = readJson(file: "${dataDir}/${f}")
    metadata[name] = data
  }
}

// ChoiceParameterDefinition is the class name for the built-in static choice parameter.
// CascadeChoiceParameter is the class name for the Active Choices dynamic choice parameter.

// begin first parameter
def fileParam = new ChoiceParameterDefinition(
  'file',
  'The JSON file to use',
)
fileParam.setChoices(metadata.keySet())
// end first parameter

// begin second parameter
def metadataJson = JsonOutput.toJson(metadata)

def scriptText = """
import groovy.json.JsonSlurper

def slurper = new JsonSlurper()
def metadata = slurper.parseText('${metadataJson}')

return metadata[file]['versions']
"""
def mainScript = new SecureGroovyScript(scriptText, true)
def fallbackScript = new SecureGroovyScript('return ["Error encountered - see logs"]', true)
def activeChoiceScript = new GroovyScript(mainScript, fallbackScript)

def versionParam = new CascadeChoiceParameter(
  'version',
  'The version to use',
  activeChoiceScript,
  'PT_SINGLE_SELECT',
  'file',
  false,
)
// end second parameter

def paramProp = new ParametersDefinitionProperty(fileParam, versionParam)

// this is the full name of the actual job which needs dynamic parameters
def job = Jenkins.instance.getItemByFullName('UNIX/deploy_to_environment/master')

// There is no "setProperty" - we need to remove and replace.
// removeProperty is safe to run even if no matching properties are set on the job.
// However, this method only removes one property per method call.
// Theoretically, this means the job could accumulate multiple conflicting
// ParametersDefinitionProperty properties.
// TODO: run removeProperty in a while loop to eliminate all matching properties for extra safety.
job.removeProperty(ParametersDefinitionProperty)

job.addProperty(paramProp)


How to storing encrypted secrets in code for a serverless.com project? With serverless.com, the easiest way to expose a secret to an AWS Lambda function is to store it in the serverless.yml file (encrypted with KMS, for example).

But committing encrypted secrets to Git is not the best thing in the world. Requires code changes when the secret has to change, for one thing.

But in terms of security alone, what are better alternatives? e.g. the secret could be stored in S3 (encrypted), with the Lambda given access to that location and KMS key, but is that actually better in any meaningful way?

There are a couple of elements to consider here:

It is valuable to have the capability to maintain configuration with a separate release cadence from the code that is being configured. Doing so encourages rotation of credentials in an automated manner, do it regularly make it painless.
It is entirely reasonable from an "Infrastructure as Code" point of view to store all configuration in Git, albeit probably in a different repository as the scope of configuration is likely to span more than one service.
Assuming both of the above statements are relevant to your situation, I would be inclined to encrypt the data with KMS and store it in S3 or DynamoDB.  I would even go so far to say that in very simple situations it might be suitable to encrypt and store the data within KMS itself.

There are several open source libraries that will support this operation:

Python: CredS3
Go: Sneaker


What does "build step" mean? The term build step is used everywhere, yet I searched both google.com and qwant.com to no avail. That baffles my mind.
I also searched for a developer terminology index. Anyone has a good explanation of this term?

In my experience "build step" is synonymous with "compilation", e.g. it's the step that builds a set of output files (binary, etc.).


Ingress or LoadBalancer on AKS I'm wondering whether to use an Ingress or LoadBalancer to expose services on Azure AKS.

I have successfully used both, but I cannot really choose. On one side, an Ingress allows us to have a single public IP address, but on the other end it factorizes all the routes inside one Ingress so deployment means impacting all the routes.

Would you recommend:

a common Ingress for all applications
an Ingress for each application;
a LoadBalancer for each service ?

just create an ingress resource along side each application, that way you can modify individual ingress resources without impacting other applications. Load balancers do not allow you to do anything meaningful with your traffic.

With ingress you can use cert-manager, external-dns, ssl offloading, etc.


Azure and DNS services I have a Virtual network (MyVPN) resource in Azure, being a class C network, MyVPN has Azure provided DNS service as DNS servers, so all virtual machines with a NIC connected to this network are able to resolve their hostname successfully, for example virtual machine myvm01 can ssh to myvm02 using only the hostname instead of IP.

However, they are unable to make a reverse DNS lookup, which makes sense since by default Azure does not provide a reverse DNS lookup zone. By following Microsoft's documentation I managed to create it, sadly I don't really know if I should delegate to Azure provided DNS service and if so, how to do it (I tried following this)

What should I do now to allow myvm01 be able to resolve the IP of myvm02 to its hostname?

There is another manual for enabling reverse lookup DNS in azure.

Will my reverse DNS records resolve from the internet?
Yes. Once you set the reverse DNS property for your Azure service,
Azure manages all the DNS delegations and DNS zones required to ensure
that reverse DNS record resolves for all Internet users.
Are default reverse DNS records created for my Azure services?
No. Reverse DNS is an opt-in feature. No default reverse DNS records
are created if you choose not to configure them.

It seems that reverse dns has to be enabled if it is used internally.


jenkins parameter behaves differently from internally defined variable I set up my jenkins job to take in the parameter payload and set its value to test\ntest.

My jenkins script is the following:

def hardset = "test\ntest"

echo "param:"
def cleanParam = jenkinsParam.replaceAll("[\\n\\t ]","Yes");
echo cleanParam

echo "----------"

echo "hardset:"
def cleanHardset = hardset.replaceAll("[\\n\\t ]","Yes");
echo cleanHardset

The output is (cleaned up to remove jenkins pipeline echos):

test\ntest

param:
test\ntest

----------

hardset:
testYestest

Why? What's different between reading in a jenkins parameter and setting one in the script body?

In practice, I'm trying to replace troublesome \n coming in from a lambda payload, but this is throwing a wrench in my plans.

doing something like def hardsetParam = jenkinsParam to set it in the script body doesn't seem to work either, sadly.

In response to a comment - doing echo jenkinsParam gives me test\ntest. To my eyes, it's identical to hardset.

It seems that the jenkinsParam is holding the output after escaping the special characters.
You are assuming that jenkinsParam = "test\ntest" and however it looks like it stores it as jenkinsParam = "test\\ntest"

See how it behaves if I use echo it these two different cases:

(I have tested this in groovy shell)

1st Case:

jenkinsParam = "test\ntest"
print jenkinsParam
test
test

** 2nd Case:**

jenkinsParam = "test\\ntest"
print jenkinsParam
test\ntest

In order to replace the payload which is stored in the jenkinsParam (2nd case behaviour), you need to replace the complete \\n from the string. Check the example below:

output = jenkinsParam.replaceAll('\\\\n', 'yes')
print output
testyestest


Real-World Validating the Integrity of a Docker Container I'm interested in information and experiences surrounding Docker security. I'd like to understand ways that a user/client could validate the contents of an image to ensure that the packaged applications/files are in compliance with their organization's/industry standard practices.

I've been reading online about security as far as establishing trust to validate the integrity and the publisher of data received from a registry(https://docs.docker.com/engine/security/trust/content_trust/), as well as Security Assurance of Docker Containers(https://www.sans.org/reading-room/whitepapers/cloud/security-assurance-docker-containers-37432)>, and I see some useful information around tools and practices to validate the security of containers and images.

Some tools I'm interested in exploring more are OpenSCAP docker tool(https://www.open-scap.org/resources/documentation/security-compliance-of-rhel7-docker-containers/) as well as Docker Bench for Security(https://github.com/docker/docker-bench-security) and Dockscan(https://github.com/kost/dockscan).

My question is, does anybody have any experience with dealing with security/compliance when deploying containers and sharing images or considering Docker as a solution for deploying applications internally/to external clients? If so, how would you recommend an organization best approach this topic?

2017 expectations

Docker is about getting some freedom from host management at the expense of reinventing all the things package management does. Docker images are just another package format. This is all very new, so I think this question is part of the growing pains of the Docker ecosystem.

basic requirements/criteria

For secure trusted containers, only use them if they have been
 1. tested by current standards
 2. cryptographically signed
 3. checked against a vulnerability database for insecure dependencies

Maybe there is a way, but I don't see a way to un-trust a signed, trusted Docker image after a researcher finds a new vulnerability. This is just one gap in the toolset cited in the question.

Are you planning to use a cloud Docker service? Are the containers' memory encrypted or at least obfuscated at the kernel (containers are just processes)?

Are your Docker hosts secure?

http://ieeexplore.ieee.org/document/7562068/
http://ieeexplore.ieee.org/document/7397244/

economy of security

In the end, you can never escape the economics of checking everything, so there needs to be some respect paid to the speed Docker provides by not worrying about all of this stuff. When you know you are handling sensitive information and you know you need to lock down the execution, it will require measures that are not available yet as commodity solutions (2017). Anything is better than nothing, so get some tools deployed and get your foot in the door.

Hubblestack?

Keep your eye on HubbleStack. It provides a Docker image which can be dropped in to run audits (and remediations) against a live database of vulnerabilities. I'd be surprised if they left Docker trust out of scope forever. Also, if you own the Docker hosts, you can use root to scan all the containers. This kind of thing belongs in the DevOps automated test and validation cycle.

get some kind of checkpoint in place to get security in the pipeline
keep a backlog of unmanaged security concerns (gaps in the tools? CVEs?)
keep the backlog prioritized by what you can get done now, at current funding
get involved in devops development and release planning
deliver software confident that you did the best with the resources you had


How can I convince someone that 100% reliability is not the right target for anything? It's a fundamental principle of DevOps and SRE that failure is normal and setting a goal of perfect reliability is misguided. But sometimes IT execs and business leaders push back on the idea. They believe the business needs 100% reliability. What are some good way to persuade them to change their thinking?

I was wondering whether anyone in this forum had new arguments they have heard or tried in person to convince executives of the need to set service level objectives at a level below 100%, hence the original question.

Here are some approaches I've seen or taken in the past to tackling this problem.  The Site Reliability Workbook has a few suggestions here, which include:

100% is not a reasonable target since every component of any system has a
non-zero probability of failure.
External components (such as an ISP) sitting between the customer and the target system aren't 100% reliable.
100% reliability implies you can never change the system, since all change introduces risk.
100% reliability implies you spend all your time reacting to reliability problems and have no time for anything else.

In addition to the above, I've also gathered the following talking points from various in-person discussions that might be useful to persuade organizational leaders:

If an IT exec sets a target of 100% reliability, they are basically encouraging their engineers to lie to them and hide problems, which means they will discover those problems in the most expensive possible way.
Nature has never evolved a 100% reliable system. Gene replication is unreliable. The human heart is not 100% reliable.  Life continues and thrives in the face of imperfect reliability, and so can technology systems and their users.
No group of humans has ever engineered a system that's 100% reliable in the long run.

We also have to acknowledge that for certain dimensions of reliability, such as data loss prevention, some regulated industries require levels of reliability so high and over long periods of time that they are virtually indistinguishable from 100%, like 99.999999%. For other dimensions like availability and latency, three or four nines of reliability are often more than adequate, and some of the arguments above may help persuade IT leadership that this is true.


Problem in integrating Travis-CI with core PHP projects I am trying to integrate my project coded in core PHP, with Travis-CI but it always fails.
Travis reports a failure even when I have only one file in my project.
PHP file Code :
<?php

phpinfo();

?>

.travis.yml File Code
language: php
php:
  - '5.4'
  - '5.5'
  - '5.6'
  - '7.0'
  - '7.1'
  - hhvm
  - nightly

You need to write some unit tests and set the script variable in your .travis.yml to actually run a file.

By default, Travis CI runs the command phpunit without any arguments. When this happens, phpunit doesn't know what you're asking it to do, and shows a help message, then exits with error code 2 (i.e. non-zero, which indicates an error occurred).

The Beginner's Guide shows you a possible solution of setting the script variable to run a Test.php file with your tests in (insert this into your .travis.yml):

script: phpunit Test.php

There is an example of a valid Test.php in the example repo that Travis provides:

<?php
    class Test extends PHPUnit_Framework_TestCase
    {
        public function testOnePlusOne() {
            $this->assertEquals(1+1,1);
        }
    }
?>


In the HashiCorp stack, where's the appropriate place to add users? I'm in the process of building some custom Linux images using HashiCorp's Packer, which will later be deployed to Azure using HashiCorp's Terraform. The VMs created from these images will need to have a set of users created; in this particular case I'm concerning myself with user accounts for employees that may need to ssh into these VMs.

For this kind of configuration, does it make more sense to add these user accounts to the base image in the Packer script, or to add them when the VM is created via Terraform? It seems to me that handling it in Packer makes more sense, but are there reasons not to do it there?

I am less familiar with what Azure offers, but other cloud providers offer a way to manage users external to the VMs the provide (normally IAM permissions).

I would not bake user into the system but look for an external way to manage them. This allows you to more easily remove users that leave your company or add new users that join.

If Azure doesn't provide a way I would look at setting up Active Directory or LDAP to manage user permissions to hosts. But I would be shocked if you needed to do this.

Note: Don't confuse infrastructure and data. Users are data, not infrastructure. Data (ie adding a user) should be able to change without changing the infrastructure. If you need to remove a user you do not want to rebuild your whole infrastructure.


Rancher, Load Balancing and own domain I have a question about rancher. I'm running my rancher on aws. I have created a new cluster with two nodes (workers) and I have deployed my docker images. Now I wanted to add Load Balancer, but I don't want to use xip.io domain but my own. I added my domain to route 53 and I wanted to make redirection from api.mydomain.com to Load Balancer. So, I added in route 53 new record and I used IP of my ec2 machine (I checked where the load balancer has been deployed in rancher)

The question is: What if I will want to change my nodes? My domain will stop working because I'm using the ip of ec2 machine. How should I point dns to my LB correctly (because I'm thinking that using the ip of ec2 machine is a not correct solution).

Your first step is to setup and configure the Route53 service from the rancher catalog.

After that, the load-balancer will automatically update its DNS in Route53.


Why is Prometheus not a good choice for data with high cardinality? I have a background in relational databases and new to Prometheus. I wonder why Prometheus is not a good choice for high cardinality data? Why do I need to use low cardinality data? It's exact opposite from SQL DBs.

What are the technical reason for that?

As far as I know, Prometheus doesn't mind high-cardinality data. What Prometheus doesn't like is high-cardinality labels.

Let's start with Prometheus official documentation, it gives a good high-level explanation why:

  CAUTION: Remember that every unique combination of key-value label
  pairs represents a new time series, which can dramatically increase
  the amount of data stored. Do not use labels to store dimensions with
  high cardinality (many different label values), such as user IDs,
  email addresses, or other unbounded sets of values.

https://prometheus.io/docs/practices/naming/

The important part is really that unique combinations of key-value creates new time series in Prometheus.

For instance, if you store a gauge-type metric registration_complete that records the time a user took to complete his/her registration form, Prometheus won't have any issue with that. You'll have one time series that have hundred of thousand of values: one value for each user that registered (how long it took). You'll be able to graph that metric over time, get the p95, etc.

If you want to add some cardinality to it, you could add a label like region: US-east, Asia-Pacific, etc. You'll be able to graph all regions and compare them or maybe group them all. The number of regions is probably low (<10) and it is certainly bound. If on AWS, there's a fixed number of regions and they don't change much over time. Sure, AWS might add, remove or rename a region, but that's not changing by the minute and there's not thousands of them.

So to come back to what Prometheus advise against: you should not create high-cardinality labels. You should not add a user_id label to your registration_complete metric. If you do so, you'll have hundred of thousands of different time series, one for each user!, and they will all have only one data point. That's really the worst case scenario.

In this case, in order to graph the registration_complete metric over all labels, Prometheus will have to query all the individual time series (the X thousands of them) and aggregate all of them.

You said you come from a SQL background, so I'll attempt an analogy. Unique combination of key-value label pairs creating new time series in Prometheus amounts to individual tables in SQL. Having a label user_id equals to having one table per user_id.

Note: not all TSDB work the same way and I can't speak for them all.


What are the implications for Docker Community Edition of the Docker acquisition by Mirantis? November 13, 2019, Mirantis Acquires Docker Enterprise Platform Business. The same day, Docker Inc. claims they will now focus on docker Desktop and Docker Hub.

Given those changes, will Docker CE continue to share the same codebase as Docker Enterprise, or should we consider it now as a fork like MySQL vs MariaDB?

I'd ask the question the other way around, rather than will Docker CE continue to share the Docker Enterprise codebase, it's whether Docker Enterprise will continue to share the upstream Docker CE / Moby codebase. They've always maintained an internal closed source fork for the Enterprise code, which has additional features. However, the development process of CE has always been open source, and features added to CE were mirrored from CE to the Enterprise codebase. The acquisition of Enterprise didn't change that it was already a fork of CE. And the acquisition did not include the CE code or any of the other open source projects, those continue to be managed by Docker, which they will maintain as a core component to their business (CE is embedded in the various Desktop products, and Hub has little value without it).

The biggest downside I see to the acquisition is the loss of staff from Docker to Mirantis. I'm sure many people had a dual role working on Enterprise and CE, and a significant chunk of the staff went to Mirantis. That said, the CE repos are open source, Mirantis staff can submit PR's for changes they'd like usptream, and they can pull down new releases of CE into the Enterprise product. The end result for users may end up looking identical to what we have today, just two close companies coordinating, rather than two sides of the same company. From an open source perspective, having more outside contributions is actually a good thing, since it forces governance and planning into the open, that in the past may have happened with internal meetings.

Disclaimer: I have been exposed to private discussions from the Captains program on this topic (not much since they are still working this out) and have done my best to keep my answer to only the public knowledge.


How are declarative distros different than Kickstart? NixOS and GuixSD both bill themselves as "declarative" and DevOps-friendly. How and in what ways are these declarative distros different than Kickstart and/or a Linux distro managed by a configuration management tool?

What advantages and/or disadvantages are there to this style of declarative linux configuration as opposed to these currently existing technologies?

AFAIK Kickstart is only usable for the initial installation of the OS, but not for subsequent package maintenance upgrades/downgrades.

Unless I miss something (quite possible, I don't have much experience w/ them) the Linux distros managed by configuration management tools require a minimum level of health from the OS (and some package requirements, at least in some cases) to be able to operate those tools, say to perform a rollback. At least NixOS appears to not have such requirement (or rather it is already fullfilled at the OS level). From About NixOS:

  Rollbacks

  Because the files of a new configuration don’t overwrite old ones, you
  can (atomically) roll back to a previous configuration. For instance,
  if after a nixos-rebuild switch you discover that you don’t like the
  new configuration, you can just go back:

$ nixos-rebuild switch --rollback

  In fact, all old system configurations automatically show up in the
  Grub boot menu. So if the new configuration crashes or doesn’t
  boot properly, you can just roll back by selecting an older
  configuration in the Grub boot menu. Rollbacks are very fast: it
  doesn’t involve lots of files having to be restored from copies.


How can I expose port 3389 using a secure approach Within our legacy infrastructure we need to keep port 3389 exposed so that we can remote desktop into this server (note that we can't go back and utilize IaC to prevent this).

We currently restrict the IP range that can access this port. Is this enough for security? This article mentions never globally exposing this port: https://www.grc.com/port_3389.htm

You can add, if it's not already the case, a frontal, rev proxy, to prevent direct access to the TS. This could also add an authentication layer...
I am not a lot in Microsoft stuff so this is really a basic advice.
If TS allow the use of pubkey instead of passwd this would be a good addition too.


GitHub Payload in Jenkins I'm using the GitHub plugin with Jenkins, and I'm trying to figure out how I can access the webhook payload. I'm aware that the Generic Webhook Trigger has this ability, as well as the GitHub Pull Request plugin, but this plugin seems more suitable to an automated workflow.

On that note, what I'm looking at is importing the class and parsing the payload, similar to what is done in the troubleshooting section here.

This sample works, but only prints the class:

import org.kohsuke.stapler.StaplerRequest
import org.jenkinsci.plugins.github.webhook.GHEventPayload
import org.jenkinsci.plugins.github.webhook.GHEventPayload.PayloadHandler

payload = new PayloadHandler()
println(payload)

This sample wants more info, but I'm not handy enough with Java to know what it's asking for:

import org.kohsuke.stapler.StaplerRequest
import org.jenkinsci.plugins.github.webhook.GHEventPayload
import org.jenkinsci.plugins.github.webhook.GHEventPayload.PayloadHandler

payload = new PayloadHandler()
println(payload.parse())

According to the source code, that's the method I need, but it produces the following error:

hudson.remoting.ProxyException: groovy.lang.MissingMethodException: No signature of method: org.jenkinsci.plugins.github.webhook.GHEventPayload$PayloadHandler.parse() is applicable for argument types: (java.lang.Class) values: [interface org.jenkinsci.plugins.github.webhook.GHEventPayload]
Possible solutions: parse(org.kohsuke.stapler.StaplerRequest, java.lang.annotation.Annotation, java.lang.Class, java.lang.String), parse(org.kohsuke.stapler.StaplerRequest, org.jenkinsci.plugins.github.webhook.GHEventPayload, java.lang.Class, java.lang.String), use([Ljava.lang.Object;), use(java.lang.Class, groovy.lang.Closure), wait(), grep()

Has anyone tried something like this before?

GitHub plugin doesn't pass payload data to any variables available from inside the pipeline.

Both Jenkins tracker and GitHub plugin project have a number of open and cancelled feature requests requesting this, for instance, https://issues.jenkins-ci.org/browse/JENKINS-24291.


Using Code Climate for Test Coverage alongside Jenkins My company is currently trying to implement Code Climate for test coverage. However, I do not have much idea about it and new to using code climate.

I tried to integrate code climate with Jenkins so that once the tests are successful, I can simply move on with other Jenkins Job.

I consulted the documentation itself, and as per documentation there are things like but there is not much explanation about it.

./cc-test-reporter after-build -t simplecov --exit-code $? || echo  “Skipping Code Climate coverage upload”

I executed this, but returned there is no such file simplecov. I later found that the -t flag is used to provide the file which will be used for coverage. Below is my Jenkins job.

#!/bin/bash
echo "---------------------------------------"
echo "| Downloading Code Climate "
curl -L https://codeclimate.com/downloads/test-reporter/test-reporter-latest-linux-amd64 >> ./cc-test-reporter
chmod +x ./cc-test-reporter
echo "| Download Complete "
echo "---------------------------------------"

CODECLIMATE_REPO_TOKEN=${CODECLIMATE_REPO_TOKEN}
echo "Running tests"
./cc-test-reporter before-build.

Once I execute this, I get the following output.

---------------------------------------
| Downloading Code Climate
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100   178  100   178    0     0    933      0 --:--:-- --:--:-- --:--:--   936

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100 12.1M  100 12.1M    0     0  10.0M      0  0:00:01  0:00:01 --:--:-- 15.8M
| Download Complete
---------------------------------------
Running tests
Finished: SUCCESS

Why am I not getting any code coverage or things like that in the console. How do I properly integrate it in Jenkins.

UPDATE
I have successfully created the job without issue.
Now the problem I am facing is

$ ./cc-test-reporter format-coverage -t lcov coverage/lcov.info -d -o codeclimate.json
DEBU[0000] coverage path coverage/lcov.info
DEBU[0000] using formatter lcov
DEBU[0000] checking search path coverage/lcov.info for lcov formatter
DEBU[0000] couldn't load branch from ENV, trying git...
DEBU[0000] couldn't load commit sha from ENV, trying git...
DEBU[0000] couldn't load committed at from ENV, trying git...
Error: could not find coverage info for source files

user@host:app$ pwd
app
user@host:app$ ll
coverage
user@host:app$ ll coverage
lcov-report
lcvo.info

The issue has been resolved.
This is for nodejs

Need to install necessary dependencies either using yarn or npm.
Once done, run

nyc --reporter=lcov yarn unit unit
$ ./cc-test-reporter after-build -t lcov --exit-code $?


Advice needed - automate everything or only new/current projects I watched Klaus Ita Keynote on FreeCodeCamp YouTube channel about devops and got inspired.

In previous job we had very good deployment system. We begun with Jenkins and pulling source from SVN, and then migrated over GitHub and used CircleCI. We also managed to use only single source code for many clients using multi tenancy approach, so everything was simplified to the bare minimum.

Now I started to work in another company, digital agency, and 80% of current projects are Wordpress.

I don’t think that we would be able to merge all clients under one source code (or we could try with multi tenancy, by managing seperate databases, seperating out plugins and themes), but at the moment all the projects are just copied to server via FTP. At least, GIT is used to keep versions.

I have in my mind, that I would really like to improve this deployment process by automating it as much as possible, at least with Jenkins and git-ftp, but th question is how to start doing this? I mean, move one-by-one project to Jenkins or only those that requires future upgrades? Or only new projects, that might come in? Is it worth that effort? What should we do with rest of 20% of custom projects?

I hope that you might share your experience, your failures and know-how’s to help one more soul to sleep better at night :)

You can spend a lot of time, to automate something, which is done monthly and do routine staff manually on a daily basis - not the best way.

If you are time/resource limited, then find the bottleneck in the software delivery process and improve it, after that bottleneck is not a bottleneck anymore, find a new one and so on. So if you are spending 80% of your development time/resources on the new project, automate it first, and leave the old ones for the next step or vice a versa, depending where the main bottleneck is.


Should I struggle for minimalistic context in GitLab CI jobs? I'm designing a build & deploy pipeline for our project. We have two jobs:

build -- builds a docker image with our app and pushes it to a container registry.
deploy -- communicates with an orchestrator, which deploys the new version.

There's one detail that I'm not sure about. Which choice is better:

Get the full repository in the deploy job, while I need only one static config (deployment spec.).
Use artifacts to pass the required config file from build job to deploy job, and set GIT_STRATEGY to none.

These two options differ in the deploy job environment:

The deploy has access to the same context as build (full repository):
Pros:

The default behavior (requires no extra code and actions for this option to work).

Cons:

The environment in deploy job is dirty.
The true context of the deploy job is implicit.
Someone could misuse it in the future.

The build job publishes an artifact with the required config for the deploy job:
Pros:

The deploy job explicitly has no access to anything unnecessary.

Cons:

Need to configure the deploy job not to fetch the repository.
Need to configure the build job to publish the artifact.
Need to configure the deploy job to consume the artifact.
The artifact always contains an unchanged file from the repository (I'm afraid that I misuse artifacts in that case).

Please, help me choose.

Your use case is very simple, so relying on GIT_STRATEGY=fetch is probably sufficient and as you said the default behavior.
You should be cautious using artifacts unnecessarily as they are uploaded to the gitlab server.
The answer to your question generally depends on a few things, like the size of your repository, network connection, runner executor type, and so on. One option is to configure a cache (in this case at the pipeline level and for the deploy just set the GIT_STRATEGY=none:
default:
  cache:
    key: "$CI_COMMIT_REF_SLUG"
    paths:
      - deployment.spec.config

build:
  stage: build
  script:
    - build something

deploy:
  stage: deploy
  variables:
    GIT_STRATEGY: none
  script:
    - push-deployment ./deployment.spec.config

An added benefit here, depending on your build process, is the possibility of caching your dependencies when running multiple pipelines against the same branch. Note that you can only cache files relative to the project build directory.


Use Sqlite database on ECS I have an application which uses SQLite database and want to migrate it to AWS ECS service. It's dockerized and have access to DB file via mounted volume.

Is there a way to make this work with ECS? (I cannot use EBS as my launch type is Fargate). Should I go with something like EFS or just get rid of SQLite altogether?

Seems like EFS is now supported on Fargate

Blog post
Reference


Mitigating Docker Compliance Issues - Can I put everything in the `daemon.json`? I am going through compliance tasks to harden our Docker setup and I am seeing many things like

Favor using docker run with docker run --pids-limit 100 to prevent forkbombs
Favor using docker run with docker run --security-opt=no-new-privileges to be safe

etc...

I want to know if I can apply these settings across the board by, say, putting

...
"pids-limit": "100",
"security-opt": "no-new-privileges",
...

in /etc/docker/daemon.json

I prefer to just have a hardened config on all my machines I am hardening.

Tips and tricks?

Thanks.

Check out the dockerd options.  This will apply to all containers managed by the Docker Daemon.

Docker also has quite a bit of documentation on how to enforce different standards and compliance here.


What is the best way to do REST with puppet? I have tried making REST calls with puppet and it was... well... awful. I didn't really want to use pure ruby to interact with my load balancer and my concern was that for REST calls, every time the puppet run occurs I would be idempotently slamming my web server with a ton of requests - first checking if a particular item is out of spec and then possibly resetting it. Using curl via an exec and an unless statement was arduous to say the least.

Furthermore, the idempotent philosophy of puppet doesn't always play well with REST APIs that are not idempotent.

Are there any simple ways, ready-made methods or plugins for dealing with REST APIs via puppet?

Note: one feature request to watch: https://tickets.puppetlabs.com/browse/PUP-6658?filter=-2

While it may not be the answer you are looking for, puppet's DSL lacks verbs on purpose.  Puppet's idempotent philosophy is a huge reason it is so useful for some use cases but for this type of need you will want to use a tool which is more targeted to that role.

When you hit limitations due to the idempotent philosophy of Puppet it is a good indication that another tool is required for that need.

I would look for publicly maintained tools for the task maybe in Ansible or Terraform.

We leveraged the "puppet kick" functionality of puppet and then tried to extend it after it was deprecated and the pain of trying to ignore their design philosophy is an expensive one.


Access service in remote Kubernetes cluster using ingress I'm attempting to access a service in an existing kubernetes cluster deployed in a remote machine. I've configured the cluster to be accessible through kubectl from my local mac.

$ kubectl cluster-info
Kubernetes master is running at https://192.168.58.114:6443
KubeDNS is running at https://192.168.58.114:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

The ingress configuration for the service I want to connect is:

kind: Ingress
apiVersion: extensions/v1beta1
metadata:
  name: gw-ingress
  namespace: vick-system
  selfLink: /apis/extensions/v1beta1/namespaces/vick-system/ingresses/gw-ingress
  uid: 52b62da6-01c1-11e9-9f59-fa163eb296d8
  resourceVersion: '2695'
  generation: 1
  creationTimestamp: '2018-12-17T06:02:23Z'
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: >
      {"apiVersion":"extensions/v1beta1","kind":"Ingress","metadata":{"annotations":{"kubernetes.io/ingress.class":"nginx","nginx.ingress.kubernetes.io/affinity":"cookie","nginx.ingress.kubernetes.io/session-cookie-hash":"sha1","nginx.ingress.kubernetes.io/session-cookie-name":"route"},"name":"gw-ingress","namespace":"vick-system"},"spec":{"rules":[{"host":"wso2-apim-gateway","http":{"paths":[{"backend":{"serviceName":"gateway","servicePort":8280},"path":"/"}]}}],"tls":[{"hosts":["wso2-apim-gateway"]}]}}
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/affinity: cookie
    nginx.ingress.kubernetes.io/session-cookie-hash: sha1
    nginx.ingress.kubernetes.io/session-cookie-name: route
spec:
  tls:
    - hosts:
        - wso2-apim-gateway
  rules:
    - host: wso2-apim-gateway
      http:
        paths:
          - path: /
            backend:
              serviceName: gateway
              servicePort: 8280
status:
  loadBalancer:
    ingress:
      - ip: 172.17.17.100

My list of services are:

My /etc/hosts file looks like below:

##
# Host Database
#
# localhost is used to configure the loopback interface
# when the system is booting.  Do not change this entry.
##
127.0.0.1   localhost
255.255.255.255 broadcasthost
::1             localhost

172.17.17.100 wso2-apim-gateway wso2-apim wso2sp-dashboard

What is the URL I should use to access this service from my local browser? Should I do any more configurations?

https://wso2-apim-gateway/ is the URL that will point to the gateway service according to your configuration. This is defined by the host: wso2-apim-gateway part of your configuration. Note that your load balancer (172.17.17.100) is a private IP address, so you will need to be on the same network to access it.

I can't see any further configuration required, except perhaps TLS certifications - if you receive a 5xx response, you may want to check the logs of the ingress-nginx deployment/replicaset.


Is it possible to set role in FreeIPA to use for applications as Jenkins, Github? LDAP in FreeIPA, it can create groups, roles and set permissions. Applications such as Jenkins, Github also has roles feature.

2 questions:

Can roles feature be used in every application? (At least Jenkins and Github may use groups feature in LDAP)
If want to control several users can access Jenkins, several users can access Github, is it possible to control it in LDAP side? If deploy multiple LDAP server for different applications will realize it, but that's complex.

Based on your title, no.

It's impossible to quantify if every application has this capability. Many do.
Users/Groups in Jenkins OSS map back to LDAP but you control the permissions via Jenkins.

As you can see above Users/Groups have specific permissions that are specified/controlled in Jenkins. If this group exists in LDAP (xxx) the users in this group get the permission you setup in Jenkins. This makes it easier then managing individuals.


Terraform and Kubernetes' StorageClass with node affinity Im able to make a number of kubernetes constructs in terraform and even have done so for my ephemeral stateful data that requires low latency disk performance.  I can terraform up a kubernetes_persistent_volume but trying to do so with node_affinity has got me a bit perplexed as it requires a lists of lists (nodeSelectorTerms and matchExpressions), where the inside list has key/value pairs, and just not sure how to terraform all that.  Heres the working yaml version:

apiVersion: v1
kind: PersistentVolume
metadata: # PVs do not have namespaces
  name: nodeconfig
spec:
  capacity:
    storage: 25Gi
  volumeMode: Filesystem
  accessModes:
    - ReadWriteMany
  persistentVolumeReclaimPolicy: Retain
  storageClassName: nodeconfig
  local:
    path: /mnt/nodeconfig
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - myhost

Has anone dealt with this?

Well of course its always the same - as soon as you actually do ask for help you figure it out...

resource "kubernetes_persistent_volume" "nodeconfig" {
  metadata {
    name = "nodeconfig"
    labels = {
      a = "a"
    }
  }
  spec {
    capacity = {
      storage = "25Gi"
    }
    storage_class_name               = "nodeconfig"
    persistent_volume_reclaim_policy = "Retain"
    access_modes                     = ["ReadOnlyMany"]
    persistent_volume_source {
      local {
        path = "/mnt/nodeconfig"
      }
    }
    node_affinity {
      required {
        node_selector_term {
          match_expressions {
            key = "kubernetes.io/hostname"
            operator = "In"
            values = [var.config.RGs.Primary.clustervmhost]  # must mnanually put this one in
          }
        }
      }
    }
  }
}

So the docs were a bit off of the lists of lists, it was a map of maps of lists.


Docker swarm container monitoring I have been on a typical journey of:

deploying software on a machine
deploying software in a container
deploying software using docker swarm

The need for this journey is that swarm removes a large portion of micro-managing resources and machine allocation etc.

There is, however, a hurdle I am trying to overcome.

For point one and two, we have heavy monitoring of our software using a tool (Zabbix) which has given us great insights into any problems or metrics.

As we migrate to point three, how does one monitor an individual container?

To give a real-world illustration:

I have software that is "buggy", every now and then it dies. The health check would typically fail and a new container would start with no service interruption. My question is,

How am I aware that this has occurred in order to investigate a potential problem ?
One potential solution is to monitor the docker swarm event stream; is this the only solution?

The easiest way is not to handle this information)) You will get tired of such useless information like containers restarts.

Anyway, there is couple of tools to monitor docker with prometheus:

Google cadvisor
Docker experimental prometheus metrics

But it requires installation of prometheus, alert-manager, find metrics for triggering and set up alert rules.

Or maybe try to find zabbix tool for monitoring docker containers.


jenkins build overview page not showing stages in webui I'm using the image: jenkinsci/blueocean from Dockerhub
The build overview page isn't showing the stages of a build.  i.e. the square boxes that represent the stages are not appearing.
Am I missing a plugin?


Ansible authorized copy from remote source to remote destination How to use ansible authorized_key to authorize a ServerA (not the controller machine) to access Server B.

Scenario: Need a playbook to execute from a ansible controller that should append id_rsa.pub of a specific user from a remote ssh ServerA (no the controller machine ) to ServerB.

In our case the ServerA count is 20 while ServerB count is 200.
What i have tried so far and failed.

---
# tasks file for passwordless
- name: Play to setup Passowrdless
  hosts: destnode
  remote_user: "{{ ssh_user }}"
  vars_prompt:
    - name: "ssh_user"
      prompt: "Please specify ssh user name"
      private: no

  tasks:
  - name: Copy pub key from SourceNode to Destnode
    authorized_key:
      user: "{{ ssh_user }}"
      state: present
      key: "{{ lookup('file', lookup('env','HOME') + '/.ssh/id_rsa.pub') }}"
    become: yes
    become_user: "{{ ssh_user }}"
    delegate_to: "{{ hostvars.sourcenode }}"
    when: ssh_user is defined
    register: copyout

  - name: Print copyout value
    debug:
      msg: "{{ copyout }}"

To append id_rsa.pub of a specific user from a remote ssh ServerA (no the controller machine ) to ServerB.

1) Fetch the public keys from ServerA

- host: ServerA
  vars:
    public_keys_dir: <PUB_KEYS_DIR>
    specific_user:
      - user1
      - user2
      - userN
  tasks:
    - name: Fetch pub keys
      fetch:
        src: "/home/{{ item }}/.ssh/id_rsa.pub"
        dest: "{{ public_keys_dir }}/{{ item }}-ServerA.id_rsa.pub"
        flat: yes
      loop: "{{ specific_user }}"

2) Configure authorized keys at ServerB

- host: ServerB
  vars:
    public_keys_dir: <PUB_KEYS_DIR>
    my_remote_user: admin
  tasks:
    - name: Set up authorized_keys
      authorized_key:
        user: "{{ my_remote_user }}"
        key: "{{ lookup('file', '{{ item }}')}}"
      with_fileglob:
        - "{{ public_keys_dir }}/*-ServerA.id_rsa.pub"

(not tested)


Can I run command from a program thats based on the root host from a containerized application? I'm in a situation where I have a Django application, Nginx webserver and Postgres database that I want to dockerize, and that's no issue I can do that. But the Django application has code that calls a subprocess to run a command from a installed program on the root host, and obviously if I run that command it's not recognized since it's not installed on the docker container.

The program can not be dockerized as far as I know, because it uses docker itself. Is there any way I can make the Django Application be able to run commands from that specific program on the root host, even though the Django application is containerized?

Can this be done through some kind of network setup of the docker container?

I find it a little hard to explain, so please do ask questions if something isn't clear enough.

This StackOverflow question has answers that could be useful for you, in particular, mounting parts of the hosts filesystem containing the host application.

But really, it doesn't seem to make much sense to run an application in a container and then try to directly access the host system. It seems to defeat many of the reasons to containerize an application in the first place.

It'd probably much better to put the required application in the container, along with Django. I see no reason why your container wouldn't be able to run docker itself (even though for some CI/CD use cases sibling containers are preferred over docker-in-docker approach).


How To Determine Min/Max Resources For Kubernetes Pods? Say I have 2 nodes for a kubernetes cluster. Both has

2 CPU → 4 Total
2G RAM → 4 Total

I have 4 pods to shcedule-- no replicas yet, 1 major pod that'll use 80% of resources, other use 5% each.
What resource limits should I specify for these pods?

CURENT USAGE STATS

# kubectl top pod --all-namespaces
NAMESPACE       NAME                        CPU(cores)   MEMORY(bytes)
default         pod1-64c455bcf8-mzxqc           1m          122Mi
default         pod2-6f7b44478b-pn4tt           1m          33Mi
default         pod3-867c956697-mf6zw           0m          48Mi
default         pod4-844bd559dd-z96fh           0m          35Mi

The DevOps toolkit 2.3 book by Victor Farcic goes into some detail on this.

What You Should Understand

You as a developer specify whatever requests/limits you like initially.
These values have no baring on reality.
In order to get "real" values, you must monitor your application's real resource usage.
This is best done through a monitoring tool like prometheus, instana, datadog etc.
E.g. in instana, I can see that I have 512MB set up for a pod's requests, 2G for its limits, but the pod only has one container currently taking 57MB.
But I can't just use 60MB as a value now... I need to load test my application and see how it behaves under real/stressed circumstances.
Then you review these values and you can adjust the requests/limits accordingly.

Determine Usage Without a Monitoring Tool

You can use some kubectl commands to get these values as a one off (it's annoying at scale).  Here is an example of a different app I have running.

Notice that requests/limits are set to 1/4-2 CPUs and 4G-8G memory.  But the container is really using almost no CPU and 1.198G memory.  If I load tested this and those values did not move up, I would need to change them.  But I have load tested it and know that they would go up towards these realistic requests/limits, so it's okay.

$ kubectl describe pod -n gazelle gazelle-ws-0-0-1-6d67cffcf-lvlhk

...
    Limits:
      cpu:     2
      memory:  8Gi
    Requests:
      cpu:      250m
      memory:   4Gi

$ kubectl top pod gazelle-ws-0-0-1-6d67cffcf-lvlhk -n gazelle --containers

POD                                NAME         CPU(cores)   MEMORY(bytes)
gazelle-ws-0-0-1-6d67cffcf-lvlhk   gazelle-ws   5m           1198Mi


Provision and configure Azure VM in private subnet (file, remote-exec, cloud-init, etc.) (Originally posted to discuss.hashicorp.com)
I’m setting up a basic multi-tier infrastructure composed of two Azure VMs:

Web server (VM1) with public IP and private IP (subnet1)
Data server (VM2) in private IP only (subnet2)

Both VM-s are based on the vanilla Ubuntu Linux images available on Azure.
Obviously, it’s very easy to provision (apt-get, Bash scripting, etc.) and configure the Web server using the file or remote-exec provisioners. I have also experimented with the clout-init for that purpose.
Questions:

What is the recommended method to provision and the data server?
Is cloud-init the only option?

AFAIU, the Terraform provisioners running via SSH are not an option because the data server (VM2) is not available at public IP.
What I need to do on the VM2 is:

install some packages
upload/edit some configuration files
download some data files
start some services

I’m running Terraform on Windows host, so I did not try the Ansible yet, but AFAIU it operates via SSH, similarly to the Terraform file or remote-exec provisioners.

Hashicorp recommends using provisioners only as a last resort. They recommend using the Azure VM custom_data method, or cloud-init where available.
Ansible would likely be overkill unless you will need ongoing in-place configuration management. But there are ways to make VM2 securely accessible to Ansible or provsioners if necessary.


Is it true to say that both a "shared server environment " and a "VPS environment" can be either IaaS or PaaS? Say I rent a DigitalOcean 20$ "VPS environment" (IaaS) or a SiteGround 10$ "shared-hosting environment" (PaaS).

As far as I know, in both cases I share hardware with other customers (both constellations aren't "dedicated" - either dedicated "on-perm" or dedicated in any other hosting model).

From what I just described, I get the impression that the notion of a "shared hosting environment versus a VPS environment" is misleading because in both cases there is direct sharing of one system (one hardware system - "shared" or "VPS").

Hence I ask, is it true to say that both a "shared server environment " and a "VPS environment" can be either IaaS or PaaS?

Usually, Only a VPS hosting can be either.

While an IaaS is always VPS, what's commonly called "shared hosting plan" is PaaS in which a client shares the same hardware and webserver software with different customers (due to sharing a webserver, it's not a VPS --- Virtual Private Server).

In a PaaS (of the subtype Pass;VPS) only the hardware is shared (so it's indeed a "shared server", but only in that sense).


Error Vagrant Provisioning with Virtual Box on Fedora I'm Provisioning Vagrant Box with Virtual-box on Fedora 26.

$ vagrant -v
Vagrant 1.9.1
$ virtualbox -v
virtualbox 5.1

Below is the error;

muhammadtaqi@localhost etc]$ vagrant up
Bringing machine 'default' up with 'virtualbox' provider...
==> default: Importing base box 'ubuntu/trusty64'...
There was an error while executing `VBoxManage`, a CLI used by Vagrant
for controlling VirtualBox. The command and stderr is shown below.

Command: ["import", "/home/muhammadtaqi/.vagrant.d/boxes/ubuntu-VAGRANTSLASH-trusty64/20170918.0.1/virtualbox/box.ovf", "--vsys", "0", "--vmname", "ubuntu-cloudimg-trusty-vagrant-amd64_1506105327930_50139", "--vsys", "0", "--unit", "6", "--disk", "/home/muhammadtaqi/VirtualBox VMs/ubuntu-cloudimg-trusty-vagrant-amd64_1506105327930_50139/box-disk1.vmdk"]

Stderr: 0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100%
Interpreting /home/muhammadtaqi/.vagrant.d/boxes/ubuntu-VAGRANTSLASH-trusty64/20170918.0.1/virtualbox/box.ovf...
OK.
0%...
Progress state: VBOX_E_FILE_ERROR
VBoxManage: error: Appliance import failed
VBoxManage: error: Could not create the imported medium '/home/muhammadtaqi/VirtualBox VMs/ubuntu-cloudimg-trusty-vagrant-amd64_1506105327930_50139/box-disk1.vmdk'.
VBoxManage: error: VMDK: Compressed image is corrupted '/home/muhammadtaqi/.vagrant.d/boxes/ubuntu-VAGRANTSLASH-trusty64/20170918.0.1/virtualbox/box-disk1.vmdk' (VERR_ZIP_CORRUPTED)
VBoxManage: error: Details: code VBOX_E_FILE_ERROR (0x80bb0004), component ApplianceWrap, interface IAppliance
VBoxManage: error: Context: "RTEXITCODE handleImportAppliance(HandlerArg*)" at line 886 of file VBoxManageAppliance.cpp

Try to start with a clean sheet by removing the .vagrant.d folder or start to remove the corrupted zip. When the file or folder has been removed, one could run vagrant up again and check whether it works.


Can an in-memory filesystem be created from inside a docker container with no special permissions? I'd like to create an in-memory filesystem within a docker container running on a cluster where I can't assign the container any special permissions (e.g. the container can't have CAP_SYS_ADMIN capability necessary for using mount).

Is it possible to create an in-memory filesystem under docker? Will one of the user space filesystems provide this functionality?

Reference article(s):

User Space File Systems
How do I mount --bind inside a Docker container?
FUSE inside Docker
An in-memory filesystem (ie, RAMDISK) using FUSE
Riofs - fuse device not found

UPDATE: After finding and reading the references added above it's looking more and more like the answer is a firm no. I'll leave this open a bit in case anyone wants to elaborate or correct me.

Kubernetes can mount a ram disk for you. The answer is right there in the standard RTFM guide.

https://kubernetes.io/docs/concepts/storage/volumes/#emptydir

Create an emptyDir volume and set emptyDir.medium to "Memory" to build, what I interpret from the documentation to be, a dynamically sized memory file system which counts against the containers memory limit.

If you're not running under kubernetes, whatever system that deploys the docker container would need to create the ram disk and mount that into the container, or give the container the special permissions discussed in the various reference articles in the question.


Does local cache solve the problem with locked files when deploying to Azure App Service? During the deployments, occasionally I experienced issue with files being in use by App Service and thus not overwritten with newer versions.

There are multiple recommendations to overcome this problem:

Run your app from the ZIP package directly
Stop your app or enable offline mode for your app during deployment (using App_Offline.htm)
Deploy to a staging slot with auto swap enabled.

I am exploring the local cache option, to fix a different problem, and it is not compatible with running from ZIP feature, as discussed here and here

If I understand correctly, when using local cache, the ZIP with code is deployed to Storage Account, and then is loaded into memory on App Service restart.

Does this mean, the issue with locked files is eliminated by using local cache, since the code is not run from the same place where it's deployed to?

The answer is no, if you are using the recommended setup, which is:

App Service should have Staging and Production slots
Staging slot has Local Cache disabled
Production slot has Local Cache enabled
The following settings are configured as "Slot setting" aka "Sticky setting" - WEBSITE_LOCAL_CACHE_OPTION, WEBSITE_LOCAL_CACHE_SIZEINMB
The code is deployed to Staging slot
Staging slot is swapped with Production slot

For deployments to Staging, I created the following steps:

Shut down Staging slot by introducing App_Offline.htm (POST to SCM URI or can be part of artifact being deployed)
Wait for slot to stop before cleaning (probe the URL of Staging slot)
Cleanup wwwroot (post to SCM URI) - this is to clean unused files. Deployments are not incremental, so if it's important to have clean old files, you can introduce this step.
Deploy to Staging slot using az webapp deployment source config-zip
Remove App_Offline.htm (POST to SCM URI)

For swapping slots:

Check Staging slot is up and ready (probe the URL of Staging slot) - this one is useful when triggering deployment environments in Atlassian Bamboo when deployed to Staging. You need to wait for Staging to be up and loaded before swapping, otherwise issues may occur
Swap the slots using az webapp deployment slot swap

If you deploy directly to Production slot, and restart it right after (because of Local Cache enabled), then yes, there should be no file locks when using this method. But this creates the downtime during deployment, in contrast with deployments to Staging slot and swapping it with Production.


network ansible command error: "msg": "network os centos is not supported" How do I get the ansible config for rolly below?
remote:
[nsaunders@rolly ~]$
[nsaunders@rolly ~]$ lsb_release -a
LSB Version:    :core-4.1-amd64:core-4.1-noarch
Distributor ID: CentOS
Description:    CentOS Linux release 8.2.2004 (Core)
Release:    8.2.2004
Codename:   Core
[nsaunders@rolly ~]$
[nsaunders@rolly ~]$ ansible --version
ansible 2.9.13
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/home/nsaunders/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python3.6/site-packages/ansible
  executable location = /usr/bin/ansible
  python version = 3.6.8 (default, Apr 16 2020, 01:36:27) [GCC 8.3.1 20191121 (Red Hat 8.3.1-5)]
[nsaunders@rolly ~]$
[nsaunders@rolly ~]$ show config
bash: show: command not found...
[nsaunders@rolly ~]$
[nsaunders@rolly ~]$ ansible show config
usage: ansible [-h] [--version] [-v] [-b] [--become-method BECOME_METHOD]
               [--become-user BECOME_USER] [-K] [-i INVENTORY] [--list-hosts]
               [-l SUBSET] [-P POLL_INTERVAL] [-B SECONDS] [-o] [-t TREE] [-k]
               [--private-key PRIVATE_KEY_FILE] [-u REMOTE_USER]
               [-c CONNECTION] [-T TIMEOUT]
               [--ssh-common-args SSH_COMMON_ARGS]
               [--sftp-extra-args SFTP_EXTRA_ARGS]
               [--scp-extra-args SCP_EXTRA_ARGS]
               [--ssh-extra-args SSH_EXTRA_ARGS] [-C] [--syntax-check] [-D]
               [-e EXTRA_VARS] [--vault-id VAULT_IDS]
               [--ask-vault-pass | --vault-password-file VAULT_PASSWORD_FILES]
               [-f FORKS] [-M MODULE_PATH] [--playbook-dir BASEDIR]
               [-a MODULE_ARGS] [-m MODULE_NAME]
               pattern
ansible: error: unrecognized arguments: config
[nsaunders@rolly ~]$

local:
nicholas $
nicholas $ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 20.04 LTS
Release:    20.04
Codename:   focal
nicholas $
nicholas $ cat first_playbook.yml
---

- name: Network Getting Started First Playbook
  connection: network_cli
  gather_facts: false
  hosts: all
  tasks:

    - name: Get config for VyOS devices
      vyos_facts:
        gather_subset: all

    - name: Display the config
      debug:
        msg: "The hostname is {{ ansible_net_hostname }} and the OS is {{ ansible_net_version }}"
nicholas $
nicholas $ yamllint first_playbook.yml
first_playbook.yml
  15:81     error    line too long (97 > 80 characters)  (line-length)

nicholas $
nicholas $ ansible all -i rolly.foo, -c network_cli -u nsaunders -k -m centos_facts -e ansible_network_os=centos
SSH password:
rolly.foo | FAILED! => {
    "msg": "network os centos is not supported"
}
nicholas $

referencing:
https://www.ansible.com/blog/getting-started-writing-your-first-playbook

This error is due to invalid value of ansible_network_os:
The playbook you try setup is meant for network devices.
And since centos is not a network device OS the playbook will fail.
Here you can find supported network OS.
Ansible Documentation: ansible_network_os values
If you don't have any device running a supported network OS I recommend you to follow the regular user guide.
Ansible User Guide
My suggestion is to setup a easy playbook that will run on the environment you have.
Get the understanding of following concepts.
Control node
Managed nodes
Inventory
Modules
Tasks
Playbooks
Here is a short example.
This will only execute hostname do confirm it execute on expected host.
Let's say nicholas is your control node and rolly is the managed node.
On nicholas:
Create a inventory as before with your control node and managed node.
inventory.txt:
[local]
localhost ansible_connection=local

[managed_node]
rolly ansible_host=<rolly ip> ansible_user=<rolly user> ansible_ssh_pass=<rolly user password>

myplaybook.yml:
---
- name: This will get hostname localy on control node
  hosts: localhost
  tasks:
    - name: get hostname
      command: hostname
      register: result

    - name: print hostname
      debug:
          var: result

- name: This will get hostname remote on manged node
  hosts: rolly
  tasks:
    - name: get hostname
      command: hostname
      register: result

    - name: print hostname
      debug:
          var: result

And then execute on the control node:
ansible-playbook -i inventory.txt myplaybook.yml


How to clean up orphaned AWS EC2 snapshots? We end up with a fair amount of AWS EC2 snapshots where the AMI has been deleted, but the snapshot is left behind to rot. I'd like a non-manual way of identifying and deleting these orphans to save us money and space.

Ideally I'm thinking a bash script leveraging the CLI, but my AWS-fu is weak. I assume someone's done this before but I can't find a script that actually works.

In the best-case scenario this will also check volumes and clean those as well, but that may be better suited for a second question.

Largely inspired by the blog posts and gist already linked in the other answers, here is my take to the problem.

I did use some convoluted JMESpath functions to get a list of snapshots and not require tr.

Disclaimer: Use at your own risks, I did my best to avoid any problem and keep sane defaults, but I won't take any blame if it cause problem to you.

#!/bin/sh
# remove x if you don't want to see the commands
set -ex

# Some variable initialisation with sane defaults
DRUN='--dry-run'
DO_DELETE=${1:-'no'}
REGION=${2:-'eu-west-1'}
ACCOUNTID=${3:-'self'}

# Get two temporary files
SNAP_FILE=$(mktemp)
IMAGE_FILE=$(mktemp)

# Get the snapshot list and the volume list
aws --region "$REGION" ec2 describe-snapshots --owner-ids "$ACCOUNTID" --query 'Snapshots[*].[SnapshotId]' --output text > "$SNAP_FILE"
aws --region "$REGION" ec2 describe-images --owners "$ACCOUNTID" --filters Name=state,Values=available --query 'Images[*].BlockDeviceMappings[*].Ebs.[SnapshotId]' --output text > "$IMAGE_FILE"

# Check if the outputed command should be dry-run (default) or not
if [ "$DO_DELETE" = "IAMSURE" ]
then
 DRUN=''
fi

# count each snapshot id, decrease when a volume reference it, print delete command for those with no volumes
awk -v REGION="$REGION" -v DRUN="$DRUN" '
FNR==NR { snap[$1]++; next } # increment snapshots and get to next line in file immediately

{ snap[$1]-- } # we changed file, decrease the snap counter when a volume reference it

END {
 for (s in snap) { # loop over the snapshots
   if (snap[s] > 0) { # if we did not decrese under 1 that means there is no volume referencing this snapshot
    cmd="aws --region " REGION " " DRUN " ec2 delete-snapshot --snapshot-id " s
    print(cmd)
  }
 }
}
' "$SNAP_FILE" "$IMAGE_FILE"
# Clean up the temp files
rm "$SNAP_FILE" "$IMAGE_FILE"

I hope the script itself is commented enough.

Default usage (no-params) will list delete commands of orphaned snapshots for the current account and region eu-west-1, extract:

aws --region eu-west-1 --dry-run ec2 delete-snapshot --snapshot-id snap-81e5856a
aws --region eu-west-1 --dry-run ec2 delete-snapshot --snapshot-id snap-95c68c7e
aws --region eu-west-1 --dry-run ec2 delete-snapshot --snapshot-id snap-a3bf50bd

You can redirect this output to a file for review before sourcing it to execute all the commands.

If you want the script to execute the command instead of printing them, replace print(cmd) by system(cmd).

Usage is as follow with a script named snap_cleaner:

for dry-run commands in us-west-1 region

./snap_cleaner no us-west-1

for usable commands in eu-central-1

./snap_cleaner IAMSURE eu-central-1

A third parameter can be used to access another account (I do prefer to switch role to another account before).

Stripped down version of the script with awk script as a oneliner:

#!/bin/sh
set -ex

# Some variable initialisation with sane defaults
DRUN='--dry-run'
DO_DELETE=${1:-'no'}
REGION=${2:-'eu-west-1'}
ACCOUNTID=${3:-'self'}

# Get two temporary files
SNAP_FILE=$(mktemp)
IMAGE_FILE=$(mktemp)

# Get the snapshot list and the volume list
aws --region "$REGION" ec2 describe-snapshots --owner-ids "$ACCOUNTID" --query 'Snapshots[*].[SnapshotId]' --output text > "$SNAP_FILE"
aws --region "$REGION" ec2 describe-images --owners "$ACCOUNTID" --filters Name=state,Values=available --query 'Images[*].BlockDeviceMappings[*].Ebs.[SnapshotId]' --output text > "$IMAGE_FILE"

# Check if the outputed command should be dry-run (default) or not
if [ "$DO_DELETE" = "IAMSURE" ]
then
 DRUN=''
fi

# count each snapshot id, decrease when a volume reference it, print delete command for those with no volumes
awk -v REGION="$REGION" -v DRUN="$DRUN" 'FNR==NR { snap[$1]++; next } { snap[$1]-- } END { for (s in snap) { if (snap[s] > 0) { cmd="aws --region " REGION " " DRUN " ec2 delete-snapshot --snapshot-id " s; print(cmd) } } }' "$SNAP_FILE" "$IMAGE_FILE"
# Clean up the temp files
rm "$SNAP_FILE" "$IMAGE_FILE"


Why can I build this image from Docker for Mac but not on Docker CE for Ubuntu? Here's my Dockerfile:

FROM php:5-fpm

RUN apt-get update
RUN apt-get install -y php5-mysql
RUN apt-get install -y php5-xdebug
RUN apt-get install -y php5-mcrypt
RUN apt-get install -y php5-gd

When I run from Docker for Mac everything works great. But from my Linux machine I get:

E: Package 'php5-mysql' has no installation candidate
The command '/bin/sh -c apt-get install -y php5-mysql' returned a non-zero code: 100

Shouldn't these behave the same?

UPDATE 1

Thanks to @caylorme for recommending I check the base image on both hosts. They are indeed different image ids.

On the Linux host:

$ sudo docker images php
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
php                 5-fpm               9b179c4d385e        7 days ago          367MB

On the Mac host:

$ docker images php
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
php                 5-fpm               707631468b8b        11 months ago       362MB

But how is that possible? I had assumed (wrongly I guess) that Docker image tags are immutable so you could guarantee reproducibility and rebuild/run on any Docker host. What can I do get the image that's on my Mac host on the Linux host?

It is possible that the base image is different on each host. Can you confirm the images for php:5-fpm on each host are identical?

If the image from the Mac host is older, try updating it. You may need to update the package name in the dockerfile.

If you want to preserve the same behavior on your newer host, make sure you are pulling the image with the same hash or version-specific tag. It’s quite possible the newer version of the php image is on a newer version of Ubuntu. The package name may have changed between repos.


Jenkinsfile update using version control I've a Jenkinsfile in the master branch of the repository with following code
pipeline {
    agent any
    stages {
        stage('A') {
            steps {
                sh 'echo "Step A"'
            }
        }
    }
}

Now I want to add a new stage B for which I created a git branch update/jenkinsfile and in that branch the Jenkinsfile is now
pipeline {
    agent any
    stages {
        stage('A') {
            steps {
                sh 'echo "Step A"'
            }
        }
        stage('B') {
            steps {
                sh 'echo "Step B"'
            }
        }
    }
}

How do I make Jenkins to use Jenkinsfile from branch B not from master in order to testing the change?

It sounds like you want a when statement.
As I understand it, you want to execute stage B when you are on branch update/jenkinsfile. In order to do this, you need to add a condition to stage B:
pipeline {
    agent any
    stages {
        stage('A') {
            steps {
                sh 'echo "Step A"'
            }
        }
        stage('B') {
            when {
              branch 'update/jenkinsfile'
            }
            steps {
                sh 'echo "Step B"'
            }
        }
    }
}

Note that you also need your jenkins to discover this new branch - this will depend on how you have configured the job. The above assumes that you have a multibranch pipeline job which is configured to include that branch.


Release/file sharing service At my company we use FTP to share releases and e-mail to notify customers of the availability.

I'm looking for a service offering:

Easily upload of a file
Users need to sign in to download

and keep a record of which user have downloaded what

proper UI

Any ideas?

What you're after is an Binary repository manager

Quoting from Wikipedia with added links:

  Notable Universal package managers include:[6]

  Apache Archiva
  Jfrog's Artifactory
  Inedo's ProGet
  Sonatype Nexus

I know for sure Nexus and Artifactory match your requirements (even if proper UI is a bit subjective and you mileage may vary).


How do I set up certbot, with Nginx, both in a Docker container? Title says it all I guess. I didn't even know what Ubuntu/Nginx/Docker/etc was a week ago but I've managed to set up Docker/Docker Compose, PHP-FPM, PHPMyAdmin + Gitlab on a reverse proxy through Nginx with somewhat difficulty. My Github repo documenting my steps.

Basically managed to do everything I wanted to do set-up wise. Last step is setting up SSL certs for my site. It wasn't as straightforward as I was hoping, I haven't been able to find any clear guides on what to do regarding setting up Cerbot and having it on automatic renewal through a docker container. Lot of the shorter tutorials seem to use custom Certbot docker images which I'm not sure why they're different, or they have custom shell scripts that make dummy certs and delete them for some reason. The Certbot docker hub page is kind of devoid of any official instructions as well...

Any pointers or instructions would be extremely appreciated, thanks.

Answered my own question, documented my steps on my Github repo.

Recap:

Add Nginx config to desired domains:

location ~ /.well-known/acme-challenge {
        allow all;
        try_files $uri $uri/ /index.php;
    }

Use certbot staging to try out test certificates before running the real deal.

sudo docker run -it --rm -v /some/place/to/save/letsencrypt:/etc/letsencrypt -v /some/place/to/save/lib:/var/lib/letsencrypt -v /some/place/to/have/html:/data/letsencrypt certbot/certbot certonly --webroot --register-unsafely-without-email --agree-tos --webroot-path=/data/letsencrypt --staging -d example.com

Run the real thing once ready

sudo docker run -it --rm --v /some/place/to/save/letsencrypt:/etc/letsencrypt -v /some/place/to/save/lib:/var/lib/letsencrypt -v /some/place/to/have/html:/data/letsencrypt certbot/certbot certonly --webroot --email someemail@account.com

And then add in Nginx configs redirecting from http to https with ssl cert locations, updating nginx in docker-compose.yml to have access to certs. Go a tiny bit more in-depth about it in my repo


How to setup testkitchen with ansible-provisioning from localhost for different vagrant-vms? As pointed out here, testkitchen is in principable usable with ansible if the right plugins are installed.

Now I've done all the installing and am at the point, where I wonder, what I missunderstood:

As a driver I use vagrant and I thought it might be a good idea to use the vagrant-host as the machine which does the ansible-provisoning since the ssh-access is obviously cared for by vagrant and python is installed on my boxes per default. Now, when I look at my logs, I see:

  Setting up ansible (2.4.1.0-1ppa~xenial) ...

How can I prevent the installing of ansible? How can I tell testkitchen to use localhost as the machine which provisions the vagrant vms?

To be more clear, I am trying to find a configuration for testkitchen (as seen e.g. here under "Example .kitchen.yml file") that can accomplish this.

So far I got this configuration in my .kitchen.yml:

[...]
provisioner:
    name: ansible_playbook
    hosts: localhost
    require_chef_for_busser: false
    require_ruby_for_busser: false
    playbook: server.yml
[...]

This still installs ansible on the vagrant boxes, which I don't want.

According to this documentation it is possible to install ansible and run a ansible playbook by using the following snippet:

Vagrant.configure("2") do |config|

  #
  # Run Ansible from the Vagrant Host
  #
  config.vm.provision "ansible" do |ansible|
    ansible.playbook = "playbook.yml"
  end

end


micrometer exposing actuator metrics vs kube-state-metrics vs metrics-server to set pod request/limits micrometer exposing actuator metrics to set request/limit to pods in K8svs metrics-server vs kube-state-metrics -> K8s Mixin from kube-promethteus-stack Grafana dashboad
It's really blurry and frustrating to me to understand why there is such big difference between values from the 3 in the title and how should one utilize K8s Mixin to set proper request/limits and if that is expected at al.
I was hoping I can just see same data that I see when I type kubectl top podname --containers to what I see when I open K8s -> ComputeResources -> Pods dashboard in Grafana. But not only the values differ by more than a double, but also reported values from actuator differ from both.
When exposing spring data with micrometer the sum of jvm_memory_used_bytes is corresponding more to what I get from metrics-server (0.37.0) rather then what I see on Grafana from the mixin dashboards, but it is still far off.
I am using K8s: 1.14.3 on Ubuntu 18.04 LTS managed by kubespray.
kube-prometheus-stack 9.4.4 installed with helm 2.14.3.
Spring boot 2.0 with Micrometer. I saw the explanation on metrics-server git that this is the value that kubelet use for OOMKill, but again this is not helpful at all as what should I do with the dashboard? What is the the way to handle this?

Based on what I see so far, I have found the root cause, renamed kubelet service from old chart to new that can get targeted by serviceMonitors. So for me the best solution would be grafana kube-state-metrics + comparing what I see in the jvm dashboard


Dynamic YAMLs for Docker Compose Currently I find myself managing a set of embarassingly similar YAMLs for environment profiles. I see there a scaling trouble if there will be lots of dynamic environments (i.e. fluid acceptance test configurations, with disposable environments you can do that!)

Has anybody done a working implementation to build YAMLs dynamically on scale based on some generic represenation ( kind of CasC level 2 beyond some naive scripting), or if this makes no sense, why and which way is to prefer then.

One could use the same docker-compose file for dev, test, acceptance and production (DTAP) systems. Things that deviate per environment, e.g. passwords could be defined in an .env file.

If .env files are not a solution for some reason then one could create different docker-compose files, e.g. docker-compose.override (dev), docker-compose.test, docker-compose.acc and docker-compose.prd.

I personally prefer the first option, e.g. using .env files and a docker-compose.override for dev.


Get tagging to work on AWS CLI I am trying to tag commit hash and app version onto the ECS service to make it easier for developers to identify the service with the code.

But I get this error

+ aws ecs tag-resource --resource-arn arn:aws:ecs:ap-southeast-1:12344556677:service/master-web-dev-WebECSServices --tags key=COMMITHASH,value=85491def key=VERSION,value=0.2.4
An error occurred (InvalidParameterException) when calling the TagResource operation: Long arn format must be used for tagging operations

Is there any problem with the ARN? Why do I always get this issue?

I think this is because your ECS service was created before tagging was available for ECS services.

See this blog post by AWS on how to migrate to the new ARN format. https://aws.amazon.com/blogs/compute/migrating-your-amazon-ecs-deployment-to-the-new-arn-and-resource-id-format-2/


Watch folder, when direct changes are made push to repo So I have an azure devops repo and pipeline that copys the repo to my server when changes are pushed. However there are times when some of our developers may make a direct edit to the production server. Is there a way for me to have the azure agent that is installed on the server watch the files for any additions/changes and if/when this happens it pushes the files to the repo?
I have tried to find a solution for this but have been unsuccessful.

First, I'd highly recommend not doing this - a better approach would be to have a single source of truth (that is your repo). Force developers to commit & push changes and let the pipeline do the rest. If it's not in repo, it didn't happen.
If you absolutely have to, then I'd go with "brute-force" way - have a pipeline that triggers periodically (every couple of hours or so) and does a diff of server files against repo files. If something changed, send a threat email to the developer who did this sync back to repo.


Azure DevOps Build and Release pipeline artifact name variable Trying to do something simple. Either access the name of the artifact from my release pipeline via some predefined variable, or create a variable for both my build and release pipeline for the artifact name. I don't want to hard-code it.
I would like my build and release pipelines decoupled.

Publish Build Artifacts YAML:

- task: PublishBuildArtifacts@1
  inputs:
    PathtoPublish: 'dist'
    **ArtifactName: $(Storage.ArtifactName)**
    publishLocation: 'Container'
  displayName: 'publish artifact'

Output:

##[section]Starting: publish artifact
==============================================================================
Task         : Publish build artifacts
Description  : Publish build artifacts to Azure Pipelines or a Windows file share
Version      : 1.158.1
Author       : Microsoft Corporation
Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks/utility/publish-build-artifacts
==============================================================================
##[section]Async Command Start: Upload Artifact
Uploading 10 files
File upload succeed.
**Upload '/home/vsts/work/1/s/dist' to file container: '#/3928702/$(Storage.ArtifactName)'**
Associated artifact 10 with build 58
##[section]Async Command End: Upload Artifact
##[section]Finishing: publish artifact

I needed to add:

variables:
- group: front_end-storage

At the top of my pipeline for the pipeline to pick up on the variables and authorize the group.


How to enable Google Maps and generate an API key using Terraform? I've been trying to spin up a project from scratch over the Christmas break to automate the creation of my home network, one of those components is the Ubiquiti Controller which requires a Google Maps API key to plot devices onto maps.

I've been able to automate the creation of the project using Terraform:

resource "google_project" "slaterfamily_network" {
  depends_on      = ["data.google_organization.org"]
  name            = "Slater Family Network"
  project_id      = "slaterfamily-network"
  org_id          = "${data.google_organization.org.id}"
  billing_account = "${var.billing_account}"
}

resource "google_project_iam_binding" "account_owner" {
  project = "${google_project.slaterfamily_network.project_id}"

  # It's important to specify ALL members, as this replaces them:
  #   https://twitter.com/devopsreact/status/840233052194320384
  members = [
    "user:richard@slaterfamily.name",
    "serviceAccount:${data.google_service_account.terraform_service_account.email}",
  ]

  role = "roles/owner"
}

This works fine for configuring Google Cloud DNS, however, I can't seem to find a way of enabling Google Maps and generating an API key, the following statements enable the Service Usage API, however it fails enabling the Maps API:

resource "google_project_service" "service_usage" {
  project = "${var.project_id}"
  service = "serviceusage.googleapis.com"
}

resource "google_project_service" "maps" {
  depends_on = ["google_project_service.service_usage"]
  project = "${var.project_id}"
  service = "maps.googleapis.com"
}

With this error:

google_project_service.maps: Error enabling service: failed to issue request: googleapi: Error 403: The caller does not have permission, forbidden

Which after tracing the HTTP calls gives me the following, formatted as cURL for ease of reproduction:

curl -H 'Host: serviceusage.googleapis.com' -H 'content-type: application/json' -H 'authorization: Bearer {{ BEARER_TOKEN }}' -H 'user-agent: google-api-go-client/0.5 Terraform/0.11.7 (+https://www.terraform.io) terraform-provider-google/1.20.0' --data-binary "{}" --compressed 'https://serviceusage.googleapis.com/v1beta1/projects/slaterfamily-network/services/maps.googleapis.com:enable?alt=json&prettyPrint=false'

Which results in:

{
  "error": {
    "code": 403,
    "message": "The caller does not have permission",
    "errors": [
      {
        "message": "The caller does not have permission",
        "domain": "global",
        "reason": "forbidden"
      }
    ],
    "status": "PERMISSION_DENIED"
  }
}

I faced the similar problem recently, but fixed it by providing additional roles to the service account. I am not sure what permissions you need, you need to look it up. In my case, I was trying to create an appengine project, So I had to add "Appengine Admin" role to the service account and enable "appengine.googleapis.com" via terraform resource "google_project_service"


How to add job dependency to gerrit trigger in Jenkins pipeline? We do have this configuration in our GUI based Jenkins project, trying to convert it to Jenkins pipeline, couldn't find how to add precurse-build as dependent build as shown in red below.

I added below below pipeline code to trigger the build whenrun_job is added as gerrit comment, and it works good, would anyone please show me how to add Other jobs on which this job depends to this?

pipeline {
    triggers {
         gerrit dynamicTriggerConfiguration: 'true',
                triggerConfigURL: 'https://some/url/path/trigger.txt',
                serverName: 'gerrit-server',
                triggerOnEvents: [
                    commentAddedContains('^run_job$')
                ]
    }

    stages {
       ...
       ...

The "correct" way to determine the names for Pipeline step parameters is to use the Pipeline Snippet generator in the web UI:

https://www.jenkins.io/doc/book/pipeline/getting-started/#snippet-generator

However, based on browsing the source code, I am guessing the name of the parameter you are looking for is dependencyJobsNames.


Run JMeter Test in Jenkins with Parameters I have a JMeter test I would like to run in JMeter that have some parameters.  I installed the performance-plugin for Jenkins to enable JMeter testing as described here:

http://jenkinsci.github.io/performance-plugin/RunTests.html

However, the instruction is not clear on how to enter command-line parameters which is essential for my test.  Here is what I got so far:

Is it possible at all to enter parameters to run my test?  Or does it have to be completely parameter free.  IE, I need to hardcode all parameters into the test itself?

Yes, you can easily trigger your JMeter jmx suite via jenkins coupled with performance plugin. For triggering the build:

If you're using windows execute the build as windows batch command, giving the same commands that you would give if you were running your JMeter script from your command line.
To give parameters you can conveniently select the option in jenkins configuration 'This project is parameterised' and pass the parameters ensuring the same reference is used in your JMeter jmx file.
You can use the __P() function to make ramup, hostname or number of threads configurable and refer them using J command.


How can container user and host user both have access to same files in volume? I'm using Docker namespaces in an effort to allow files created within a volume to be arbitrarily read and written to by both the container user and the host user (please let me know if this is even a valid approach to what I'm trying to do!).

My host username is aryzing. So I'm running the docker deamon as

dockerd --userns-remap=aryzing

The volume maps my host's ~/workspace/app to the volume's /usr/src/app.

From the host system I have no issue modifying the files, but from within the container, I have access errors. Specifically, I'm running NPM's install command from within the container, npm install, and it's unable to install the packages due to a permissions error:

Error: EACCES: permission denied, access '/usr/src/app/node_modules'

As per this post I've also tried running the container with the --user flag, and ran into the same issue.

Any suggestions on how to achive the functionality I'm after?

I was tripping myself up: when I was trying the --user flag, I still had --userns-remap set. Running the daemon without --userns-remap did the trick.