Is it possible to set a minimum Python version in Pygradle? Pygradle is used to build a python project.

Problem

The problem is that the CI runs a different Python version, resulting in different build outcomes, e.g. the build succeeded on the laptop, but failed on CI.

CI

============================= test session starts ==============================
platform linux -- Python 3.6.3, pytest-2.9.1, py-1.4.29, pluggy-0.3.1
rootdir: /home/travis/build/user/project, inifile:
plugins: cov-2.2.1
collected 1 items

Laptop

============================= test session starts ==============================
platform linux2 -- Python 2.7.5, pytest-2.9.1, py-1.4.29, pluggy-0.3.1
rootdir: /home/user/dev/project, inifile:
plugins: cov-2.2.1
collected 1 items

Discussion

To ensure that the build fails both locally on CI and on the laptop, the same python version should be used.

Question

Is it possible to set a minimum Python version in Pygradle?

Side note: personally I dislike using a minimum version: you never know beforehand how the language will evolve and if a certain future version will still be compatible with your code. For example many 2.7-compatible scripts can't actually work with python 3.X, setting 2.7 as the minimum version won't cut it.

I prefer to explicitly list as allowed the (existing) versions that I can test and confirm to be compatible. Any future version can be added later on, after such confirmation is obtained.

If you agree with the above then the com.linkedin.python plugin can apparently perform such version checks (it doesn't looks like it supports a minimum version):

  Default and allowed Python version

  This plugin enforces a set of default and allowed Python versions. For
  example, you can specify pythonVersion = '3' and you will get
  whatever the default Python 3 version is. Similarly pythonVersion =
  '2' gets you whatever the default Python 2 version is.

  This plugin also enforces a set of allowed Python versions. If you
  choose a Python version that is not allowed, you will see an error
  messages such as:

Python 3.2 not allowed; choose from [2.6, 2.7, 3.4, 3.5, 3.6]

  If you see this error message, you must adjust your pythonVersion
  setting to one of the allowed values.

  Consumers of this extension can change the default Python 2, Python 3,
  and allowed versions by making a call on the PythonDetails object,
  either the one returned by PythonExtension.getDetails() or on any
  PythonDetails instance you create, e.g. in Groovy:

pythonDetails.setPythonDefaultVersions('2.7', '3.6', ['2.7', '3.5', '3.6'])

But I'm not familiar with pygradle, I can't really tell exactly how the plugin works and/or is configured to do the job.


Does Buildbot support complicated build/deploy scenarios? I need a tool/framework for CI/CD which can:

Support multi-repository project
Build and ship only changed part (frontend/backend) using revison hashes to decide
Ship some branches (release candidate brances) to test environment and some to production (master)
Get version number from repository based on branch name for RC
Get version number from repository based on last merged branch in master for production
Create build version in bug tracker and mark issues as fixed in created version based on issue status and planned version
Notify about new version uploaded on test/production with issues list from bug tracker
Get new database migration scripts (based on diff with previously uploaded version using revison hashes) and notify about them & (maybe) execute them

I wrote a script to do this but it lacks interface to do manual build/ship or to monitor build status (only notification at the end of build/deploy)

I have found Bouldbot but not shure it can handle these requirements.

If you know a framework which can handle these requirements without pain, please, name it.

Jenkins supports scripted pipeline which can handle my every requirement.
Buildbot can handle these requirements with custom steps, i suppose, but it's too hard if compared to Jenkins' scripted pipeline.


How to grant all resources in an AWS account the ability to assume role? I have an organization with a master account (account A) and a second account (account B).
I want to create a role so all instances in account B will have read permission from everything in account A.
For example, EKS in account B will be able to pull from ECR in account A. Or a resource in account B will be able to read data from an RDS or S3 in account A.

How can I create a role and apply it to all current and future resource in account B so they will have complete read assess to resources in account A?

There are several ways to enable access, and your question touches on more than one.

First method, is giving EC2 instances access to do API calls with the AWS API in another account. For this, the instances would need the appropriate IAM credentials. You can assign an IAM Role to an instance. The relationship is then "IAM Policy" -> "IAM Role" -> "IAM Instance Profile" -> "EC2 Instance". This Policy can give permission for this instance to do the sts:AssumeRole API call in the account of this IAM/EC2, and assume a role in the second account. https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2_instance-profiles.html

I have not tested this, but as far as I know, while the above description is correct - it will not work with the sts:AssumeRole API since it requires non-temporary (non-session) credentials. And the credentials provided to the EC2 instance are in fact sts:GetSessionToken type of credentials provided to it via the EC2 metadata service.

The same applies to worker nodes, and pods, running in EKS. There is a way to enable this for EKS that is described in details in the documentation. https://docs.aws.amazon.com/eks/latest/userguide/security-iam.html

On the other hand, if all you need is to access RDS and talk to your MySQL/PostgreSQL database - you don't need any IAM credentials. Just a clear path through the network where security groups allow these instances to access the RDS database. In EC2 and RDS, security groups allow to include foreign-account security groups as a filter for "source IP addresses". So when you choose to do something like that, you don't really need to specify the IP address of your EC2 instance where the application is executed, but rather just specify which security group is being used by that instance, with the addition of the account number. https://docs.aws.amazon.com/vpc/latest/peering/vpc-peering-security-groups.html

The S3 service does require IAM credentials to write/read data from it, but in this case it also allows to specify a permissions policy on the S3 bucket itself. So while the above method of sts:AssumeRole credentials might not work, you can create S3 buckets that do allow access from a peer account. https://aws.amazon.com/premiumsupport/knowledge-center/cross-account-access-s3/

Very similar to this, the ECR service also has an option to allow the GetLogin API call from another account. https://aws.amazon.com/premiumsupport/knowledge-center/secondary-account-access-ecr/

And naturally the simplest way, is to provide credentials directly to EC2 instances. This can be done either by writing them into a file. For example the aws CLI is using the file ~/.aws/credentials, and it allows to specify multiple named profiles so you can specify which account needs to be controlled. https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-profiles.html

Another alternative, is to create a service that is a "Credentials ATM Machine". One excellent tool for doing this is HashiCorp Vault https://www.hashicorp.com/resources/deep-dive-vault-aws-auth-backend. AWS even has a QuickStart CloudFormation template to deploy Vault in your account. https://aws.amazon.com/quickstart/architecture/vault/


Using Prometheus to monitor Spring Boot Applications in Kubernetes Cluster I have spring boot powered microservices deployed in my local kubernetes cluster. The microservices are using micrometer and prometheus registry but due to our company policy the actuator is available on another port:

8080 for "business" http requests
8081/manage for actuator. So, I can access http://host:8081/manage/prometheus and see the metrics when running the process locally (without kubernetes).

Now, I'm a beginner in Prometheus and have a rather limited knowledge in kubernetes (I'm coming with a Java developer background).

I've created a POD with my application and succesfully run it in kubernetes. It works and I can access it (for 8080 I've created a service to map the ports) and I can execute "business" level http requests it from the same PC.

But I haven't find any examples of adding a prometheus into the picture. Prometheus is supposed to be deployed in the same kubernetes cluster just as another pod. So I've started with:

FROM @docker.registry.address@/prom/prometheus:v2.15.2

COPY entrypoint.sh /
USER root
RUN chmod 755 /entrypoint.sh

ADD ./prometheus.yml  /etc/prometheus/

ENTRYPOINT ["/entrypoint.sh"]

entrypoint.sh looks like:

#!/bin/sh
echo "About to run prometheus"
/bin/prometheus --config.file=/etc/prometheus/prometheus.yml \
                --storage.tsdb.path=/prometheus \
                --storage.tsdb.retention.time=3d \
                --web.console.libraries=/etc/prometheus/console_libraries \
                --web.console.templates=/etc/prometheus/consoles

My question is about how exactly I should define prometheus.yml so that it will get the metrics from my spring boot pod (and other microservices that I have, all spring boot driven with the same actuator setup).

I've started with (prometheus.yml):

global:
  scrape_interval: 10s
  evaluation_interval: 10s

scrape_configs:

  - job_name: 'prometheus'
    metrics_path: /manage/prometheus
    kubernetes_sd_configs:
      - role: pod
    bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_label_app]
        action: keep
        regex: sample-pod-app(.*)|another-pod-app(.*)

But apparently it doesn't work, so I've asking for the advices:

If someone has a working example it would be the best :)
Intuitively I understand I need to specify the port mapping for my 8081 port but I'm not exactly know how
Since prometheus is supposed to run on another port, am I supposed to expose a kubernetes service for port 8081 at the kubernetes level?
Do I need any security related resources in kubernetes to be defined?

As a side note, at this point I don't care about scalability issues, I believe one prometheus server will do the job, but I'll have to add Grafana into the picture.

P.S. Originally asked this question on SO but was suggested to ask this here as well.

The best way to deploy prometheus to kubernetes is with the helm chart:

https://github.com/helm/charts/tree/master/stable/prometheus

If you haven't used helm yet, its a very simple install to your laptop/etc and the cluster (and the newest version doesn't even require a cluster side install).  It takes like 30 seconds.  It's a package manager for kubernetes and it allows you to install, upgrade, list, and delete "charts" which are basically simple wrappers around kubernetes YAML files (with some nice templating).  It would even be good to wrap your spring boot apps in a helm chart =).

Once you've got prometheus running well, there is a good tutorial here on setting up actuator's prometheus endpoint in spring-boot and getting it picked up by prometheus by editing its values.yaml.

https://www.callicoder.com/spring-boot-actuator-metrics-monitoring-dashboard-prometheus-grafana/

# my global config
global:
  scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  # - "first_rules.yml"
  # - "second_rules.yml"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'
    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.
    static_configs:
    - targets: ['127.0.0.1:9090']

  - job_name: 'spring-actuator'
    metrics_path: '/actuator/prometheus'
    scrape_interval: 5s
    static_configs:
    - targets: ['HOST_IP:8080']

There's also some info at the end of the tutorial about surfacing all this in Grafana for slick visualization.


How to decide between Rolling Releases versus LTS Releases? There are software packages that offer a choice between either of those types of releases:

a Rolling release.
a Long Term Support (= LTS) release.

Consider the available Ubuntu releases as an example of this. And imagine that:

some software requires a Linux server,
whereas the decision is made (for whatever reason) to go for Ubuntu,
the decision makers do know what these release types actually mean,
it turns out that any release starting from 16.04 and above will do,
but whatever release is selected, all Dev / QA / Operational servers must run the very same release.

Question:

What are the criteria one should use to decide about which type of release to select?

I don't think there's a good answer that applies to all software packages or situations.

Rolling releases are more risky as there is no predictability around when things will change. You may have installed the thing last night and already there's an update to deal with. But what does that update do? Is there enough information to know if you want it? Sometimes, but it depends entirely on the developer/company releasing the package.

Some things to consider before going with a rolling release package:

Who are the primary users of the package, and in what environment?
What is your tolerance for risk?
What is your process for performing upgrades?

LTS releases are intended to be used for longer periods of time and often have published lifetimes that enable you to plan and test migrations to the next LTS release well before updates to the package/system end.

My recommendation is that servers (with the exception of sandbox/R&D machines) should use LTS releases. On desktop machines it depends on the end user and what that individual wants.

In fact, I would go so far as to say unless there are specific and very good reasons to go with a rolling release, always use LTS releases.

Note:

When dealing with higher-ups, appealing to their desire not to spend money is always a good tactic. Rolling releases are more expensive in the long run when you consider the amount of time spent by employees testing, installing, dealing with bugs from new versions, etc. While you'll have to do some of this with LTS releases, their longevity means it's significantly less.


How do I make ansible-playbook log its output in a machine readable format like xml or json? Looking for information on how to output machine readable output form ansible-playbook.

This is done using one of the configuration options. For a general overview of configuring Ansible, see the configuration documentation, in particular the callback plugin configuration.
Callback plugins change how Ansible provides feedback on the progress and state of tasks. There are several options for JSON output:

JSON output to stdout
JSON to syslog

Setting these configuration options can be done either in a configuration file (ansible.cfg) or with environment variables (ANSIBLE_XXX), depending on what you prefer.
To quote the current documentation:

Changes can be made and used in a configuration file which will be searched for in the following order:

   ANSIBLE_CONFIG (environment variable if set)
   ansible.cfg (in the current directory)
   ~/.ansible.cfg (in the home directory)
   /etc/ansible/ansible.cfg

Ansible will process the above list and use the first file found, all others are ignored.

Assuming you want to send the output to stdout then, your ansible.cfg could look like:
[defaults]
callback_whitelist=json
stdout_callback=json

Using environment variables:
 ANSIBLE_CALLBACK_WHITELIST=json ANSIBLE_STDOUT_CALLBACK=json ansible-playbook ...


What practices or tools enable Continuous Deployment of Databases? Continuous Delivery or Continuous Deployment of infrastructure and code is comparatively simple compared to trying the same approaches for databases, specifically RDBMSs.  Code and infrastructure will not change or evolve once deployment has completed.  Databases, however, will have new data added to them making the data if not the schema inherently mutable components.

I am aware that there are practices such as only adding database objects, i.e. tables and columns, never modifying or removing them - this purely additive way of approaching database schemas has the advantage of ensuring that schemas are backwards compatible at the cost of successively more complicated schemas.

Equally there are products such as Flyway and Ready Roll which assist in writing migrations to be written between versions of a schema.

What other practices and tools exist currently to allow for continuous deployment of database schemas into a production environment where data integrity is of a concern?

Pramod Sadalage and Scott Ambler wrote a book Refactoring Databases: Evolutionary Database Design that is an incredibly solid primer to the subject of DBs in a CD org/team.


How to set Deployment Type using ECS First Run Wizard The AWS ECS First Run Wizard provides no option for choosing the 'Deployment type' of the ECS Service:

Whereas the ECS Service creation screen does:

Why does the First Run Wizard not allow for setting the same properties as the regular ECS creation screen does?  There's also no way to edit an ECS Service created via the First Run Wizard to change the Deployment  Type.

How do I set the Deployment type of an ECS Service created via the First Run Wizard?

I think you want to setup Blue/Green deployment type for ECS. As Blue/Green deployment, there should be one environment (Blue) should exist before deploying another environment called Green. As it's your first project or first ECS services, there is no prior environment thus we cannot choose Blue/Green deployment in deployment type.


The Phoenix Project and Military service I have recently been reading "The Phoenix Project".
While I can relate to most aspects of the story, I am struggling to understand the frequent references to the military background of many characters (e.g. Bill himself, Stefe, Erik, ...).

Since I am not familiar with US culture and army in particular, I am pondering 2 questions:

Question 1:
Are these reference to the army background an essential part of the message of the book or just story-telling (e.g. trying to give some depth to the characters by telling their history)?

Question 2:
If an essential part, how does the way the US army organises its operations relate with a DevOps culture? Should it be embraced or avoided in the workplace?

My take on the military references in the book, as a veteran, has to do with Man, Machine, Method, Measure; and, with drilling.

The military is entirely other compared to civilian operational models. Setting aside the chain of command and the Military Justice systems; consider that the way any service member mops, waxes and buffs a floor is detailed in a procedure outlining what (exact) equipment is to be used, how it is to be calibrated, how the person is to operate said equipment and measures by which to determine compliance with the procedure. The same goes for brewing coffee, making Kool-Aid, or launching aircraft from a carrier deck.

Secondly the military runs "drills" constantly so emergencies aren't chaotic and reactive; they are methodical and responsive. Hence these would be recommended practices in a DevOps context. Practice well-documented and consistently executed procedures often to increase reliability and responsiveness in the face of the unknown.

In Phoenix project they apply both these practices to tame the chaos that Bill and the team have been subject to throughout the book.


How to export Cucumber Reports via Jenkins Pipeline I'm using the Cucumber Reports Plugin for Jenkins and I want to be able to export the HTML reports so they can be posted on an external portal. I can see that there is a folder named cucumber-html-reports in the build directory in Jenkins, but how do I access this? Is there some way to access these files from within a pipeline?

Working off what @chupasaurus said, here's what I came up with:

node('master') {
    dir("../builds/${BUILD_NUMBER}/") {
        sh "cp -r cucumber-html-reports $WORKSPACE"
    }

    archive "cucumber-html-reports/*"
}

Obviously all this does is archive the report for that build, but you can easily extend this to copy it somewhere else where it can be hosted or manipulated. Just make sure you put this after the Cucumber Reports plugin step.


How to help DevOps Engineers feel less like a lone wolf? I have just been speaking to a DevOps guy who raised some really good points about the struggles of being a DevOps Engineer and feeling like a one-man army sometimes, even though he is in a team of 16 Engineers.

He wears lots of different hats, but sits in the Development team doing infrastructure work. He loves the cool tech that he gets to work with- loads of automation, cloud, containerization etc. But he struggles that he is the only person doing ops, in a dev team. He reports to the Development Manager, but works more closely with the Infrastructure Manager.

This seems to be the case with lots of DevOps professionals I speak to. What could be done to help DevOps Engineers feel less like a lone wolf?

My first thought is "why is the he the only person doing ops, on a dev team,  especially when he gets to work with loads of automation?".  I think there's an opportunity there to address the lone wolf syndrome; particularly in a dev environment, infrastructure-as-code provides a great framework for sharing the load.  Ops people should be experts in determining and defining infrastructure needs for application, but they should also be able to build a platform to let dev roles do as much as they can independently.

It sounds like a silo within a team; old habits die hard.  A coder may not feel comfortable spinning up and hardening a server, but in a pure devops model, they should have the tools to do so.  An ops person in a devops team shouldn't responsible for delivering infrastructure for the app itself, but they should provide some insight into what is needed and some guidance on how the app developers can do it themselves.  It's almost a meta-infrastructure model; ops engineers build infrastructure that can build infrastructure on demand when requested by the development team.

Consultation, communication, and blending of responsibilities are all crucial to the success of a devops team.


Is there a premium version of Google translate that is available for purchase to integrate into business applications? It looks like there may have been at one time a version of Google translate called "Business" that did not employ the free API used by the version that comes up in my browser. If there is, how can I get more information about it (I've already googled it and cannot find anything)? If not, is there some kind of replacement that can be used in its stead?

Yes !

There's an API available on Google Cloud & you can find more details about it on https://cloud.google.com/translate/


error validating data: ValidationError(Deployment): unknown field "spec" in I know that yml files are temperamental and I still struggle to figure it out. Can anyone help out here to understand what is it about my client-deployment.yml file that makes Kubernetes want to issue the above error?

apiVersion: apps/v1
kind: Deployment
metadata:
  name: client-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      component: web
  template:
    metadata:
      labels:
        component: web
    spec:
      containers:
        - name: client
          image: ldco2016/multi-client:v1.2
          ports:
            - containerPort: 3000

I got the above error running kubectl apply -f k8s

k8s being one of the folders in the project with the yml configuration files.

Sure enough, everything and including the last spec: needed to be tabbed to the left once:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: client-deployment
spec:
  replicas: 3
  selector:
    matchLabels:
      component: web
  template:
    metadata:
      labels:
        component: web
    spec:
      containers:
        - name: client
          image: ldco2016/multi-client
          ports:
            - containerPort: 3000

Unfortunately, you can't tell the difference with how yaml is formatted when pasted here, so it would not have been an easy catch, but if you would be looking at the original file configuration you would have noticed it like I did when I looked a third time.


Expose metrics for Kotlin using JMX_Exporter to prometheus I am trying to use JMX_Exporter for my kotlin code to expose metrics to prometheus in order to show it in Grafana. I have gone through many articles and tried to understand how could it be done. I have found below two links useful and trying to achieve using those.

https://github.com/prometheus/jmx_exporter
https://www.openlogic.com/blog/monitoring-java-applications-prometheus-and-grafana-part-1
What i did so far is, created a folder 'prometheus-jmx' in root directory and added mentioned JAR and config.yml file in that folder. Then, i added below parameter to my dockerfile.

CMD java -Xms128m -Xmx256m -javaagent:./jmx_prometheus_javaagent-0.12.0.jar=8080:./config.yml -Dconfig.file=config/routing.conf -cp jagathe-jar-with-
    dependencies.jar:./* com.bmw.otd.agathe.AppKt

My prometheus is running in my OpenShift cluster along with my application. I could scrape metrics for my other applications/deployments like Jenkins, SonarQube etc without any modifications in deployment.yml of Prometheus.

My application is running properly now on OpenShift and from application's pod, I can scrape metrics by using below command.

curl http://localhost:portNumber

But on promethus UI, I cant see any JVM or JMX related metric.

Can someone please tell me where and what am i doing wrong? Any kind of help would be appreciated.

After trying many things, I came to know that I needed to expose the port of my application's container in order to let Prometheus or other deployments to know. After exposing the port, I could see my application under targets on Prometheus and I could scrape all JMX and JVM metrics. Hope this would help someone in future...


Relation/interplay of deployment pipeline stages (dev-stage-prod) with blue/green deployment mechanisms? I'm building some new infrastructure in AWS and plan on having separate accounts for dev, stage, and prod environments. I'd like to implement a blue/green deployment strategy at least for production, and I've seen a few different ways this could be done, an obvious one being CodeDeploy's built-in blue/green deployment capability.

Someone suggested to me that the stage/prod environments could be thought of as the blue and green environments. They are, after all, supposed to be identical environments, but I hadn't thought of stage being something that could act as a production platform. I think stage is supposed to be for end-to-end testing, final verifications by QA, demos, etc.

So, I believe stage should continue to be for internal purposes only. But we may need two production accounts. At any given time each of them would be expected to contain a combination of inactive and active services (blues and greens). DNS (Route 53) would be in a shared account, and when a release occurs it would switch routings from resources in one account to resources in the other one.

Then, it kind of seems like having stage is less useful when you have blue/green. When a new release is in the inactive state and going through the most comprehensive tests, it's essentially what stage does. And we don't even really have QA--developers and our customers are the QA. So can we get rid of it? I have heard praises sung of having a straight-to-production strategy whenever automated testing is taken seriously enough.

Is this sensible?

Yes.  With no QA and wanting to have an internal environment separated from production this makes sense.  However, a couple comments:

You should go for whatever is going to make your business money instead of going for an enterprise standard.  While some of the common practices may not be sensible yet, keep in mind that as your company grows (e.g. a dedicated QA role), they might eventually be very useful.
If you get rid of any stages, you are essentially eliminating a development environment.  Straight to production can be great, or greatly harmful.


Developing inside a Docker image? Every time I onboard a new person on my team I have to go through the same routine of installing everything, figuring out what changed since last time we onboarded someone etc. Not fun.

I've heard about people maintaining docker containers with the development environment, but I'm not sure how to accomplish this.

Let's say you work on a repository containing a React webapp. Our IDE of choice is IntelliJ. It'd be cool if a new dev would just get a container with all the necessary dev stuff installed.

How do I make this happen?

The onboarding experience should be almost as simple as telling your new developer to just clone the repo and run docker-compose up. Personally I wouldn't bother worrying about IDE integration because people might prefer to use different IDEs.

Every project/application (if you have multiple) should be able to run separately and each project/application should have it's own codebase. That's a concept from The Twelve-Factor App methodology.

In the root of each project add a Dockerfile and a docker-compose.yml file.

  NOTE: I created Craft Kits which you can use to automatically generate these files for the framework/language you're using with one command (e.g. mason craft nodejs), you don't even need to know Docker.

The Dockerfile will build your application into an image, installing and dependencies your application needs to run. This image is the image that runs your app.

FROM node:8.8-alpine

# Set our workdir
WORKDIR /app

# Install and cache app dependencies
COPY package.json /app
RUN npm install

# Add project files
COPY . /app

# start app
CMD [ "npm", "start" ]

The docker-compose.yml file will spin up your environment. It will define all the services Docker should start for your application

version: '2'
services:
  web:
    build: .
    volumes:
      - ./:/app
    ports:
      - "3000:3000"

  mongodb:
    image: mongodb
    volumes:
      - ./storage/data/mongodb:/data/db
    ports:
      - "27017:27017"

When you onboard a new developer, you can just tell them to git clone your repo and to fire up their development environment by running:

docker-compose up -d

The advantage of this approach when you are developing locally, running docker-compose up will mount your code as a volume.

Then when you want to deploy, when you build your Dockerfile it will copy your code into your image so everything is packaged up nicely and easily moved around.

We use the same approach for apps on Codemason, my service for building, managing and scaling container-based applications. When you push new code, we automatically build an image from your Dockerfile. Since that image contains everything your app needs to run, it can be easily launched on any server and scaled up/down across multiple servers.


AWS ELB Application Load balancer, SSL not working I've setup a "Application Load balancer" in AWS. It has two Listeners, one on HTTP (port 80) and one on HTTPS (port 443).
It also has one target group that with only one FARGATE instance that I've registered in ECS.
In the Image below you can see the two listeners and the SSL Cert. I've attached to the HTTPS listener. You can also see that they have the same rule defined. Both of them target the same TargetGroup, over HTTP (port 80). From what I've understood that's an accepted practice to have a Load balancer listen to HTTPS but forward the request over HTTP since it's in the same VPC.
But for some reason only the HTTP listener is working. I've visited the site over HTTPS both directly through the LB's DNS Name and the domain I've attached to the SSL cert. both I both of the give the same result, it loads a while until it gets a timeout.
Anyone have a guess what could be wrong?

I solved it, I had forgotten that I had to add port 443 to the security group that my load balancer group belongs to.


Back-filling prometheus (and related system) metrics? Projects like Thanos, which are based on prometheus storage formats, seem to say that it is impossible to back-fill data in a prometheus based system.

Can anyone explain why and if there are any possibilities for changing this?  Lack of ability to backfill would make it difficult to switch  metrics systems without impact (when previous systems can already retain multi years' data).

Answering my own question (after a few days of research).
I have yet to test this but https://github.com/Cleafy/promqueen seems to be designed for this purpose:

PromQueen made possible to record prometheus metrics offline. PromQueen can, > therefore, backfill the recorded data inside a native prometheus database.
PromQueen is composed of two primary tools:

promrec tapes the metrics in a specified output file.
promplay backfills the prometheus database from scratch.

It doesn't have a lot of contributors but it comes from another POC repo and it was updated in the last 6 months, so I suspect it'll work fine.
In any case, it proves that you can plausibly back-fill prometheus, so it's not impossible.
Roadmap - Prometheus
You can also find this in the prometheus road-map, so it's clearly slated to be built-in eventually.
https://prometheus.io/docs/introduction/roadmap/

Backfill time series
Backfilling will permit bulk loads of data in the past. This will allow for retroactive rule evaluations, and transferring old data from other monitoring systems.


Are NixOS and GuixSD the incoming Linux distros of DevOps? The main advantage of these distros is the declarative approach which allows for setup to be done in the same way as you would configure a Vagrantfile or Dockerfile.

In the way that Docker replaced/is replacing VMs for reproducibility, is there a chance these distros will eventually replace typical distros that are used in a DevOps setting due to their atomic and functional nature?

Some articles, including this one give some arguments why docker is popular:

  In a nutshell, here's what Docker can do for you: It can get more
  applications running on the same hardware than other technologies; it
  makes it easy for developers to quickly create ready-to-run
  containered applications; and it makes managing and deploying
  applications much easier. Put it all together and I can see why Docker
  is riding the hype cycle as fast as I can recall ever seeing an
  enterprise technology go.

  Moreover, for once the reality is living up to the hype. Frankly, I
  can't think of a single company of any size that's not at least
  looking into moving their server applications to containers in general
  and Docker in specific.

My personal view is that a tool could be successful if it will solve a (major) problem. For example, a half year ago I gave a presentation of docker in the company, the developers had some doubts until they experienced a issue that it was not possible to run all microservices on their local machine.

The same is applicable to the question:

  Are NixOS and GuixSD the incoming Linux distros of DevOps?

It depends whether these distributions will solve a (huge) problem.


How many executors can I have for a Jenkins Windows build node? I have set up Windows build nodes in awhile, but I do remember when I did I was only able to set 1 executor per node. Is this still true?

You can specify the executors of your node in his configuration.
Manage Jenkins -> Manage Nodes -> Configure Node -> Number of build processors
In environments where we only run scripts etc. we have one executor per core.
But if you have large build jobs and the performance is important for you, you should read this article from the jenkins documentation.
https://www.jenkins.io/doc/book/scaling/architecting-for-scale/#Calculating-how-many-jobs


shell script to pass values properties file in java I have a properties file that looks like this:

mysql.username=USERNAME
mysql.pass=PASS

I need to change USERNAME and PASS with variable values passed to shell script. I cannot use sed since that will work fine the first time to replace USERNAME and PASS with "values" but once they are replaced, sed will not find the variable names in the file the second time the script runs and thus an issue.

How can this be handled?

You can obtain old parameters from the properties file before using sed, e.g.

# get old parameters
$ export oldUsername=$(grep mysql.username $FILE | cut -d'=' -f2)
$ export oldPass=$(grep mysql.pass $FILE | cut -d'=' -f2)

# set new parameters
$ export newUsername=NEW_USERNAME
$ export newPass=NEW_PASS

# change parameters
$ sed -i "s/$oldUsername/$newUsername/g" $FILE
$ sed -i "s/$oldPass/$newPass/g" $FILE


Does Docker images works without client application I recently installed Docker in Ubuntu 18.04 and with that, I have downloaded the mariadb image like this
docker run --name mariadb \
-p 3306:3306 \
-v /opt/mariadb_data:/var/lib/mysql \
-e MYSQL_ROOT_PASSWORD=lopaz \
--restart always \
-d mariadb:10.1

this is pretty cool method that I used to download from the official mariadb image and it's working well. What my question is that I want to use to login to the MariaDB ? when I type
mysql -h <myhost> -u root -p
MySQL or MariaDB not found on my local Ubuntu machine. What happened ? how my pc knows that I installed mariadb using docker? then I installed mysql client app then only I can use mysql commands on my local pc! please give me a clarification about why it happens when install using docker?

When you download mariadb image, mysql client tool is only available inside container. It is not getting magically installed on your local - that is why you had to install mysql client app.

Alternative way to connect to db without installing mysql app is to use docker exec and through this get to mysql tool available on container. So you would do something like:

docker exec -it your_maria_db_container_id mysql -h localhost -u root -p

Hope this helps.


Problem with Vagrant : Nothing happens on vagrant up command Versions
Vagrant: 1.9.5
Provider: Virtualbox 5.2.30
Guest: precise32
Host: windows 7 -32bit, build:7600
Powershell: 2.0

Vagrant destroy & up output
nothing happens here...just an empty directory " .vagrant " is being created

Expected behavior
virtualbox should be picked up by vagrant

Actual behavior
Cursor blinks ..that's it
nothing happens on -> vagrant up and on -> vagrant global-status ->it shows "There are no active Vagrant environments on this computer! "

Possible reasons for this behavior
I think there are some version dependecies between vagrant, virtualbox & powershell. I don't know I'm just guessing.

NOTE
I have read numerous official docs and similar previous issue, nothing worked. when vagrant up command is issued, it does nothing, the cursor keeps blinking forever.

NOTE
I know this is dumb question, I can use latest releases but is there anything I can do with this setup

If it's an Intel CPU, you may have to enable Intel VT-x in the bios. Sometimes the virtual machine will tell you this, and sometimes not. Here's a link that might help (ignore the Windows 10 part since you're on Win7). Also getting everything to the newest version would be a good idea if you can.


What are best practices to implement Gatling in CI? Gatling creates an HTML report:

Reports generated in 0s.
Please open the following file: /path/to/target/gatling/recordedsimulation-1496161958749/index.html
[info] Simulation RecordedSimulation successful.
[info] Simulation(s) execution ended.
[success] Total time: 31 s, completed May 30, 2017 6:32:57 PM

What are best practices to read the output and check whether it is acceptable, i.e. if a certain value is larger and smaller than X? Should one create a custom script or does Gatling itself provide such a functionality?

https://gatling.io/docs/2.3/extensions/jenkins_plugin/

https://wiki.jenkins.io/display/JENKINS/Gatling+Plugin

  Configuration

Install Gatling Plugin (via Manage Jenkins -> Manage Plugins)
Configure your project to execute Gatling simulations, for example using the Maven plugin (see Maven plugin documentation)
Configure your job :
    For a maven job: add "Track a Gatling load simulation" as a new post-build action. Beware that neither your Jenkin's path nor your

  job's name should contain any space.
          For a pipeline job: add a line to your pipeline script: gatlingArchive()

  Optional configuration: You can publish Gatling results with the
  Jenkins JUnit plugin. Your Gatling simulation needs to have some
  assertions, as each Junit testcase is a different Gatling assertion.

Configure the Jenkins plugin with the following Test report XMLs: target/gatling/assertions-*.xml
Execute Gatling with the following option: -Dgatling.useOldJenkinsJUnitSupport=true, for example: mvn gatling:execute -Dgatling.useOldJenkinsJUnitSupport=true


How to setup Docker mirror proxy Repository in Nexus Repository ManagerOSS 3.19.1-01? So,

We are going to setup local proxy repository. Which directs to further docker hub and our another remote Docker Hosted repository hosted on different location.Which is requested through Docker(group) in Nexus.

I want to create a mirror for our remote repository which we are accessing through docker proxy in Nexus Repository manager in our local repository and caching its contents when somebody requests a image that is hosted in remote repository and next time when somebody fetches it he is served through local caching.

  So if anybody pushes updated docker image on our remote repository it
  should be automatically mirrored in local repository

unlike when somebody requests updated image then local cache is invalidated and request again goes to remote repository for fetching the updated image.

Bottom line.

  How to do docker repository mirroring between two repository manager based on Nexus version i mentioned in question or any other way?

Any help appreciated.Thank you

In Nexus, go to the Admin Backend.

Create a blob store if you haven't already. Guidance can be found here: https://help.sonatype.com/repomanager3/configuration/repository-management#RepositoryManagement-BlobStores

Once you have a blob store, continue like so:

Choose "Repositories", hit "Create Repository". Choose "docker (proxy)" as the recipe.

Give it any name (for example "docker-hub").

Under "Remote storage" enter https://registry-1.docker.io.

As "Docker Index" select Use Docker Hub.

Select a blob store for it (which you have created above).

Hit "Create repository". No more configuration is needed for the proxy repo.

Next up: A group repository, from which you will pull:

Go back to "Repositories" and hit "Create repository" again. This time select "docker (group)".

Call it anything you like (for example "docker-group").

For "HTTPS" enter any port you like. (we have 10500)

For blob store select any blob store (for example the one you have created above).

Under "Group" select for "member repositories" the repository you have created above.

You can also add a "docker (hosted)" repository as well if you have one set up. (I will explain later)

Hit "Create repository".

Now open your Docker cli and do a pull of some Docker Hub image but don't pull it from Docker Hub, pull it through the HTTPs endpoint of your docker (group) repo that you have created above like so:

$ docker pull my.repository.com:10500/phpmyadmin/phpmyadmin

This will create a pull request to your Nexus OSS, which will proxy the request to Docker Hub. The image from Docker Hub will be cached in your Nexus and will be delivered to you.

After pulling browse your docker (proxy) repo that you have created above and check the images inside. You will find a phpmyadmin/phpmyadmin inside (or any other image you have checked out) with a current timestamp.

Hope that helps!


How to add a static slave to Jenkins running in Kubernetes? I've installed a Kubernetes cluster and deployed Jenkins in it which dynamically starts slave machines using Kubernetes plugin for Jenkins.
Some builds require a specific build machine with GNU make and specific kernel packages and for that I have a dedicated static virtual machine.
How can I automatically add the static virtual machine to Jenkins as a permanent slave? so when a Jenkins pod goes down and another starts it would automatically connect to the static slave?

Since I'm using Jenkins official helm chart which contains JCasC plugin (Jenkins Configuration as Code), I was able to configure the static slave attachment using it including the creation of relevant credentials.
Here's the relevant section of the values.yaml file I use to install/upgrade the Jenkins helm chart:
JCasC:
    defaultConfig: true
    configScripts:
      credentials: |
        credentials:
          system:
            domainCredentials:
              - credentials:
                  - usernamePassword:
                      scope: GLOBAL
                      id: "CentOS_7.1_k4.14.11_ssh_password"
                      username: 'jenkins'
                      password: ${decodeBase64:BASE64ENCODEDPASSWD}
                      description: "Username/Password Credentials for CentOS_7.1_k4.14.11"
      permanent-nodes: |
        jenkins:
          nodes:
            - permanent:
                name: "CentOS 7.1 k4.14.11"
                remoteFS: "/home/jenkins"
                numExecutors: 2
                labelString: "CentOS_7.1_k4.14.11"
                mode: EXCLUSIVE
                retentionStrategy: "always"
                launcher:
                  SSH:
                    jvmOptions: "-Xms256m -Xmx1024m -XX:MaxPermSize=1024m"
                    credentialsId: "CentOS_7.1_k4.14.11_ssh_password"
                    host: "10.1.66.120"
                    port: 22
                    sshHostKeyVerificationStrategy:
                      manuallyTrustedKeyVerificationStrategy:
                        requireInitialManualTrust: false

Now whenever I re-deploy or upgrade Jenkins helm chart, it automatically configures these credentials and add the static slave.


How to create the smallest working docker image every time? Aim: to create the smallest working docker images every time

Current

REPOSITORY          TAG       IMAGE ID            CREATED             SIZE
a-docker-image      latest    x                   42 minutes ago       1.92 GB

Attempt

Adding a cleanup step at the end of the Dockerfile:

#clean
RUN apt-get purge -y wget
RUN rm -r a-build-dir
RUN apt-get purge -y a-package

reduced the image size a little bit:

REPOSITORY          TAG       IMAGE ID            CREATED             SIZE
a-docker-image      latest    y                   2 minutes ago       1.86 GB

Discussion

I have built various docker images. Every time I try to decrease the size of the created image, but I always feel it is too large. I am looking for a script that has already been created by someone on github that removes all superfluous packages from the image so the size of the created image will be as small as possible.

As I said I always try to reduce the size of the image, but I want to apply this consistent so that every image that I create from now on will be as small as possible.

Question

How to create the smallest working docker image every time?

A Dockerfile creates a new layer for each of the commands in the file. Since layers are well, layered on top of each other - you cannot remove files that a previous layer added. This is why when you install packages, or download files, or create builds each in a separate command - these are still there in the image, even if in a future layer you removed them.

So if you just change this:

RUN apt-get update -y
RUN apt-get install -y wget a-package
# ...
RUN apt-get purge -y wget
RUN rm -r a-build-dir
RUN apt-get purge -y a-package

To this:

RUN apt-get update -y \
    && apt-get install -y wget a-package \
    && mkdir a-build-dir \
    && wget http://some-site/very-big-source-code.tar.gz \
    && tar xzvf very-big-source-code.tar.gz \
    && do-some-compilation \
    && apt-get purge -y wget \
    && cd .. \
    && rm -rf a-build-dir \
    && apt-get purge -y a-package

You will get a much smaller image.

Another option, is to squash the image after you built it.
Q: How does the new docker --squash work?

Yet another option, is to choose a slim base image. For example, images that use Alpine Linux as their base instead of Debian, take just 10-15mb instead of 180-250mb. And this is before adding your own application and data. Many official base images on Docker Hub have an alpine version.


Accessing attributes from within recipes Let us say I do knife node edit mynode and within the normal section I edit the JSON to add "myattribute": "myvalue",. I save this and verify that it has taken effect by issuing knife search node -a myattribute mynode, and I see it there. How would I then use this value from within a recipe? I would have expected to just be able to say node['myattribute'] when running the recipe on mynode but that just returns nothing. The usual Ohai attributes such as node['hostname'] obviously work as expected.

I think that according to chef documentation you should use normal with node data.

  A normal attribute is a setting that persists in the node object. A
  normal attribute has a higher attribute precedence than a default
  attribute.

you can read about it here:

https://docs.chef.io/attributes.html

https://docs.chef.io/knife_node.html

some example:

  "normal": {
    "ebs": {
      "devices": [
        {
          "mount_device": "/dev/xvdf",
          "encrypt": true,
          "volume_mapping": [
            {
              "device": "/dev/xvdf"
            }
          ]
        }
      ]
    },

In you recipe:

if node.ebs.devices[0]['mount_device'] == '/dev/xvdf' and node.ebs.devices[0]['encrypt'] ...

Another example with node['myattribute']

   "normal": {
        "myattribute": "myvalue"
        },


Git & Jenkins: get latest green commit on branch We're just starting to push for CI-CD and as a baby step we're going to try updating a stack with the latest green develop once every couple of hours. I am fairly new to Git/Bitbucket, and can't figure out how to ensure the checkout that Jenkins makes gets the last commit to have been marked green by Jenkins, rather than just "the last commit" as a blanket statement.

We have the Bitbucket Build Status Notifier plugin installed, so Bitbucket does track which commits are green after our unit tests run. Is there a way to leverage this info to make sure the right commit is picked?

You don't make any mention of the scripting language you want to use, so I will talk specifically about the HTTP requests to the BitBucket API:

Assumptions

If you have a BitBucket Repository that has three commits in in it the first and the last are failing the build, the middle is passing:

4768815 ❌
49d7110 ✅
42d357f ❌

Get the list of commits

You can get the list of commits by calling the following API method:

  https://api.bitbucket.org/2.0/repositories/{{owner}}/{{repo_slug}}/commits

  owner: RichardSlater
  repo_slug: greencommitproofofconcept

The response looks like this:

{
  "pagelen": 30,
  "values": [
    {
      "hash": "4768815fdc27abf4be17096e7c460f7f68f5d39b",
      "repository": { ... },
      "links": {
        ...
        "statuses": {
          "href": "https://api.bitbucket.org/2.0/repositories/RichardSlater/greencommitproofofconcept/commit/4768815fdc27abf4be17096e7c460f7f68f5d39b/statuses"
        }
      },
      "author": { ... },
      "parents": [ ... ],
      "date": "2017-04-10T11:38:18+00:00",
      "message": "README.md edited online with Bitbucket",
      "type": "commit"
    },
    {
      "hash": "49d7110b98616358d16055960a4abdf2926b890d",
      ...
    },
    {
      "hash": "42d357f1df7a7d7bcf1f10a9f3a5a40d85d5b11c",
      ...
    }
  ]
}

If you parse the JSON and loop over the responses you can extract out the statuses from:

values[n].links.statuses.href

Where n is the index, i.e. 0, 1 or 2 in the above example.  If you were to construct this from scratch it would be in the following format.

Get the list of statuses from the commit

  https://api.bitbucket.org/2.0/repositories/{{owner}}/{{repo_slug}}/commit/{{sha}}/statuses"

  owner: RichardSlater
  repo_slug: greencommitproofofconcept
  sha: 4768815fdc27abf4be17096e7c460f7f68f5d39b

Note: this is a Hypermedia API which means the urls could change so I would recommend using the links from the previous response rather than trying to generate them from scratch.

The response from the above HTTP request will be something like:

{
  "pagelen": 10,
  "values": [
    {
      "key": "POC-01",
      "name": "Build #1",
      "repository": { ... },
      "url": "http://devops.stackexchange.com/q/809/397",
      "links": { ... },
      "refname": null,
      "state": "FAILED",
      "created_on": "2017-04-10T13:04:28.261734+00:00",
      "updated_on": "2017-04-10T13:04:28.261759+00:00",
      "type": "build",
      "description": "Changes by Richard Slater"
    }
  ],
  "page": 1,
  "size": 1
}

From this response you can extract the state using:

values[n].state

Again where n is the status - there could be many of them if one commit resulted in many builds.

If the state for the build you care about is SUCCESSFUL then you have your answer and you can immediately return the sha for the commit.

Loop over all of the commits from the first phase, if you run out of commits follow the next page link that is included in the call to /commits.

Complete Flow Diagram

At a high level the flow will look like this:

Don't forget this is a Hypermedia API so wherever possible have your code follow the links in the API rather than trying to "guess them".


Why don't the instances in the auto-scaling group get assigned public ip addresses unless it's a default vpc? I'm trying to migrate our deployment process over to aws code deploy (by following instructions here:https://pionect.nl/autoscaling-aws-laravel-codedeploy-opsworks-chef-automation/).

When I'm creating the auto-scaling group, I selected the same vpc as my other instances belong to. However when I do that, it says that it the ec2 instance will not be assigned a public ip. How come? My other instances have the same vpc id and are assigned an ip address just fine.

i got the answer from aws support:

he reason why you are seeing that notification is because it is in accordance to your launch configuration (staging-1-deploy) settings. During the creation of a launch configuration, the assignment of public IP addresses to EC2 instances defaults to Only assign a public IP address to instances launched in the default VPC and subnet. (default) [1]. To support automatic assignment of public IP addresses to your EC2 instances as they are launched, you will have to change the launch configuration settings. Since a launch configuration cannot be edited [2], you can create a new one by copying your current launch configuration. Follow the steps below to achieve this:

1 - Go to your AWS console, select Launch Configurations, select staging-1-codedeploy
2 - At the bottom of the details, click the grey Copy Configuration button on the right.
3 -  ClickEdit Details link, then click Advanced Details on the resulting page.
4 -  Go to IP Address Type and select the second option, Assign a public IP address to every instance.
5 - [optional] Change the name of the launch configuration if need be and
6 - Save launch configuration

You can now proceed to use your new launch configuration for your auto-scaling group and all your EC2 instances will be assigned public IP addresses as desired.
I have attached a few screenshots to show some of the steps described above.


How do I prevent service A from being placed in the same node as service B in Docker Swarm? From the docs, I could only find information about placement restrictions based on node attributes, likes labels and its role. However, I couldn't find how to configure placement based on the services already running on a given node.

How can I do that?

At present the answer is "no", swarm mode does not support affinity or anti-affinity between a service and other containers/services running on a node. However, depending on your goal there are other options.

One option is to reserve resources like CPU and memory for your container. This will ensure two resource intensive applications do not schedule themselves on the same node if there are insufficient resources.

The other option, as you've mentioned, is to use labels and constraints. You can give the same label to multiple nodes, which is often done to specify a type of node, or assign nodes to a specific team/project/environment.

Both of these options will reduce your fault tolerance, particularly if you assign a label to one or a small handful of nodes. If those nodes with the label go down, docker will be unable to reschedule your service on another node.


Best practices for app and infrastructure code repositories We're looking for guidance on how to structure application code versus infrastructure code (IaC). More specifically, are there typical advantages/disadvantages to storing the two types of code in the same repository versus in different repositories?
A bit more background: We're on the journey to improve our ability to deliver systems more effectively. At the moment, we have application code in one repository and infrastructure code in a separate repository. We've newly begun having the app team change app-level infrastructure code. With that separation of repos, keeping the changes in sync introduces overhead. We think that bringing in the app-specific infrastructure code into the app repository would be advantageous.
Primarily, we're looking for info that helps to answer the question of which is the more effective option, one repo or two.

The correct answer, like almost everything in IT, is, "it depends." It depends on the way you work, the type of company you are working for, your requirements, your non-functional requirements, and possibly a lot of other factors.
tl;dr -- read the final section, it's the most important.
Layers
In this, admittedly long answer, I'll talk about the 3 layers of an organisation's IT operations -- infrastructure, platform, and application. These 3 layers are blurred at the boundaries and there is a continual debate about their delineations, but if you're not familiar with this model, roughly:

Infra: machines and VMs, security and resource groups, auto-scaling groups and elastic load-balancers -- stuff you typically provision with Terraform
Platform is Kubernetes, OpenShift, IIS, etc. -- stuff you typically provision with Packer, kubectl, or helm (and, this is the first blurring, because EKS, AKS, GKE, are often provisioned with terraform too)
Application is the business logic and software that runs your business, confusingly this is also often provisioned with kubectl or helm, and possibly packer if you're into the wonderful immutability model

Monorepos
I think the question you're asking relates to the concept of 'monorepo', where all the code needed to build and deploy your application is contained in a single repo and versioned together. In this answer, I'll mainly address the benefits, techniques, and drawbacks of monorepos, though you don't have to go deep into monorepo culture for this to apply, simply maintaining a single version for all your application, infra, and platform will have the same effect.
When making your decision you need to factor in whether you are working with an app per cluster, as is becoming more common in modern K8S systems, or a shared cluster. Another factor is which team owns the various layers of the architecture and how they will distribute their output.
Teams & Architecture
My experience comes mainly from large banks. In these organisations, we have hundreds (even thousands) of development teams and it is impractical and inefficient to expect each team to develop its own infrastructure and platform layers as well as their application. Distributing this effort to each team would be a governance and compliance nightmare and hiring that many people with the right skills would be impossible.
Instead, we have centralised teams for infra and platform, and they distribute their output to the development teams and we typically work with the cluster-per-app model. In modern orgs we work with a cross-functional team that has some skills in each of the areas we need, including DevOps, and each team member is also a member of a group that is focused on their speciality where they receive groupwide instruction.
We have 2 main models to provide access to the output of the centralised infra and platform teams:

The infra and platform teams can provision clusters on behalf of a dev team and provide the pipelines needed to deploy applications to it. This works well when a team has little or no DevOps capability, but it is far from being the best solution as it creates a dependency from the dev team to the infra and platform teams which usually causes impedance and added complexity.

The infra and platform teams can provide access to their git repos, and the dev teams can fork them into git submodules within their own mono-repos. This model is by far the most common and currently represents the best solution for highly-regulated industries. One of the biggest benefits here is that a dev team can make any changes they need to make to the infra and platform and issue pull requests for the respective teams to manage, without having to coordinate the schedules of the teams and thus reducing impedance.

Both solutions have their problems and complexities, but both are manageable given good engineering leadership and tool support.
Solution 1 would require separate repos for infra, platform, and app, and this usually results in a need to synchronise the backlogs of the respective teams, which quickly becomes a logistical nightmare as the number of teams increases. Solution 2 would require the implementation of 'initial trust' base infra with something like Open Policy Agent (OPA) rules to ensure that dev teams are complying with organisational governance requirements (e.g. verifying that a dev team isn't running a bitcoin mining operation, or publishing pro-trump/fascist/racist propaganda to our public-facing websites). OPA is an amazing tool when decentralising responsibility for the lower layers of an app while maintaining central control and I can't recommend it highly enough.
The app per cluster model allows you to use a monorepo that can completely rebuild your app from the ground up with minimal human interaction. The alternative architecture, multi-tenanted, completely disallows the monorepo approach and you must have separate repos for each of the layers.
Versioning & Configuration Management
The benefits of a monorepo are manifold, but one of the major benefits is having a single version for all layers of your application. One common problem in IT is called Configuration Management -- I refer here to the ITIL meaning, where we talk about which version of what software/hardware work together, rather than the folk-meaning which relates to configuration files.
This is such a hard problem to solve that many books have been written on the subject and ITIL has an entire discipline dedicated to it. Many of the larger banks I have worked for have entire departments dedicated to this discipline and to do it right usually requires the Enterprise Architecture department to keep continually updated models of every component and relationship in an organisation's entire IT estate.
The configuration management problem is largely solved with a monorepo. You'll never hear anyone say "I tried to deploy your software but you've written a helm 3 chart and our cluster only has helm 2 installed," or, "we patched the OS with the latest security updates and your application crashed." With a monorepo, as long as you have sufficient testing in place, your helm version will always be in sync with your helm files and your software will have always been tested with the latest security patches.
Configuration files
I think the OP mentioned config files in one of the comments so I'll address that briefly here too.
If you are building software that a client will run, then configuration files per client or installation are often necessary. In all other cases, they are not needed and should be avoided. Service discovery should be the responsibility of a service-mesh (my preference is Consul), not a configuration file. Secrets should be provided via dynamically generated properties files housed in an in-memory file-system only accessible to a single instance of your service. All other 'configuration' data should be managed by the platform, and it should be exactly the same for every environment. The days of placing the database name and port, and the database credentials, in your database.properties file were over a long time ago and you should move on now.
It isn't always possible to completely avoid configuration files. Where this is the case the files should preferably be 'owned' by the environment layer, not the application layer. These can be used to enable/disable platform services and control feature flags so as to customise a platform for your team's needs. These should mostly be interpreted by the deployment mechanism to ensure that only the things you actually use are deployed, rather than having things enabled/disabled at runtime, which is a big security no-no. You could use these configurations to down-scale a test environment so that its not as large as a prod environment, or to scale one up for load testing, or even to deploy to a less than six-sigma HA configuration for basic dev work.
Application-specific configuration is considered by many to be a code smell. It is often used to reference database names, ports, and the like. Your app should run in a container and should always run on port 80, so that doesn't need to be configured. Your database should always have a meaningful name and service discovery (DNS) should route you to the appropriate instance for your environment. Your memory and CPU constraints belong to the container, which is in the platform layer, not the app layer. You may have some truly app-level configuration points but that normally only true where you are deploying many instances of your apps for different clients (even if they are in your own organisation), in which case these are probably ok, but even then you should work on deriving them from the context in which you app runs, rather than hard-coding them in a config file unless absolutely necessary.
Environments
Another major benefit from the monorepo approach is that you gain the ability to 'print' new environments. With an application that is designed to take advantage of this (autoscaling, auto-healing, service-mesh, etc.) this will allow you to have a test environment that is 100% aligned with your production environment. Further, if time-to-market is a prominent enough factor of your business plan, you can spin up multiple test-environments and run your tests in parallel rather than sequentially.
This also means that developers can have a production-like environment to experiment with whenever they need one. They simply have to run the provisioning scripts for the app and away they go.
Of course, there is a cost to 'printing' environments and not everyone has the unlimited resources of a major bank to fall back on, so this capability must be managed, but if you engineer your environments to be automatically torn down when they're not being actively used then you will go a long way towards reducing these costs. OPA rules can also be used to ensure that boundaries are not overstepped and costs are managed efficiently without human interaction.
Team Dependencies
Personally, I believe that the major factor in the productivity of individual teams is the relationships they necessarily have with their peers in different layers. If there are 300 dev teams, an infrastructure team will be continually inundated with requests for changes that must be understood, managed, prioritised, and scheduled -- this is a major effort in large orgs and can leave a team waiting months, or longer, for changes that support their needs, leading to project delays.
There are no winners here -- if the centralised team is working on your problem then they're not working on someone else's, so either you are delayed or the other 300 teams are. It is a lose/lose situation and it simply does not work well at all. Monorepos with git's submodules (or similar structures) help fix this problem. Dev teams can make their own changes to their forked codebase and simply have it reviewed by an authorised representative of the centralised team, and pull requests can be used to manage this process.
Feature Flags
Of course, nothing is perfect. We don't yet have a silver bullet in software engineering and monorepos are no exception -- here be dragons. One fatal but unfortunately common misconception centres around the concept of feature-flags. These flags show what is finished, production-ready, and can be released, and what is a work in progress and should not yet be released to production. Many articles written by engineers suffering from the Dunning-Kruger effect suggest that these flags should be used to ensure that unfinished code should not be executable in a production environment. In reality, these flags should stop code being released to production at all if it is not yet ready. Non-production code should not be in production -- it is a huge security hole if it is released and this must be avoided.
No choice really
As far as I understand it though, the structure of your repos and software architecture tends to follow the structure of the organisation you work for. Ultimately, the deciding factor may not be related to software at all, but instead is related to your organisation's politics and maturity/sophistication/nature. You may experiment with counter-structures, but usually, these experiments fail to gain traction and are destined for the scrapheap.
If your organisation is split into separate teams with separate responsibilities then your repos will be split the same way. If you go for a cross-functional team approach then your repos will tend to be merged. If you just have one big team that's responsible for the success of the organisation, even if it has specialities then you will naturally end up with a monorepo and life will be good.
If your organisation is poorly structured, following the Harvard method espoused by various MBAs, of splitting the organisation into departments and assigning each of them a vice-president and then grouping them whenever there are more than 7 under a senior VP, then your path to working software will be fraught with problems and progress will be slow. I'm not trying to bash MBAs here, it's just that their models conflict with the primary activity of a lot of modern organisations, which is software development, and I feel that they need to modernise their approaches to take into account the opportunities provided by modern technology and big-data, rather than continuing a model that was developed by candlelight.
Wow, this has turned into a bit of a lecture. Sorry, I don't have time to make it shorter, but as I said up-front, this is a complex subject with a lot of variables. The last section is the most important though, so read that first -- it could save you a lot of pain.


using copy module with wildcards I need to copy a database backup file from a primary server to a standby server. Since I can't use synchronize, I was following this method:

copy file to local ansible server from primary,
then, copy it from local to standby.

The name of backup files are different on every machine. for example:

TEST1.0.db2inst1.DBPART000.20190729162630.001
TEST3.0.db2inst1.DBPART000.20180729172631.002

The playbook I was using looked like this:

 - hosts: primary
   remote_user: root
   tasks:
   - name: copy backup to local file
     fetch: src={{item}} dest=/data1/backup
     with_fileglob: /data1/{{dbname}}.0.db2inst1.*
  tags: transfer

 - hosts: standby
   remote_user: root
   tasks:
   - name: copy backup to standby
     copy: src={{item}} dest=/data1 mode=0755 owner=db2inst1
     with_fileglob: /data1/backup/{{dbname}}.0.db2inst1.*
   tags: ransfer

(The playbook is run with the command: ansible-playbook -i hostfile transfer.yml --extra-vars "dbname=TEST1")

However, this doesn't work; I get this output:

primary  : ok=1    changed=0    unreachable=0    failed=0    skipped=1    rescued=0
standby  : ok=1    changed=0    unreachable=0    failed=0    skipped=1    rescued=0

I think the name of files are problem. What should I do?

with_fileglog creates a loop with the fileglob lookup. As all other lookups it runs on the controller machine, not on the remote server.

Since I guess you don't already have those backup files on the controller in the path you referenced, your loop is empty.

As you have guessed, this approach will not work for your scenario. I suggest you have a look at the find module to create the list of files to fetch from your primary server.


Secure Jenkins X on kubernetes I have installed Jenkins X on kubernetes. I want to change the default installation and secure the Jenkins url, so that it will be not accessible to the public.

Can I change it to some port, or some way to have a firewall for only few IPs allow to access the Jenkins URL.

In my experience, the easiest way to do this is to put Jenkins behind an SSL-terminating proxy, such as Apache or Nginx.  You can also configure SSL within Jenkins itself, but I've found that using a proxy is usually a lot easier to set up and manage with regards to SSL.


Azure Security Center with Ubuntu 19 Has anyone successfully gotten the Azure Security Center agent to work with Ubuntu 19? My omsconfig.log is full of messages like

This event indicates that failure happens when LCM is trying to get the configuration from pull server using download manager null. ErrorId is 1. ErrorDetail is cURL failed to perform on this base url: uks-agentservice-prod-1.azure-automation.net with this error message: Stream error in the HTTP/2 framing layer. Make sure cURL and SSL libraries are up to date.

I am convinced that they are up to date

# openssl version
OpenSSL 1.1.1b  26 Feb 2019
# curl --version
curl 7.64.0 (x86_64-pc-linux-gnu) libcurl/7.64.0 OpenSSL/1.1.1b zlib/1.2.11 libidn2/2.0.5 libpsl/0.20.2 (+libidn2/2.0.5) libssh/0.8.6/openssl/zlib nghttp2/1.36.0 librtmp/2.3
Release-Date: 2019-02-06
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtmp rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: AsynchDNS IDN IPv6 Largefile GSS-API Kerberos SPNEGO NTLM NTLM_WB SSL libz TLS-SRP HTTP2 UnixSockets HTTPS-proxy PSL

In omsadmin.sh I see

#This is a temporary fix for Systems with Curl versions using HTTP\2 as default
    #Since 7.47.0, the curl tool enables HTTP/2 by default for HTTPS connections. This fix runs curl with --http1.1 on systems with version above 7.47.0
    #Curl http2 Docs Link: https://curl.haxx.se/docs/http2.html
    CURL_VERSION_WITH_DEFAULT_HTTP2="7470"
    CURL_VERSION_SYSTEM=`curl --version | head -c11 | awk '{print $2}' | tr --delete .`
    if [ $CURL_VERSION_SYSTEM -gt $CURL_VERSION_WITH_DEFAULT_HTTP2 ]; then
      CURL_HTTP_COMMAND="--http1.1"
    fi

But this doesn't seem to be honoured by OMSConsistencyInvoker which is a binary file. The above error in the omsconfig.log coincides every 15 minutes with that command being invoked from cron.

Problem solved, I think: the agent comes with its own libcurl.so.3 in /opt/oms/lib, it doesn't use the system one at all. Once I replaced that, it seems to be succeeding. Previously:

$ cd /opt/microsoft/omsconfig/Scripts
$ ./PerformRequiredConfigurationChecks.py
instance of OMI_Error
{
    OwningEntity=OMI:CIMOM
    MessageID=OMI:MI_Result:1
    Message=cURL failed to perform on this base url: uks-agentservice-prod-1.azure-automation.net with this error message: Stream error in the HTTP/2 framing layer. Make sure cURL and SSL libraries are up to date.
    MessageArguments={}
    PerceivedSeverity=7
    ProbableCause=0
    ProbableCauseDescription=Unknown
    CIMStatusCode=1
    OMI_Code=1
    OMI_Category=0
    OMI_Type=MI
    OMI_ErrorMessage=A general error occurred, not covered by a more specific error code.
}

But now:

$ ./PerformRequiredConfigurationChecks.py
instance of PerformRequiredConfigurationChecks
{
    ReturnValue=0
}


How to pass groovy output to pipeline variable I have the below code in the jenkisn pipeline:

   stage ("amd_distribution_input_transformation"){
        steps{
            script{
                    amd_distribution_input_transformation url: params.DOMAIN_DESCRIPTOR_URL, secret: params.CERDENITAL_ID
                }
            }
        }

groovy code:

def call(Map parameters)
{
    def CREDENITAL_ID = parameters.secret
    def DOMAIN_DESCRIPTOR_URL = parameters.url
    sh '''
        python amd_distribution_input_transformation.py
      '''
    }
}

in the amd_distribution_input_transformation.py some code is running, and at the end, it returns JSON object named 'artifacts_list'

my question is, how can I access this object in my next stage in the pipeline:

stage ("create bundle"){
            steps{
                script{
                    amd_distribution_create_bundle credential_id: artifacts_list
                    }
                }
            }

You can send the output of your python script to stdout and capture it in a environment variable and that variable can be passed to other stages in the pipeline. Also, the environment variables can be used inside the scripts.

You can do something like this:

def call(Map parameters)
{
    def CREDENITAL_ID = parameters.secret
    def DOMAIN_DESCRIPTOR_URL = parameters.url
    env.artifacts = sh(script: "python amd_distribution_input_transformation.py", returnStdout: true )

    }
}

Here, the output of the script amd_distribution_input_transformation.py will be stored in the variable artifacts and will be available further.


Is there a DevOps methodology aimed towards academia and students? I am still learning about DevOps, I wonder in Academia the workloads can often be quite heavy, I noticed there are variations on the DevOps methodology, e.g. DataOps and DevSecOps.

I wonder if there is something like StudentOps or similar to this especially targetted to the more scientific and computer science like subjects? I have noticed a similar question: DevOps in research and science projects? . But this does not explain why additionally one has not been developed for academia.

Is there a DevOps methodology aimed towards academia and students? No.


Organizing the build steps on AzureDevOps I am new on DevOps world and I am making my first build/release pipeline for a .net web application using Azure DevOps. On the picture bellow, I just add some build steps and create some tasks groups to avoid duplications on the builds (that are similar). I have the Run on agent that I called it as Qa Agent Job  which runs over the local build agent hosted on my build server (as a windows service).

My question is related with organization. I have some build steps here and I would like to organize it by sessions, for sample, Nuget Restore, Versioning, Build, Testing, Building services, Publishing artifacts etc. Is that a problem to organize it using the Run on agent item on build step? Should I lost any performance like "connecting on the server again" or "start a new process" or something like that?

Thank you.

You can separate the steps, especially as it would let you do divide the work among different agents. You can also leave them in a single agent if you wish.

The primary advantage I see is leveraging different pools for different sets of work.

With releases last I checked variable scope was limited (as of 2018) to the "phase".

I wouldn't necessarily worry about performance due to connections unless you've observed this issue and have more specific detail you can add.


What are the best logging/metrics reporting tools for use with OpenShift? OpenShift includes EFK and I'm looking into using that or ELK, but there are so many options (found a link with 50+ options) that I'm trying to narrow down. Ideally something free/cheap. What is the best logging and metrics reporting tool?

We are running a handful of microservices within OpenShift.

Some simple use cases of what we are looking to do:

As an admin, I need a report showing how many times a service was called in a given time frame.
As an admin, I need a report showing how many times a service was called successfully in a given time frame.
As an admin, I need a report showing the average time a service takes to process requests.
As an admin, I need a report showing the lifecyle of a specific message (UUID).  The lifecycle will show each service called.

The question hasn't had an answer in seven months so I will promote my upvoted comment to an answer:

Finding the lifecycle of a specific message across services is known as "distributed tracing". Distributed tracing requires more than just capturing logs. OpenShift uses Kubernetes which is part of cncf.io/projects which governs opentracing.io and has one tracing solution jaegertracing.io. As Jaeger is a sister project to Kubernetes you are likely to find it widely supported. Here is an openshift commons video about it.

I should mention that the question is likely to gather a lot of answers of the type "the only tool I have used is the best tool ever!". Experts will avoid answering such questions. On many sister StackExchange sites, often questions that get such answers get put on hold by moderators as been unhelpful.

I recommend that you don't look at new tools before fully exploring the tools that are near to hand or "most obviously stable and well supported". As EFK (sometimes more amusingly know as FEK as it sounds naughty) ships with OpenShift its worth really digging into what it can do. Then ask questions about how to do the things that it appears it cannot do. That is the style of question that helps everyone and that experts will gladly answer and you may then find that the most stable and widely support tool meets your needs. That is the best outcomes as it is going to be the easiest to get running, the least likely to break on upgrades, and the least likely to itself go down causing you late nights fixing it.


How to avoid "retry storms" in distributed services? A "retry storm" is caused when clients are configured to retry a set number of times before giving up, a retry policy is necessary because of packets loss will occur in normal operation of a service.

Take this example:

If for example the services as a whole were scaled to support 80,000 requests per seconds and run at about 80% of capacity, a spike in traffic that caused the service to receive 101,000 requests per second would cause 1,000 of those requests to fail.

When the retry policies kick in, you end up with additional 1,000+ requests, depending on where the failure was detected which would push the service as a whole up to 102,000 requests per second - from there your service goes into a death spiral doubling the number of failed requests every second.

Other than massive over-provisioning of services beyond the projected peak transaction, which would be inefficient.  What strategies can you employ to avoid "retry storms"?

It depends on what you're trying to avoid.

If you are trying to avoid any service interuption of something which is a genuinely critical service (I'm thinking in terms of "people will die if my API call is not appropriately served") the you need to just budget for the huge inefficiencies that come from vastly over provisioning dedicated resources. And yes they have to be dedicated, none of this allowing for traffic spikes stuff, multiple services spiking would thus cause an outage.

In the far more likely scenario that you're service going down would be inconvenient you can tackle the problem both from the client and server sides. Although it's worth noting that it's logically impossible to actually solve the problem of to much traffic because without processing the traffic (which consumes resources) you can't know if it's a retry, if it's a retry for a request that was successful but incorrectly handled by the client, if it's a DDOS, etc. But you can mitigate impact.

In the client code write sensible retry logic which has an upper limit and a mechanism for gracefully failing. That way you don't stick your users in an infinite loop of failing requests and you just give them an error telling them to try whatever they just did in little while.

For your server side infrastructure the simplest solution is to throttle. Hard limits on requests, especially if you can try and spread them logically based on your specific use case (ie. If you have a centralised service make some hard decisions, do you want to start blocking geographically distant requests which might be resulting in threads hanging server side? Or do you want to distribute your inevitable yet minor outage evenly? etc) It basically boils down to the fact that returning a 503 intentionally from a gateway is a hell of a lot cheaper than letting the request go through and sending a 504 anyway. Basically force clients to behave based on what you can currently provide and provide the correct responses so that clients can react appropriately.


What makes a difference between high-cost cloud providers and low-cost ones? I need to decide, for a small client and a small budget (around $500 a month) the proper Cloud Provider to have some compute instances to hold a web server, a database and an Apache server. I was wondering, seeing companies such as DigitalOcean, that offer compute instances on a very low price compared to Oracle, Azure, AWS, Google...

Why do companies choose these big corporations instead of the cheaper options when it comes to infrastructure? What is the big difference between such high and low prices?

@jasper, and welcome to DevOps SE!

While your statement can be confirmed by other voices, it's very hard to x-ray price policies of a company. It's a complex gameplay always driven by many factors and decisions and a question about pricing policies of Digital Ocean vs. AWS might be a good and on possibly well on topic question at Economics SE.

For example, one could argue to find facts supporting the evidence of Digital Ocean currently being cheaper that:

Docker, Inc. is in a situation where it cannot afford losing customers, so maybe they a cautious and prefer lower margin to attract customers.
Amazon is the market's prime with top notch customers from overall industry and government: stability, seriousity and credability are way more important than money, especially given that expenditures on information technology is only a (small) part of industrial budget planning. Also, Amazon has many more data centers and is as whole investing billions to AI technology ($0.8b in 2017, almost complete estimated market value of Docker Inc.), custom data center hardware, buying startups etc.

Please consider making your question more specific in the DevOps SE site context to get more answers.

This sort of decision is possibly best to elaborate together with the customer, collect their business requirements if price is not the only one requirement and assess risks.

Further reading/related business topic: due diligence.

Additional note: one might think, AWS would have way more customers than Digital Ocean but at least as of 2017 this was not the case.

Digital Ocean: 1 million users in 2017
AWS: more than 1 million users in 2017


Release based on restricted timeline with CI/CD There is possibility that client may want certain functionality or feature to be available based on certain season or specific date/time/zeo location. This could grow more complex based on code logic or database changes spreading or impacting to other screens/logical layers.

How can a team handle above situation with CI/CD or DevOps as a process?

Branch-by-abstraction and feature-flags/feature-toggles are the typical techniques allowing development of any combinations of features in various degrees of readiness in the same branch. For a proper CI/CD approach that should be the main/master branch. Each feature being developed can be tested independently by turning the respective flag on or off. "Releasing" the feature in that branch simply means that the respective flag is enabled in the source code

To be effective, all the relevant combinations of the feature flags' values should be subjected to the respective CI/CD QA verifications in this branch. The complication, of course, is the number of such combinations which can cause cost/time "explosions" in these verifications, especially in large scale projects with many features.

Grouping multiple feature flags under a single "super-flag" is an effective method of reducing the number of such relevant combinations. Turning on (in the code base) these super-flags are often time-driven according to a release schedule. Which can be either periodical or based on certain specified/seasonal time/dates, it doesn't really matter (as long the respective features meet the expected quality level by that time, of course, but that's a different discussion).

Turning on the super-flags is all that's needed for the software products that are continuously deployed. But for the products which can't be continuously deployed (like OSes or most of the embedded software products) release branches in which only the super flag(s) (or directly feature flags) applicable to the respective releases are enabled is the typical way to keep the release time/costs under control. They can also serve as the CI/CD branch vehicles for subsequent hot fixes, service packs and/or minor releases.

Release branches would be pulled off the main/master branch (or parent major release branch) whenever the quality level is good enough for the very next release and trouble to support the delta to the 2nd next release remains acceptable (as soon as the branch for the next release is pulled the parent branch can diverge, heading towards the 2nd next release).

And no, release branches aren't (necessarily) evil: they are not feature branches and they can completely avoid any massive branch sync/reparent/merge operation, thus remaining true to the CI principles.


Cassandra DB: store binary file record via shell While the dev team works on a REST service to manage binary data in Cassandra, ops would like to store local blobs directly from shell.

Is it actually possible via CQL/bash without a Cassandra driver? I could find so far only an example for Ruby.

The only way to do it with cqlsh is wrap binary file into CSV file and run cqlsh --some_args -e 'COPY table_name FROM binary_wrapped.csv', but in order to do so you have to generate a new primary key to use in CSV-file.

IMO: Since cqlsh is a wrapper around python client, your ops should just write simple python script for this operation, it would be much easier in many ways.


Impact on policy after deleting IAM user I have created one IAM User named "test" with S3 and Glacier Permission.

Now with user "test" i created one life cycle policy to move data from S3 to Glacier at every 5 days.

Now, i delete IAM user "test" because i don't need it anymore.

So my question here is, After deleting user "test", S3 Life cycle policy will work or not? OR will there be any impact on life cycle policy?

The policy will still exist.

The documentation lists what is deleted when a user is deleted, any resources that the user has created will persist.


Directory structure - how do I upload it as-is to an artifactory? I am new to Artifactory, and I wonder how to upload a directory structure containing many different binaries -- keeping the structure intact -- to an artifactory?

This can be done through the Jrog CLI. The CLI has a recursive option that looks like it will do what you hope.
jfrog rt upload --recursive artifactory-mirror/* artifact-repo/dir-struct/

The CLI also has some performance improvements for uploads that doing it through curl or a browser don't seem to get. Also, scriptable!
To include full directory structure you will want to use the flag --flat=false
jfrog rt upload --flat=false --recursive artifactory-mirror/* artifact-repo