Gitworkflow and testing short lived feature branches in Openshift/Kubernetes Recently my team started adapting git workflow for agile-based delivery, we've been using JIRA, TeamCity and various other CI tools to complete the pipeline. The workflow works great for monolith or SOA applications in JAVA.

The problem comes with microservices delivery pipeline targeted for enterprise environments such as Openshift.

With git-workflow, the expectation is feature branches once developed and tested will be merged to the develop branch, however, when there are numerous features being developed a developer needs a platform to test the service feature he just built before submitting a pull request.

With openshift/K8s we expose a service, let's assume each service built out of the feature branch looks like below.

myapplication-MIM-158848-feature1.myco.internal.au
myapplication-MIM-158849-feature2.myco.internal.au

Internally openshift should be able to allocate dedicated pods to test the services and they should be disposed once merged with develop branch, another problem with this approach is running integrated test or if services have dependency with consumer apis. Modifying the service name based on feature branch is not a good idea.

Does anyone know a better way of handling this delivery model in microservices based platform. Thanks

I'm not altogether sure if there actually is a problem you're trying to solve, or whether you are looking for confirmation that you are on the right track. Some thoughts:

  Internally openshift should be able to allocate dedicated pods

Sure. Add a post-build step in your CI pipeline which does the following:

Fashion a .yaml or .json description of that pod (or rather, DeploymentConfig). How/where you do that depends on your architecture - if each of your microservices has exactly the same port (i.e., all listening on 80 or 443), then you only need a central template for that, and basically only inject the name (containing the microservice name and the feature branch name) and the git source URL for it. The file can of course contain multiple OpenShift objects (i.e., the DeploymentConfig and a Route, maybe).
oc apply it, OpenShift will build an image and start a pod for it.

  and they should be disposed once merged with develop branch

Add a post-merge step which does the same in reverse, i.e. oc delete ... for every component of the original .yaml. Probably this is easiest if you give a branch-based label to each of the components, so you don't need to keep track of which you have applied in the first place.

  another problem with this approach is running integrated test or if services have dependency with consumer apis.

If a service involved in an integrated test has state, then you indeed need to fire up a new pod/volume for each test run. If your integration test depends on external state, then you also need to manage that somehow. Both of these aspects are true anyways, no matter how you architect your solution.

In these cases, you would probably do well to not start deployments right out of your individual CI pipelines, but do a "big" deployment when you start your integration test (i.e., deploy all individual microservices like described above, but also inject an ID for your current integration test run into all their names, so everything stays unique).

  Modifying the service name based on feature branch is not a good idea.

Why not? If you wish to have a pod running for every feature branch, then you need to do it that way, period.

It would certainly force you to do some kind of dependency injection (i.e., if you are testing one service A which needs another service B, then you have to tell A which B-feature it should take. But you need to do that anyway, in some form or fashion; as far as I can understand, you are trying to make the services (their branches) independent from each other, and I also assume from your question that you have one service per git repository (i.e., one service per branch) so you need to have them running per-branch.


GitLab CI/CD job without pulling LFS files? When our GitLab CI/CD job runs, it first pulls a Docker image and then git clones the entire repo.  Afterwards, the Build commands are executed in the Docker image, etc.

Is it possible to only clone the non-LFS files from the repo?  ... Or perhaps clone the HEAD of the master branch from only a specific subdirectory and it's subdirectories, etc.?

Yes, full cloning of LFS files can be restricted!  By default, GitLab will clone your repo into the CI/CD build directory.  To limit the clone from downloading the LFS files, tell it not to do it.  You do this by setting a variable in .gitlab-ci.yml like this.

# Other declarations etc above the specific job
jobname:
  variables:
    GIT_LFS_SKIP_SMUDGE: 1
# More job declarations, etc.

The "smudge" process replaces the link/pointers to the LFS files with the actual files.  By telling GitLab's CI job to skip the smudge, the LFS files aren't downloaded.

Placing GIT_LFS_SKIP_SMUDGE: 1 in a variables section at the top may make it apply to all of the defined jobs.


Is it possible to design conditionally executed tasks in Bamboo? For example, by branch name you might want to execute different tasks (regress testing in release branches instead of massive integration tests for example).

Is there a mechanism to do that in Bamboo's UI?

Atlassian Labs have now provided this functionality as an add-on: https://marketplace.atlassian.com/apps/1219706/conditional-tasks-for-bamboo?hosting=server&tab=overview

Only available for versions 6.7.1+


How to create high availability databases? With modern techniques, it's relatively trivial to create a clustered, high-availability solution for webservers by using docker (or comparable) to handle the front-end serving of pages to users.

e.g. for a LAMP stack, docker would handle the "A" and "P" parts.

Is there any way to make the database (e.g. mysql) work in a similar high-availability system with docker or comparable.

There are docker images for running opensource databases in high availability mode such as https://info.crunchydata.com/blog/an-easy-recipe-for-creating-a-postgresql-cluster-with-docker-swarm

You can also find examples of Microsoft SQLServer container for Linux being run with high availability on Kubernetes https://docs.microsoft.com/en-us/sql/linux/tutorial-sql-server-containers-kubernetes

Your question is missing an important concept from the devops perspective. If we run a particular version of a database (MySQL, PostresQL, SQLServer) on either a physical server, or a VMware ESX virtual machine server, or an openstack Xen virtual server, or in Docker in a laptop or in an http://cri-o.io container runtime in Kubernetes it’s “the same software”. If the high availability is based on running two or three instances of the database, and them opening tcp sockets to exchange data, and perform leader failover, then as “it’s the same software” it doesn’t matter whether we are running the database cluster as physical, or on a virtual machine hypervisor, or in a container orchestrator. That means you can learn how to configure the software for HA as containers on your laptop to then know how to configure it for HA on physical servers.

Obviously you might get different performance on different hardware or virtual machines or container runtimes. Typically  it’s disk bandwidth and memory that makes the biggest difference for a database not how the process is being run (within reason). Yet from the perspective of making a high availability setup if the software you are using to only needs normal disk and normal tcp connections you can expect it to work anywhere that lets you access disk and network. It is then a matter of making sure that you provide enough cpu, memory, disk and network of sufficient reliability to meet your NFRs and SLAs.

A lot of companies might setup virtual machine farms or container orchestrators while not thinking about the demands of relational databases running in high availability modes. For example if you configure a HA setup of two Postgres instances but they get placed in two different virtual machine on the same physical server then you have a very poor setup with respect to the likelihood of one machine fault killing both database servers. If you configure all disk to be NAS network drives then that might be fine to run application servers but terrible to run databases under load. Often people are only thinking about scaling up application servers and having many of them crash and restart on different machines as the aim without the same disk being attached at startup. Anyone trying to run a database cluster in such an environment is going to have many sleepless nights. This means it’s important to ensure that wherever you run your databases is sympathetic to the needs of databases to have relative stability. If the environment is optimised and operating as though everything hosted is a stateless application server workloads that is not fussy about being moved around and restarted at random you might regret setting up a database cluster there.


Why is the AWS EC2's spot price greater than the on-demand price? I was trying to provision spot instances via Ansible yesterday, and almost all my requests failed, even when I put my spot price == the on-demand price of that instance.

So, when I had a look at the spot pricing graph, I found something very interesting:

The spot price of the instance in us-east-1a is more than the on-demand price, which confused me. [in fact, ~5x times higher]

Aren't spot instances preferred for the low cost? If yes, then why is the price higher than the on-demand price?

According to AWS's docs:

  Spot instances provide you with access to unused Amazon EC2 capacity
  at steep discounts relative to On-Demand prices.

Also, does this mean that people are bidding over the on-demand price?  If yes, then why so?  Aren't they better off with an on-demand instance?

Or did I understand the concept of spot instances wrong?

This is actually a great example of people slightly abusing spot. People are saying 'Our workload is really important but we don't want to pay full on demand price', so they set a bid price higher than on-demand on the assumption that it is very unlikely to be terminated, but still want to get the 'cheapest possible' spot price on offer.

There have been cases where people enter, for example, $1000 (I've been told of at least one time this happened) because they want the benefit of the spot market. Of course naturally at some point the demand comes in and the spot price SOARS to make people pay higher than on-demand.

The way the spot market works is that Amazon have X instances spare capacity, and they count from the top down until they fill the need for all X instances. The 'price', then, is the lowest price at which they can fulfill those X instances.

So imagine Amazon have 10,000 instances - well they will count down to (say) $0.43 until they've got those 10,000 instances fulfilled. But if that supply suddenly drops to 100 instances, then maybe a few people put bid prices of $10,000 for their 100 instances, suddenly they'll be paying that $10k per hour.

Tl;dr Understand how spot works, and set a cap you are prepared to pay.


GUI over GitLab CI Environment Deployments With Multiple Instances? Let's say I have a SaaS style project (similar to Databricks).  We may make release 2.0.0 of this project, have automated testing and deployment to dev, automated promotion to nonprod, etc.

In production, I may have 200 instances of it running though.

What is the best way using GitLab CI to manage 200 instances/production environments? Obviously you wouldn't want 200 branches with deployments controlled by merges/etc... I assume.

How can you get a good overview over all of these environments and ensure they're deployed completely/view their versions and roll-out progress, ensure smoke tests completed for a successful pipeline/etc?

Its been implied to me that Spinnaker may be a good tool to plug this gap.  Can someone explain how that is if so?  Are there any better options?

Check out the GitLab Operations Dashboard feature (https://docs.gitlab.com/ee/user/operations_dashboard/)

There is a screenshot on that page, too, so you can get an idea of it.


Running additional commands on Docker image startup As far as I know, it's a good practice to use containers from Dockerhub instead of making my own ones. Most of those contain docker-entrypoint.sh script that is executed on startup. However, sometimes I need to do something on startup in addition to what is already described in docker-entrypoint.sh. Namely, I'm setting up a RabbitMQ container, I want to use a bind mount to save configs, but the config directory has to be owned by rabbitmq user. I wish I could execute chown -R rabbitmq /etc/rabbitmq on startup in addition to docker-entrypoint.sh, but I don't know how to do that.

I understand that I can simply clone the repo with Dockerfile and entrypoint from git, add something to the entrypoint and build docker image manually, but it looks like a crutch. I can also just attach to the container and execute that command manually which also looks like a crutch.

Simply write a Dockerfile which uses that image as a base. Then you can replace individual files (like the entrypoint) easily.

Also, instead of replacing, you could rename the original entry script, create your own, which does whatever it needs to do, and then just calls the original.

Somewhat like this:

Dockerfile:

FROM fancy-base-image:1.2.3
RUN mv docker-entrypoint.sh docker-entrypoint-original.sh
ADD docker-entrypoint.sh

docker-entrypoint.sh:

#!/bin/bash
echo Do something.
./docker-entrypoint-original.sh
echo Do something else

Obviously this is just a fake example, you'll have to adjust for directory and file names and such.


Rundeck ignoring node I'm trying execute a job on know node by rundeck server, but when execute it was ignored and job run on server local (Rundeck Server).
Anyone went through this situation?

Rundeck 3.1.3-20191204
Node configured in Project Setting
It's possible see the host in Node menu of Project
Rundeck server can access via ssh the node.

to solve, after make all setup to Rundeck know and access de node you need go to Project Setting > Edit Configuration > Default Node Executor and change to SSH. The problem occured because my project was created like LOCAL default node executor.


Jenkins Tool vs Slave local tool What is the difference between using maven as a tool versus having a slave that has maven installed?

I am using Jenkins on OCP and it came with a slave image for maven and not setup as a tool. I now need to install gradle and I am wondering if I should add it as a tool or make a slave with it installed.

Installing it as a tool allows Jenkins to manage the binary and ensure it's available for your build while also providing supporting pipeline syntax withEnv(). Having an agent with the binary allows you to shell out and run your commands using sh without withEnv(). It also allows you to decide where these binaries come from depending on your agent type (static, vm, container, pod).

Personally, I prefer the latter because I can control the updating of said binaries on the agent in a more automated fashion.


Triggering specific pipeline builds for monorepos in Jenkins I'm in the process of converting multiple repositories into a single repository, our CI tool of choice is Jenkins due to the conversion of multiple repository structures into a single one 2 main issues have arisen.

Build/test times have increased significantly as all the builds/tests have to be run for every single commit. This is partially alleviated by using a build tool, in our case we have gone with using Buck.
After all the tests associated with the committed code are run, I have a deployment Jenkinsfile for each project. How will I be able to only trigger the Jenkinsfiles for projects that need to be re-deployed? And if I am able to do so, is this a correct practice?

You can use the "when" block combined with the built in "changeset" condition to conditionally run only certain stages of your monorepo's pipeline. From the when.changeset documentation:

changeset-
Executes the stage if the build’s SCM changeset contains one or more files matching the given string or glob. Example: when { changeset "**/*.js" }

Here is an example Jenkinsfile using this strategy:

pipeline {
    agent any
    stages {
        stage('build matchengine') {
            when {
                changeset "**/matchengine/*.*"
            }
            steps {
                echo 'building match engine'
            }
        }
        stage('build posttrade') {
            when {
                changeset "**/posttrade/*.*"
            }
            steps {
                echo 'building post trade'
            }
        }
    }
}

, applicable to the monorepo project structure shown below:

 .(my-project)
   |-- Jenkinsfile
   |-- matchengine
   |-- posttrade
   |-- serverless
   |-- ui

This strategy won't scale past small codebases because it would be hard to keep track of which modules depend one each other. You would be better off using a build system like Bazel. Your CI job would simply issue a bazel build //... (build everything), and Bazel would calculate what actually needs to be built, and what needs to be tested. Further, there even exist bazel rules such as rules_docker and rules_k8s which can calculate which of your containers need to be rebuilt and pushed to a container registry, and which of your applications need to be redeployed to Kubernetes.


Determine the URL for a SCM trigger from inside the build My Jenkins is affected by a known Jenkins bug where it will trigger jobs that have an SCM poll set if a change to the shared library happens. I can't have this, so I thought I would get the trigger info from inside the job and if the URL was from the library repo I would abort the build. This has proven to be extremely difficult to do.
This doesn't work...
println ("Cause: " + currentBuild.causes[0])

or
print currentBuild.rawBuild.getCause(hudson.triggers.SCMTrigger$SCMTriggerCause)

or
def causes = currentBuild.rawBuild.getCauses()
for (cause in causes) {
    print cause
}

The first example barfs
org.jenkinsci.plugins.scriptsecurity.sandbox.RejectedAccessException: unclassified field org.jenkinsci.plugins.workflow.support.steps.build.RunWrapper causes
    at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.unclassifiedField(SandboxInterceptor.java:397)
    at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onGetProperty(SandboxInterceptor.java:381)

but doesn't allow me to whitelist it on the script security page.
The last 2 print thing hudson.triggers.SCMTrigger$SCMTriggerCause@3 but it seems the only thing I can do is call getShortDescription() and it prints Started by an SCM change.
If you know how to print more information about an SCM trigger such as the URL please share =).

Some bad news and some good news :)

Bad news:

not a bug. this is a feature (default behaviour)

I don't understand why this bug is still open, either regression, or miscommunication.

Good news:

During setup of Global Shared Libraries jenkins allows to mark whether you should trigger user jobs upon changes in the library.
And, you can also mark this during library usage in the job's groovy.

Please refer to:

https://issues.jenkins-ci.org/browse/JENKINS-41497
https://stackoverflow.com/questions/46819344/jenkins-shared-groovy-library-git-merge-triggering-jobs


Best way of binary repository for .NET projects I am very to new to .NET projects and I am wondering which is the best tool to store binaries of .NET project. How good to store binaries of .NET projects in artifactory and why? can some one answer generally Any comments can be helpful.

I'd recommend ProGet. If you're familiar with Artifactory you might find this comparison helpful: https://inedo.com/support/kb/1115/a-comparison-artifactory-vs-proget


jenkins fatal: ambiguous argument, unknown revision or path not in the working tree In jenkins pipeline, execute git command :

git log --format=%B --no-merges origin/master..origin/${currentBranch}

jenkins output error :

fatal: ambiguous argument 'origin/master..origin/XOPS-42_AddResultToJIRA': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'

but in my local computer i execute above git command, it work well, how to resolve above error in jenkins

From the git documentation around the command git rev-list:

git rev-list A...B

Try your command syntax like the above (i.e. origin/master...origin/${currentBranch})


Terminate single processing server based on SQS queue condition, but keep/ reinstate autoscaling size at 1 I have a processing server that processes requests in an SQS queue, only about 10 per day, each taking around 2 minutes to process.

Sometimes the process gets "stuck" and it would be ideal to terminate the single server in the autoscaling group, but keep the autoscaling size to 1, as the problem is often resolved with a fresh server created by autoscaling.

Can this be done with autoscaling configuration alone, or do I need to create a separate process with step or lambda functions?

Disclaimer: this is certainly not the best solution for backend workers, I'm just trying to be creative within the given constraints.

First, you'll need to be familiar with SQS's visibility timeout (documentation) for this to work.

Let's assume the following, then I'll walk through how it might work:

We only have 1 request (henceforth "message") coming in at a time (10 per day, hopefully that's feasible)
We can take up to 12 minutes to process a message in the worst-case scenario

Next, some configuration changes:

We configure the ASG to scale desired count to 0 when ApproximateNumberOfMessagesVisible = 0 for 5 minutes.
We configure the ASG to scale desired count to 1 when ApproximateNumberOfMessagesVisible >= 1.
We set all messages VisibilityTimeout to 10 minutes.

A failure scenario might work out like this:

At 0 minutes, 1 message is being processed by the worker. There are now 0 other messages in the queue, ApproximateNumberOfMessagesVisible = 0
At 2 minutes we are now assuming that the worker is now stuck processing its in-flight message. No action is taken yet.
At 5 minutes our ASG scale-down event triggers, and desired capacity is set to 0 (because there is no pending work)
At 10 minutes, SQS automatically re-enqueues the message due to the visibility timeout. ApproximateNumberOfMessagesVisible = 1
Shortly after, our ASG scale-up event triggers, and desired capacity is set to 1 (because there is pending work)

The 3-minute and 5-minute gaps are somewhat arbitrary, they merely highlight the need for a brief waiting period for such a fickle system.

Reference:

https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-visibility-timeout.html

https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-available-cloudwatch-metrics.html

HOWEVER...

The best worker is one that exits its process instead of getting stuck. That would allow for process monitoring on EC2, or for containerization. Were your worker an ECS service, it could scale up when there is work, recover from failures, and scale down afterwards. Refactoring your worker might be less effort than ongoing manual intervention for my hacky solution, and save you money!


Using DevOps pipelines to create distribution bundles of applications We want to create bundles of our applications in order to distribute to clients etc. An application can have more than one component. Ex: an application can have a spring server and a scheduler so in order to distribute this application both jars will have to be given to the client.

We want to use Docker/Docker compose to accomplish this. Ex: we give a client a bundle containing a Docker compose file, and he/she just runs docker-compose up to start the application.

We currently have a Jenkins pipeline that when run will produce a zip file containing .tar files of all the Docker images of the components. There will also be a start.sh (or start.bat) file that, when run, will load all the Docker images and run docker-compose up.

Some of these components have properties that the client needs to configure according to their environment (ex: SCM URL). If the component is Spring/Java based, we would externally mount the application.properties file into the docker image in the docker-compose.yml

This all works, I just want to know if the current process we are using is compliant with industry standards or is there a better way of meeting the requirement? Any open source tools that would simplify the process?

Add a config file with parameters used in install/run script. You can look at Harbor (Github, install docs) as an example of doing just that (and they have a configuration migrator).


Azure managed NFS shared storage for Azure VMs What is a reliable way to implement Azure managed NFS shared for multiple Azure VMs?
The idea is an alternative to AWS EFS for multiple EC2.
I only found the following solutions with disadvantages:

Azure NetApp files - minimum of 4TiB and it asks to contact sales.
Azure shared disks - it has "maxShares limit".
Azure files NFS - It's in preview.
SoftNAS on Azure - It doesn't make sense to provision 2 SoftNAS for HA for a small amount of storage.

So how do companies get an alternative solution to AWS EFS on Azure which makes sense?

You can use Azure Files
https://docs.microsoft.com/en-us/azure/storage/files/storage-files-introduction
Limits for it can be found from here
https://docs.microsoft.com/en-us/azure/storage/files/storage-files-scale-targets


Automate Application Upgrade in Azure VMs Sorry if this question is on the wrong Stack Exchange or too vague. I am really new to DevOps and got a big task and don't want to start on the wrong direction.
We have many Linux-based (Redhat) VMs in Azure and we have Splunk installed on them and I am looking for a way to automate Splunk upgrade.
I don't want to upgrade Splunk every-time there is a new version, but I was wondering if I can selectively upload a new version somewhere in Azure and have a Runbook that detects the new version and execute the Runbook to upgrade the Splunk app on each VM?
Upgrading Splunk requires a custom script (to make sure the new version is compatible with our environment) which makes this automation harder to design I suppose.
Is there a better way or option to perform Splunk upgrade? Preferably in a rolling fashion.
Any help would be appreciated.
Thanks.

Do you think this might help you?
https://docs.microsoft.com/en-us/azure/automation/update-management/update-mgmt-overview#:~:text=You%20can%20use%20Update%20Management,and%20in%20other%20cloud%20environments.&text=For%20multiple%20Azure%20VMs%20by,page%20in%20the%20Azure%20portal.
If your VM is disposable and you can replace it with a new VM then you can also use VM Image galleryfor Linux where you can have images for a VM or multiple VM types and you can update it at one place using Azure DevOps and then update the underlying VM image
https://docs.microsoft.com/en-us/azure/virtual-machines/linux/image-builder-gallery-update-image-version


Amazon Web Service (AWS): Spot request active followed by cancelled terminating 5/31/2018, 5:21:51 PM   fleetRequestChange  cancelled
5/31/2018, 5:15:46 PM   fleetRequestChange  cancelled_terminating
5/31/2018, 5:15:46 PM   fleetRequestChange  active
5/31/2018, 5:15:36 PM   fleetRequestChange  submitted

I am trying to create a Spot instance. However, it's cancelled right after it's being activated. Tried several times.

What is the problem??

Seems like the initial limit for instances is set to 1. Solution is to request a limit increase which is not intuitive by the provided instance cancellation.

AWS Service Limits

  You can take the following steps to request an increase for limits.
  These increases are not granted immediately, so it may take a couple
  of days for your increase to become effective.

  To request a limit increase

  Open the AWS Support Center page, sign in if necessary, and choose
  Create Case.

  For Regarding, choose Service Limit Increase.

  Complete Limit Type, Use Case Description, and Contact method. If this
  request is urgent, choose Phone as the method of contact instead of
  Web.

  Choose Submit.


How to serve RESTful APIs? Background:
  I am developing a Web application for Employee Management. I use Laravel for building a RESTful API, and (maybe) ReactJS for the frontend.

My problem:
  I am confused on how to deploy my application. My current plan is to use the command Laravel artisan serve for the RESTful API, and use Nginx for serving HTML files. But is Laravel artisan serve possible for production? How to serve RESTful API?

Some other thoughts:
  As far as I know about Node.js applications. We can use ExpressJS for building a RESTful API, then pm2 start app.js for production. But Node.js is good at serving dynamic contents. If I build the RESTful API with Laravel or Spring Boot, or Ruby on Rails (Sinatra), I don't think a command like pm2 start app.php (app.java or app.rb) will work (here pm2 is only for convenience, I know it is only for a Node.js app) because PHP, Java or Ruby is quite different from Node.js. I am afraid I am missing something fundamental here.

The confusion might come from the concepts behind the tools you are using. So let's start with a little definition.

  Laravel and Artisan

Laravel

Framework for building a full MVC Web application in PHP. It is the equivalent of Spring-boot with Java or Rails with Ruby.

Artisan

Laravel's console driven interface for development. It provides tools such as the web server coming with PHP 5.4+. When using the "serve" command a simple web server will start. This is perfect for development, but I would suggest you use a server that gives you proper configuration flexibility in production such as Apache or nginx.

  Your other thoughts

Java/Ruby/PHP/Javascript

Are programming languages. Node.js is like an ecosystem that allows you to run client/server/middleware/tools Javascript based applications. Someone did a great job of comparing it with a Java ecosystem. Right here.

  But is Laravel artisan serve possible for production?

In theory, yes. It would require your development setup to be in production as well. Hence from this assertion, the answer becomes self evident for security, operability, configuration and performance to name a few. You will need a different configuration for production.

  How to serve RESTful API?

No matter which language you use, choose a library or a framework that will give you the tools to implement your APIs. Since Laravel is your framework, you can take a look at this tutorial to get started.


devops - Optomizing gitlab cicd pipeline's caching I have a gitlab yaml I've been working on for a bit and while everything works it seems like each job takes quite a bit longer than it should. I tend to think the culprit is that npm files are fetched every time rather than sharing the node_modules cache between jobs.
Short of creating my own docker image, how can I optimize this yaml in a way that reduces the bottlenecks ?
gitlab .yaml
image: node:10.19.0 # https://hub.docker.com/_/node/
# image: node:latest

cache:
  # untracked: true
  key: my-project
  # key: ${CI_COMMIT_REF_SLUG} # per branch
  # key:
  #   files:
  #     - package-lock.json # only update cache when this file changes (not working) @jkr
  paths:
    - .npm/
    - node_modules
    - build

stages:
  - prepare # prepares builds, makes build needed for testing
  - test # uses test:build specifically @jkr
  - build
  - deploy

# before_install:

before_script:
  - npm ci --cache .npm --prefer-offline

prepare:
  stage: prepare
  needs: []
  script:
    - npm install

test:
  stage: test
  needs: [prepare]
  except:
    - schedules
  tags:
    - linux
  script:
    - npm run build:dev
    - npm run test:cicd-deps
    - npm run test:cicd # runs puppeteer tests @jkr
  artifacts:
    reports:
      junit: junit.xml
    paths:
      - coverage/

build-staging:
  stage: build
  needs: [prepare]
  only:
    - schedules
  before_script:
    - apt-get update && apt-get install -y zip
  script:
    - npm run build:staging
    - zip -r build-staging.zip build
  # cache:
  #   paths:
  #     - build
  #   <<: *global_cache
  #   policy: push
  artifacts:
    paths:
      - build-staging.zip

deploy-serv1:
  stage: deploy
  needs: [build-staging]
  tags: [linux]
  only:
    - schedules
  #   # - branches@gitlab-org/gitlab
  before_script:
    - apt-get update && apt-get install -y lftp
  script:
    # temporarily using 'verify-certificate no'
    # for more on verify-certificate @jkr: https://www.versatilewebsolutions.com/blog/2014/04/lftp-ftps-and-certificate-verification.html
    # variables do not work with 'single quotes' unless they are "'surrounded by doubles'"
    - lftp -e "set ssl:verify-certificate no; open myserver.org; user $LFTP_USERNAME $LFTP_PASSWORD; mirror --reverse --verbose build/ /var/www/domains/dev/projects/my-project/build/; bye"
  # environment:
  #   name: staging
  #   url: http://dev.myserver.org/projects/my-project/build
  when: manual
  allow_failure: true

build-production:
  stage: build
  needs: [prepare]
  only:
    - schedules
  before_script:
    - apt-get update && apt-get install -y zip
  script:
    - npm run build:staging
    - zip -r build-prod.zip build
  # cache:
  #   paths:
  #     - build
  #   <<: *global_cache
  #   policy: push
  artifacts:
    paths:
      - build-prod.zip

deploy-serv2:
  stage: deploy
  needs: [build-production]
  tags: [linux]
  only:
    - schedules
    # - master
  before_script:
    - apt-get update && apt-get install -y lftp
  script:
    - sh deploy-prod
  environment:
    name: production
    url: http://www.my-project.org
  when: manual
  allow_failure: true

I highly recommend using npm ci instead of npm install. The slow part of installing npm modules is figuring out everything that needs to be installed. If you commit your package-lock.json and use npm ci, it skips that part entirely and just looks at your lockfile to download what's needed. It's significantly faster.


Vault `SSH-CA` Elliptic Curve rather than `RSA` Is it possible to use ECDSA(Elliptic Curve Digital Signature Algorithm) or EdDSA( Edwards-curve Digital Signature Algorithm) instead of RSA when configuring SSH-CA on HashiCorp Vault?

I think the answer should be yes to ecdsa but no to ed25519, but that it is currently no to both.
You can externally generate an ecdsa keypair and load it into ssh/config/ca but vault-1.4.3 cannot sign using it. I haven't tried ed25519.
Golang's ssh code (which vault uses) doesn't appear to know how to sign with ed25519 but should be able to sign with ecdsa. However, I think there's an issue with an inappropriate default being set elsewhere that means it is always being told to use an rsa-based signing algorithm and hence doomed to fail for ecdsa keys. See:
https://github.com/hashicorp/vault/issues/8414#issuecomment-656145432
If I am right, it looks like an easy fix so it may be available soon.
This is possible as of August 2020, see https://github.com/hashicorp/vault/pull/9824, tested with vault 1.6.2


Access private DockerHub repo in Docker Swarm I am trying to deploy a docker swarm using Digital Ocean's docker-machine driver.

This is currently a single node that contains several public DockerHub images, and one private.

I am having difficulties in configuring it to accept the private image properly.

I have SSHed into the node, and used docker login. After that, I was able to properly docker pull the private image.

But, when trying to run docker stack deploy -c myapp.yml myapp, I am getting this warning:

image me/myapp:latest could not be accessed on a registry to record
its digest. Each node will access me/myapp:latest independently,
possibly leading to different nodes running different versions of the image.

What do I need to do to properly login to a private registry in docker swarm mode?

You need the --with-registry-auth flag, e.g.:

docker stack deploy -c myapp.yml --with-registry-auth myapp

From the docker stack deploy --help output:

--with-registry-auth     Send registry authentication details to Swarm agents


Can I rename an RDS instance? I have inherited a collection of RDS MySQL instances with random (useless) DB Instance names.

I often have to dig into the tags section to find out which project they belong to.

One of the settings I can modify in an RDS is the DB Instance Identifier.

If I change this, what will happen besides for the name change? Will it affect anybody's code? Or is it a harmless change?

These are live systems that I cannot afford to bring down.

Yes, you can rename an RDS instance... at your own peril.

  When you rename a DB instance, the [DNS] endpoint [hostname] for the DB instance changes, because the URL includes the name you assigned to the DB instance.

  http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_RenameInstance.html

Your application code requires configuration changes to be coordinated, for this reason.

See the link above for the other implications.  This is a supported and generally safe operation, but only in your maintenance window, because there will necessarily be a brief disruption as the DNS is changed and your app code is cycled in with the new database hostname.


Should I build my own pipeline or use GitLab for Kubernetes? My company is moving from an architecture of "many projects on a single webserver" to a Kubernetes cluster with multiple different hosted web applications in a single cluster.

In order to make integration of new customers easy, I want an UI to clone a 'template' code repository and give it a new life. When creating a new client from this interface, it would also create a Kubernetes deployment in a staging cluster, clone a database and possibly do other automated tasks.

I thought of rolling my own interface that would manage my Kubernetes deployments so I could tailor it exactly to the requirements, then I looked up GitLab and figured somebody was probably doing just that already.

Having never used GitLab, would it suit my use case?

Once the initial setup is done, I would actually use it for its intended purpose so I know it'll be good for that at least.

Question 2: Can I edit Kubernetes deployment configs from GitLab, such as changing replica count or RAM limit while it's running?

One could just run kubectl apply -f <file>. When one updates an application, one could just run this command. A CI is basically able to run scripts. As GitLab is just a CI and a Git repository one could just run kubectl in a gitlab-ci.yml pipeline.


What are the correct volume maps for zabbix running on Docker? I want to run zabbix inside Docker and there are images for MySQL, Nginx and zabbix server ready to go.

I know that to keep the containers stateless I need to map a volume on the host to the container. I know that for MySQL the important container volume to map is

/var/lib/mysql

But what are important parts of zabbix to backup so that the container can stay truest stateless?

From what I can tell it seems to be all the directories under /var/lib/zabbix.

There is a list of all the important paths Zabbix containers use: https://www.zabbix.com/documentation/3.0/manual/installation/containers
(scroll down a bit until you find "Volumes" part).

If I were you, instead of bind mounting these directories I would build a derivative image from the official Zabbix image and run statements like "COPY my_alertscript.sh /var/lib/zabbix/alertscripts/" in the build phase.

If, for some reason, baking your changes to the image itself is not possible, then you may still want to prefer named volumes instead of bind mounting. See: https://docs.docker.com/storage/volumes/ to read about its advantages.


What are DevOps & Ops user stories, or elements in scrum context? What type of user stories should Operations write down while being asked to be a stakeholder in a scrum project?

Following this template:

"As Operations engineer, I want to.. in order to.."?!

My first idea is for example to come up with for example expectations to quality standards i.e. acceptance criteria to deliverables, like "as an operations engineer, I want to get deliverables accompanied with installation and validation automation scripts in order to enable fast and transparent deployments", but is there more to it?

Make sure that your stories are not centered on Operations.  Remember that DevOps is a culture not a role.  Is an "Operations Engineer" really the stakeholder for the story?
Think of the features and business value they provide.  As a stakeholder I want feature so that business value. If you are struggling to identify the business value why are you doing it?
Stories are supposed to be "vertical slices" as opposed to "horizontal slices".  This means that stories should focus on an entire feature, not a layer.  This fits very well in a DevOps culture as your stories can cut across the traditional Development and Operations layers.

You already have some interesting features such as the quality standards. Now you just need to identify who the end stakeholder is, what the business value of this would be, and ensure that you write the story in such a way that people with development, operations, and possibly QA skill sets can work on it.


Listing filesystem differences between a Docker image and one of its ancestors I asked a similar question a few minutes ago, and I found the docker diff command that shows the differences in the file system between a container and its base image. But I realized afterward my need is more complex than that.

I have a Docker image, and I know that Docker image was build from a given base image. Is there a way to list the filesystem changes between those two images?

Here is an example:

sh$ sudo docker image ls
REPOSITORY                    TAG                 IMAGE ID
rogueimage                    latest              77a21f689dcd
ubuntu                        18.04               47b19964fb50

I know rogueimage:latest is based on ubuntu:18.04 but I would like to know the filesystem difference between the two images. How to do that?

There was already a stackoverflow question about that topic, but the answers didn't really help me. The container-diff tools was the most promising solution, but I would have preferred not relying on a third-party tool.

Finally, since docker can export an archive of a container filesystem, I end up with this solution:

BASE_CID=$(sudo docker create ubuntu:18.04)
TARGET_CID=$(sudo docker create rogueimage:latest)

sudo docker export "${BASE_CID}" | tar tv > base.filelist
sudo docker export "${TARGET_CID}" | tar tv > target.filelist

Then using your favorite diff tool you can examine the difference:

vimdiff {base,target}.filelist


How to scale-down in a multi-tenant environment? Cloud environments in AWS allow for multi-tenancy managed by the user himself, classic example are container orchestrators such as ECS or Kubernetes.

When you have two services, one needs memory another cpu and you put these in a single cluster. Then scaling-up is relatively trivial. Each time you need more capacity in terms of either cpu or memory, add more capacity. Since EC2 capacity means units in cpu and memory both.

Scaling up based on a single metric can very easily achieved using CloudWatch Alarms.

When scaling down, in order to reduce cost it requires to take into account both memory and cpu limits and not let any of the two drop below the required amount.

Since unfortunately CloudWatch Alarms do not allow to use boolean logic or take into account multiple metrics.

What is a good way to implement scale down of capacity for an auto scaling group?

Autoscaling is a good case for machine learning

This is a hard problem to do well.

What you really want is something like Nest Thermostat for your EC2 infrastructure.

There are (aforementioned) multiple dimensions of resource demand/limitations.

CPU
memory
disk space
disk IO
network IO
concurrency/latency/queue depth

There are multiple indicators of demand (in addition to the above).

concurrent unique visits/sessions
pages per visit/session
rate of engagement/interaction feature usage

There are multiple common patterns of changing demand over time.

daily user demand cycles
weekly user demand cycles
monthly ...
annual ...
special event/days
DDoS load
media/marketing exposure traffic spikes

There are multiple financial decision factors.

does revenue scale with traffic? How? (How conservative do you need to be?)
is there a hidden cost to control (transaction costs, limits)?
what's the cost model of scaling? (things in a pipeline scale together, things in a load-balanced cluster scale independently)

Before long, if you try to hand-optimize on hand-selected features you're going to have a monster of technical debt that is possibly more complicated than any other logic in your site. Amazon makes more money when you err (with a large margin) on the side of caution, so their tools will probably never get close to what you want.

Instead, choose an architecture/technology stack that can grow/scale so you don't have to get it exactly right the first time. Then pick a few factors which you think are obvious. Then try to come up with a way to sort multiple representative possibilities in order of preference. Then collect some real world data covering all those points. If you're lucky, a simple obvious hand-coded solution will jump out at you from looking at the data. If not, code up something that will give you an approximate model f(x1,x2,x3,x4) --> y * app nodes, using an appropriate algorithm.

I bet you didn't think this one was going to be so much fun!


Using tmux to host a server So I have some API server that I need to be up 24/7. Is it a bad idea to open up a tmux window and run the server application? When you run a binary in tmux, the binary stays running even when you close the tmux window. The binary also stays running when I disconnect from the SSH session. I like running the server application in tmux because I can easily just attach to the tmux window and have access to a shell where the executable lives. The server constantly prints log messages to standard output, meaning I like to be able to read those messages in my tmux window. However, there is something that just seems off about this method of running a server application. I would prefer if the server could somehow run in the background, but I also want to be able to see the logs that the terminal is printing out at any given time. So, to summarize, is it a bad idea to run a server application in tmux? If so, what do I do instead so that I can still read the server application's logs?

What you want can be done. I've seen it done with GNU screen when Tmux wasn't around yet. However, I would not consider it good style to do it.

There are better ways to do this. Daemons are simply "normal" processes being run in a particular way. That particular way usually means to fork "twice" (so that the parent/child relationship gets broken) and plugging stdout, stdin and stderr to some file (or /dev/null if no logging is desired).

There are tools aplenty out there to achieve what you want while also ensuring that there is a policy in place if the process unexpectedly goes down. After all a service should be available as close to 100% of the time as possible.

Read up on:

daemontools
supervisor
daemonize
runit

Arguably your system will already have one of these or a very similar tool installed. But you don't give enough information for me to provide more detail.


Host a docker registry native I want to host my own docker-registry server to be independent from docker-hub.
I want to run it native on my server, but I can only find documentations, which are using running it inside a container.

My question:
Is it possible to install docker registry over a package or do i have to run it in a container?

I take for granted you are referring to the registry server released by docker.com. The documentation clearly states on the very first line:

  Before you can deploy a registry, you need to install Docker on the host

Since the registry is the main product of the docker/distribution opensource project, you could fork and see how you can modify this as a standalone install not requiring docker, but I have no clue if it is possible and that would probably end up being a bad idea.

There are other existing products you can install 'standalone' that will let you manage your own registry. But they all come with some degree of requirements (java, ruby...). I have experience with 2 free products I use daily:

Sonatype nexus3 oss is a multi format repository manager supporting docker registries (requires java).
gitlab-ce has support for container registries in each projects if you configure it (ruby required).

The latest one is probably an overkill if you only need a basic registry and not all of the git, wiki, etc. ressources.

I recently came across a fairly recent review article listing other products I have no experience with:

Harbor from VMware
Portus by Suse

It also mentions as a bonus the use of Openshift as a registry, but even minishift for something as light as possible will require to install docker.

All of these solutions can probably meet your 'native run' requirements, but they are all much more complex to install and maintain than simply poping-up a registry:2 container.


Problem Restore EF Core 2.1.1 in VSTS, Hosted VS2017 I have a project with Asp.Net Core 2.1.1, so I have to force the runtime to 2.1.1, when I added the EF Core 2.1.1 packages (SqlServer, Design and Tools). Locally the project performs well, however in the process of Build in VSTS using the Hosted VS2017 the build process fails, it does not restore the packages by dependency conflict. I installed the packages he asks for:

Microsoft.EntityFrameworkCore After New Failure
Microsoft.EntityFrameworkCore.Abstractions and then
Microsoft.EntityFrameworkCore.Analyzers ...

This is the way? Go running and failing build a build until you discover all the packages that need to be placed as dependencies? Anyone have any tips to solve this another way?

My build process is like the .net core of VSTS.

This is my .csproj file

<Project Sdk="Microsoft.NET.Sdk.Web">
  <PropertyGroup>
    <TargetFramework>netcoreapp2.1</TargetFramework>
    <AutoGenerateBindingRedirects>true</AutoGenerateBindingRedirects>
    <RuntimeFrameworkVersion>2.1.1</RuntimeFrameworkVersion>
  </PropertyGroup>
  <ItemGroup>
    <PackageReference Include="Microsoft.AspNetCore.App" />
    <PackageReference Include="Microsoft.EntityFrameworkCore" Version="2.1.1" />
    <PackageReference Include="Microsoft.EntityFrameworkCore.Abstractions" Version="2.1.1" />
    <PackageReference Include="Microsoft.EntityFrameworkCore.Analyzers" Version="2.1.1" />
    <PackageReference Include="Microsoft.EntityFrameworkCore.Design" Version="2.1.1" />
    <PackageReference Include="Microsoft.EntityFrameworkCore.SqlServer" Version="2.1.1" />
    <PackageReference Include="Microsoft.EntityFrameworkCore.Tools" Version="2.1.1" />
    <PackageReference Include="Microsoft.EntityFrameworkCore.Tools.DotNet" Version="2.0.3" />
  </ItemGroup>
</Project>

I also could not restore packages after upgrading to 2.1.1. I got multiple errors similar to this:

Detected package version outside of dependency constraint: Microsoft.AspNetCore.App 2.1.0 requires Microsoft.EntityFrameworkCore (= 2.1.0) but version Microsoft.EntityFrameworkCore 2.1.1 was resolved.

I solved my problem by explicitly specifying version 2.1.301 in the build definition.


Log monitoring/metrics dashboard with images/video I have a system that generates both textual log/metrics data, as well as images. I'm looking for some tool that would allow me to track the log data (a la Kibana or Grafana), but also let me embed the images, ideally in some way that was easily correlated with the logs based on time. That is, I'd have the normal charts derived from the log data, and if I click a point on a plot, it would show me the image from that time.
Does any system already exist that would let me do this? Free obviously preferred since it makes it easier, but this is ultimately a corporate project, so a paid tool also works.

In terms of embedding in an external page, you have some options out of the box with both Kibana (docs here) and Grafana (docs here).
If I understand correctly, you are trying to generate realistic clickable thumbnails - in that case, you should try Grafana first since it better supports server-side image rendering


Software/devops engineer's personal dashboard recomendations Can you recommend a free tool or GitHub project that is designed for an individual to keep up/aware of their daily tasks and operations?

For example:

Show perf metrics from running azure VMs in graphs or tables
Show events from application insights
Ability to configure workflows that execute some code (like javascript on node) with a click of a button
Call a rest API with parameters with a click of a button
List my work items, PRs, build status from VSTS
Show a filtered view of my mailbox on Office 365 that I configure to show important mails from the team or boss

I'm hoping that this tool would either be browser based or work as a Windows client.

I'm sure no tool exists that meets all these requirements. So if I had to build something myself then are there any good platforms that allow easy customization to make these things happen?

Any pointers?

For all of your read-only dashboard needs, there is Grafana with multitude of plugins, data sources and visualizations (tables, charts etc). You can also program your custom data sources pretty easily with official "simple-json-datasource" plugin.

By default, Grafana does not support buttons or other kind of form/action stuff on its own. Grafana is all about visualizing data, not modifying it. Yet, while I haven't tried it myself, Grafana supports Text/HTML/Markdown panels, I think you can implement any JavaScript logic inside those panels.


How to use Jenkins to manage a repository from Github with a submodule from Bitbucket? For my Jenkins build, I need to manage a Github repository to automate building a project. However, the issue is that this repository needs to pull a submodule from Bitbucket. This requires two difference instances of authentication, one for each service. As of now, Jenkins is only allowing the use of a single set of credentials, using a username and SSH key. Is there a way to specify a different set of credentials for each pull?

It is a convenience that Jenkins will pull from git for you. With such a complex setup as needing to pull from two repos you can simply use a pipeline job that runs "sh" to explicitly git checkout the code and pull the submodules:

sh "git checkout ${custom_env_var} ${custom_parameter}

The good news with this slightly manual approach is that you can debug the sh commands to run on a dev server and cut paste them straight into a pipeline. Just start a “sh” session (usually it runs bash) and test the commands.


What is the proper way to fix `error: did not detect an --insecure-registry argument on the Docker daemon` while running `oc cluster up`? According to this blog one has to change the docker daemon:

/usr/bin/dockerd --insecure-registry 172.30.0.0/16 -H fd://

but I wonder why the error occurs and how to fix it properly?

This error happens because the oc cluster up command spawns a Docker registry. The 172.30.0.0/16 IP range is probably corresponding to your Docker bridge network. As the registry will run in a container with an unpredictable IP address, you must indicate the IP range (--insecure-registry 172.30.0.0/16).


What is the difference between the traditional Development and Operations Model and Site Reliability Engineering? "SRE is what happens when you ask a software engineer to design an operations team." – Site Reliability Engineering

Since Google's Site Reliability Engineering Book was released, on more than one occasion I have been told that SRE is an extension of the existing Operations or Application Support model.

We've had a couple of questions that defined differences between Sys. Admins, DevOps Engineers and Site Reliability Engineers:

What is the difference between Sysadmin and DevOps Engineer?
What is the difference between SRE and DevOps?
What could be a valid definition of DevOps to introduce it to a novice?

However none of these questions or their answers describe the differences between a Systems Administrator and a Site Reliability Engineer.

In broader terms: what are the key differences between Google's practice of Site Reliability Engineering and the traditional separated Development and Operations functions within a business.

Thankfully, since Site Reliability Engineering developed internally at Google and only recently has started to make its way into the broader community, it is fairly well-defined.  What isn't, though, is web operations (or "systems administration" - as an example of the lack of clarity, you use both in your question).  It's difficult to discuss the differences between two things when you're not altogether sure what one of them is.

But I'm an adventurous fellow, so I'll give it a shot.

In very traditional shops, developers and sysadmins are very siloed from each other.  The devs build an app, then consider their job complete as soon as their code has been committed.  The sysadmins take the build artifacts (which may be just the code, if it's an interpreted language) and deploy it to production servers.  It's the sysadmins' job to keep the application running smoothly, and in general manage the production environment.  However, often performance problems come from architecture issues in the app; the sysadmins don't have the programming knowledge to know what the app is doing, and the developers don't know how the app acts in the production topology with production traffic, so no one is equipped by themselves to solve the problem.

Additionally, the developers are usually judged on how quickly they can produce new features, while the sysadmins are judged on how infrequent the app breaks in production.  Since change is one of the leading causes of breakage, this puts the two departments at odds with each other - an old rivalry that hurts the business and the people involved.

At some point, some developer-centric companies got so annoyed at this that they began practicing "NoOps" - they eliminated their operations departments and the perceived roadblocks that came with them.  In reality, this meant that developers took on operations roles, but maintained their old titles.

In a discussion surrounding NoOps, John Allspaw, then VP of Technical Operations at Etsy and an editor of the well-respected Web Operations book, defined roles at Etsy this way:

  Etsy Operations is responsible for:

  Responding to outages, takes on-call
  Alerting systems thresholding, design
  Architecture design and review
  Building metrics collection
  Application configuration
  Infrastructure buildout/management

  Etsy Development is responsible for:

  Responding to outages, takes on-call
  Alerting systems thresholding, design
  Architecture design and review
  Building metrics collection
  Application configuration
  Shipping public-facing code

  Neither of those lists are comprehensive, I'm sure I'm missing
  something there. While Etsy Ops has made production-facing application
  changes, they're few but real (and sometimes quite deep). While Etsy
  Dev makes Chef changes, they're few but real. If there's so much
  overlap in responsibilities, why the difference, you might ask? Domain
  expertise and background. Not many Devs have deep knowledge of how TCP
  slow start works, but Ops does. Not many Ops have a comprehensive
  knowledge of sorting or relevancy algorithms, but Dev does. Ops has
  years of experience in forecasting resource usage quickly with
  acceptable accuracy, Dev doesn't. Dev might not be aware of the pros
  and cons of distributing workload options across all layers1-7, maybe
  only just at 7, Ops does. Entity-relationship modeling may come
  natural to a developer, it may not to ops. In the end, they both
  discover solutions to various forms of Byzantine failure scenarios and
  resilience patterns, at all tiers and layers.

In his world, developers and ops engineers had very similar high-level skill sets and responsibilities; where they differed was in their expertise.  Their differing specialties encouraged them to work together to solve problems, and their common base-level skills gave them a language in which to do that.

This is generally the definition of web operations that I land on for most cases.  So it's the one we're going to continue along with.

So then, what is Site Reliability Engineering?

The Google SRE book opens with a definition of SRE... and then another one... and then spends a chapter continuing to define the role and an entire book covering the specifics.  Even when developed in one organization, it seems that it's difficult to condense the job down to one single agreed-upon definition.

To start with, we need to walk back to 2003, when Ben Traynor joined Google and founded what came to be the first Site Reliability Engineering team.  Recall that a few paragraphs ago we were in the early 2010s; but in 2003, the industry was still pretty set on the sysadmin/developer divide as the natural way of things.  So when Ben says that SRE was what would happen if a software engineer created an operations team, this was a much more radical melding of the two worlds than it appears now.

The definition given in the preface emphasizes each of the three words individually:

Engineering - the use of computer science and engineering concepts to solve problems
Reliability - a focus on making systems more scalable, more reliable, and more efficient
Service - the later evolution of "site", emphasizing that SREs are responsible for networked services

The introduction chapter lists the tenets of Site Reliability Engineering as:

Ensuring a durable focus on engineering - taking pre-emptive action to avoid frequent pages and other "toil"
Persuing maximum change velocity without violating a service's SLO - a subject that can easily have its own several-hundred word answer, but roughly summarized as helping developers make changes, as long as they don't cause too many issues
Monitoring - automatic alerts when things go wrong
Emergency response - fixing things when they're broken
Change management
Capacity planning
Provisioning
Efficiency and performance - ensuring that a service performs at an expected level - bottlenecking hurts users, but excess capacity costs money

I'd categorize Site Reliability Engineering as a specialized subset of modern Web Operations.  An SRE organization focuses heavily on automating everything, to a degree that is only cost-effective in fairly large companies.  Ideas like error budgets can only work when your service has many, many requests, as otherwise you lose granularity (for a smaller service, a particular error could affect 0-20% of your requests, depending on the minute).  Related areas like security are absent from the SRE definition because companies large enough to have true SRE teams have dedicated teams for security.

The SRE program, as defined by Google, is web ops developed for the specific needs of Google, and not necessarily applicable elsewhere.

However, Site Reliability Engineering has been expanding in broader industry use recently.  My current job title is an SRE, even though I work at a much smaller company and my job description fits pretty well with John Allspaw's 2012 Etsy web ops definition.  My theory is that we've been progressing through titles as a shorthand for espousing the evolution of a single field:

We started as sysadmins.
Then as web sites became more of a "thing", job postings started to refer to web operations engineers to distinguish sysadmins who specialized in the web from those who also handled general office IT.
Then DevOps was supposed to separate out those who were comfortable using programming to reduce their web ops workload.
But as DevOps got muddled by the lack of a clear definition, we adopted Site Reliability Engineering to specify that we're looking for people who are on-call supporting production services.

So what is the difference between a sysadmin and an SRE?  The year in which they received their title.  What is the difference between traditional operations and site reliability engineering?  SRE is merely the current incarnation of ops, using new tools (hello, containers!) and, as networked programs continue to become more larger and more important, an increased focus on practices that allow one engineer to do more.


How do Sensu handler notification plugins determine the path of their JSON config files? So, we're transitioning from Slack to MSTeams. I'm tasked with updating our Sensu configuration so that alerts get sent to MSTeams channels instead of Slack channels.

In the /etc/sensu/conf.d/handlers.json, we have Slack notifications configured as follows:

...
"slack" : {
    "type": "pipe",
    "command": "handler-slack.rb -j slack",
    "filters": [
        "occurrences"
    ]
},
....

That -j slack tells the handler to load a JSON config file which is located at /etc/sensu/conf.d/slack.d/slack.json.

The Teams handler is configured similarly:

...
"microsoft-teams" : {
    "type": "pipe",
    "command": "handler-microsoft-teams.rb -j microsoft-teams",
    "filters": [
        "occurrences"
    ]
},
...

But when Sensu tries to pass a notification to the Teams handler, I get an error that leads me to believe that the JSON config wasn't loaded:

/opt/sensu/embedded/lib/ruby/gems/2.4.0/gems/sensu-plugins-microsoft-teams-2.0.0/bin/handler-microsoft-teams.rb:104:in `get_setting': undefined method `[]' for nil:NilClass (NoMethodError)
    from /opt/sensu/embedded/lib/ruby/gems/2.4.0/gems/sensu-plugins-microsoft-teams-2.0.0/bin/handler-microsoft-teams.rb:28:in `payload_template'
    from /opt/sensu/embedded/lib/ruby/gems/2.4.0/gems/sensu-plugins-microsoft-teams-2.0.0/bin/handler-microsoft-teams.rb:108:in `handle'
    from /opt/sensu/embedded/lib/ruby/gems/2.4.0/gems/sensu-plugin-2.7.1/lib/sensu-handler.rb:89:in `block in <class:Handler>'

payload_template is the first setting that handler-microsoft-teams.rb attempts to load. It is not specified in my JSON, and the code is written to treat it as optional.

So, my question is this: how does the Slack plugin go from the string passed to the -j argument (slack) to the path /etc/sensu/conf.d/slack.d/slack.json? And how do I determine the equivalent path magic for the Teams plugin? I don't know if this magic is inherent in Ruby (in which I'm a non-expert), Sensu, sensu-plugins, or something else. I've been digging in code for a while now and I'm coming up empty.

(Yes, I'm aware of Sensu Go and would like to upgrade; that's not an option on my current timeline.)

Okay, so it turns out this wasn't my problem, but I know the answer to the question I asked:

The file /opt/sensu/embedded/lib/ruby/gems/2.4.0/gems/sensu-plugins-microsoft-teams-2.0.0/bin/handler-microsoft-teams.rb defines a function get_setting, which looks like this:

def get_setting(name)
  settings[config[:json_config]][name]
end

config[:json_config] is the string passed with the -j switch.

On to config: the file /opt/sensu/embedded/lib/ruby/gems/2.4.0/gems/sensu-plugin-2.7.1/lib/sensu-handler.rb requires sensu-plugin/utils, which defines the following:

module Sensu
  module Plugin
    module Utils # rubocop:disable Metrics/ModuleLength
      def config_files
        if ENV['SENSU_LOADED_TEMPFILE'] && File.file?(ENV['SENSU_LOADED_TEMPFILE'])
          IO.read(ENV['SENSU_LOADED_TEMPFILE']).split(':')
        elsif ENV['SENSU_CONFIG_FILES']
          ENV['SENSU_CONFIG_FILES'].split(':')
        else
          ['/etc/sensu/config.json'] + Dir['/etc/sensu/conf.d/**/*.json']
        end
      end

      def load_config(filename)
        JSON.parse(File.open(filename, 'r').read)
      rescue
        {}
      end

      def settings
        @settings ||= config_files.map { |f| load_config(f) }.reduce { |a, b| deep_merge(a, b) }
      end
...

So settings is going to load all JSON files nested anywhere under /etc/sensu/conf.d/ and deep-merge them. Beyond that, the path is not relevant to Sensu's ability to load it using the string slack; that's the name of the entry in the JSON payload. This mechanism is common to all Sensu config, not just handler plugins. Documentation is here: https://docs.sensu.io/sensu-core/1.8/reference/configuration/.

I looked at my log files, and it does seem to be finding and loading the files I had optimistically placed under /etc/sensu/conf.d/teams.d/, but it's not working.

Before I found the answer documented here, I took a stab and renamed the directory from teams.d to microsoft-teams.d, and the configuration object defined within from teams to microsoft-teams. I don't know why that would have had any effect, but it's working now. Maybe I had fat-fingered or copypasta'd something and resolved that inadvertently in the process of adding the microsoft prefix.


Is it possible to use a JenkinsFile in one repo to build and deploy a related repo? Our team has two separate repositories for a frontend/backend system.  They would like to have these two repositories deployed to the shared team environment together.

My understanding of the JenkinsFile is that it will only work on the repository it committed to and only run when the SCM system sends a request to Jenkins informing it of new changes.

Is it possible to have the JenkinsFile in the two separate repositories communicate with each other so that when one is built the other is built as well and only after both are built, they will then be deployed?

My main goal here is to avoid creating a separate Jenkins job within the Jenkins UI.

Yes, this is pretty easy with Jenkinsfiles with no need for any third-party plugins or anything along those lines: use the built-in Pipeline build step.

I use this to trigger builds of projects in a dependency chain, so that after one project builds successfully, other projects that depend on it will pull in the updated dependency and build against it.

Here is what this looks like in a Jenkinsfile:

build repoName

where repoName is a variable containing the name of the repository you wish to build.

Things can get a little more complicated, such as if you're using folders or Multibranch Pipelines:

build "${projectName}/${repoName}/${branchName}"

This example builds a job within a Multibranch project where the variables are:

projectName - the name of my Multibranch project
repoName - the name of the repository
branchName - the name of the branch to build, usually master

Or this example builds a job with parameters and also triggers the build asynchronously (parent job won't wait for child job to complete before moving on to the next step):

build(
  jobName,
  wait: false,
  parameters: [
    [
      $class: 'BooleanParameterValue',
      name: 'someBooleanParameter',
      value: true,
    ],
    [
      $class: 'StringParameterValue',
      name: 'someStringParameter',
      value: 'some string value',
    ],
  ],
)

As you can see, the invocation of build can get relatively complex.  See the documentation linked above for more complete information.


What processes enable software vendor/supplier management in a DevOps way of working? Within a traditional separate Development and Operations organisations, procedures and processes are put in place to ensure that relationships between software vendors and suppliers are maintained. These relationships exist so that when for example a vulnerability is discovered operations teams are able to identify the business impact and mitigate appropriately.

If that organisation was built around ITIL service management principals then they would likely be doing one or all of the following:

Supplier Management
Vendor Management
Risk Management

With the assumption that the Operations role doesn't exist as a separate entity within a DevOps enabled organisation, how is an organisation collectively or severally meant to maintain supplier/vendor relationships so that when a vulnerability is publicised the teams can respond by reaching out to the vendor to understand remediation and mitigation actions.

Conceivably we could:

Ask all of the Scrum teams to own their vendor relationships and act accordingly, this seems scalable but at the cost of a high overhead across an organization.
Maintain a central function responsible for these relationships, this central function may not have the full context the the Cross-functional teams have.
Use a 3rd Party automated service to scan infrastructure and applications to maintain a dynamic inventory which can be assessed for known vulnerabilities, this however doesn't actually maintain a relationship with vendors and is only as good as it's database of known vulnerabilities.

What strategies do you use?

Since you reference the handling of vulnerabilities specifically in your question, I'll speak to this regarding how we handle this within our organization and group of companies.

We have an Information Security department who are responsible to monitoring and detecting vulnerabilities. You could think of this team as being responsible to technical information security management. When vulnerabilities are detected using regular scan with tools such as Nessus or published US-CERT by the information security community, this team passes these vulnerabilities to each Information Security Officer within the group.

The Information Security Officer, who is responsible and accountable for risk management, is tasked at a high level with:

Establishing Information Security Policy, Standards, Procedures and Guidelines
Promoting Information Security Awareness
Completing Information Security Assessments and Audits

Having received the information regarding vulnerabilities, the Information Security Officer evaluates the vulnerability, determines the criticality of the vulnerability and it's impact on the business applications and environment. This information is then given to the appropriate Development Manager.

Development Managers are responsible and accountable for supplier and vendor management. Business Applications are allocated by business domain to various Development Managers. The Development Managers work with either a vendor and/or their own Scrum team to resolve the vulnerabilities withing their domain. Should specialist advice be required, this is provided by the Information Security Officer.

Regarding your possible options.

Scrum teams might well lack the organizational clout to negotiate and compel vendors to address vulnerabilities quickly. It's also unlikely that the scrum teams themselves would have been in a position to ensure that all the correct legal clauses have been put into the vendor and supplier contracts to ensure that there is an obligation to handle vulnerabilities within a given SLA.
Maintaining a central function ought to work to a degree, however as you point out they would lack the full context. Three tiers of management, covering three different aspects will address this short coming.
A 3rd Party automated service will be able to alert you to vulnerabilities, but it will not necessarily be able to correlate those results to a specific business application, and as a consequence a given vendor or supplier. You will still need individual people to manage this aspect.


reboot server during Ansible tasks I have 3 scripts that must be run on server one by one. for example:

- name: find config
  shell: /data/scripts/config1
  become: yes
  become_user: root
  tags: find_config

- name: change config
  shell: /data/scripts/config2
  become: yes
  become_user: root
  tags: change_config

- name: confirm config
  shell: /data/scripts/config3
  become: yes
  become_user: dbusr1
  tags: confirm_config

in second script config2 there is a reboot command and then the rest of script will be run at start up. something like this:

some variables
.
.
commands before reboot
.
.
reboot command
.
.
commands that run at start up

does this reboot affect ansible tasks? how i can write tasks that wait until commands that run at start up be run then execute config3? is this even possible in ansible? what modules should i use?

Use this source code for reboot your client machine

---
  - name: System Reboot
    hosts: debian
    become_method: sudo
    become_user: root
    become: true
    tasks:
      - name: reboot nodes     # Reboot client side debian machine
        shell: sleep 2 && shutdown -r now "Ansible reboot"
        async: 1
        poll: 0
        ignore_errors: true
      - name: wait for the server to come back  # when server timeout it through error. That error is handle by ignore_errors.
        local_action: wait_for
        args:
          host: "{{ inventory_hostname }}"
          port: 22
          state: started
          delay: 120
          timeout: 200
        ignore_errors: true


Do AWS backups of large empty volumes take longer than small ones? AWS provides backup functionality for RDS databases and snapshots functionality for EBS volumes.

Given a small amount of data, would the backup and restore time change depending on whether it's stored on a large disk or a small one?

In case you are asking yourself why one would put a small amount of data on a large disk: EBS gp2 volumes offer more IOPS (IO operations) with more storage (3 IOPS per 1 GB). So if you need a disk with plenty of IO operations but little persistent data, you might end up with a few megs of data sitting on a 1000 GB disk.

If my assumption that backups might take longer for larger disks is true, then it might be worth deciding for the more expensive io1 disk type instead, which can scale with up to 50 IOPS per GB.

Firstly I would argue that backup time does not matter,  as it usually doesn't correlate with the restore time.

In regards to restore time,  snapshots restore on volumes almost instantly,  as data is only copied onto the volume from S3 when requested.

So as long as you only request actually used data from the volume, it would only matter how much data is used on the volume to determine the time to restore.

This link explains the process well https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-restoring-volume.html


If I set up a Travis CI agent, will it be independent of Travis's existing infrastructure? Is this correct that if I decide to setup an own TravisCI agent, I will be enabled to be independent of the existing distributed Travis computation network?

Travis CI doesn't offer running your own workers, you use their infrastructure unless you mean the on-prem Travis Enterprise. You might be thinking of GitLab CI?


What is recommended books reading order offering thought leadership to read on DevOps to go from beginner to expert? For sure there is partly an opinion question, but many colleagues around me have little knowledge about books offering thought leadership on DevOps. So this thread might be of great convinience.

One fantastic read that offers great thought leadership on this issue and is specifically tailored to the non-technical is The Phoenix Project which is available in paperback, a Kindle E-book and as an Audiobook. This book offers an excellent starting place to educate on the who, what and why of DevOps, wich is often seen as nebulous buzzword. The official synopses states,

  Bill is an IT manager at Parts Unlimited. It's Tuesday morning and on his drive into the office, Bill gets a call from the CEO.

  The company's new IT initiative, code named Phoenix Project, is critical to the future of Parts Unlimited, but the project is massively over budget and very late. The CEO wants Bill to report directly to him and fix the mess in ninety days or else Bill's entire department will be outsourced.

  With the help of a prospective board member and his mysterious philosophy of The Three Ways, Bill starts to see that IT work has more in common with manufacturing plant work than he ever imagined. With the clock ticking, Bill must organize work flow streamline interdepartmental communications, and effectively serve the other business functions at Parts Unlimited.

By novelizing this story the authors are able to expertly illustrate and explain in a detailed and entertaining manner:

What DevOps means and entails
What teams and individuals are involved in DevOps
How DevOps can benefit a companies' workers and business
Challenges and weaknesses of the traditional model

While this will not explain or recommend how to do DevOps on a technical level, it will make and explain the business case and method for doing so, thereby taking you from beginner to intermediate.

The authors have also published a followup tome, the DevOps Handbook which bills itself as a how-to on "Creating World-Class Agility, Reliability, and Security in Technology Organizations," however one reviewer, FA Calkins cautioned,

  Dont buy this book if you expect it to teach you how to build a CI/CD DEVOPS environment. It is basically a review/summary of a bunch of research papers that the authors have read, but does not provide any specifics details of how to set up a CI/CD pipepline (only theory). You can get most of the content of this book by reading articles from the web. Also, you can get more specific details of how to implement CI/CD with specific tools, programs, and scripts from other sources. Dont waste your time trying to read this book if you want to deploy CI/CD.

Though I have not yet read it (it is on my reading list!), the description claims that the book includes

  More than 40 DevOps case studies, including Amazon, Etsy, Capital One, Google, Facebook, Intuit, Nationwide Insurance and many more
  More than 400 pages of DevOps applications, lessons and 'how-to’s'.
  DevOps data gathered from more than 25,000 data points.

These resources should at least get you started on the business/philosophy piece.


Is there a way to run with_items loops in parallel in Ansible? I'm running Ansible 2.2, but can upgrade if it helps.

I saw this and was pretty excited, but it doesn't seem to be in this (or any) version of Ansible documentation.

The problem I'm trying to solve is I've got 1000 users that I need to manage on a Centos box.

It takes quite a while to run this task serially.  And even more annoying, everything shows up as changed because the "expires" command on the user module always marks the thing as changed.

this also looked promising, but it took the same amount of time to run each command in the with_items loop and didn't go any faster (I never bothered to wait long enough to get to the end).

Skipping tasks is fast now (a lot faster than it was in Ansible 2.0), if I can't figure out how to make this work in parallel, I think I'll go back and figure out how to skip pointless tasks and if all else fails, I'll write my own module.  But it seems like I should be able to do all this faster in Ansible.

This is what I want to run in parallel,  host_authorizations is a list of usernames and other data.

  - name: Create/modify OS user accounts
    user: name={{ item.username }} group=sshusers shell=/bin/bash home="/home/selinux-modules/{{ item.username }}" state=present expires={{item.expiredate|default(omit)}}
    with_items: "{{ host_authorizations }}"
    tags: full_maintenance

As @webKnja mentioned this is possible with async mode. I have recently discovered it myself and learned that you can use it in 3 different ways depending on your needs.

Execute and poll the results, notice the poll:5, This will poll the results every 5 seconds. You may save some time with this method.

- name: My long runing task
  some_module_name:
    ip: "{{item.fabric}}"
    username: "{{user}}"
    password: "{{password}}"
    secret: "{{secret}}"
  loop: "{{zoning_list}}"
  register: _alias_vc_0
  async: 60
  poll: 5

Fire and forget poll: 0, This is very quick option since Ansible it's just shooting out those tasks. The down side is that we don't know what was the outcome of the task i.e. changed: True/False. Of course it's a downside if you care about the feedback ;).

name: My long runing task
some_module_name:
  ip: "{{item.fabric}}"
  username: "{{user}}"
  password: "{{password}}"
  secret: "{{secret}}"
loop: "{{zoning_list}}"
register: _alias_vc_0
async: 60
poll: 0

Fire and forget with async_status, the syntax for the task is the same as example 2 whowever it will require additional task async_status. This is my favorite since it's relatively fast (faster then normal looping or the execute and poll) and allows you to capture the feedback although will need to deal with new register for your async_task.

retries: 20 -- how many attempts before failing.

delay: 2 -- how many second to wait between polls.

- name: My long runing task
  some_module_name:
    ip: "{{item.fabric}}"
    username: "{{user}}"
    password: "{{password}}"
    secret: "{{secret}}"
  loop: "{{zoning_list}}"
  register: _alias_vc_0
  async: 60
  poll: 0

- name: Wait for My long running task to finish
  async_status:
    id: "{{ item.ansible_job_id }}"
  register: _jobs_alias_vc_0
  retries: 20
  delay: 2
  until: _jobs_alias_vc_0.finished
  loop: "{{_alias_vc_0.results}}"

A word of caution, depending on the task yo may not be able to use the async option. I had examples where I was interacting with system which was not able to handle multiple requests for the same resource. I found async option best working if I have to perform the same task across multiple hosts. That's where I was able to "save" the most time.

Since you posted the link to Ansible documentation in the question I'm not going to do that.


What is the purpose of running PHP-FPM in its own container instead in the same container as apache2/nginx? I'm fairly new to Docker so excuse any obvious ignorance or misunderstandings. That said, I've been coding and configuring web applications for a long time now. I have recently been dabbling with some more sophisticated (Docker and "traditional") web/application server setups and experimenting with performance enhancements and simplifying deployments.
My personal favorite configuration to use for most projects thus far is nginx as a (mostly) "static file" web server &&|| caching mechanism &&|| Load Balancer in a reverse proxy config with Apache v2.4+ as the "backend" which runs PHP (also Perl &&|| Python) in a "FastCGI" (PHP-FPM) configuration (with mod_php disabled) and mpm_events (instead of mpm_prefork). We can always add in something else also such as REDIS or memcached, where applicable (but I rarely see a significant performance gain when using Cloudflare caching).
I've been dabbling with a few different ways to keep my "favorite web server composition" flexible and "ephemeral" enough to accommodate any and all possible options for further performance gain or resource load management. What I keep seeing is that there seems to have been a shift towards using PHP-FPM in its own separate "stand-alone" Docker container sometimes around late 2019.
Why?
While I can appreciate keeping resources isolated and separate for an easier to debug/config/secure configuration(s), I don't fully understand what the benefits are to having PHP-FPM in a separate Docker container that is implicitly REQUIRED by the application that the Docker containers are comprising.
If anything, having a separate PHP-FPM container seems like additional configuration, additional resource consumption and an even more complicated build/deploy consideration in a CI/CD pipeline.
I can even get onboard with "simple preference", such as instead of using Apache2 and nginx on the same Ubuntu/Debian or RHEL/CentOS container, break off nginx into its own container(s) and simply config your hostname(s) &&|| IPs and ports appropriately.
But what is a practical use case and advantages for having PHP-FPM separated from Apache or nginx containers for any additional benefit beyond perhaps using more Dockerfile &&|| docker-compose.yaml and less ba.sh scripting to get the same job done? If any part of the application fails or falters, then functionality will suffer. The additional network &&|| docker-network communications and hostname resolutions seems like it would just be a performance penalty over combining certain aspects of the docker-composer-ed application together, rather than splitting them up simply for the sake of splitting them up.

Im heavily investigating this, and like many people, im discovering that in 2020 there doesn't seem to be so much logic in separating tightly coupled webserver + PHP + app process/code from each other:
Two main arguments for separation are scalability and separation of concerns (one process per container).
Logically the webserver should only act as webserver and distribute traffic to PHP nodes behind a service that uses a LB.
This way one container/service serves webserver purpose and serves static files, while proxying the php requests to the php pod which only concerns with PHP.
And webserver and php pods can scale differently - ie you may need 50-100 PHP pods for a workload, but only 3-4 webserver nodes. So you save on resources. Great!
Except in respect to separation of concerns...
If the webserver is not going to serve requests to different apps, there is little logic for separation of concerns to different containers - leave aside services. If the app is coupled with the webserver, you just end up spawning extra webserver pods, with accompanying service and load balancers to go with them.
On top of that you will have to distribute the app code to both containers or mount them from a NFS bind, or non NFS mount. You will have to deal with all the relevant access and permission stuff, if you put the app code in both pods you will have to deal with static + dynamic file separation, or even end up having to distribute both type of files to both pods. You could put two containers and make one of them mount directories on the other and that would work, but that actually puts two containers in the same logical host in regard to network, host etc so you are still combining them. This still provides some separation of concerns though. But the communication in between the two containers in the same pod is a concern - it has to either happen through intra-host network over loopback, which uses network stack and is less performant. Maybe sockets could work.
Even worse if your app may require webserver being present in the same container - for example if you are doing ssl termination and have to carry over the visitor's actual ip to the php pod, you may end up having to jump through some hoops. While both webserver and php are available in the same container, this can be done through various means.
When everything is in one container, none of these is necessary. Everything is neatly packed in the same place, no need to track different deployments, services, app codes, replication controllers.
This kind of singular container/deployment is very advantageous today especially if you are using ingress-nginx or nginx-ingress and hosting many different apps/sites in the same cluster. Each app/site is contained in its own container (imagine different WordPress sites), each site is completely contained in its permissions, access, deployment, version, requirements etc.
So much that you could even allow apps/sites to customize their runtime environments by providing configuration files for webserver or php from NFS shares or other mounts - ie .htaccess, php.ini etc. Which is a must for hosting different apps since people tend to have different requirements.
In respect to performance
Today NGINX or Apache require very little resources with event based request handling. For example in my benchmarks, i saw Apache 2 with event mpm requiring only 3-4 MB memory, and a grand total of ~2-3% cpu (out of 1000m requested cpu, ie 1 vcpu) while handling 50-100 concurrent requests per second while serving WordPress.
The weights of webserver vs PHP pods would differ depending on the app or website of course, and there could definitely be apps which would require such a separation, but for most common web workloads this does not seem necssary.
So spawning a few Apache pods and then spawning hundred or more PHP pods does not seem to bring much gain. Aside from creating internal cluster network traffic as pods need to communicate with each other and also you create more complication in the app configurations (deployments, services, load balancers etc).
Another disadvantage of separating containers is that you create two times the services per app. This will reflect on the limits of the cluster you create since services, pods require ip addresses.
When webserver + PHP +app are in the same container, the webserver can just communicate with php-fpm via file sockets. Which is faster, and creates less network overhead even on loopback interface inside the container, leave aside creating network traffic inside the cluster. Internal network and cpu load saved.
In conclusion...
For specific purposes, with one Apache 2 or NGINX server distributing requests to different apps on different PHP-FPM clusters, separating concerns would likely be a necessity.
But in the era of ingress-nginx, this really doesn't seem necessary.
Especially reduced complexity, portability, saved performance and internal cluster noise, better customization and other benefits gained from packaging tightly coupled server + PHP + app deployments in one single container seem to be too good to pass.
Furthermore, this even enables shared-hosting like formats in Kubernetes clusters, which seems to open many possibilities.
I would very much appreciate any input from anyone who had experience in dealing with this choice in between selecting singular container for tightly coupled app vs multiple cointainers, or anyone who did benchmarks on these.


Multiple apps (microservices) and one proxy (nginx) docker-compose configuration/architecture Having the following architecture:

Microservice 1 + DB (microservice1/docker-compose.yml)
Microservice 2 + DB (microservice2/docker-compose.yml)
Proxy (proxy/docker-compose.yml)

Which of the following options would be the best to deploy in the production environment?

Docker Compose Overriding. Have a docker-compose for each microservice and another docker-compose for the proxy. When the production deployment is done, all the docker-compose would be merged to create only one (with docker-compose -f microservice1/docker-compose.yml -f microservice2/docker-compose.yml -f proxy/docker-compose.yml up. In this way, the proxy container, for example nginx, would have access to microservices to be able to redirect to one or the other depending on the request.
Shared external network. Have a docker-compose for each microservice and another docker-compose for the proxy. First, an external network would have to be created to link the proxy container with microservices.docker network create nginx_network. Then, in each docker-compose file, this network should be referenced in the necessary containers so that the proxy has visibility of the microservices and thus be able to use them in the configuration. An example is in the following link https://stackoverflow.com/a/48081535/6112286.

The first option is simple, but offers little felxibility when configuring many microservices or applications, since all docker-compose of all applications would need to be merged to generate the final configuration. The second option uses networks, which are a fundamental pillar of Docker. On the other hand, you don't need all docker-compose to be merged.

Of these two options, given the scenario of having several microservices and needing a single proxy to configure access, which would be the best? Why?

Tnahks in advance.

This is going to be an opinionated answer as both options are clearly possible.

I would go with the first option you described for the following reason:

It seems that at the moment you don't have a lot of microservices (otherwise you wouldn't be using docker-compose and would likely be doing k8s or at least docker swarm). Now, if your application grows a lot - you would likely outgrow compose in any case, so it would be wasteful if you optimize for compose at this stage anticipating future complexity.

So my take would be to go with option 1 for now which is much easier to implement. And when you need extra flexibility, likely you'd have other reasons to switch from docker-compose to something else.


Infrastructure Stability Report We are using Azure IAAS to host our applications, and these are hosted on a number of virtual machines for different tiers
Our client have recently asked us to provide monthly report on the Infrastructure Stability, so that they could know the health of the infrastructure and its availability during a given time period.
Are there any standardized norms on how these reports should look like and what information it should contains, as we are not very clear on this and even client is not sure about the exact specifications.

We are currently using Zabbix for monitoring systems and thinking of using the same for these reports. Please suggest if there any tools available for Infrastructure Stability reporting and how we should approach it.

Its very common to provide a StatusPage (like Statusy.co or StatusPage.io). There are numerous examples of major providers having them:

https://status.github.com/messages
https://status.bitbucket.org/
https://status.aws.amazon.com/
https://azure.microsoft.com/en-us/status/
https://api.twitterstat.us/

You could provide the same status page to your customers. When there is a service disruption, you could manually update the status while providing detailed, customer-facing information. Or you could hook it into your Zabbix monitoring to have the status auto-updated.

This also lets your customers see a history of incidents and average uptime of your service.

There are also self-hosted, open-source solutions that you could implement:

https://cachethq.io/
http://www.stashboard.org/

Although, you'll really want to host those outside your existing infrastructure so that your StatusPage can always be reached, even if you're down. If you go the self-hosted route, I recommend hosting the status page at a different provider: Digital Ocean, AWS, etc...