Looking at a Career jump into DevOps and advice or suggested reading I've been in various facets of Software development over the past 10+ years. I started in the development side of the Application Lifecycle Management(= ALM) cycle, and more recently I've been working in the QA side of ALM.

One thing that has consistently bothered me was all the different companies that I've worked for tried to embrace an effective ALM process but in the end it usually turned out to be a half-baked implementation of a scrum methodology that had all the different team leads debating about the right and wrong ways to implement ALM processes.

I've been currently researching DevOps in hopes that I can shift my career to a direction that will allow me to better contribute to a companies ALM process.

I have quite a bit of experience in Dev, QA, and General IT but I wouldn't consider myself an absolute expert in either area.

Any suggestions, advice, recommended reading, etc?

If you recommend a book could you include the ISBN number (it's easier to search for it that way).

I do not think that there is a single correct piece of advice for such a broad question, but here are my thoughts your situation:

As a starting point look at your current workflow, processes, and technologies being used at the company and then research some tools
(Jenkins, Kubernetes, Docker, Grafana, Ansible, Kubana etc)
or techniques(writing your own scripts or programs) that might be useful to your company or any projects you are working on.

I would also suggest maybe attending local DevOps meetups or talks in your area for more exposure.


Testing Jenkins jobs and pipelines Currently, we have a fair number of Jenkins jobs and pipelines for builds, testing, deployments and other automated activities.

Every time we change or add a new job, we only test it manually - e. g. going over the "happy path" (when job is done with no errors), testing a couple of negative test cases when a job or pipeline fails - checking the error code and notifications.

This approach clearly is not reliable and does not scale well. How can we improve this process? Is there a place for test automation when it comes to checking how Jenkins jobs and pipelines work?

I'm posting this here not because I endorse these solutions (in fact, I've never tried them), but just because they are a potential answer to your question:

You can start with JenkinsPipelineUnit, a unit testing framework for Pipeline scripts.

There is also a project called jenkinsfile-runner which executes your Jenkinsfile in a transient, headless Jenkins instance.  Supposedly, this can be used for integration testing Jenkinsfiles and Pipeline shared libraries.  However, as of summer 2018 there is no documentation on how to use this tool for integration testing, and I have been unable to find any examples of anyone using this tool "in the wild".

See also the related bug report on the Jenkins JIRA: "Test framework for Jenkinsfile".


How to prevent Azure Traffic Manager from routing over high latency connections in a fail-over scenario? We have a service hosted in four regions, two in North America and two in Europe.  We are using Azure Traffic Manager to provide:

Performance-based routing - that is routing consumers to the service that has the lowest latency.
Fail-over - if one region is down, traffic is routed elsewhere.

As it turns out if a service goes down, then Traffic Manager will spread the load across all other endpoints to reduce the chance of a cascading failure.

For us, however, we would rather run each region with enough capacity to be able to handle the traffic of two regions than force our users to make transatlantic requests.

Is it possible to configure Traffic Manager to always route to the next-nearest region in the event an endpoint is down?

Disclaimer: I have not setup a working example of this

I believe you are looking for Nested Traffic Managers

If you setup a main traffic manager that uses performance to select between North America and Europe child traffic managers.  As long as one of the services in East or West North America is functioning the child traffic manager will report its status as available and the main traffic manager will continue to send traffic to the child based on performance

Docs:
https://docs.microsoft.com/en-us/azure/traffic-manager/traffic-manager-nested-profiles


CI pipelines, concurrency and build results possibly overwriting each-other I'm currently using gitlab-ci to generate docker images for some repositories, however i dont feel as though this solution is as reliable as it should be.

I can forsee a issue with multiple builds happening close together and build #100 overwriting the newly built docker image produced by build #101, if for whatever reason jobs dont finish sequentially (eg: multiple build hosts, etc...).

Currently the tags i'm hoping to use follow this pattern:
namespace/application:branch-name

The crux of the question is as follows:

How do other services solve this problem? (presumibly running builds sequentially per branch?)
How should this issue be resolved in gitlab environments in a totally watertight manner

I understand this problem would go away if i use the SHA of the commit for the build tag, but i'd like to stick to the branch name for the following reasons:

orphaned / old images will be grabage collected by the docker registry
branch name is easier to use

I am using AzureDevOps and the trick we use when naming containers is to add build number to the label

namespace/application:branch-name-bulidNumber

So in your case, you could try gitlab variable CI_CONCURRENT_PROJECT_ID

After comment

for images clean-up you can use:

  docker image prune --all --filter until=48h

As docker system prune and docker image prune have the until filter. So docker image prune --all --filter until=48h would remove all (not just dangling) images that were created more than 48 hours ago. Hopefully that helps. source here


Where to Store Container Configuration (I reviewed this question but the one answer doesn't offer practical solutions.)

The general wisdom when it comes to containers appears to be:

"Make them immutable and identical."

I take this to mean that every container for a given application function (a web server, for example) should be bit-for-bit identical.  The code, the internal config, and the file system should be built the same way every time for a given version.

This begs the question: where to store the config that makes a given app container a DEV, TEST, QA, or PROD container?  If the environment config is stored inside the container, that makes them different.  This seems to run contrary to the identical/immutable goal.

A better model would be to somehow keep environment-specific configuration outside the container.  There are two methods I can think of to do this:

Store environment-specific configurations on a mounted volume, and make the volume mapping a deployment variable
Use configuration service such as in the External Configuration Store pattern

My particular use case is for .NET web applications, where the web.config is a file stored with the application.

Both options appear to have potential pros and cons.  Is one of these options better, or am I missing a potential best option?

we do this with .conf files and loading them via compose

env_file: /path/to/our.conf

they can also be used with

docker run --env-file /path/to/our.conf ourcontainer:version

In some cases we use the variables defined in our conf files directly, meaning the code we are running reads them direct, in others we need to modify other files on a per environment basis so we use sed or other tools to make changes to a file we have turned into a template


Quest for a missing release of Apache Cloud Stack 4.7.1.3 Is there any public evidence of Apache Cloud Stack release 4.7.1.3?

Namely, what I can find so far, there is only 4.7.1.1, but an instance I can observe reports to be a 4.7.1.3.

See also:

https://github.com/apache/cloudstack/issues/3084

As it came out, the instance is not Cloud Stack but Cloud Platform by Citrix, (a commercial fork?) so they have slightly different version numbers.

I.e.:

4.7.1.3 of Cloud Platform depends on CS 4.7.1.1.
4.11 of Cloud Platform depends on CS 4.10..?


How to pass HashiCorp Vault secrets(k/v passwords) to Terraform EC2 instance How can we pass HashiCorp Vault secrets, k/v username/password and feed Terraform while its creating Ec2 instance.

For example, i don't want to use in plain text

variable "myvariable" {
default = "mypassword"
}

Using token yea, but what resource should call Vault and ask for secrets which would be injected on creation time.

The simplest case is to use the Terraform Vault Provider (https://www.terraform.io/docs/providers/vault/index.html) - which, caveat, I've not used myself.

Example from the Terraform site:

provider "vault" {
  # It is strongly recommended to configure this provider through the
  # environment variables described above, so that each user can have
  # separate credentials set in the environment.
  #
  # This will default to using $VAULT_ADDR
  # But can be set explicitly
  # address = "https://vault.example.net:8200"
}

resource "vault_generic_secret" "example" {
  path = "secret/foo"

  data_json = <<EOT
{
  "foo":   "bar",
  "pizza": "cheese"
}
EOT
}

Optionally you can write a script in whatever language you are comfortable with to make the call out to Vault and invoke via local-exec in Terraform.

Example of invoking a simple command from the Terraform site:

resource "aws_instance" "web" {
  # ...

  provisioner "local-exec" {
    command = "echo ${aws_instance.web.private_ip} >> private_ips.txt"
  }
}


Configure AWS cli inside chef cookbook I am working on setting up something with S3 and Chef, I have AWS CLI installed and working but I need to figure out how to specify the access ID and the secret access key. I know there are cookbooks out there for this but I have tried a few with no luck so I am not using anyone else's cookbooks for this project anymore.

Suggestions on how I can run aws configure and then output the correct value for each of the 4 things it asks?

Code in my script to set env variables:

export AWS_ACCESS_KEY_ID=<key>

export AWS_SECRET_ACCESS_KEY=<secret_key>

export AWS_DEFAULT_REGION=<region>

Have you tried setting the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables? By setting these variables before you run your cli commands, you should not have to run the aws configure command.

  Environment Variables

  Environment variables override configuration and
  credential files and can be useful for scripting or temporarily
  setting a named profile as the default.
  The AWS CLI supports the following environment variables.

  AWS_ACCESS_KEY_ID – AWS access key.

  AWS_SECRET_ACCESS_KEY – AWS secret key. Access and secret key
  variables override credentials stored in credential and config files.

  AWS_SESSION_TOKEN – Specify a session token if you are using temporary
  security credentials.

  AWS_DEFAULT_REGION – AWS region. This variable overrides the default
  region of the in-use profile, if set.

  ....


The best way to share volume with config data between multiple Docker containers I have Docker image which contains config files of VirtualBox and I want to use this image for running multiple containers in parallel.

Config files of VM are being stored on host OS, however, I am afraid of issues with config data of VMs which may occur because of parallel accessing by multiple containers.
All containers will be started by Jenkins automatically.

Therefore, could you please tell what is the best way to share data between containers to avoid data corruption because of parallel usage?

Please note: I can not run FreeBSD container under Linux and I can not setup FreeBSD as host OS to use FreeBSD containers as long as Docker is an experimental thing in FreeBSD and Kubernetes is not supported under FreeBSD.

One could use read-only volumes.

docker run -d \
  -it \
  --name=nginxtest \
  -v nginx-vol:/usr/share/nginx/html:ro \
  nginx:latest

If one would like to use the same volume in multiple containers then one could specify the following when running each container:

-v vol:/usr/share/nginx/html:ro

Note: one could use ro to ensure that the volume is read-only to prevent that multiple containers will write at the same time to the same volume and prevent issues.


Push Jenkins Build status to Github and Trigger Jenkins build from Github Webhook FYI - New to DevOps. Please bear with me. Thanks

I have a webhook created by Admin on Github Repo to trigger Jenkins build.
I created a token from my Github(I am not admin on the repo) and used it in Github Connection in Jenkins Configure.

Now if I commit when the token credentials are selected in Github in Jenkins Config, the Jenkins build is not triggered automatically. When I click on Build now the job is triggered and job status is pushed to Github.

If I don't select any credentials in Github in Jenkins Config, the jenkins build is triggered on commit but the build status is not reflecting in Github.

There is a manage hooks options also in Github in Jenkins Configure but that probably would require a token from Admin on repo?

Can you guys guide me on what should I do now to
1. Trigger the Jenkins Build on commit.
2. Push the build status to Github.

Thanks

Triggering Jenkins using GitHub webhook should be a straight forward process. This article would help you.


Jenkins: What is a build? I'm coming from Ops and I don't have a heavy software engineering background. In Jenkins, what is a "build" exactly and why is it named such? When I build a shell/powershell script, I'm not really compiling anything, am I? Then why is Jenkins "building" the source code. I thought building from source to binary is for compiled languages, unless Jenkis is using the word "build" in a more generic manner.

Please help me understand what's going on objectively.

As you guessed, build not necessarily means "compile". Building is a process of creating software from sources, which might include stuff like gathering dependencies, compiling, archiving, transforming code in any way, but also testing, deploying to different environments and promoting the artifacts between them.


How to filter label_values in Loki? I want to create a query variable in grafana using Loki that returns all the possible label values of the label baz such that they only include label values that exist where the label foo equals "bar".
Looking at the docs, I came up with the following query for the variable:
label_values(rate({foo=~"bar"}, baz)
However this returned all the possible baz label values and not only those that exist while foo="bar", basically the results are the same as:
label_values(baz)
What am I doing wrong? thank you.

After researching this, i've found an open issue on grafana's github. At the time of writing its still in the backlog.


Interference of Docker CMD with su: works with su-exec but not with su? In this bash script under Linux Alpine they have (line 8):

exec su-exec "$ZOO_USER" "$0" "$@"

As far as I know, su-exec is however an Apache httpd dependecy - can I avoid it installing it under Ubuntu and use some more plain alternative?

I have tested so far:

 su -c "$0 $@" $ZOO_USER

But then the call through Docker CMD shows that I might have some sort of escaping/quotes error,

CMD ["zkServer.sh", "start-foreground"]

resulting in the following output; obviously there is a misplacement for the username field in the command sequence.

No passwd entry for user 'start-foreground'

But what is then the difference to the original (su-exec), which works fine? And how to get it right with su?

====

UPD: su-exec is here a third-parthy Alpine Linux package
https://github.com/ncopa/su-exec

wrong argument order to su

$ZOO_USER needs to be the first argument to su with the "$@" coming afterwards.

su -c "$0" "$ZOO_USER" "$@"

It is ok for the options to come before the username.  It is a good idea to keep them in double quotes so that there aren't issues if someone accidentally puts spaces into it.

Your error message reinforces that this is an order of arguments problem.

even better

Based on comments this would be even better:

su -c "$0" "$ZOO_USER" -- "$@"

su man page excerpt

  Additional arguments may be provided after the username, in which case they are supplied to the user's login shell. In particular, an argument of
  -c will cause the next argument to be treated as a command by most command interpreters. The command will be executed by the shell specified in /etc/passwd for the target user.

  You can use the -- argument to separate su options from the arguments supplied to the shell.

busybox docs excerpt:

In a comment the OP asked about why this wouldn't work in Alpine Linux.  Since alpine is based on busybox I looked up the busybox docs.
 According to the docs busybox doesn't support sending additional arguments to su:

su
su [OPTIONS] [-] [username]

Change user id or become root

Options:

        -p, -m  Preserve environment
        -c CMD  Command to pass to 'sh -c'
        -s SH   Shell to use instead of default shell


How to create ALB that can handle traffic on 2 ports I want my HTTP traffic on port 3701 and my WS (websocket) traffic on 3702. For a moment I thought that using 2 target groups was the answer, but there doesn't seem to be a way to differentiate between WS and HTTPS traffic using an ALB.

So perhaps using a NLB (network lb) will allow me to differentiate between the two? How can I have one load balancer forward traffic to 2 different ports?

Add listeners to the ALB

1 listener for 3701
1 listener for 3702

update the ALB action for each Listener to point to the the correct Target Group


DevOps in space related context? If you compare the "mainstream" DevOps if there is such, is a DevOps practice in a spacecraft related context much different, or not? If this is too broad, take avionics software context as example.

  Avionic systems include communications, navigation, the display and
  management of multiple systems, and the hundreds of systems that are
  fitted to aircraft to perform individual functions (Wikipedia).

The key difference between DevOps in the context of Avionics and, aircraft/spacecraft merely has to do with deployment velocity due to FAA certifications, Aircraft re-grooms and the way in which the Airline industry releases their changes.

But ultimately, DevOps is not exclusive to High Velocity delivery methods (like Agile). And can be done in the context of Waterfall methods or other similar methods.

In some ways, the airline industry is often factory oriented, which aligns well with Kim, Behr, and Spaford's depiction and conceptualization of DevOps.

Development on LRUs and racks can easily be done using an Agile methodology using a DevOps mentality. These many small changes can be stored up and released periodically to undergo final rack testing and certifications (such as AS9100). Airlines could be doing much more to leverage automation for software deployment to fleets and have yet to capitalize on these developments.

At the Avionics company I worked for for several years, we were seeing several successes and challenges with implementing DevOps. I simply can't emphasize enough however that challenges were internal and political, not technical or methodological challenges. These are also not systematic issues resulting from industry regulations, policies and standards. Avionics companies who are able to innovate and provide better value, higher reliability and cheaper costs will easily be able to out-innovate their competitors and gain traction and market share. The Avionics industry is ripe for this kind of shakeup and DevOps is well suited to be a cause of changes.


How can I experiment with Cloud (Azure, AWS, Google, etc) without going broke? Cloud technologies are very hot right now, but they can be expensive.  What are the best strategies for learning/trying cloud services without racking up a huge bill?

The three major cloud platforms you mentioned have either free trials or a free tier; these all include spending and resource caps which prevent you from spending more than your free trial allows:

Azure

Azure offer £150/$200/€170 for the first one month of usage, in addition many of Azure's services have a free or low cost tier:

App Service Free Tier
SQL Azure Basic Instance @ £0.005/hour
Azure Machine Learning Free Tier
Azure Service Bus @ £0.04/million operations
Azure Functions @ £0.15 per million executions

If you have a MSDN or Visual Stuido Online account you can also get £100/$150/€130 per month of credit just because you are a subscriber to one of these services.

If you are a startup you may qualify for $120,000 of Azure Credits to host your application through Microsoft's BizSpark programme.

Amazon Web Services

Amazon have a Free Tier which includes:

750 Hours of EC2 which equates to about a month of a t2.micro instance
750 Amazon RDS
5GB S3
250,000 AWS IoT Messages
... and more

In addition there perpetually free usage allowances on most services:

AWS Lambda: 1 million free requests a month
DynamoDB: 25GB Storage
Amazon SNS: 1 million publishes
... and more

Google Compute Platform

Google offer a $300 free trial for 12 months in a similar deal to Amazon. In addition they have free usage tiers for many of their core offerings:

Google App Engine: 28 instance hours / day
Google Cloud Database: 1GB Storage
Google Compute Engine: 1 f1-micro instance / month
Google Pub/Sub: 10GB messages / month
Google Cloud Functions: 10 million executions / month
... and more

If all of that wasn't enough for you most of the cloud providers will compensate you in credits if you blog about their services in detail, you do have to put in a substantial effort up front and "appear" on their radar by writing blog posts, presenting at meetup and conferences but once they see you they will gladly put $100 a month towards your further learning.


How to bind ports for docker container in bitbucket pipelines? I'd like to test an application using bitbucket pipelines with a custom docker image running some services. I have a dev docker image which runs all the services I need in order for tests to pass.

docker run -it -p 3000:3000 -p 6379:6379 -p 8983:8983 my_dockerhub/image
./start_services.sh
bundle exec rspec
# everything passes

I can't seem to find a way to start the container with the right ports bound so the tests to pass.

I tried using the container directly using this in my bitbucket-pipelines.yml:

image:
  name: my_dockerhub/image
  username: $DOCKER_HUB_USERNAME
  password: $DOCKER_HUB_PASSWORD
  email: $DOCKER_HUB_EMAIL
  # bind ports here somehow?

pipelines:
  branches:
    '{master, develop, bitbucket_pipelines}':
      - step:
          name: Test
          script:
            - ./start_services.sh
            - sleep 30
            - bundle exec rspec

But I get the following error at the bundle exec rspec step

# Errno::ECONNREFUSED:
#   Connection refused - connect(2) for "127.0.0.1" port 8983

I tried running the docker container directly within the bitbucket pipeline

pipelines:
  branches:
    '{master, develop, bitbucket_pipelines}':
      - step:
          name: Start Docker
          script:
            - docker login -u $DOCKER_HUB_USERNAME -p $DOCKER_HUB_PASSWORD
            - docker run -t -p 3000:3000 -p 6379:6379 -p 8983:8983  my_dockerhub/image
            - ./script/start_services.sh
            - sleep 20
            - bundle exec rspec

But the step froze at the docker run step.
I assume this is because I used the -t flag so the command doesn't exit. I was thinking I could possibly use docker exec to send commands to the container without running it  but then how would I check the tests have passed?

Try running with -d detached flag

docker run -td -p 3000:3000 -p 6379:6379 -p 8983:8983 my_dockerhub/image
 #       here ^

Containers started in detached mode exit when the root process used to run the container exits. This will prevent the step to freeze at docker run stage.

Docker docs -d flag


How to check application responding in Kubernetes I am a newbie in k8s. I have a .net framework application written in C# running in K8s orchestration. I simply want to check in fixed interval (lets say in every 300sec) that my application which is running inside pod is responding or not.
Can anyone suggest how to implement this ?

Kubernetes supports TCP, HTTP, and shell-exec probes for an application startup, readiness, and liveness health checks. If your application is a web application, you can use HTTP based liveness healthcheck as shown in the following code snippet:
apiVersion: v1
kind: Pod
metadata:
  labels:
    test: liveness
  name: liveness-http
spec:
  containers:
  - name: liveness
    image: k8s.gcr.io/liveness
    args:
    - /server
    livenessProbe:
      httpGet:
        path: /healthz
        port: 8080
      initialDelaySeconds: 3
      periodSeconds: 300


How to create replicas in docker-compose when `scale` is deprecated? According to docker-compose issue #5586, scale is deprecated. The replacement is deploy.replicas but that latter is not handled by docker-compose, only by docker stack implying you have to run Docker in swarm mode.

As alternate solutions, you can scale services from the command line with docker-compose:

docker-compose up --scale app=3

I'm not too fond of this solution since in my understanding docker-compose.yml is supposed to be self-contained and should not depend on command line options.

Another option is to run docker-compose in --compatibility mode:

docker-compose --compatibility up

In that case, docker-compose will translate some deploy entries into their version 2 equivalent. But because of the name of that option, and because of the warnings in the docs, it does not look like a mid- to long-term solution.

What would you suggest to replace the docker-compose v2 scale option?

@decoomanj on github indicated that scale has to be replaced by deploy in conjunction with replicas.

According to mrampant-nist on github one has to use --compatibility as well.

Let's test this. First of all all docker-compose.yml is needed:

version: '3.3'
services:
    nginx:
      image: nginx
      scale: 6

and issuing:

docker-compose up

results in:

ERROR: The Compose file './docker-compose.yml' is invalid because:
Unsupported config option for services.nginx: 'scale'

Applying @decoomanj's suggestion, i.e. replacing replicas:

version: '3.3'
services:
    nginx:
      image: nginx
      deploy:
        replicas: 6

results in:

WARNING: Some services (nginx) use the 'deploy' key, which will be ignored. Compose does not support 'deploy' configuration - use `docker stack deploy` to deploy to a swarm.

Running docker-compose --compatibility up starts 6 services:

CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS               NAMES
f19aa54634d7        nginx               "nginx -g 'daemon of…"   5 seconds ago       Up 4 seconds        80/tcp              6233_nginx_5
749cff4ac1c3        nginx               "nginx -g 'daemon of…"   5 seconds ago       Up 3 seconds        80/tcp              6233_nginx_3
b86ed7b217fe        nginx               "nginx -g 'daemon of…"   5 seconds ago       Up 1 second         80/tcp              6233_nginx_6
0b92e5ef4a0e        nginx               "nginx -g 'daemon of…"   5 seconds ago       Up 2 seconds        80/tcp              6233_nginx_2
e414bca52b53        nginx               "nginx -g 'daemon of…"   5 seconds ago       Up 4 seconds        80/tcp              6233_nginx_4
e9f33dc634a8        nginx               "nginx -g 'daemon of…"   6 seconds ago       Up 5 seconds        80/tcp              6233_nginx_1

In conclusion, replicas has to be replaced by deploy and replicas and --compatibility is needed to solve the issue.


Do any services other than EC2s consume ip addresses in AWS? SCENARIO:
I am running out of free ips in my AWS subnets. I would like to talk to my infra team and request them to create new subnets. But they are asking me list of all ips I have consumed so far and to show how each ip is used

TOTAL IP CAPACITY:
So I see there are a total of 384 ips available across 8 subnets in 2 VPCs.

FREE IPs:
Now I can see from the VPC page that there are a total of 126 free ips available.

TOTAL IPs CONSUMED:
So this means 384-126 = 258 ips have been consumed in this account.

Total # of EC2s:
I also got a list of all EC2s from the "Tag Editor" page on AWS Console. That list shows a total of 72 EC2s.

DOESN'T ADD UP
So 258 ips have been consumed as per #3 above
But there are only 72 EC2s created totally as per #4 above

So who consumed the remaining 258-72 = 186 ips?
Do any other services other than EC2 consume ips?
Where is the disconnect?

There are many AWS services which requires a network interface such as AWS load balancers, Lambda, NAT gateway, VPC endpoints etc. For each AWS load balancer, it may use up to 2 IPs and 1 IP for 1 lambda function.
I would recommend to check the network interface section under EC2 service which will show you all the IPs which are currently being used and attached to EC2, Lambda, load balancer or some other service.
NOTE: For each subnet, 5 IP addresses will be reserved by AWS. So, if you have a subnet with /28 subnet mask (16 available IP addresses ), in AWS, you will get only 11 IPs which can be used.


How can Agile methodologies, CI/CD be applied to massive, monolithic software projects with 100s/1000s of developers? Using continuous integration represents an (almost) mandatory requirement for a software project to be agile. And without CI there can be no continuous delivery/deployment.

But CI can face scalability challenges, see How can continuous integration scale for very large projects/teams?.

Splitting work on multiple smaller pre-integration branches would allow work on those branches to be agile, but then merging of those pre-integration branches needs to be handled. And those would be massive, unpredictable merges, rather difficult to plan and incorporate into an Agile framework. Until the merge is actually performed one can't really tell if it'll take 2 hours, 2 days or 2 entire sprints to complete.

So how can Agile methodologies still be applied to massive, monolithic software projects with 100s/1000s of developers?

Well, you won't be able to apply an Agile methodology to a large team. One of usual principle is to work on Pizza Team (Less than 10 persons which can share a large pizza for diner together) because the lack of formalism advocated by Agile process makes it hardly applicable to a large team, the information would be scattered and some would be lost in the crowd.

So before thinking about the CI system, you'll have to review the organization around the software you build.
This mainly need to refactor a large project in smaller independent pieces, this could be separate softwares (and you'll take the path of microservices) or it could just be an architecture design within the software, where a class/object/library is given to a small team. This team has to document its service input and output (service contract/service orientation).

Once this is done you can split the responsibility of each independent piece to a smaller team which will be able to apply an Agile process.
This may solve the CI/CD problem, splitting the code base would allow each team to follow it's own pace, as long as it's interface is properly versioned and that the input/output are fixed for a type of call you can make parts of the product evolve while some other part keep a previous behavior.

The main drawback is that the service contracts from each part must be compatible between versions in use, which need a dependency system to know which part use which version of what before dropping an unused version of an API. This brings problems to make breaking changes and usually need a careful transverse planning between producer and consumers.


Reset time vs alerting window size in alerting strategies for SLOs In https://landing.google.com/sre/workbook/chapters/alerting-on-slos/#pros_and_cons_of_alerting_based_on_burn_r I don't understand how the reset time can be 58 min when the alerting window size is one hour.

How can the reset time be shorter than the alerting window size?

Consider a service using the alerting rule from the book:

job:slo_errors_per_request:ratio_rate1h{job="myjob"} > 36 * 0.001

The burn rate is the rate at which we are exhausting our error budget. A burn rate of 1 means that we'll have exactly exhausted our error budget at the end of our reporting period. A burn rate of 2 means that we'll have exhausted our budget half way through our reporting period.

For a 99.9% SLO, an error rate of 0.1% constitutes a burn rate of 1. An error rate of 1% means a burn rate of 10, and a 100% error rate means a burn rate of 1000

If we want to alert when we've exhausted 5% of our 30 day error budget in 1 hour, then that means we are burning error budget at a rate of 36, which translates to an error rate of 3.6% for a 99.9% SLO.

Imagine a service with a constant rate of requests, that starts serving 100% errors at t=0 minutes

Here our SLI counts the total number of "events".

SLIs written in terms of some kind of discrete events, where an event can either be good or bad, have many nice properties: They range from 0-100%, where 0 is "nothing is working" and 100% is "nothing is broken"; I find this quite intuitive. By making all of your SLIs behave the same, you simplify tooling, alerting rules, and understanding.

With a request-based system, events are usually individual requests. So 0 errors, 1 request in minute at t=-2. I used 1 request per minute for simplicity. The result is the same if you have 1 request per minute or 1 million; it's the ratio of errors to total requests that matters.

t=-2, errors=0, total=1
t=-1, errors=0, total=1
t=0, errors=1, total=1
t=1, errors=1, total=1
...

If our monitoring system evaluates its rules every minute, we get:

at t=-1, our 1h average error rate is 0
at t=0, the 1h error rate is 1/60 = 1.666%
at t=1, the 1h error rate is 2/60 = 3.333%
at t=2, the 1h error rate is 3/60 = 5%

Our alert fires 2 minutes after the event started.

Let's assume our incident continues until t=60.

at t=60, our 1h error rate is 100%: a burn rate of 1000, for a 99.9% SLO
at t=61, the 1h error rate drops to 59/60 = 98.333%
at t=62, the 1h error rate is 58/60 = 96.666%
at t=118, the 1h error rate is 2/60 = 3.333%

At this point, our 1h error rate is below the alerting threshold, and so the alert is no longer in the firing state, 58 minutes after the end of the event.

at t=120, our 1h error rate finally drops back to 0%, as no errors have been observed in the last hour.


Configuration of AWS CodePipeline for Android CI/CD I'm trying to set up a basic CI/CD pipeline for an Android project using AWS CodePipeline and I have 3 steps in the pipeline as follows:

Source: Pull project from GitHub repo
Build: Build the android app using a docker image (futurenda/ android-sdk)
Test: Run instrumentation tests (espresso tests)

Currently the Source and Build steps work fine (both are successful). From my understanding of running tests in CodePipeline, you can use CodeBuild and simply invoke the tests as you normally would from the command line. In this case, to invoke Espresso tests I can run sh gradlew connectedAndroidTest on my local machine.

I've set the pipeline up with the Build and Test steps being 2 seperate CodeBuild projects, each with their own buildspec.yml file. For the Build step (which is working), the file is named buildspec-build.yml, and is found by CodeBuild without issue. However, for the Test step (not working) my buildspec.yml file is not found. They are both located in the root of my project (and confirmed to be pushed to my remote GitHub repo).

The error I receive is: stat /codebuild/output/src712781592/src/buildspec.yml: no such file or directory. This happens during the DOWNLOAD_SOURCE step in the CodeBuild project. I've checked to CodeBuild project to ensure it is indeed expecting a file named buildspec.yml.

Any ideas?

Here are both of my buildspec.yml files, if it matters:

buildspec-test.yml (found by CodeBuild):

version: 0.2
phases:
    build:
        commands:
          - $ANDROID_HOME/tools/bin/sdkmanager "build-tools;27.0.3" "platforms;android-27"
          - sh gradlew assembleDebug
    artifacts:
         files:
             - app/build/outputs/apk/debug/app-debug.apk
             #some other stuff here.

tester-buildspec.yml (not found by codebuild):

version: 0.2
phases:
  build:
    commands:
      - sh gradlew connectedAndroidTest
artifacts:
  files:
    #other stuff that doesn't really matter here

Just to recap: I have a pipeline set up with 2 CodeBuild projects as steps within the pipeline. The first CodeBuild project finds the buildspec.yml file, but the second one doesn't.

The indentation looks incorrect for the artifacts section in your buildspec-test.yml file. It is nested under the phases section. That way it is not being output properly and can't be picked up by the next build.

It should be the same as in your second file.

version: 0.2
phases:
  build:
    commands:
      - $ANDROID_HOME/tools/bin/sdkmanager "build-tools;27.0.3" "platforms;android-27"
      - sh gradlew assembleDebug
artifacts:
  files:
    - app/build/outputs/apk/debug/app-debug.apk
  #some other stuff here.


Trigger Jenkins job from Gitlab CI with parameters There is supposed to be a way to trigger a Jenkins job via GitlabCi, using the respective plugin.

My question is whether there is a way:

a) to trigger a parameterized Jenkins job

b) to pass parameters when triggering the job

Here's the way I do it: no plugin required, just triggering Jenkins api from gitlab-ci.

Gitlab-CI

I will assume you have a gitlab-ci runner installed and configured.

First, you need to have a .gitlab-ci.yml file in your project having a basic structure such as:

stages:
- my-jenkins-trigger
variables:
 MY_VARIABLE: "EVERYTHING_IS_AWESOME"

my-jenkins-trigger-job:
 stage: my-jenkins-trigger
 script: curl -i -X POST --user JENKINS_USER:JENKINS_TOKEN JENKINS_JOB_URL/buildWithParameters?MY_JENK_PARAM=${MY_VARIABLE}

In the above, I also assume

You have a Jenkins job somewhere at URL JENKINS_JOB_URL
this Jenkins job has a build parameter called MY_VARIABLE
JENKINS_USER, JENKINS_TOKEN are defined [*]

That simple?

Well yes, but no...

That is the rough structure. That script will merely trigger a Jenkins job and forget about it. You need to work a little more to monitor the job and feed its status back in Gitlab-CI, manage security and possibly get some commit info from gitlab to inject into your job.

Monitoring

In order to have a proper monitoring, I recommand to write a full trigger + monitor + return value script [** ] (in whatever language available or you're familiar with).

Just start by triggering the job as I stated above.

Then, run a while loop (don't forget to put it to sleep [***]) on

curl --silent --user JENKINS_USER:JENKINS_TOKEN JENKINS_JOB_URL/lastBuild/api/json | grep result\":null > /dev/null

until the result of this command is not 0.

Once the Jenkins job is finished, you would probably want to fetch the job's console in Gitlab

curl -i -X POST --user JENKINS_USER:JENKINS_TOKEN JENKINS_JOB_URL/lastBuild/consoleText

Finally you may curl once more on JENKINS_JOB_URL/lastBuild/api/json but this time you grep it on UNSTABLE, SUCCESS or FAILURE.

Discussion

By following the guidelines above, you can fully orchestrate Jenkins jobs from Gitlab-CI. I've posted a long discussion on why and when should you do this.

I hope this will help you.

[*] Your Gitlab project Settings > CI/CD > Secret vaiables
[** ] Of course I mean by that to craft a script nicely with params, functions, nice variable names, meaningful logs... You name it.
[***] I found a sleep of 20 seconds worked for me


Docker Container capacity planning I'm running my application on 4 Virtual Machines with 8 3.2 GHz Virtual CPUs and 32 GB, however, I will be splitting up the processes into separate containers.

I am not sure how many containers to run per host. What are the typical numbers?
 For example how the VM to bare metal server ratio is 1:10 generally, any links on attributes to consider, decision frameworks to consider or experiences would help!

The same as any Linux load, consider no more processes in wait state than the number of CPU.
The load on 1, 5 and 15 mins given by uptime should ideally be 1 less than number of cores.

Containers are roughly isolated processes, leaving a core for orchestration avoid congestion.

That doesn't mean only 7 containers on a 8 cores machine, it's a matter of simultaneous load, so the best way is to measure it and adapt consequently.


How to create a centralised dashboard to monitor multiple cloud (Amazon and Google)? I'm looking for a single product or service which can be used as a centralized dashboard to monitor cloud infrastructure provide by Amazon Web Services and Google Cloud Platform in addition to the application.

I'm looking at few options like sensu which support all of the services in Amazon Web Services, but I didn't find any plugins to monitor Google Cloud Platform.

Are there any other services or products that allow me to create this, my preference is Open Source?

Note: Everyone has access to GCP and AWS and if anyone launches something it will be easy to have one dashboard for monitoring.

I think what you are looking for is an Open Source project that can take inputs from both Amazon CloudWatch and Google StackDriver; there isn't a huge amount out there at the moment, but I will detail what I know.

I have made the assumption that you know how to import your application telemetry into the solutions below.

Open Source

Open source solutions to this problem have been lagging behind as late due to favourable prices of SaaS and built-in solutions.  The products below are constantly evolving so if they don't do what you need now they may be in the future.

Elastic Stack, formerly known as the ELK stack is made up of three individual products from Elastic.co: ElasticSearch, LogStash and Kibana.  It is possible to ingest data from CloudWatch with a plugin and to export StackDriver data to Google Pub/Sub and then ingest into LogStash.
Grafana, is slightly better than Kibana at displaying time-series data. However, dosn't have as robust an ecosystem around it.  Grafana has built in support for Amazon CloudWatch, Google StackDriver is not supported as yet. However, someone is working on it in the form of a plugin.

Software as a Service (SaaS)

Software as a Service monitoring solutions is a multi-billion dollar market, as large enterprises will readily pay for a SaaS or off-the-shelf solution to their dashboarding challenge.

New Relic, a very mature product with extensive cloud support, including ingesting of CloudWatch metrics and Google Cloud Storage Metrics.
Splunk, a bit more focused on logs than metrics, however, has good support for AWS CloudWatch and Google Cloud Platform.
Google Stackdriver, obviously has good support for Google Cloud Platform, also it is capable of reading performance metrics in from AWS.
DataDog is my personal favourite, it has a vast range of integrations including support for Amazon Web Services and Google Cloud Platform.

Single Pane of Glass

In the last three years, various products have gone to market providing a "Single Pane of Glass".  The advantages of these products are:

Heterogeneous Platforms - you can use many different monitoring tools and products and then unify them with a Single Pane of Glass.
Data Normalization - most monitoring tools have idiosyncrasies in the way they present data a Single Pane of Glass helps to normalise these so that metrics from Google, Amazon and your App make sense when displayed in the same graph.
Business View - a Single Pane of Glass usually has the capability to integrate with non-technical data sources.  As an example displaying data from financial tools to show a measure of business success.  This view makes for some compelling C-Level dashboards that demonstrate how technology is providing value to the firm.


Downloading multple files from remote using chef resource I am wondering if there is a resource to download multiple files from remote using chef resource. I want to use:

remote_file 'Download remote file' do
  path /opt/
  source http:///xxx
  mode '0644'
  action :create
end

...for one file. What if I want to download multiple files in a same place with different urls? Or how can I modify this resource to make multiple downloads which have different URLs?

This is how I added attributes in recipe:

node['file']['name'].each do |pkg|
    remote_file "path/plugins/#{pkg}" do
        source "node['file']['url']/#{pkg}"
        action :create
    end
end

in attributes

default['file']['name'] = %w(
    xx-2.0.jar
    xx-2.jar
    xx.jar
    )

what about something like that:

%w{
        mysql-community-common-5.7.16-1.el7.x86_64.rpm
        mysql-community-libs-5.7.16-1.el7.x86_64.rpm
        mysql-community-client-5.7.16-1.el7.x86_64.rpm
        mysql-community-server-5.7.16-1.el7.x86_64.rpm
}.each do |pkg|
        remote_file "/tmp/#{pkg}" do
          source "https://s3.amazonaws.com/tmp/mysql/#{pkg}"
        end

       rpm_package pkg do
        source "/tmp/#{pkg}"
        action :install
      end
end

another way:

urllist = {
  { 'url': 'http://some.url1/', 'path': '/some/path1/', 'filename': 'some.file' },
  { 'url': 'https://some.url2/', 'path': '/some/path2/', 'filename': 'another.file'}
}

urllist.each do |urlinfo|
  remote_file "#{urlinfo['path']}/#{urlinfo['filename']}" do
    source "#{urlinfo['url']}/#{urlinfo['filename']}"
    owner 'someowner'
    group 'somegroup'
    mode 0755
  end
end


How do you test changes to Jenkins plugins before deploying them? If you ever been bitten by a plugin upgrade that broke some functionality you must have given some thought about this problem: What should be the Jenkins plugins upgrade policy? How do you test changes before deploying them?

Has anyone gone as far as having a test instance running dummy jobs to test the new versions or do you just pray that upgrading the versions won't break anything?

According to the policies of the company where I work we have dev, preprod and prod environments (on some service dev can be missing). And the path of new version preprod->tests->validation->prod.

In our case jobs in preprod are heavy and complex enough to be sure we do not need to pray when implement in prod :)

Note: We use svn to maintain and deliver configuration. We do not make changes inplace.


Sonar Quality Gate behavior in Jenkins pipeline is unstable I have some pipelines in Jenkins, with the SonarQube stage (Version: 6.7.1, build 35068)
In some builds it is aborted due to time out:

12:19:01 Sleeping for 3 min 0 sec
[Pipeline] timeout
12:22:10 Timeout set to expire in 25 min
[Pipeline] waitForQualityGate
12:22:10 Checking status of SonarQube task 'XXXXXXXXXX' on server 'Sonar'
12:22:10 SonarQube task 'XXXXXXXXXX' status is 'PENDING'
12:47:10 Cancelling nested steps due to timeout

and in some other builds it pass:

13:56:47 Sleeping for 3 min 0 sec
[Pipeline] timeout
13:59:47 Timeout set to expire in 25 min
[Pipeline] {
[Pipeline] waitForQualityGate
13:59:47 Checking status of SonarQube task 'XXXXXXXXXX' on server 'Sonar'
13:59:47 SonarQube task 'XXXXXXXXXXXX' status is 'SUCCESS'
13:59:47 SonarQube task 'XXXXXXXXXXXX' completed. Quality gate is 'WARN'

Trigger command in bitbucket file:

    stage('Sonar Quality Gate') {
        sleep 180
        timeout(time: 25, unit: 'MINUTES') {
            def qg = waitForQualityGate()
                    if (!(qg.status == 'OK' || qg.status == 'WARN')) {
                        error "Pipeline aborted due to quality gate failure: ${qg.status}"
                    }
        }
    }

what might be the reason for it? how can I keep it stable? can it be network isssue?

It was related at the end to Sonar DB performance (after getting AWR reports):
https://stackoverflow.com/questions/59982345/how-to-maintain-cleanup-of-the-sonar-table-project-measures


How good are heartbeat, elasticsearch and kibana as a monitoring/alerting solution as a replacement for Nagios? Since we already have filebeat, metricbeat and heartbeat configured and running.

I wonder if people have successfully implemented the ELK stack for actual monitoring with alerting/notifications instead of using traditional software as

Nagios
Icinga
Zabbix

Especially since ElasticSearch now has had watchers and alerting for a while.

I haven't tried it myself, but in Alarms management with Elasticsearch and Kibana someone recommended Elastalert


What is the cheapest way to get CI for a Qt project? To build up my portfolio I'm writing an application to showcase all the best practices I'm aware of, including unit tests and continuous integration.

The project was originally started in C++, and I took advantage of buddy.works to compile it and run my tests there. Managing CMakeLists was a pain but I endured for the greater good.

After a while I noticed that Qt would have been a better fit for my purposes. Being this a non-commercial open source project I'm allowed to use Qt Creator for free. However, buddy.works (and almost all other free CI tools) only allow bare GCC build environments. In my case I'd need a fully configured Qt environment, as when I install the IDE, plus Android SDK and NDK.

I see buddy.works allows to run commands in a "private container", but I'm not sure if building a container myself (something I'd start looking into if I knew this would work).

I guess the safest choice would be to rent an Amazon EC2 instance, install on that everything I need (Qt, SDKs, Jenkins) and configure it to communicate with my GitHub or Bitbucket, but it would cost me (and I can't say how much exactly). I expect the CI process to only run like thrice per week, with an average build time that at most would take half an hour. I also liked the idea of the green build badge in the readme file, something buddy.works offered and with an EC2 instance shutting down could be trickier to achieve (if I was to purchase an instance that boots every day, runs CI and shuts down). However, a free alternative would be much appreciated.

Any hint about this? Thank you very much.

One of the major advantage of the container technology is standardisation: easily replicating the same environment wherever a specific container is used, be it, for example, on your own machine during development or on the CI machine during the execution of the CI/CD pipeline.

Building and using custom containers is a pretty standard way of including OS-level dependencies (like libraries and toolsets, for example) which aren't included in a service provider's default container images. Depending on the provider the choices for the base image you'd use might be limited.

If you have a CI/CD provider which accepts custom-built containers (not all of them do, at least not yet) and you can meet their custom container restrictions (if any) then building the custom container for your case is IMHO a valuable investment. Not only for the CI execution, but even for developing of your app and ultimately for showcasing the container technology as a development technology itself :)


Building a CI/CD pipeline on GCP I have a containerized web application on GitHub and I would like to build a CI/CD pipeline on GCP consisting of a trigger stage, a build stage and a deployment stage and be able to do continuous integration and deployment. What GCP tools can I use for this task? I have successfully accomplished the same task on AWS with Jenkins, where my GitHub commit triggers a multi-stage pipeline and deploy the app to EC2.

This is what I have been able to do on AWS:

For building use Google Cloud build. For deployment you can you use Google Cloud Deploy.
https://cloud.google.com/deployment-manager/docs/
https://cloud.google.com/cloud-build/docs/


Separate SQL Server or just separate SQL Database for test and production? I'm new to SQL Server, so this may be more of a SQL Server management question.

I'm creating test and production databases for a service, and I imagine I'll be wiping the test database quite a lot. Additionally, I'm going to want different replication and audit logging strategies.

Does it make sense to have both SQL Databases on the same SQL Server, which seems to be what the Azure Portal makes easiest to manage, or does it make more sense to create a separate logical SQL Server for the test and production databases?

Go with a separate logical server:

It is going to cost the same because you are charged on the tier of the database not the database server.
By providing separate servers you provide isolation between the two databases protecting one from the other.
Unless you use contained users your users will be shared between Test and Production, if you happen to use the same user for both databases a change of the password in production would also require the same change in Test.

You might want to use the same server for a database if:

You are creating a very large number of Databases across many Dev/Test environments as there is a limit to the total number of SQL Servers.
You have two databases that need to be deployed as part of the same ARM template.


can't access my instances by public IP in a VPC (aka: what is wrong with my terraform)? As stated in the title of the question, I have a Terraform file (attached below), but the infrastructure it builds doesn't allow me to connect by SSH (I get "Connection timed out"). So either the packets aren't getting to the EC2 instances or the responses aren't getting out. Any ideas? For the purposes of this question the DNS and ALB can be ignored (the former works fine, the latter almost certainly doesn't yet).

provider "aws" {
  region = "us-west-2"
}

resource "aws_vpc" "vpc" {
  cidr_block = "10.0.0.0/16"
}

resource "aws_internet_gateway" "gw" {
  vpc_id = "${aws_vpc.vpc.id}"
}

resource "aws_subnet" "subnet_a" {
  vpc_id = "${aws_vpc.vpc.id}"
  cidr_block = "10.0.1.0/24"
  availability_zone = "us-west-2a"
}

resource "aws_subnet" "subnet_b" {
  vpc_id = "${aws_vpc.vpc.id}"
  cidr_block = "10.0.2.0/24"
  availability_zone = "us-west-2b"
}

resource "aws_route_table" "rt" {
  vpc_id = "${aws_vpc.vpc.id}"

  route {
    cidr_block = "0.0.0.0/0"
    gateway_id = "${aws_internet_gateway.gw.id}"
  }
}

resource "aws_route_table_association" "rt_a" {
  subnet_id = "${aws_subnet.subnet_a.id}"
  route_table_id = "${aws_route_table.rt.id}"
}

resource "aws_route_table_association" "rt_b" {
  subnet_id = "${aws_subnet.subnet_b.id}"
  route_table_id = "${aws_route_table.rt.id}"
}

resource "aws_security_group" "alb_sg" {
  vpc_id = "${aws_vpc.vpc.id}"

  ingress {
    from_port = 0
    to_port = 80
    protocol = "tcp"
  }

  ingress {
    from_port = 0
    to_port = 443
    protocol = "tcp"
  }
}

resource "aws_alb" "alb" {
  internal = false
  load_balancer_type = "application"
  security_groups = ["${aws_security_group.alb_sg.id}"]
  subnets = ["${aws_subnet.subnet_a.id}","${aws_subnet.subnet_b.id}"]
}

resource "aws_route53_zone" "primary" {
  name = "REDACTED.REDACTED"
}

resource "aws_route53_record" "alias_route53_record" {
  zone_id = "${aws_route53_zone.primary.zone_id}"
  name    = "REDACTED.REDACTED"
  type    = "A"

  alias {
    name                   = "${aws_alb.alb.dns_name}"
    zone_id                = "${aws_alb.alb.zone_id}"
    evaluate_target_health = true
  }
}

resource "aws_key_pair" "kp" {
  key_name = "kp2019y09m25d001i"
  public_key = "ssh-rsa REDACTED"
}

resource "aws_instance" "instance_a" {
  ami = "ami-08ba47c10a42c61be"
  instance_type = "t2.micro"
  availability_zone = "us-west-2a"
  subnet_id = "${aws_subnet.subnet_a.id}"
  associate_public_ip_address = true
  key_name = "${aws_key_pair.kp.key_name}"
}

resource "aws_instance" "instance_b" {
  ami = "ami-08ba47c10a42c61be"
  instance_type = "t2.micro"
  availability_zone = "us-west-2b"
  subnet_id = "${aws_subnet.subnet_b.id}"
  associate_public_ip_address = true
  key_name = "${aws_key_pair.kp.key_name}"
}

output "instance_a_ip" {
  value = "${aws_instance.instance_a.public_ip}"
}

output "instance_b_ip" {
  value = "${aws_instance.instance_b.public_ip}"
}

Your default security groups and newly created security groups include default rules that do not enable you to access your instance from the Internet.
Add this:

resource "aws_security_group" "allow-ssh" {
  vpc_id = "${aws_vpc.vpc.id}"

  ingress {
    from_port = 0
    to_port = 22
    protocol = "tcp"
  }
}

Then add the line vpc_security_group_ids = [aws_security_group.allow-ssh.id] to the instance definitions.


Helm/ YAML indentation issue I am etting weird YAML indentation with some conditional stuff, effectively I'm doing:

resources:
{{ toYaml .Values.resources | indent 10 }}
{{- if .Values.filebeat.enabled -}}
      - name: elastic-logger

But the final line is indented by 10, not 8 as it should be. in fact I can indent it by 0, 10, 100 - but it's always indented 10. any ideas? (edited)
.Values.resources is for one container, and if filebeat.enabled then I'm adding another container to the array
output:

 resources:
 {}
    - name: elastic-logger

Turns out -}} is a special syntax which says 'eat up any following whitespace'.  Removing the - fixed this.

I get a blank line, but don't really care, considering!


Delivery in a SalesForce environment In an organization with a hundred developers, and several thousand salespeople using SalesForce to create their application. Given the restrictions of SalesForce environment and deployment options.

What would be the process and which tools to use to reach a delivery pipeline where changes can be delivered to production many times per day?

The basic flow is:

salespeople think of ideas for improving their flow (bugs, new dev).
product managers distill the requirements and pass onto developers.
developers write code and create configuration changes in git.
magic happens (currently takes about 3-4 months).
new changes are in production and salespeople can use it.

I would recommend to create an issue about CD in the ticket system of salesforce. I have googled a little bit and this seems to be their issue tracker.

I would prevent to implement CD in salesforce as that will be very cumbersome and will take a lot of time. In my opinion this issue should be solved by SalesForce itself.


Accessing AWS Secrets Manager from Docker All the tutorials I've seen for AWS Secrets Manager have secrets being retrieved by the CLI. How is this done in production?

Can I mount a set of secrets to a docker container as a file?

Access to AWS Secrets Manager requires AWS credentials. Those
  credentials must have permissions to access the AWS resources that you
  want to access, such as your Secrets Manager secrets. The following
  sections provide details on how you can use AWS Identity and Access
  Management (IAM) policies to help secure access to your secrets and
  control who can access and administer them.

so you need to provide the aws credentials to docker which has access to the manger

you can follow on of these methods:

providing the credentials with run command:

docker run -e AWS_ACCESS_KEY_ID=XXXX -e AWS_SECRET_ACCESS_KEY=XXXX myimage

or providing the file ~/.aws/credentials and then create your container:

docker-machine create --driver amazonec2 --amazonec2-open-port 8000 --amazonec2-region us-west-1 aws-sandbox

see this


How to get code from Bitbucket via Octopus? Is there a way that I can tell Octopus to take its code from Bitbucket and not GitHub (without using Bitbucket's pipelines)? I couldn't anything under community steps that would let me do that (somewhat surprisingly).

It appears that Octopus supports a push from BitBucket, but not a pull (https://octopus.com/blog/bitbucket-pipes-and-octopus-deploy)
Github supports either option: https://octopus.com/docs/packaging-applications/package-repositories/github-feeds


EC2 EBS Free Tier - Is Storage Reusable? Can I delete an EC2 instance and start off from scratch on my EBS free tier usage? For instance, I was running a Red Hat server with 10 GB of EBS being used. However, I deleted that EC2 within a few hours. I then spun up another machine with 8 GB of EBS. Do I now have 22 GB of storage left for the month or just 12 GB, after combining the 8 with the 10?

I assume, given that AWS doesn't reference hourly usage on their free tier category for EBS that the 30 GB is just a flat fee. Again, I simply don't understand if I can replete an instance, then start from scratch to zero EBS on a new instance.

While I can't find it explicitly written anywhere, I do know that textbooks frequently reference that you can use the free tier to avoid costs, and that they frequently have you create and destroy instances between chapters.  So, I find it extremely hard to believe that you would get billed for deleting and creating new EBS volumes.

So, I believe that if you create a 30GB one, destroy it, and create another, you should be fine.  You should definitely make sure to remove the volumes and any related snapshots though just to be safe.

They definitely don't seem to explicitly confirm that though... e.g. in text like this:

  Notable offers for the first 12 months following your AWS account sign-up include some level of free usage for Amazon Elastic Compute Cloud (EC2), Amazon Elastic Block Store (EBS), Amazon S3, Amazon Relational Database Services (RDS), and many other useful services.

They use the free tier to get customers hooked.  They want you to like it and not be frustrated so you're more likely to use the real services and so that you may advocate them in your businesses.  So, by that logic, I similarly find it hard to believe they would "trick you" by billing you in this situation without making it more obvious.  That would be bad for sales.


What is a glob in salt-stack? I was going through targeting minions in salt. There I came across a term called glob. Can someone explain with an example what is a glob in Salt-Stack?

Globbing is a term used for matching file name patterns with wild cards.  You can see a more detailed definition here https://searchsecurity.techtarget.com/definition/globbing

As it relates to Salt Stack, it's finding the minion id using wild cards.  Additional details here: https://docs.saltstack.com/en/latest/topics/targeting/globbing.html


Disabling HTTP Basic auth for requests from specific ips in k8s ingress Currently I have a webpage which is freely accessible from certain networks an requires HTTP basic auth otherwise. This behaviour is set in the nginx config. It looks something like this:
geo $authentication {
    default "Authentication required";
    SOME-IP "off";
    SOME-NETWORK1 "off";
    SOME-NETWORK2 "off";
}

...
auth_basic $authentication;
...

Now I want to move this webpage to a k8s cluster under ingress-nginx ingress controller. As I can see from the docs it has support for basic auth but provides no ways to allow access from specific ips/networks. Is there a way/another ingress controller/tool to configure said behaviour?

The thing I was looking for was external-auth in ingress-nginx. I created a simple service with nodejs which took over all the basic-auth logic. Then I used an external link to this service in k8s annotations for restricted resources.
annotation snippet:
  annotations:
    nginx.ingress.kubernetes.io/auth-url: http://service-name.default.svc.cluster.local:3000/auth


Continuous Integraton using Jenkins Pipeline for a .NET project The project:
A .NET standalone application consisting in:

The main application that is a Windows Service containing a web server used for the web interface. Project is 90% .NET (Framework, Standard and Core) plus a few C++ libraries
Windows Agents that will be deployed naturally on Windows machines
ASP.NET Core Agents that will be deployed on Linux machines
WIX used for packing and creating the installer

We don't have a dedicated and experienced Devops team member, but we currently have in place a sort of CI/CD pipeline in VSTS. Nothing fancy, just a bunch of build definitions that run the unit tests for every commit and nightly the integration and Selenium tests split into several builds across multiple VMs (due to MSTest parallelization at the assembly level in order to decrease the run time), and another one that is manually started to create the installer.

In the near future we may need to switch to Jenkins and after spending some time messing with builds in it I grew really fond of the Pipeline plugin (especially with the Blue Ocean one). On the other side, I discovered that if you're not developing in the Java world, you're kind of screwed with even the most basic tools required for .NET like MSBuild and MSTest or VSTest lacking Pipeline support.

Only to be able to run the tests and display the results we had to do a bunch of steps, mostly consisting in batch scripts and add extra tools to be able to build, perform dotnet restore, nuget restore, run the tests, transform the .trx file into JUnit supported ones to be able to display them.

Considering all the info above, I humbly ask the following:

What should an "upgrade" of the existing build system look like?
Are there any buried resources/guides/starting points in how we could make use easier of the Pipeline in the .NET world?
Anything else that I may be missing?

Specifically in answer to your 3rd question, if you are willing to look outside of the Jenkins Ecosystem there are alternatives out there that might be of value to you.

For my clients who use the Microsoft Stack and have fewer than four teams, I have been recommending the use of AppVeyor it is highly tuned for the .NET Stack and integrated very naturally with msbuild and Wix.

There are two ways of configuring AppVeyor, either through the web-based user interface or via the appveyor.yml file checked into the root of the git repository.  I would strongly recommend the latter, you are welcome to start from this template:

version: 3.0.{build}

build:
  parallel: true
  project: EVEMon.sln

cache:
  - packages -> **\packages.config

install:
  - nuget restore

The major drawback of AppVeyor and it's cousins TravisCI and CircleCI is they don't play particularly nicely with on-premise source code management solutions, you do really need to be using GitHub or BitBucket.


Which cloud region could have the fastest time for a docker image push? TLDR
How to know what region in the cloud nearest to docker hub?

Full details
From my localhost my network is super slow to push the docker image onto docker hub.

My workaround is to use a cloud machine to do that docker image push hopefully that the internet there would be faster.

So my question is which region on the cloud is nearest to docker hub for me to choose?

My google search results little helpful

p.s.
I also asked on stackoverflow here

TLDR

From cloud instance, just go push to docker hub; you will be routed to the best location automatically.

Full details

DockerHub is a massive scale and very critical global service.  So, you almost definitely don't have to think about this at all.

There are techniques on cloud hosts (AWS, Azure, GCP) to replicate & host content/services in regions across the globe and route traffic to the closest entry point automatically.  The cloud hosts make this quite easy to do with their managed services (while without them it would be very difficult).

Also, there are many third party providers of technologies like CDNs which do similar things at scale (and they are really good at it).  They are extremely frequently used in these cases.

So, again - just go to docker hub and don't think about it.  You will be routed to the best location automatically.


Using Jenkins gitlab token to make an API call From inside a Jenkins pipeline, I want to create a merge request on gitlab

def toJson = {
    input ->
    groovy.json.JsonOutput.toJson(input)
}
def body = [
    private_token: "...",
    source_branch: "...",
    target_branch: "...",
    title: "..."
]
httpRequest httpMode: 'POST', requestBody: toJson(body), contentType: 'APPLICATION_JSON', url: 'https://my.gitlab.domain/api/v4/projects/1234/merge_requests'

My problem is with the private_token, I don't want to put it in clear in the Jenkinsfile.

I didn't found a way to get it from Jenkins credentials, is there one ?

Try something similar to the following:

environment {
  GITLAB_API_TOKEN = credentials('gitlab_api_token').
}

or

withCredentials([[
  $class: 'com.dabsquared.gitlabjenkins.connection.GitLabApiTokenImpl',
  credentialsId: 'gitlab_api_token',
  variable: 'GITLAB_API_TOKEN'
]])

See https://jenkins.io/doc/pipeline/steps/credentials-binding/
and  https://github.com/jenkinsci/gitlab-plugin/issues/536 for more information.


Good IDE or plugins for typical DevOps functions? I'm starting up the DevOps initiative at my company and trying to decide what IDE and other development tooling to invest the team in for our various repositories.

Language wise we have a lot of Ruby and Groovy, but that's hardly the entire picture.  We really don't even know fully yet all the systems we will be working with in the near future, and there are some DSLs in the mix:

Akamai DSL
CloudBees / Jenkins logic in Groovy and Jenkinsfile
Chef
Ansible, maybe
Docker, hopefully
One of several release automation products, not yet chosen
I would like to use ServerSpec or something like it
SaltStack DSL
Kubernetes DSL
And so on.

Are there any IDEs with plugins to make working with this stuff a little easier?

that understand the relationships between these entities
that can mock out DNS and network connections to make unit tests possible for these DSLs
that can validate DSL syntax or even detect certain logical errors considering multiple of these?

This reddit thread is very discouraging to me as it seems a lot of folks out there are using basic editors with syntax highlighting plugins, which I view as woefully inadequate.

I personally really like PyCharm (same guys who made IntelliJ and Android Studio).

I mostly use Ansible, Bash, and Python.  Has good plugins + syntax highlighting for Bash (doesn't everything at this point?), and Ansible.  Since it's a native Python IDE, it's also very convenient when writing or troubleshooting python scripts, especially since you don't even need to exit the IDE to run them.

It also has integrations through plugins:

Chef: https://plugins.jetbrains.com/plugin/7548-chef-integration

Ruby: https://plugins.jetbrains.com/plugin/1293-ruby

Both should in theory allow full IDE-like experience with both languages (i.e. resource completion, syntax highlighting, jumping to references, etc).

Really, you can't go wrong with anything by JetBrains if you choose to use any of their other IDEs like IntelliJ.


Three solutions to structure a deployment with Ansible? I am currently deploying a new product and came across some problems to structure my Playbook and Roles. I have three different solutions and hope to get some input which path might be the best.

The product is a webapp on a windows server. The following steps are needed to deploy (high level):

install javajdk
install tomcat
install win_service
configure java regkes
configure tomcat
install webapp
start service

I use static inventories for prod and staging, group_vars, roles (created with ansible-galaxy) so there is some kind of structure.

Solution 1
Put everything in one huge playbook. That does not sound well, but has the advantage that you know the variables from previous installations e.g. tomcat needs to know where JAVA_HOME is... that was with the javajdk installation, due to more than one java on the machines there is no global JAVA_HOME...

Solution 2
Break all out in small roles. Sounds great but roles should be independent from each other. I did not found a method to decouple tasks. Would it make sense to have single roles for e.g. install_tomcat, config_tomcat and set up a playbook which has the mentioned roles in sequence?

But how does one role know about, e.g. a path name which is needed in the next role. -> Installation sets the path, config needs to know.. I an really unsure to make roles independent.

Solution 3
Kind of solution 2. Have a main playbook which contains only roles, split as much as possible to roles. Have independent variable names in ./roles/vars/main.yml

Somehow(?) find a place higher in the variable precedence with the playbook where all needed var definitions go. Internally either path a variable to the role or simply over-rule a role variable.

E.g. a product has a service, the service name is generally part of a path name for tomcat. The tomcat name should not depend on the service, so there should be a tomcat_install_path in the role but also a service_name with the product. So in case the playbook for the product is called the service name should be part of the name, the tomcat_install is called with a total different deployment there should be no fiddling with a service name.

I would go with Solution 3 but I have not yet found a way to set that up. What is the experts opinion, is solution 3 doable, is one of the other once better or is there a 4th solution?

I would recommend to structure the code as defined in the Ansible Best Practices document:

and to follow the structure that major contributors to the ansible galaxy platform use, like geerlingguy.

If one will follow a significant different structure then it will be really hard in the future to reuse roles from the ansible galaxy community and one has to reinvent the wheel.


AWS Storage Cost for a custom AMI How does AWS calculate the cost of a custom AMI?

If I would create an AMI which is based on Ubuntu 16.04, do I need to pay for the complete size of AMI (e.g. 4GB) or just the differences of my custom AMI to the base AMI (e.g. 0.4GB)

AWS stores whole AMI as a single non-differential snapshot thus they charge for a full size of files in an image.


Find relevant domain/hosted zone under which to host subdomain Context
In Terraform, to create a new domain record, I first need to find the zone under which it should be hosted.  E.g. If I want to create a DNS record for mysite.example.com, I'd get the zone example.com and create a record for mysite under it, like so:
variable zone {
    type = string
}
variable subdomain {
    type = string
}
variable targets {
    type = list(string)
    default = ["devops.stackexchange.com"] # just defaulting this stuff to reduce code in this sample
}
data aws_route53_zone this {
    name = var.zone
}
resource aws_route53_record this {
    zone_id = data.aws_route53_zone.this.zone_id
    name = "${var.subdomain}.${var.zone}"
    type = "CNAME"
    ttl = 100
    records = var.targets
}
output fqdn {
    value = aws_route53_record.this.fqdn
}

If I wanted to create devops.mysite.example.com then I'd either call this module with:
zone = "example.com"
subdomain = "devops.mysite"

Or, if I had a hosted zone for mysite.example.com, I'd call it with:
zone = "mysite.example.com"
subdomain = "devops"

It's possibly that I could go with the fist approach (e.g. where example.com is my only hosted zone) only for someone to later add a hosted zone for the mysite subdomain.  If that occurred I wouldn't want to have to change my code; since the requirement is still the same; I'd just want Terraform to see that it was still possible to fulfill my requirement (i.e. to create a DNS CName record pointing devops.mysite.example.com at devops.stackexchange.com) and adjust accordingly.
Question
Is there a way to have Terraform determine the appropriated hosted zone for a given record?  E.g. So I could call my module passing devops.mysite.example.com and have the system work its way through the hosted zones recursively until it finds the correct one, then creates the appropriate entry there (or if no appropriate zone was found, error)?

Short answer: I don't believe that is possible.
In my opinion, I suspect this problem has more to do with process and responsibilities than it does with Terraform.
You said:

It's possibly that I could go with the fist approach (e.g. where example.com is my only hosted zone) only for someone to later add a hosted zone for the mysite subdomain. If that occurred I wouldn't want to have to change my code...

Terraform is best when at least most of its associated resources are Terraform-managed. That means you should be managing your zones, too, not just your records.
Which begs the question: why isn't your zone in Terraform?
If your zone was Terraform-managed, the impact of that change should be evident in a terraform plan to anyone who might come along to change it. Even if it references a zone in a different remote state file, a tool like runatlantis.io could show that diff in the Github PR before merging.
Let's say this is the worst-case scenario: you managed the records, it's a different team that creates and manages the zones, and they don't want to use Terraform. In that case, perhaps your DNS records shouldn't be in Terraform, either. Terraform is arguably too stateful for that. Perhaps a few python boto3 scripts can do everything you need it to.


Sprints section shows tasks as unparented I am not sure if this has been answered before, I have looked all over Google and can't seem to find any real answer. Apparently Microsoft allows you to view parent items to tasks in the Sprints section, and I've seen this done at my workplace when we view the Taskboard.
In my own personal DevOps configuration, I've made a custom work item type called "Goal", and assigned "Task" items as children to these "Goals". However, when viewing the Taskboard, all of my tasks show as "Unparented". My goals and tasks are on the same area path and iteration path.

In my Backlog levels settings, I've created a top level portfolio backlog named "Goals" that only uses the "Goals" work item type. When I change the Iteration backlog to include "Goal" work items, it puts the work item type on the same iteration as "Task" work items, and the Taskboard does not properly populate.

Is there something I'm missing? I've tried to read multiple documentation pages on Microsoft's website to no avail.
https://docs.microsoft.com/en-us/azure/devops/boards/backlogs/resolve-backlog-reorder-issues?view=azure-devops
https://docs.microsoft.com/en-us/azure/devops/boards/backlogs/set-up-your-backlog?view=azure-devops#taskboard-types
https://docs.microsoft.com/en-us/azure/devops/organizations/settings/select-backlog-navigation-levels?view=azure-devops
https://docs.microsoft.com/en-us/azure/devops/boards/backlogs/organize-backlog?view=azure-devops
https://docs.microsoft.com/en-us/azure/devops/organizations/settings/work/customize-process-backlogs-boards?view=azure-devops&viewFallbackFrom=vsts#add-or-edit-portfolio-backlogs

I was able to contact DevOps Support and get an answer. The issue was that I could only create Portfolio Backlogs, not Requirement Backlogs. By changing the default Requirement Backlog to include my custom work item, those showed in the Sprints section.


Jenkins - load scripted parameters from a SCM repo I am using all kind of parameters like "Active Choice Parameter", "Active Choice Reactive Parameter", "Active Choice Reactive Reference Parameter" that have a lot of Groovy code within them. My problem is that now I need to audit changes so I can link regression to code commits. I don't see an option to load the code from a SCM, how do I version this?

As far as I can tell, there is no native functionality in Active Choice Parameters to load the Active Choice code from SCM.  There are a couple ways to build this functionality yourself, however they're both clunky solutions that can be difficult to debug.  (Unfortunately I can't copy any example code from my work right now because I'm at home on my personal computer.  Remind me after the weekend is over and I might be able to find some example code snippets to go along with these suggestions.)

Your Active Choice scripts can clone a repo, read in a file, and eval it.  This is made quite difficult by the fact that Active Choice scripts don't have any access to the Pipeline DSL, so you can't use any of the steps you would normally use in Pipeline code to clone a repo like scm or git or withCredentials.  There are also many other considerations you'll need to take into account which you normally can ignore in Pipeline code, like ensuring that you run the clone inside a temporary directory and so on.  Ultimately I found this too difficult to do so I have been using the second solution instead:
You can write a separate "meta" job that updates the parameters of your actual job and injects the Active Choice groovy scripts into its parameter configuration. I set this job to run on a hook on the repo which contains my Active Choice scripts, so every time I push to that repo, it updates the Active Choice groovy scripts.  This does rely quite heavily on the Jenkins Java API (which is available in Active Choice groovy scripts), so you'll need to get familiar with the ins and outs of that API.  That API can be quite a maze, especially for Pipeline-related endpoints (since Pipeline source code is heavily modularized and spread out across many repositories and API docs).


Failure to build with docker-compose: Invalid signal: SIGQUIT PS D:\>docker-compose up
Building app
Step 1/7 : FROM php:7.4-fpm-alpine
—> 5ae58b49885c
Step 2/7 : RUN apk add --no-cache autoconf curl dpkg-dev dpkg freetype-dev file g++ gcc giticu-dev jpeg-dev libc-dev libmcrypt-dev libpng-dev libjpeg-turbo-dev libxml2-dev libzip-dev make mariadb-dev postgresql-dev pkgconf php7-dev re2c rsync unzip wget zlib-dev
ERROR: Service ‘app’ failed to build: Invalid signal: SIGQUIT

Running:

docker run -ti 5ae58b49885c apk add --no-cache autoconf curl dpkg-dev dpkg freetype-dev file g++ gcc git icu-dev jpeg-dev libc-dev libmcrypt-dev libpng-dev libjpeg-turbo-dev libxml2-dev libzip-dev make mariadb-dev postgresql-dev pkgconf php7-dev re2c rsync unzip wget zlib-dev

seems to work except for the error at the end:

[...]
OK: 392 MiB in 105 packages
time="2020-05-23T11:54:06+05:00" level=error msg="Error waiting for container: failed to
shutdown container: container 4e49e4bb8234f213ed580978bc3a4f4cacff549f91affcf82783a2703f341ddb
encountered an error during hcsshim::System::waitBackground: failure in a Windows system
call: The virtual machine or container with the specified identifier is not running.
(0xc0370110): subsequent terminate failed container 4e49e4bb8234f213ed580978bc3a4f4cacff549f91affcf82783a2703f341ddb
encountered an error during hcsshim::System::waitBackground: failure in a Windows system
call: The virtual machine or container with the specified identifier is not running. (0xc0370110)"

Version: 19.03.8
OS/Arch: windows/amd64
Experimental: true

Switching containers doesn’t help. What can I do or how to debug?
Thank you.

Latest edge release 2.3.1.0 fixed this and many others problems


Change variable inside of project. Jenkins, docker, kubernetes Because I am new in the DevOps, I don't know where I need to apply the changes to correctly run the project.

The structure:

Git → Jenkins → Docker build -> Kubernetes (deploy)

Is it better to make the change of variables in the project, (example: password, database connection, etc.) to Jenkins, Kubernetes, or Docker?

Example:

With docker-compose I use an environment variable to change the variable inside my code.

environment:
  MONGO_INITDB_ROOT_USERNAME: user
  MONGO_INITDB_ROOT_PASSWORD: user

I want to change this variable and run correctly in the production environment.

I don't know where it is the best to change the variable, Jenkins, Kubernetes with Ansible, etc.

As @Harith mentioned, the use of secrets simplifies deployments in various environments using environmental variables. To load in a secret directly use:

  $ kubectl create secret generic < NAME OF SECRET > --from-literal=< KEY >=abc123!:?

NOTE! This will put the secret in the Default namespace, add a namespace flag for specific resources e.g.:

  $ kubectl create secret generic mongodbsecret --from-literal=mongoUser=abc123!:? -n mongoNamespace

You can then reference this secret in the deployment.yml or pod.yml or wherever you put your image for deployment on the kubernetes cluster.

env:
- name: MONGO_INITDB_ROOT_USERNAME <name of the env var in the code>
   valueFrom:
     secretKeyRef:
       name: mongodbSecret <NAME OF SECRET>
       key: mongoUser <KEY>

However, if all you need is to change 'development' to 'production' you could put an environment var directly in the (deployment).yaml.

env:
- name: NODE_ENV
  value: production

You can also update the Jenkinsfile to send in the environment by for instance:

def resource_env = BRANCH_NAME == 'master' ? 'production' : 'development'

sh "kubectl set env ${kubeImageName} NODE_ENV=${resource_env} --namespace ${kubeNamespace}"

On your current pipeline idea: The safest place for sensitive env vars would be putting it in kubernetes in a secret in your situation. However, for simple selection of deployment environments then using a Jenkinsfile simplifies things a bit like 'prod' vs 'dev'. Putting sensitive, variable or private things on git is a bad idea. Keep your docker images precise and task-orientated.

Hope that helps


Does my organization need adopt Agile Soft. Dev. before adopting DevOps? Agile software development is the methodology of choice for software shops today. But there are still organisations which are not practicing Agile in their software development, who might be interested in adopting DevOps.

When I say Agile Software Development, I mean any offspring that came out of the Manifesto for Agile Software Development. Such as Extreme Programming, Scrum, Lean Software Development and others.

Is Agile software development a mandatory prerequisite of adopting DevOps on an organisation level?

Agile Software Development is not required to do DevOps, but I believe the argument can be made that the value proposition for DevOps is often a lot lower without Agile.

DevOps is a lot of things, but automation is a central theme.  The value of automation increases in direct proportion to the frequency with which Development creates new releases.  Frequent deployment has a positive impact on certain types of products, specifically consumer applications.  High velocity all the way through the software delivery lifecycle returns value for each iteration (anyone seen the CA ad where the zombies want new features in their apps?)

Without Agile, high-frequency releases are extremely difficult, if not impossible.  If the Development team is releasing software once a quarter, or twice a year, DevOps can still automate the process, but then what is the point?  The investment in time, training, and resources to adopt DevOps may be partly returned in quality, however, the best value is in maintaining high velocity throughout the delivery lifecycle.

One could also argue that if you're going to adopt DevOps, why wouldn't you also adopt Agile?  The principles that make them both work, work well together.  Practicing DevOps by itself, without Agile, could create an imbalance between Ops and Dev, in which Ops is outperforming Dev for service delivery.


Remote starting/stopping Tomcat from Jenkins I like to stop Tomcat before redeployment, manually clean out the old wars and all the logs, then restart the server. To do that from Jenkins, I have been using the SSH plugin and call sudo service tomcat stop/start and it works... sometimes. Other times the stop works but it fails to restart but the job still erroneously succeeds.

Is there an alternative to restarting Tomcat to SSH, especially when invoked from Jenkins. E.g. can I set up something like xinetd to make these admin functins somewhat of web services. Automating SSH connection from Jenkins is sort of a crapchute.

If you install the manager webapp with a default set up of tomcat then it is possible to use this to install and restart applications, see

https://tomcat.apache.org/tomcat-7.0-doc/manager-howto.html#Reload_An_Existing_Application

I feel that you are asking the wrong question however.  The problem is not how to restart Tomcat but how to be sure that it has restarted

Set a host, tomcat_port and check path then use shell code like this

while [ "$response_code" != "200" ]
do
        echo "Checking whether tomcat (${host}:${tomcat_port}/$check) is alive..."
        response_code=`curl -sL -w "%{http_code}" "http://${host}:${tomcat_port}/${check}" -o /dev/null`
        echo "Received response code $response_code"
        if [ "$response_code" != "200" ]
        then
                echo "Waiting for a bit..."
                sleep 15s
        fi;
done;
echo "Tomcat is alive!"

This will wait until the Tomcat comes back - or the containing Jenkins job will time out eventually


What strategies exist to transition DB Migrations with Stored Procedures to Continuous Delivery? Our company has broken up most of our large Monolith application into smaller Microservices and we have a clear plan on how to get new services on Kubrenetes into a CI/CD pipeline.

However, we still have some parts of our applications which have not been broken up because of their database dependencies and the fact that they require downtime to deploy because of DB Migration issues. Part of the problem is that we have a large number of stored procedures which contain business logic in them.

I've seen plenty of information about how to refactor code and APIs so that we can have Blue/Green/Canary deployments. But how do we this for DB changes or migrations without downtime?

It is quite difficult to migrate the databases without downtime, however, using some approaches you can do it with very minimum impact, nearly a zero downtime.

The main idea in this is to create new databases with new schema. Then make changes in your code. First migrate the data from the old database to the new database. Then make changes in your code such that it starts writing to both the databases, old and new. You need to decide a time (time when there will be least writes to the database). Then stop all the writes. Remove the old database and make sure all the writes should now happen to new database from now on.
Then start the writes back.

In this way, you can achieve DB migration with minimal downtime.

You can check this for a detailed information on how this can be done.


How can I manage secrets in .tf and .tfstate? I would like to use the Terraform MySQL Provider to keep a list of mysql users and grants handy for creating new test environments. The .tf and .tfstate files both seem to want to store the MySQL passwords in plaintext.

Concerning .tf:

It is my understanding that .tf files live in revision control and are maintained by a team.  How does that practice differ when secrets are in the .tf? It is possible to encrypt these values at all?

Concerning .tfstate:

I can store the .tfstate securely somewhere after running Terraform apply, but it would be preferable for this use case to not store it at all?

Terraform supports adding an additional file with variables during invocation.

documentation: https://www.terraform.io/intro/getting-started/variables.html#from-a-file

We are using that feature to provide a secrets.tfvars file on each invocation of Terraform. We also use a script to wrap the command so that its invocation is consistent, and all team members avoid having to make the same mistakes. The wrapper synchronizes .tfstate with S3 before an execution, and pushes .tfstate back to S3 at the end. I also hear of people doing the same thing with state stored in Consul, even adding a kind of semaphore in consul to prevent two people from starting Terraform at the same time.

When you avoid setting a default value in a variables.tf file, it forces the user to input the value. It can be either entered manually or using the -var-file command option like described above. Not setting a default on your secrets is a good way to enforce changes that require a change in secrets.

The secrets.tfvars file is a symbolic link to one of the files with secrets which are not stored in version control. We have several, one per environment, like so secrets-prod.tfvars, secrets-dev.tfvars, secrets-stg.tfvars, etc...

An even better practice would be to generate these secrets files during the wrapper script based on data in Vault or some other way to share secrets. Since currently when the format of secrets changes, or secrets themselves, we need to communicate it to the team outside the version control channel - and this doesn't always work well, to be honest. But secrets do change infrequently.


why use multiple From instructions in multistage build? I'm new to Docker, sorry if my question sounds dumb.
below is a dockerfile from a textbook shows how multistage build works:
FROM diamol/base AS build-stage
RUN echo 'Building...' > /build.txt

FROM diamol/base AS test-stage
COPY --from=build-stage /build.txt /build.txt
RUN echo 'Testing...' >> /build.txt

FROM diamol/base
COPY --from=test-stage /build.txt /build.txt
CMD cat /build.txt

My questions is:
Why use multiple From instructions? why we don't do sth like:
FROM diamol/base AS build-stage
RUN echo 'Building...' > /build.txt

COPY --from=build-stage /build.txt /build.txt
RUN echo 'Testing...' >> /build.txt

COPY --from=test-stage /build.txt /build.txt
CMD cat /build.txt

First of all, you don't need the COPY instructions or multiple RUN instructions if you have only one stage:
FROM diamol/base AS build-stage
RUN echo 'Building...' > /build.txt && echo 'Testing...' >> /build.txt
CMD cat /build.txt

Of course, the textbook example is just that - an example to illustrate the concept.
In reality multistage builds are used to create smaller final images, while keeping the entire build process inside the container.
Generally speaking one may install all development and runtime dependencies in a build stage, then copy just the final distribution package to the final stage, where only runtime dependencies are installed to reduce the final image size.
For example:
FROM ubuntu:latest AS base
RUN apt -y install some-library

FROM base AS build
RUN apt -y install some-library-dev
COPY . /src
WORKDIR /src
RUN make && make install

FROM base
COPY --from=build /usr/bin/app /usr/bin/app
CMD /usr/bin/app


Scaling issues with Nagios as CaSC If Nagios server would use a CasC source of configuration in a larger organization is there a design flaw in this approach which will limit scaling up?

To avoid broken configuration, test instances of Nagios could be validated in a CI environment.

Usage scenario: imagine a central Nagios service with more users than you want to be able to break it, who would like to add their services for monitoring.

Personally I don't think using CasC itself will have any negative scalability impact.

Fundamentally CasC means the actual config files are not hand-maintaned, instead they're auto-generated following version-controlled rules - which can be a lot more reliable.

But once the config files are generated - the service using them functions just as it dit with manually-crafted config files.

If anything - using CasC should make such large nagios system more reliable - if done correctly CasC should reduce/eliminate the risk of human error in modifying the config files.


How to remove directories and files in another layer using Docker? Why this question?

The reason for posting this Q&A is that sometimes certain software is required to compile software in a docker image. Once compiled, these packages are superfluous and should be removed in order to reduce the image size. In some cases images that were more than 1.5GB where reduced to less than 300MB. The following use case does not represent a real world scenario, but provides an example of this use case.

Attempts to solve the issue

It works to remove a directory when the remove command is defined in the same layer as the creation of the folder:

FROM alpine
RUN mkdir dir && cd dir && wget http://google.com && rm -rf dir

This results in:

user@host$ docker build -t dir .
Sending build context to Docker daemon 1.336 MB
Step 1/2 : FROM alpine
 ---> 4a415e366388
Step 2/2 : RUN mkdir dir && cd dir && wget http://google.com && rm -rf dir
 ---> Using cache
 ---> c283805e687f
Successfully built c283805e687f

When the aim is to remove the directory in another layer, this fails:

FROM alpine
RUN mkdir dir && cd dir && wget http://google.com
RUN rm -r dir

And this fails:

FROM alpine
RUN mkdir dir && cd dir && wget http://google.com
RUN rm -rf dir

Another attempt was using workdir:

FROM alpine

RUN mkdir dir
WORKDIR dir
RUN wget http://google.com
WORKDIR /
RUN ls
RUN rm -r dir

Building the image resulted in:

Sending build context to Docker daemon 2.048 kB
Step 1/7 : FROM alpine
 ---> 4a415e366388
Step 2/7 : RUN mkdir dir
 ---> Using cache
 ---> aed5c75218cb
Step 3/7 : WORKDIR dir
 ---> Using cache
 ---> 715de09e7e08
Step 4/7 : RUN wget http://google.com
 ---> Using cache
 ---> c07bfb3e1133
Step 5/7 : WORKDIR /
 ---> Using cache
 ---> c6de86d0191a
Step 6/7 : RUN ls
 ---> Using cache
 ---> 9567593cce39
Step 7/7 : RUN rm -r dir
 ---> Running in 7fe7b28294bf
rm: can't remove 'dir': Directory not empty
The command '/bin/sh -c rm -r dir' returned a non-zero code: 1

mkdir dir && cd dir && wget http://google.com && rm -rf dir doesn't do what you think it does.

Let's break it down:

mkdir dir creates dir.
cd dir changes directory into it.
wget http://google.com downloads google.com inside dir/.
rm -rf dir attempts to delete a directory called dir that's in the current directory.

The problem is that there is not dir in the current directory, because you're already cded into it.  And since you've added the -f flag, the command doesn't produce an error.  This is one of the reasons why you shouldn't use force flags without being very sure you need them.

Since you didn't use the WORKDIR command, the directory change is discarded at the end of the RUN.  Thus, the next line executes in the original working directory, where it tries to mkdir dir but, as we've previously discussed, that directory still exists with the wgetted contents inside it.