Digital Ocean vs AWS EC2 for Laravel based application I'm working on a project based on Microservice architecture in Laravel, which means I'll have many small apps and Databases, I'm confused weather I should buy AWS EC2 or Digital Ocean's hosting. My concern is pricing, In numbers I'm having a 100% price difference and getting Digital Ocean's hosting in 50% of the price with more features then what I'm getting on AWS.

Please suggest me what would be the better option.

We faced exactly the same choices for our live service that deploys a laravel API and multiple react apps.

There is no generic correct answer to a question "what tool/provider is better?" as it depends on your circumstances. Cost is a particularly tricky question. Are we just talking about the cost of getting something live, or the cost of running an elastically scalable service, or the cost of running a highly available elastic service that spans data centre availability zones such that it isn't taken down by an outage in a single data centre zone?

Digital Ocean ("DO") is very highly promoted in the Laravel community so it was "the default" for the development team when I joined a startup about two months from go-live. When I asked "what about data centre availability zones" the answer at the time for Digital Ocean was that they didn't support it and we would have to set-up in two different DO data centres and work on our own mechanism (e.g. DNS) to switch between them if DO had a major outage in one place. Yet on AWS they have availability zones and also Elastic Load Balancer so its far easier to get a setup going that is "future ready" if you intend to be running a very successful large scale service a year after launch.

In the end we went live on openshift.com which is a shared kubernetes service running on AWS. So our live DNS points at the Elastic Load Balancer of a shared Kubernetes cluster were we rent space for our containers. The cluster we are on has containers by other companies and runs across many AWS EC2 virtual machines. Yet they are not dedicated to us so we only pay a small fraction of the costs. Yet if any one virtual machines crashes where our containers are running openshift kubernetes automatically starts a replacement container on another virtual machine. So we have automated high availability and crash recovery but only pay-as-we-go for the memory and CPU we currently use. We can scale up more containers anytime we want by using changing the replica count. Openshift has internal load balancing so that a single internal service IP with an internal DNS name automatically points at however many containers we want to run of a particular type. That solves discovery between our services. If you deploy a redis instance from the built-in catalog it is instantly available on a private IP with a DNS entry only viable in your project. That sort of ”discovery” feature is really important when you are running microservices at scale.

With our setup, we get a great low-cost price for a very professional, elastic, highly scalable, fully automated infrastructure-as-code setup, that has all the advantages of AWS ELB, and all the advanced features of kubernetes for microservices.

Edit: I wrote a tutorial for my experiences running laravel backend with a reactjs frontend on openshif.com that is over at https://gist.github.com/simbo1905/d27ba7218a2fd71fcb4f0fcede82fd6d


What are best practices for sharing and accessing Vault secrets? When one follows a Hashicorp's Vault demo, one has to issue vault init -key-shares=1 -key-threshold=1 at a certain moment that generates a key 1 and root token:

Key 1: ABC
Root Token: XYZ

If person A generates a secret by executing vault write secret/hello value=world how could person B check on his/her computer the value of secret/hello?

Assuming person B has a role which also allows them access to the path secret/hello, then they would simply run vault read secret/hello from their own machine.

If you're following the demo, I think the vault server will be on your local machine? So if someone else is to connect to it they would need to set VAULT_ADDR to point to the IP where the server is running.


How to clean up old images in docker registry with jenkins I have a jenkins multi branch project, that builds on every commit and pushes the image to our private registry:

docker-reg:5000/<project>/<branch>:latest
docker-reg:5000/<project>/<branch>:<build-server>-<build-number>

How can I delete the labels in the docker registry in order to let the garbage collector kick in upon:

build gets orphaned after some time

remove <project>/<branch>:<build-server>-<build-number

branch gets merged and jenkins sub project gets automatically deleted

remove <project>/<branch>:latest

We have a script which we run periodicaly which scans our docker registry for orphaned items, and then deletes them.

Unfortunately, it requires multiple API calls to get all the data required, and then to delete each image.  But basically, if an image hash is lacking a 'latest' tag or a tag that starts with 'v' it gets culled.

You could probably also try deleting based on last modified date.

We run this as a cron job in kubernetes, but you could just as easily run a script like this as a periodic jenkins job.


How to automate rolling upgrades for ec2 running linux behind elastic and/or application load balancer? Would like to ask for recommendations on ready automated solutions or templates for rolling ec2 instance upgrade on AWS, requiring a reboot with ELB/ALB.
Specifically required to perform the maintenance upgrade for the kernel on hundreds of Linux machines.

For each ELB/ALB steps are:

Disconnect ec2 instance from ELB/ALB wait for connection draining
Upgrade kernel and reboot machine (not part of automation, e.g. running external ./upgrade.sh)
Wait for instance to be online, connect to ELB/ALB, wait for health check status
Continue to next instance in the load balancer

Note: not looking for migration to the solution similar to Netflix Spinnaker or immutable deployment strategy implementation, but for general and simple on-demand maintenance tool.

AWS have a specific service to deal with this type of use case, EC2 Systems Manager

This allows making base patch sets and then checking they are applied to target systems.

  "Patch Manager is a fully-managed service (provided at no additional
  cost) that helps you simplify your operating system patching process,
  including defining the patches you want to approve for deployment, the
  method of patch deployment, the timing for patch roll-outs, and
  determining patch compliance status across your entire fleet of
  instances. It’s extremely configurable with some sensible defaults and
  helps you easily deal with patching heterogeneous clusters."

-- "Amazon EC2 Systems Manager Patch Manager now supports Linux"

You'll need to install an SSM agent on each system and add a role but other than that it should be unobtrusive


Why does chown not work in RUN command in Docker? I have the following docker-compose file:

version: '2'

networks:
  default:
    driver: bridge

services:

  yii2-app:
    build:
      context: .
      dockerfile: ./Dockerfile-app
    ports:
      - "80:80"
    networks:
      - default
    depends_on:
      - yii2-db
    volumes:
      - "./app:/var/www/app"
      - "./nginx:/etc/nginx/sites-available"
    ...

Dockerfile-app:

FROM richarvey/nginx-php-fpm

ADD app /var/www/app

RUN rm -Rf /etc/nginx/sites-enabled/*

ADD nginx/site.conf /etc/nginx/sites-available/site.conf

RUN ln -s /etc/nginx/sites-available/site.conf /etc/nginx/sites-enabled/site.conf

RUN cd /var/www/app && \
    composer install

RUN cd /var/www/app && chmod +x yii && \
    cd web && mkdir -p uploads && \
    cd /var/www && chown nginx:nginx -R app/

My last command RUN has no effect: chown does not set the nginx files owner. The folder "uploads" also is not created.

When I run docker-compose build --no-cache that step is passed:

Step 7/7 : RUN cd /var/www/app && chmod +x yii &&     cd web && mkdir -p uploads &&     cd /var/www && chown nginx:nginx -R app/
 ---> Running in 26a918bece47
Removing intermediate container 26a918bece47
 ---> 00db026a461c
Successfully built 00db026a461c
Successfully tagged passport-app_yii2-app:latest

However, when I run the workload in the "common" way using docker-compose up -d the change is not applied.

Why does my last RUN command not work?

I found the reason of the problem.
In Docker "anything after the VOLUME instruction in a Dockerfile will not be able to make changes to that volume".
I define volume "./app:/var/www/app", and after that manipulate with it, so it's not work.


What are possible implementations (or examples) of the four-eyes principle? Michael Grünewald recently posted this comment:

  A very important method you do not mention is the “four-eyes principle” which is used in the finance – either as a regulatory obligation or as a safe-guard. In the software industry it is implemented in various ways, as e.g. code reviews but also can be used to validate commands affecting live systems.

Correct me if I'm wrong, but I was taught that the "four-eyes principle" is about something that is "approved to happen", after at least 2 human beings (and/or automated processes) gave their prior blessing. Or to use the (slightly corrected) wording about the "two-(wo)man rule" from Wikipedia:

  The two-man rule is a control mechanism designed to achieve a high level of security for especially critical material or operations. Under this rule all access and actions requires the presence of two authorized people at all times.

Regulatory obligations are, pretty sure, off topic here, but in the context of "safe-guard", what are the possible conceptual implementations of this four-eyes principle, which probably could apply to any platform / OS / hardware being used?

One of the implementation on code is the Pull Request model (PR)  popularized by GitHub.

The main reasoning behind is that only a small set of maintainers of the product will be allowed to merge code into the release branch. Every new feature/bugfix will happen on a new branch and once done will be defined as a pull request.

This allows to test on the merge from actual release (master) code with the code in the PR in an automatic way (Travis being the most popular for public project actually) and give a first feed back on the code quality. Travis CI (for example) can run on the result of actual master with the  code from the pull request merged into it, so it will fail if the merge is impossible, or if the commands defined in the travisci.yml return a non zero exit code

Once the automatic tests have passed, for the 4 eyes principle, it still requires a configurable number of persons to review and approve the change before it gets merged obviously at least 1 person (who is not the PR author) to enforce 4 eyes reviewing the changes.

There's a wide range of options to merge automatically once the quorum of reviewers is met or to still need a manual merge by a maintainer.

The permissions to review and merge can be separated which helps giving more people the right to "vote" on the merge status while keeping the possibility to restrict who can actually do the merge.


What is the best way to find the Auto Scaling Group of an ECS cluster? We are looking to inspect the ASG EC2 CloudWatch metrics such as CPU and Memory, since it includes an aggregate of all the ASG instances.

Given that we know the ECS cluster, or the ECS service name. Which API calls would be needed to figure out the name of the Auto Scaling Group and its CloudWatch metrics?

Example in AWS cli that should work to get the ASG name

a=curl 'http://169.254.169.254/latest/meta-data/instance-id'; aws autoscaling describe-auto-scaling-instances --instance-ids $a --query 'AutoScalingInstances[*].AutoScalingGroupName'


Scheduling collisions possible in IaC - Infrastructure as Code - solutions vs. software defined data center? To clarify the terms first,

Infrastructure as Code (IaC) can be delivered with upcoming technologies like Docker Swarm, Kubernetes, OpenShift, OpenStack and possiby some others run on bare metal. - You can code the infrastructure.
Software defined data center (SDDC) are typically enterprise and mostly more traditional solutions from IBM, VMWare, recently also Nutanix any many others. -You can define the infrastructure through some sort of UI.

Correct me if I am wrong.

Then, typically, my assumption/observation larger companies (much larger than a s startup, much smaller than Google) would provision you with some SDDC resources and there you can play with IaC.

In such a case which I consider yet not very rare, does this setup actually make sense if you want high performance?

An SDDC solution would have some sort of scheduler, and your IaC CPUs are virtual CPUs on real CPUs, so in most pessimistic scenarios your Spark shares resources with say corporate mail server or some backup solution. And there can be priorities set between them, so the sharing is not equal. And collision cases described below from random towards a notorious DOS (Denial of Service) bias.

Collision case 1. Then your Docker Swarm, or Kubernetes scheduler would "see" a machine and say, well, the load there was not so much, so I take it - and then it finds what if was not expecting because the SDDC system scheduler has decided to take it already for something else.

Collision case 2. Just in the moment the IaC scheduer decides to compute something, the SDDC scheduler decides to unschedule it, so what would happen is kind of paralyzing spikes setting IaC to a slow motion mode.

I disagree with your definition, kubernetes, openshift and other are scheduler and as such should go in the SDDC part. IaC things are Vagrant/Terraform/Cloudformation/Vmware Cloud management. They are about writing code to define the infrastructure.

With this in mind k8s may fit both, a deployment.yml is IaC to drive the k8s SDDC.

Let take exemple for a k8s cluster running on top of vSphere infrastructure:
You have a scheduler k8s on top of another scheduler vSphere. SDDC on top of another SDDC.

K8s is concerned by its node load (which are vSphere VM), vSphere is concerned by its nodes load (ESXi hosts).
With this in mind k8s will move pods (containers) around its nodes, where vSphere will move k8s nodes around its hosts.

This indeed could lead to a spike or freeze of k8s nodes when both wish to move their loads at the same time. The solution is just an architecture solution: disable vSphere DRS on your k8s nodes so there won't be a move of your k8s nodes around while they are already moving containers.

Each time you overlay systems with the same behavior, you have to plan their definition properly to avoid race conditions.

Worst case would be running a docker swarm cluster on top of kubernetes on top of mesos on top of vSphere infrastructure on top of Nutanix partitions. (Absolutely doable but I don't think anyone would do that out of demonstration purposes)

So yes, this setup makes sense, it mainly require a careful design of all layers to achieve the best performances.


Is there a standard way to check if a web API is alive? As a developer, I have been involved in producing a number of assorted web-APIs, both for internal and public use.

Is there a standard (or de facto) way of checking if a web API is up and running?

(And possibly report back e.g. deployment info and other metadata.)

Web APIs usually don't have a handler for a GET to the root-URL, and often require authentication for the other URLs. Sometimes I have solved this by adding a specific GET http://example.com/alive or GET http://example.com/hello handler in order to enable our devops to monitor the API and to make it easier for myself to quickly check if it is running. It would be nice if there was a more... official way. (:

It would be nice if there was a more... official way. (: If this is not an official way how to check it? One could also say that if there are only POST's then why not using a POST?

Some services only have POST and these are checked by doing an automated POST every X minutes and check the output. If the POST fails then the monitor system will notify the team.


Where do I set that agent certificate for Puppet on Amazon Linux? I am using:

Puppet server version: 1.2.0
Amazon Linux 2

I have master and agent set in /etc/puppet/puppet.conf and currently I get back only the certificate for the master on the master. The agent's DNS name under [agent] and [main] is

certname = doug2c.mylabserver.com

but

puppet cert list -all   returns:

+ "doug1c.mylabserver.com" (SHA256) C2:AE:78:00:8D:EE:40:09:06:26:10:AD:82:47:58:26:C7:86:F5:A1:8A:D8:B1:BE:81:C2:AF:D1:70:F9:04:EF (alt names: "DNS:doug1c.mylabserver.com", "DNS:puppet", "DNS:puppet.mylabserver.com")

I tried useing find to locate where it was set to puppet with

find / -name doug1c

but that didn't work.

On the agent I needed to set the server ip set to puppet in
/etc/hosts

I had accidentally put it on the agents ip address and then been incorrectly assuming i'd done something wrong on the master rather than the agent.

(server ip)  (domian name)   puppet


EKS - created cluster from console with federated IAM admin - how to access I have no programmatic access for the user who created the cluster only graphical.
I tried a lot of things and I always get access denied on kubectl as I have to use another IAM identity that has programmatic access.
I assume there is some trick with STS but I don't have the access keys of the root account.
I guess if I give graphic access to the IAM identity that I want to use, it will work out, but I want to know if there is a way with my current situation. Also using the aws-cli is not an option for creation in the given scenario.
The admin who created my user won't allow my federated IAM to have access keys.

The only way to manage IAM users - using appropriate account with IAM Administrator privileges.

See AWS Identity and Access Management
If you lost your root account, you may contact an AWS support, see Recover Your AWS Account Password


lc_collate values for database "postgres" do not match: old "C.UTF-8", new "en_US.UTF-8" I am working on testing a method for postgres 9.3 to 10 upgrade. I currently am at the dry run phase and ran into a few issues. The current blocking one is lc_collate values for database "postgres" do not match:  old "C.UTF-8", new "en_US.UTF-8", I get this when I run the below command:

time ${PGBINNEW}/pg_upgrade --old-bindir=${PGBINOLD} --new-bindir=${PGBINNEW} --old-datadir=${PGDATAOLD} --new-datadir=${PGDATANEW} --old-port=${PGPORT} --new-port=$5433 --jobs=$(nproc | awk '{ {print ($1 > 4) ? 4 : ($1 == 1 ? 1 : $1 - 1)} }') --link --username=postgres --check

Any ideas on what might be happening here?

I solved this thanks to Reddit post. It was fairly simple when combined with another part of my testing.

I used the code {PGBINNEW}/initdb --pgdata="$PGDATANEW" --encoding=UTF8 --no-locale to reinit the DB AFTER I did the below command:

sudo rm -rf ${PGBASEDIR}/data10

What happened was I cleared the old DB init config and then put in the config I needed without the localization of the database


docker-compose healthcheck for rabbitMQ I'm trying to run rabbitMQ using docker-compose, but the service is always starting or unhealthy.
rabbit is running fine, so I suspect there is something wrong with my health check.
Running the healthcheck command locally does return a value.
> curl -f http://localhost:5672
AMQP    %

But docker-compose ps always says the service is unhealthy (or starting, before it runs out of time).
> docker-compose ps
docker-entrypoint.sh rabbi ...   Up (unhealthy)   15671/tcp

Here is what my docker-compose.yml file looks like.
# docker-compose.yml
version: '2.3' # note: I can't change this version, must be 2.3

volumes:
  rabbit-data:

services:
  rabbit:
    hostname: 'rabbit'
    image: rabbitmq:3.8.5-management
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:5672"]
      interval: 30s
      timeout: 30s
      retries: 3
    ports:
      - '5672:5672'
      - '15672:15672'
    volumes:
      - 'rabbit-data:/var/lib/rabbitmq/mnesia/'
    networks:
      - rabbitmq

networks:
  rabbitmq:
    driver: bridge

I have also tried using nc instead of curl in the healthcheck, but got the same result.
healthcheck:
  test: [ "CMD", "nc", "-z", "localhost", "5672" ]

From https://github.com/docker-library/rabbitmq/issues/326

You could use the command rabbitmq-diagnostics -q ping in case you just need a basic check.
healthcheck:
  test: rabbitmq-diagnostics -q ping
  interval: 30s
  timeout: 30s
  retries: 3

More information on how to run more advanced health checks could be found here


What are the pros and cons of running application testing during vs. after a docker build? Our dockerfile creates the application environment (e.g. env variables, install libraries with apt-get etc.). It also pulls and copies the python application code from its git repo.

However, we are debating whether the application tests (some unit, some integration) should be put within a docker RUN command (e.g. RUN /bin/bash -c "source activate cool_env; pytest") or after the build using the CI stack (e.g. Jenkins, Openshift) to execute the tests on the built container.

What are the pros and cons of each?

I support J.Doe's suggestion for separate Docker files for each of the build and test stage. Such approach also allows you to:

re-run the test stage for whatever reason (known intermittent failures, for example) without re-doing the build stage
run multiple different test stages in parallel using the same build - for example instead of running a single long test stage made up of serialized tests, for a potentially significant overall pipeline speeding up.


One time/set up jobs on gitlab ci? I'm setting up a pipeline that compiles and builds new docker images, and then deploys those images to testing kubernetes cluster.

I'm using a shell runner.

I need to setup my kubectl client to point to the testing kubernetes cluster. (ie. setting the cluster and context).

As I understand it - I should only need to run this once. Is there a way to get this to run once, as defined in the .gitlab-ci.yaml? Or should I just repeat the set up steps everytime in before_script?

I dealt with the same issue. Some people recommended using the Gitlab group "secrets" and using before_script. Since I knew I was also going to deploy and would need other tools on my runner like "helm", I made my own docker container. I still keep my config for my clusters in a base64 encoded group secret but I set it like this in the dockerfile.

ENV kube_config=$kube_config
RUN echo -n ${kube_config} | base64 -d > ~/.kube/config

Now just set an environment variable for your gitlab runner to point to your new image.

KUBERNETES_IMAGE: registry.gitlab.com/MY_USERNAME/MY_REPO_NAME/MY_IMAGE_NAME:latest

Don't forget to test your new container!

stages:
  - build
  - test
  - release

variables:
  CONTAINER_TEST_IMAGE: $CI_REGISTRY_IMAGE/gitlab_runner:$CI_COMMIT_REF_NAME
  CONTAINER_RELEASE_IMAGE: $CI_REGISTRY_IMAGE/gitlab_runner:latest

before_script:
  - apt-get update && apt-get install docker.io -y
  - docker login -u gitlab-ci-token -p $CI_JOB_TOKEN $CI_REGISTRY

Build Image:
  stage: build
  script:
    - docker build --build-arg kube_config=${kube_config} -t ${CONTAINER_TEST_IMAGE} .
    - docker push ${CONTAINER_TEST_IMAGE}
  except:
    - master

Test Kubectl:
  stage: test
  script:
    - docker pull ${CONTAINER_TEST_IMAGE}
    - docker run --rm ${CONTAINER_TEST_IMAGE} kubectl get deployments -n kube-system
  except:
    - master

Test Helm:
  stage: test
  script:
    - docker pull ${CONTAINER_TEST_IMAGE}
    - docker run --rm ${CONTAINER_TEST_IMAGE} helm ls
  except:
    - master

Test Docker:
  stage: test
  script:
    - docker pull ${CONTAINER_TEST_IMAGE}
    - docker run --rm -v /var/run/docker.sock:/var/run/docker.sock ${CONTAINER_TEST_IMAGE} docker images
  except:
    - master

release-image:
  stage: release
  script:
    - docker pull $CONTAINER_TEST_IMAGE
    - docker tag $CONTAINER_TEST_IMAGE $CONTAINER_RELEASE_IMAGE
    - docker push $CONTAINER_RELEASE_IMAGE
  only:
    - master

Edit: Missed this part.. I'm using a shell runner.

So post of what I said is probably not relevant. How ever I do think there is some advantages to building and deploying with a gitlab runner directly on a cluster.


Jenkins Artifactory Integration - Custom Maven plugin creating dynamic version/build I am currently working on a custom Maven plugin that generates a build artifact in the form of a zip file with a dynamic version or build number.  The format of the file is something like

target/[artifactId]_[version or build].zip

Where [version or build] parameter is dynamically generated based on a snapshot of time, (Eg. [changeset24]-[changeset57]) on a parameterized Jenkins build where the user of the build will not necessarily know the latest changeset.  The plugin uses an API to determine the latest changeset, for example changset57, and then uses the starting changeset parameter to form the unique version/build of the zip artifact.

This is meant to be a solution for streamlining the chaotic and manual process we face in our enterprise around delta based SQL scripts on hugely complicated schemas.  There is no clear promotion path up through environments for a single versioned database artifact.  It is a long story with a very immature devops organization and many frameworks and skill sets that do not mesh well, so I will spare the rest of the details here.

Ultimately we need to specify not the JAR artifact for this build to upload to Artifactory, but the ZIP file that is being created here, however Maven does not seem to allow for us to change the hard coded <version> in the POM file, and even if we could, we need the ability to:

Run the Maven build with custom plugin execution goal
Parse the output file for the unique version number that is generated
Override the ${VERSION} or ${BUILD_NUMBER} environment variables in Jenkins
Upload the artifact to Artifactory with the new environment variable for version or build number being included in build info.

The problem I have is that I seem to have trouble figuring out how to configure these 4 steps with the existing Maven3-Artifactory Jenkins plugins, nor with the Generic Artifactory Plugin using FileSpec JSON.  I don't see how I can inject a custom shell script to override environment variables here either between this process.

Does anybody have any good suggestions of how I should configure this job?

NOTE: Please don't suggest that my general approach is fundamentally bad or flawed or whatever.  That is not helpful because I am facing numerous technological and political constraints that prevent a more appropriate solution.

I figured out how to get this job to run which took a number of very specific steps:

1) The Maven3-Artifactory plugin will simply not do what I need it do here.  I still need this in my job though to actually execute my custom maven plugin from the project POM.  I configured the Build Environment for this plugin to NOT upload any build artifacts to Artifactory.

2) I added the Generic Artifactory plugin that has the responsibility of taking the zip artifact from the Maven plugin and uploading it to our Artifactory repo.  I struggled to define an upload FileSpec that worked correctly.  I ended up using the Legacy Patterns option where I could define the upload artifact and the repo location based on a number of Environment Variables that have been already defined for me previously (more info on this in future steps).  The Download Details are effectively empty.

3) The first Build Step is execution of the Maven build.

4) The second build step is execution of a custom shell that will parse the maven build artifact zip file for information that is needed to derive the version number.  It then injects this version number into a temporary property file in the root of the job workspace.

5) The third build step uses the Inject Environment Variables plugin to put the temporary property file into environment variables used by the Generic Artifactory Plugin Legacy Pattern.

This seems to work well as a solution for my situation.


Can I use docker compose to run application container and its dependencies on google cloud or digital ocean, the same as locally? I have a spring boot application, angular front end app, couchbase, and redis. They are all defined in a docker compose yaml file. I can make the application up and running locally. Is there a way to use docker compose tools/commands to do the same in google cloud or digital ocean?

If you don't want/need platform-specific integrations and transforming your services to a cloud I guess GCP Compute Engine is for you -- it's like a usual virtual machine that you can start, SSH to it and have all privileges to install and run software the same way you did it on your dev machine.


jenkins stuck "Please wait while Jenkins is getting ready to work" I've been trying to get a simple jenkins master node to start but it keeps getting stuck at the "Please wait while Jenkins is getting ready to work".

My setup is nothing exotic:
jenkins is installed inside a LXC hosted by a proxmox 6 instance.
10Go of RAM, 4CPU, 10G of hdd, debian 9

I increased the lxc specs just in case (I started with 4G of RAM and 2 CPU)

I tried a lot of different things such as:

Stop / start the jenkins soft
Stop / start the container
installing using debian package
Install using the war (see my service file below, only used for the war install style)
Installing on centos 7
Installing on qemu

Service file:

[Unit]
Description=Jenkins Daemon

[Service]
ExecStart=/usr/bin/java -jar /home/jenkins/jenkins.war --logfile=/home/jenkins/jenkins.log
User=jenkins

[Install]
WantedBy=multi-user.target

The log doesn't contain any error except a WARNING :

2019-10-03 12:05:38.516+0000 [id=1] WARNING o.e.j.s.handler.ContextHandler#setContextPath: Empty contextPath

But it always ends at being stuck at this page.

I've searched the web but my google fu seems bad today so here I am.

This sounds crazy to me, jenkins install is supposed to be easy and quite forward, but I can't make it works ...

[Edit]
Might be a connection issue, we saw jenkins trying to connect to an ec2 instance... we have some proxy, but did not found the correct options (in Java command) to get through it, any tips welcome

OK, a good week-end does its job as usual: My issue was indeed caused by my company proxy (a good thing would be to add this requirement in the jenkins doc, i might have  missed it anyway).
So the fix is actually simple, just need to add the correct options to the jenkins start command, as in:

-Dhttp.proxyHost=<your_proxy_ip_here> -Dhttp.proxyPort=<your_proxy_port_here> -Dhttps.proxyHost=<your_proxy_ip_here> -Dhttps.proxyPort=<your_proxy_port_here>

Full example:

/usr/bin/java -Djava.net.preferIPv4Stack=true -Dhttp.proxyHost=10.21.92.40 -Dhttp.proxyPort=3128 -Dhttps.proxyHost=10.21.92.40 -Dhttps.proxyPort=3128 -jar /usr/share/jenkins/jenkins.war --webroot=/var/cache/jenkins/war --httpPort=8080

tcpdump was the savior here (+ a colleague :) )


You have two mistakes:

name as parameter of file module. It is an alias of path, so you actually try to create directory add dir.
Unnecessary single quotes in path, giving you actual paths like etc/ansible/roles/webservers/'handlers'.

Try with:

...
- name: add dir
  file:
    path: '/etc/ansible/roles/webservers/{{ item }}'
...


Spread pods over cluster Current using AWS's auto-scale policy to start EKS worker nodes (min+desired 10, max 15) on a fixed time every day. There are 120 pods and 30 deployments.
Obviously, only one node will be up firstly. Then the second node. After that last node will be up.
For that reason, Kubernetes add all pods in the first nodes.
Is there any way so that pods will be evenly distributed in the scenario?

You should look into affinity and anti-affinity as that allows you to control what pods go on which node. You can have it so that with anti-affinity there is only 1 pod of each deployment on a node. This is a bit overkill and wouldn't work for you exactly though IIRC you can have multiple pods on a node just with a limit.
https://kubernetes.io/docs/tasks/configure-pod-container/assign-pods-nodes-using-node-affinity/


jenkins-x returns error 1 when I try to create a cluster I'm following this tutorial. I'm running the jenkins-x cli on a aws ec2 instance where I installed kubenetes and jx. I get to the part where I run the command jx create cluster eks --cluster-name=konsek-cloud --skip-installation=true --verbose and it errors out with

DEBUG: eksctl is already available on your PATH at /home/ec2-user/.jx/bin/eksctl
DEBUG: aws-iam-authenticator is already available on your PATH at /home/ec2-user/.jx/bin/aws-iam-authenticator
DEBUG: Dependencies to be installed:
DEBUG: kubectl is already available on your PATH at /usr/bin/kubectl
DEBUG: helm is already available on your PATH at /home/ec2-user/.jx/bin/helm
error: exit status 1

I cannot figure out how to get more information out of it or figure out what is wrong. Can someone please suggest ways to "turn up" the logging output? or know something I can try to fix this?

I just experienced the same issue. It seems the eksctl that is being installed upon first use is not working for me. I went to the eksctl site and installed it from there (https://eksctl.io/introduction/installation/)

With the jx eksctl binary :

➜  ~ jx create cluster eks --cluster-name=konsek-cloud --skip-installation=true --verbose
DEBUG: eksctl is already available on your PATH at /Users/<user>/.jx/bin/eksctl
DEBUG: aws-iam-authenticator is already available on your PATH at /Users/<user>/.jx/bin/aws-iam-authenticator
DEBUG: Dependencies to be installed:
DEBUG: kubectl is already available on your PATH at /usr/local/bin/kubectl
DEBUG: helm is already available on your PATH at /usr/local/bin/helm
DEBUG: brew is already available on your PATH at /usr/local/bin/brew
error: exit status 1

With the manually installed binary :

➜  ~ jx create cluster eks --cluster-name=konsek-cloud --skip-installation=true --verbose
DEBUG: eksctl is already available on your PATH at /usr/local/bin/eksctl
DEBUG: aws-iam-authenticator is already available on your PATH at /usr/local/bin/aws-iam-authenticator
DEBUG: Dependencies to be installed:
DEBUG: kubectl is already available on your PATH at /usr/local/bin/kubectl
DEBUG: helm is already available on your PATH at /usr/local/bin/helm
DEBUG: brew is already available on your PATH at /usr/local/bin/brew
Creating EKS cluster - this can take a while so please be patient...
...


Use Vault SSH OTP backend with Jenkins I'm setting up Vault to store secrets. SSH backend with OTP auth looks like a good way to stop managing SSH keys for access.

I have SSH backend working, and am looking at hooking up Jenkins with the SSH-OTP auth so that I can remove the deployment keys too.

I looked at Jenkins plugins for vault, but they only work for fetching secrets from Vault.

Are there plugins or any Jenkins hacks for this purpose, or if not, is it possible to write one of my own?

Note: Right now, Jenkins uses SSH keys. I want to eliminate the use of keys, and configure Jenkins to get an SSH OTP from Vault every time it needs to SSH into some host for a deploy.

I looked at Jenkins plugins for vault, but they only work for fetching secrets from Vault.

It depends how the current configuration looks like that is used to deploy apps.

https://github.com/jenkinsci/hashicorp-vault-plugin

If one uses Jenkins pipelines, then one could replace the keyId with the one that is defined in Hashicorp vault.

node {
  // define the secrets and the env variables
  def secrets = [
      [$class: 'VaultSecret', path: 'secret/testing', secretValues: [
          [$class: 'VaultSecretValue', envVar: 'testing', vaultKey: 'value_one'],
          [$class: 'VaultSecretValue', envVar: 'testing_again', vaultKey: 'value_two']]],
      [$class: 'VaultSecret', path: 'secret/another_test', secretValues: [
          [$class: 'VaultSecretValue', envVar: 'another_test', vaultKey: 'value']]]
  ]

  // optional configuration, if you do not provide this the next higher configuration
  // (e.g. folder or global) will be used
  def configuration = [$class: 'VaultConfiguration',
                       vaultUrl: 'http://my-very-other-vault-url.com',
                       vaultCredentialId: 'my-vault-cred-id']

  // inside this block your credentials will be available as env variables
  wrap([$class: 'VaultBuildWrapper', configuration: configuration, vaultSecrets: secrets]) {
      sh 'echo $testing'
      sh 'echo $testing_again'
      sh 'echo $another_test'
  }
}


Purchased AWS reserved instance in wrong region I purchased AWS reserved instance by mistake in Mumbai Region. actually, I need that in N.virginia region.

I contact AWS support center and they told "switching region of AWS reserved instance is not possible"

Is there any way to switch region? Please help.

If you have done payment of your reserved instance then you are unlucky. now you have only one option Sell your reserved instance in AWS marketplace.

But If you till not done payment then you lucky, I suggest don't do payment till next billing cycle. AWS by default cancel your reserved instance if you don't do payment after the end of the month.


Will data contained withing Persistent Volume survive cluster deletion? I have a Kubernetes cluster based on 3 hosts managed via Rancher

I've created Persistent Volume on the first host and use it via a claim by a PostgreSQL pod. My app is generating and storing there some data.

Kubernetes docs state that "Persistent volume lasts through "Cluster lifetime".

My question is - if I will delete a cluster and then create it again, and then create Persistent Volume on the same host (first one) with the same configuration will data be still there?

My confusion stems from the fact that I don't quite get what is considered as "persistence" here. "Persistent Volume" entity within-cluster will obviously get deleted with a cluster. But whats about the data?

It depends on what your persistentVolumeReclaimPolicy is. If you manually created/defined the PersistentVolume, then the default action is to keep the data. (This sounds like what you're doing.)

If you are using something like EKS on Amazon, your PV's are dynamically generated EBS volumes. In that case, they will be deleted by default.

A little bit more about reclamation: https://kubernetes.io/docs/concepts/storage/persistent-volumes/#reclaiming


Find most popular images with `zip` and `git` on Docker Hub To speed up builds, I'd like to avoid install tools that I need on every commit with apt-get update -y && apt-get install -y git zip.

Normal way is to create custom build image, register at Docker Hub and upload it there. But I am 100% sure that there is already an up to date image that contains both git and zip tools that I need. The only question is how to find this image?

UPDATE 202004: There is no indexer for DockerHub, and there is no even indexer for Docker image contents. To start with that one could get a list of files from an image by patching dive by @wagoodman, or using https://github.com/GoogleContainerTools/container-diff

I can answer half of your question: If you have a set of images you're trying to investigate, the best way would be to use which in a new container using the image in question. This way you can quickly see if a particular image has the binaries your are looking for already on PATH:

docker run --rm ubuntu:latest which zip git
# no result

docker run --rm ubuntu:latest which bash sh
/bin/bash
/bin/sh
# ^-- both bash and sh are in the image

I don't know of a way to discover the set of images in question, as there is no manifest of files that a registry will provide. That is, yes, an image is a set of tar files, which you can extract a set of files/directory names... however, you'd still need to pull the entire image to get this information, so it would be easier to run the which command above.

There are tools for exploring information that a registry exposes (skopeo, doocker-ls, reg, manifest-tool, etc) but none of them will do exactly what you're looking for, which is "find a repository within a registry which contains an image that has a particular binary in it".


Ansible Dynamic Inventory - generate inventory with ip addresses under specific tag with underscore instead of dot I am new to Ansible. When I am using ec2.py dynamic inventory to generate inventory, after the playbook is finished to run, the results are shown as list of IP addresses under specific tag with underscore instead of dot.
For example I want to run a playbook on instances with specific tag, I wonder how other people work with IP addresses?

---
- hosts: tag_test_staging
  sudo: true
  tasks:
  - name: Make sure that we can connect to the machine
    ping:

    PLAY RECAP *********************************************************************
    10_80_20_47                : ok=0    changed=0    unreachable=1    failed=0

For private address, ec2.ini:

hostname_variable = private_ip_address
destination_variable = private_ip_address
vpc_destination_variable = private_ip_address

The results from ec2.py:

 "tag_test_staging": [
    "10_80_20_47"
  ],

For Public Address, ec2.ini:

hostname_variable = ip_address
destination_variable = public_dns_name
vpc_destination_variable = ip_address

The results from ec2.py:

 "tag_test_staging": [
    "52_28_11_11"
  ],

It works if I comment out the hostname_variable, because it allow to override the inventory_name with an ec2 variable, instead of using the destination_variable

#hostname_variable = private_ip_address
#hostname_variable = ip_address
destination_variable = public_dns_name
vpc_destination_variable = private_ip_address

For Private IPs:

destination_variable = private_ip_address


CICD using jenkins which is installed in other machine Suppose I have two machine ( A & B ) and i have jenkins installed in machine A , can I deploy in machine B using Jenkins installed in machine A? I am using Jenkins Multibranch pipeline

If your question is can Jenkins deploy to machine B then the answer is absolutely!  Jenkins provides documentation on deployments here, and has an example of a multibranch pipeline deployment here.

If you want machine B to deploy to a different computer (e.g. machine C) then you can do this by configuring machine B to be an "agent", and following the deployment guides linked to above.


How to specify 'os' for free gitlab runners ('runs-on' equivalent) How can I specify the operating system in the .gitlab-ci file to indicate which underlying OS on which the gitlab-runner should be running when using the 2,000 minutes of free execution in with gitlab.com shared runners in Google Cloud Platform?

For context, Github's free account supports 2,000 free minutes per month in "Github-hosted runners" for running Github Actions in Azure and MacStadium. You can specify whether you want the build to happen in Linux, Windows, or MacOS via the runs-on key in the github workflow yaml file.

What is the equivalent method for specifying that I want to run a CI/CD job in a specific OS on GCP in the .gitlab-ci file?

You can specify that you want gitlab to execute your CI job on a windows machine using tags.

Gitlab made Windows Shared Runners available as Beta in Jan 2020. Their announcement shows how to use the tags key in your .gitlab-ci file to specify Windows machines. You can also run in a docker container in windows:

https://docs.gitlab.com/runner/executors/docker.html#using-windows-containers

For example, I was able to execute a simple "hello world" powershell script with the following job in my .gitlab-ci file:

windows:
  script: "build/windows/buildExe.ps1"
  tags:
    - shared-windows
    - windows
    - windows-1809

And here's the buildExe.ps1 powershell script reference above:

Write-Host "listing contents of C drive root"
Get-ChildItem -Path C:\ -Force

Write-Host "listing contents of cwd"
Get-ChildItem -Force

The above config produced the following output in my CI job:

[0KRunning with gitlab-runner 12.9.0 (4c96e5ad)
[0;m[0K  on windows-shared-runners-manager 6QgxEPvR
[0;msection_start:1590842482:prepare_executor
[0K[0K[36;1mPreparing the "custom" executor[0;m
[0;m[0KUsing Custom executor with driver autoscaler dev (64a348d)...
[0;mCreating virtual machine for the job...
Virtual machine created!
section_end:1590842624:prepare_executor
[0Ksection_start:1590842624:prepare_script
[0K[0K[36;1mPreparing environment[0;m
[0;mRunning on PACKER-5E557E8E via
runner-6qgxepvr-wsrm-87cf700a3349de62a18e...
section_end:1590842640:prepare_script
[0Ksection_start:1590842640:get_sources
[0K[0K[36;1mGetting source from Git repository[0;m
[0;m[32;1mFetching changes with git depth set to 50...[0;m
Initialized empty Git repository in C:/GitLab-Runner/builds/maltfield/cross-platform-python-gui/.git/
[32;1mCreated fresh repository.[0;m
[32;1mChecking out 91f0d9e3 as master...[0;m
From https://gitlab.com/maltfield/cross-platform-python-gui
 * [new ref]         refs/pipelines/151176370 -> refs/pipelines/151176370
 * [new branch]      master                   -> origin/master
git-lfs/2.8.0 (GitHub; windows amd64; go 1.12.2; git 30af66bb)

[32;1mUpdating/initializing submodules...[0;m
git-lfs/2.8.0 (GitHub; windows amd64; go 1.12.2; git 30af66bb)
section_end:1590842661:get_sources
[0Ksection_start:1590842661:restore_cache
[0K[0K[36;1mRestoring cache[0;m
[0;msection_end:1590842670:restore_cache
[0Ksection_start:1590842670:download_artifacts
[0K[0K[36;1mDownloading artifacts[0;m
[0;msection_end:1590842677:download_artifacts
[0Ksection_start:1590842677:build_script
[0K[0K[36;1mRunning before_script and script[0;m
[0;m[32;1m$ build/windows/buildExe.ps1[0;m
listing contents of C drive root

    Directory: C:\

Mode                LastWriteTime         Length Name
----                -------------         ------ ----
d--hs-       12/11/2019   2:47 AM                $RECYCLE.BIN
d--hs-        2/25/2020   8:52 PM                Boot
d--hsl       12/11/2019  10:29 AM                Documents and Settings
d-----        2/25/2020   8:16 PM                Git
d-----        5/30/2020  12:44 PM                GitLab-Runner
d-----        2/25/2020   8:16 PM                GitLFS
d-----        2/25/2020   8:34 PM                Go
d-----        9/15/2018   7:12 AM                PerfLogs
d-r---        2/25/2020   8:35 PM                Program Files
d-----        2/25/2020   8:27 PM                Program Files (x86)
d--h--        2/25/2020   8:51 PM                ProgramData
d--hs-        2/25/2020   8:10 PM                Recovery
d--hs-       12/11/2019  10:28 AM                System Volume Information
d-----        2/25/2020   8:32 PM                tools
d-r---        5/30/2020  12:43 PM                Users
d-----        2/25/2020   8:38 PM                vcpkg
d-----        2/25/2020   8:51 PM                Windows
-arhs-        2/25/2020   8:45 PM         408834 bootmgr
-a-hs-        9/15/2018   7:09 AM              1 BOOTNXT
-a-hs-        5/30/2020  12:41 PM     1073741824 pagefile.sys
listing contents of cwd

    Directory: C:\GitLab-Runner\builds\maltfield\cross-platform-python-gui

Mode                LastWriteTime         Length Name
----                -------------         ------ ----
d--h--        5/30/2020  12:44 PM                .git
d-----        5/30/2020  12:44 PM                build
d-----        5/30/2020  12:44 PM                src
-a----        5/30/2020  12:44 PM            534 .gitlab-ci.yml
-a----        5/30/2020  12:44 PM          35787 LICENSE
-a----        5/30/2020  12:44 PM           1171 README.md
-a----        5/30/2020  12:44 PM              6 requirements.txt

section_end:1590842686:build_script
[0Ksection_start:1590842686:archive_cache
[0K[0K[36;1mSaving cache[0;m
[0;msection_end:1590842694:archive_cache
[0Ksection_start:1590842694:upload_artifacts_on_success
[0K[0K[36;1mUploading artifacts for successful job[0;m
[0;msection_end:1590842701:upload_artifacts_on_success
[0K[32;1mJob succeeded
[0;m

As for MacOS, gitlab currently doesn't support it natively. If this changes, updated answers are welcome.

Other answers showing how to run MacOS inside a VM (eg using vagrant and libvirt) on a gitlab shared runner are very welcome :)


How to prevent hug of death on EC2 Instance? I've a iOS app on the app store and recently I've received a huge surge of traffic to my landing page hosted on EC2 and resulted in the page not responding, luckily I managed to recover it by restarting and upgrading the instance to a t2.medium.

Now I'm looking to hire someone to implement a technology to prevent the same death again. My experience only stretch that far allowing me to understand basic devops stuff but not enough for load balancer on AWS, I want to know what is a affordable implementation for my instance.

My landing page and iOS app backend are hosted on the same instance.

If you want something quick to get this sorted without much more knowledge I'd recommend elastic beanstalk. It's another AWS application which will handle the load balancer configuration and instance scaling for you.

There's no extra cost on top of the load balancer and instances so you can keep using t2 type instances but let elastic beanstalk scale as much as you want to help.

Auto scaling isn't instant and in times of spiking traffic will take a short amount of time, minutes normally, to be able to handle a spike but it will be much better than manually scaling the instance size and is really easy to get to grips with.


Can I change the mount path and port number of a built container? Two containers (A,B) were created from the same image.

The two containers have the same mount path (-v) and port number (-p).

The "A" container is currently in the "stop" state.
The "B" container is currently running.

Can I change the mount path and port number of the "A" container?

Containers are designed to be disposable. Delete the "A" container and recreate it with the new volume mount and port number. Note that deleting the container will delete any changes made within the container specific RW filesystem layers. If you have changes in there that you need to preserve, then you really want to make those volume mounts. In the end, the workflow looks like:

docker container rm ${old_container_id}
docker container run -v ${new_volume} -p ${new_port} ...

As for changing an existing container, there is:

docker container update ...

however, changing volumes and port bindings are not part of the settings designed to be changed from this. Going under the hood of the docker engine to try to change these settings manually is unsupported and may result in unexpected behavior or a broken engine.


Configure Eureka or Consul in Rancher I would like to know how to configure Consul or Eureka using Rancher. I stumbled in the part where I need to enter the key value.

THe issue was related to permission capabilities. Thanks


How to create a local network mapping in linux I'm deploying nextcloud via docker, however the way the configuration is set up, it creates a subnetwork at 10.10.10.0/24 on the system it's running on instead of running it on 127.0.0.1. Then the only way to access it is by navigating to 10.10.0.5 on the host machine.
However, I want to be able to access this image from every machine on my local network.
I already have a few things running, so I need to map the port as well.
Is there an intuitive way to map the network such that navigating to 127.0.0.1:2287 maps it to 10.10.0.5? I can't seem to find anything indicating how to do this.
I would think this would have to be a bidirectional mapping since it needs to send data back since it's nextcloud, but I'm unsure about this.
I did try setting up the docker container to report to 127.0.0.1 but it didn't work out. It would only report on 172.19.0.5
If you wanted to see my docker setup, it's as follows:

Create the subnetwork for docker: docker network create --driver=bridge --subnet=10.10.10.0/24 --gateway=10.10.10.1 dockernet

Run docker-compose

version: '3'

services:
  nextclouddb:
    image: mariadb
    container_name: nextcloud-mariadb
    volumes:
      - /home/username/data/mysql:/var/lib/mysql
      - /etc/localtime:/etc/localtime:ro
    networks:
      default:
        ipv4_address: 10.10.10.4
    environment:
      - MYSQL_ROOT_PASSWORD=STONGPASSWORD
      - MYSQL_PASSWORD=ANOTHERSTONGPASSWORD
      - MYSQL_DATABASE=nextcloud
      - MYSQL_USER=nextcloud
    restart: unless-stopped

  nextcloudapp:
    image: nextcloud:latest
    container_name: nextcloud-app
    networks:
      default:
        ipv4_address: 10.10.10.5
    volumes:
      - /home/username/data/html:/var/www/html
      - /home/username/data/config:/var/www/html/config
      - /home/username/data/apps:/var/www/html/custom_apps
      - /home/username/data/data:/var/www/html/data
      - /home/username/data/themes:/var/www/html/themes
      - /etc/localtime:/etc/localtime:ro
    restart: unless-stopped

networks:
  default:
    external:
      name: dockernet

Do you have network constraints to create a new subnet or what is the reason?
if it's not useful at all, you can just map the port and open the firewall of your PC so that the other machines can see the application.
version: '3'
services:
  nextclouddb:
    image: mariadb
    container_name: nextcloud-mariadb
    volumes:
      - /home/username/data/mysql:/var/lib/mysql
      - /etc/localtime:/etc/localtime:ro
    environment:
      - MYSQL_ROOT_PASSWORD=STONGPASSWORD
      - MYSQL_PASSWORD=ANOTHERSTONGPASSWORD
      - MYSQL_DATABASE=nextcloud
      - MYSQL_USER=nextcloud
    restart: unless-stopped

  nextcloudapp:
    image: nextcloud:latest
    container_name: nextcloud-app
    volumes:
      - /home/username/data/html:/var/www/html
      - /home/username/data/config:/var/www/html/config
      - /home/username/data/apps:/var/www/html/custom_apps
      - /home/username/data/data:/var/www/html/data
      - /home/username/data/themes:/var/www/html/themes
      - /etc/localtime:/etc/localtime:ro
    restart: unless-stopped
    ports:
      - 2287:80


Azure DevOps and private Docker registry When I try to add a Service connection to a private Docker registry in Azure DevOps, it seems I have to provide a Docker Hub (Docker ID) user and password.

Why is that? I just want to push to our private Docker registry.

I tried putting in my Docker ID and using the service connection in Azure Pipelines, but I get permission denied.

Do you know of any guides to using Azure Pipelines with a private Docker registry?

To use any of Docker's tooling you need a Docker account.  Part of this is that Docker will use base images from Docker Hub to build images that may be stored in a private repository.
For the Docker ID getting permission denied verify that you used the ID and not your e-mail.
Microsoft and others have put together great documentation both Azure Pipelines and private Docker registries:
Introduction to private Docker container registries in Azure (will help you make sure it is configured properly)
Accessing Azure resources from a Dockerfile (useful since you may be trying to do something similar in your build pipeline)


Error with declaring volumes in Docker-compose, 'no such file or directory, open '/app/package.json' I am a very beginner in the Docker world, while I am trying to experiment with Docker file and Docker-Compose I am facing the below error, I now am able to understand it is related to declaring volumes in Docker compose, if I remove volumes, everything works fine, may be it is trying to refer back to the package.json  file in my local drive and somehow container is not finding the package.json inside the container, yet to figure out how to solve this.

enoent ENOENT: no such file or directory, open '/app/package.json'

Here is the output from docker-compose:

Starting frontend_web_1  ... done
Starting frontend_test_1 ... done
Attaching to frontend_test_1, frontend_web_1
web_1   | npm ERR! path /app/package.json
web_1   | npm ERR! code ENOENT
web_1   | npm ERR! errno -2
web_1   | npm ERR! syscall open
web_1   | npm ERR! enoent ENOENT: no such file or directory, open '/app/package.json'
web_1   | npm ERR! enoent This is related to npm not being able to find a file.
web_1   | npm ERR! enoent
test_1  | npm ERR! path /app/package.json
test_1  | npm ERR! code ENOENT
test_1  | npm ERR! errno -2
web_1   |
test_1  | npm ERR! syscall open
web_1   | npm ERR! A complete log of this run can be found in:
web_1   | npm ERR!     /root/.npm/_logs/2019-04-22T04_10_39_226Z-debug.log
test_1  | npm ERR! enoent ENOENT: no such file or directory, open '/app/package.json'
test_1  | npm ERR! enoent This is related to npm not being able to find a file.
test_1  | npm ERR! enoent
test_1  |
test_1  | npm ERR! A complete log of this run can be found in:
test_1  | npm ERR!     /root/.npm/_logs/2019-04-22T04_10_39_236Z-debug.log
frontend_web_1 exited with code 254
frontend_test_1 exited with code 254

Dockerfile.dev

FROM node:alpine

WORKDIR /app

COPY package.json .

RUN npm install

COPY . .

CMD ["npm", "run", "start"]

Docker file contents are below

version: '3'
services:
  web:
    build:
      context: .
      dockerfile: Dockerfile.dev
    ports:
      - "3000:3000"
    volumes:
      - /app/node_modules
      - .:/app
  test:
    build:
      context: .
      dockerfile: Dockerfile.dev
    volumes:
      - /app/node_modules
      - .:/app
    command: ["npm", "run", "test"]

My directory structure as as shown below,

Executed the below statement and the corresponding response,

$ docker run -it --rm -v "$(pwd):/app" busybox ls -l /app

the input device is not a TTY.  If you are using mintty, try prefixing the command with 'winpty'

Update: I now am able to understand it is related to declaring volumes in Docker compose, if I remove volumes, everything works fine, may be it is trying to refer back to the package.json file in my local drive and somehow container is not finding the package.json inside the container, yet to figure out how to solve this.

As mentioned in the post/question, it was an issue with volumes, my PC's firewall was blocking the Docker from accessing the drives in my local machine, I reset the credentials in 'Docker for Desktop' app in the shared drives section and everything was back working once again :). Thank you everyone, in fact BMitch was suggesting the same solution, but I was a completely beginner and I did not comprehend it well. Thanks everyone for your responses.


How to automate the key exchange in WireGuard when you deploy a cluster of machines? Let's assume you want to deploy a cluster of machines on Hetzer Cloud. For simplicity let's call them worker1, worker2, worker3. They need to communicate with a server called master, which will be running on different account then the workers. Ideally, the whole setup should not be open to the internet. Unfortunately, Hetzner supports only private networks within the same account.
To make it work, you can setup your own VPN using WireGuard. Conceptually, it is not hard. You need to setup three connections (between the master and each worker). The tricky part is how to automate the key exchange. Ideally, it should not be more work if you deploy additional workers (e.g. 100 instead 3 workers).
Setting up such a VPN cluster sounds like a common problem, but I cannot find any recommendations on how to setup 1-to-n or n-to-m connections, only tutorials on how to peer two machines. I'm thinking of automating the key exchange with Ansible (generate keys, gather them, install them on the master), but wanted to check first whether there is an easier solution to the problem that I missed.
In SSH, workers could share their key, which would simplify the problem. In WireGuard, keys cannot be shared, as far as I understood. How would you automate the setup of a VPN with WireGuard, so each worker can reach the master? Or is WireGuard the wrong choice for the problem?
Clarification:

In my scenario, it is not possible to move the workers and master to the same account; otherwise, Hetzner networks would be the straightforward solution for setting up a private network.
If you are not familiar with Hetzner Cloud, it is not a problem. You can assume that you get normal Linux machines, but then you are on your own (it does not support VPC peering across accounts as AWS does). Yet you can use all Linux tools available for creating the VPN setup. WireGuard would be my first choice, but I'm open to other techniques.

(Disclaimer: As the question has not received an answer yet, I came to the conclusion that there is currently no out-of-the-box solution. Thus, I'm sharing what I ended up doing, and hope it will help others. Still, I would be interested to learn of a better solution if there is one.)

WireGuard can be used, but each of the n-to-m connections will need its own keys. Managing that by hand becomes tedious if the number of machines increases. Therefore, it is advisable to automated it:

Generate all keys in advance
Distribute the keys to the machine using Ansible (e.g. by generating systemd services wg0.network and wg0.netdev)

Note that step 1 assumes that the maximum number of machines is known in advance. If you cannot provide an upper bound, this approach will not work. Instead you will need to implement a dynamic key generation and exchange protocol.
If you end up using fewer machines then the configure upper bound, it will still work. Distributing the keys means that WireGuard will allow connections from different machines, but does not require that all peers are always available.

Scaling: In my setup, the upper bound was in the order of 100. That works without problems, but I did not test how it scales if you need to go higher (e.g. to the order of 1000 or 10,000).
What if there is no upper bound? -- That is a more difficult problem, and I don't know how to solve it best. As said, you would need to generate key dynamically and exchange them. I was considering opening SSH and using it to exchange on-the-fly generated WireGuard keys.
Note that with SSH, you can reuse the same key on different machines. That eliminates the need to exchange dynamically generated keys. But be aware that reusing the same key for all machines is a trade-off between convenience and security. If one machine gets compromised, the attacker will gain access to all other machines, too.
Maybe there is some solution that is more elegant and more secure. Perhaps directly using Wireguard is not a good idea if you need to support an arbitrary number of dynamic hosts. It is easy to introduce subtle security issues when implementing your own key exchange protocol.


Disable creation of labels in Jira? Following the base instructions I can't create new Jira item through API request due to failing because of trying to create labels. Even if I am using  add_group_labels: false it tries to create label anyway, isn't this option supposed to disable this behavior? From what I read I need to allow the user to create labels through rest api, but I just don't want the albels on the ticket.

Testing with:
curl -H "Content-type: application/json" -X POST -d '{"receiver": "prometheus-customer-non-production", "status": "firing", "alerts": [{"status": "firing", "labels": {"alertname": "TestAlert", "key": "value"} }], "groupLabels": {"alertname": "TestAlert"}}' http://xx.x.xxx.xx:9097/alert

Output:
{"Error":true,"Status":500,"Message":"JIRA request https://customer.atlassian.net/rest/api/2/issue returned status 400 Bad Request, body \"{\\"errorMessages\\":[],\\"errors\\":{\\"labels\\":\\"Field 'labels' cannot be set. It is not on the appropriate screen, or unknown.\\"}}\""}

Here is the configuration jiralert.yaml:

# Global defaults, applied to all receivers where not explicitly overridden. Optional.
defaults:
  # API access fields.
  api_url: https://customer.atlassian.net
  user: 'example@customer.com'
  password: 'customerpassword'

  # The type of JIRA issue to create. Required.
  issue_type: Alert
  # Issue priority. Optional.
  # priority: Low
  # Go template invocation for generating the summary. Required.
  summary: '{{ template "jira.summary" . }}'
  # Go template invocation for generating the description. Optional.
  description: '{{ template "jira.description" . }}'
  # State to transition into when reopening a closed issue. Required.
  reopen_state: "To Do"
  # Do not reopen issues with this resolution. Optional.
  wont_fix_resolution: "Won't Fix"
  # Amount of time after being closed that an issue should be reopened, after which, a new issue is created.
  # Optional (default: always reopen)
  reopen_duration: 0h

# Receiver definitions. At least one must be defined.
receivers:
    # Must match the Alertmanager receiver name. Required.
  - name: 'prometheus-customer-non-production'
    # JIRA project to create the issue in. Required.
    project: customer
    # Copy all Prometheus labels into separate JIRA labels. Optional (default: false).
    add_group_labels: false

# File containing template definitions. Required.
template: jiralert.tmpl

Below you can find versions info:

Package release utilized -> https://github.com/prometheus-community/jiralert/releases/download/1.0/jiralert-1.0.linux-amd64.tar.gz

cat /etc/*-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu 18.04.2 LTS"
NAME="Ubuntu"
VERSION="18.04.2 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.2 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic

kubectl version
Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.1", GitCommit:"b7394102d6ef778017f2ca4046abbaa23b88c290", GitTreeState:"clean", BuildDate:"2019-04-08T17:11:31Z", GoVersion:"go1.12.1", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.3", GitCommit:"5e53fd6bc17c0dec8434817e69b04a25d8ae0ff0", GitTreeState:"clean", BuildDate:"2019-06-06T01:36:19Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}

helm list prometheus-operator --tiller-namespace=monitoring
NAME                    REVISION        UPDATED                         STATUS          CHART                           APP VERSION     NAMESPACE
prometheus-operator     3               Thu Apr  9 16:44:37 2020        DEPLOYED        prometheus-operator-8.12.12     0.37.0          monitoring

Based on the github reply I received, it looks only way is to allow it in Jira as it will spam same alert otherwise:

  Labels are how JIRAlert matches alerts to tickets. Without them, it
  would simply create a new ticket every time it gets called by
  Alertmanager for the same alert, i.e. once every few minutes.

  If I understand the error correctly, it looks like you'll have to
  enable labels for your JIRA project. I don't have admin access to a
  JIRA instance, so I can't tell you exactly how to do that.


Create backend Address pool for basic load balancer azure arm template I'm attempting to create a backend address pool via an ARM template for a load balancer with the following code:
            "type": "Microsoft.Network/loadBalancers/backendAddressPools",
        "apiVersion": "2020-05-01",
        "name": "[concat(variables('engineLbName'), '/LoadBalancerBEAddressPool')]",
        "dependsOn": [
            "[resourceId('Microsoft.Network/loadBalancers', variables('engineLbName'))]"
        ],
        "properties": {}

The loadbalancer it is targetting is using a basic SKU. When i attempt to deploy this arm template through the azure portal, I am given the following error message:
{
"error": {
    "code": "BackendOperationOnlyAllowedForStandardLB",
    "message": "Operation on backend pool /subscriptions/****/resourceGroups/*******/providers/Microsoft.Network/loadBalancers/LB-engine/backendAddressPools/LoadBalancerBEAddressPool not allowed. This operation is only allowed for standard SKU load balancers.",
    "details": []
}

}
Obviously, this indicates that i need to make my LB/LBIP of a standard SKU, however i specifically need this done on a basic SKU.
I am very confused as to why this error occurs, as we have many basic LB's that have backend configuration, and you are freely able to create a BE pool through the portal on a basic LB - I am unsure on how to do this via an ARM template.

I had the same issue, I downloaded the Template from Azure which effectively had my Load Balancer's Backend Address Pool as a "dependsOn" and as a "property".
This is under the load balancer's "dependsOn" list :
    "[resourceId('Microsoft.Network/loadBalancers/backendAddressPools', 'LB-db-phi-dbphipri', 'LoadBalancerBEAddressPool')]"

And the property under the Load Balancer's property list:
    "backendAddressPools": [
      {
        "name": "LoadBalancerBEAddressPool",
        "properties": {}
      }
    ]

Lastly, I had a child definition of the backendAddressPool as well
{
  "type": "Microsoft.Network/loadBalancers/backendAddressPools",
  "apiVersion": "2020-05-01",
  "name": "LB-db-phi-dbphipri/LoadBalancerBEAddressPool",
  "properties": {}
}

You don't need both. In my case I removed the "child" and the "dependsOn" and just have the property defined under the load balancer.


NetworkPolicy: Why are pod selectors used over "service selectors"? My understanding of containers is that they should be considered to be short-lived. In addition, we can scale containers (and if using a container orchestrator like k8s, via pods) by creating additional instances on one or more machines.
In order to allow for the ephemeral nature of containers, as well as to allow users to access an application from only one address (say, www.myapp.com), we must abstract away the container. From now on, I'm going to assume we're using k8s as the container orchestrator and speak only of pods. In k8s, we do this by using Services to expose pods (either to other pods which comprise the application, or directly to users). Services allow us to load balance and allow for deployment scenarios such as blue/green and zero-downtime deployments while providing a consistent and abstract interface from which to access the underlying pods.
With this in mind, I have a question about the NetworkPolicy resource in k8s. When creating a NetworkPolicy, a podSelector is used to determine to which pod(s) a network policy applies. However, since a Service must be created in order to expose a pod, why is it, then, that a NetworkPolicy uses a podSelector instead of a "serviceSelector"? Wouldn't the concept of a "serviceSelector" be more general/abstract, and better reflect to what the policy is really allowing access? Is there a reason I'm unaware of for why network policies are "applied" to pods versus services? (Not to say that you would never want a NetworkPolicy to apply directly to a pod, but, generally speaking, most times we apply them to pods which are exposed by a Service—or maybe I'm totally off base here?)

Pods is the deployment unit in Kubernetes and it is also the unit that gets a single unique IP address within the cluster.
You can access Pods without going via a Service. A Service is mostly there to get a "stable" address for an application, where the traffic is then load balanced to any Pod representing the Service. Both Pods and Services gets a DNS name in the cluster.
A Service has a selector to identify what Pods belong to the Service, but a Pod can receive traffic from multiple Services. In the same way a NetworkPolicy has a selector to select the Pods where it is applied.
Since a Pod is the smallest unit in Kubernetes, that gets its own unique network address it also makes sense to add the network security (e.g. Network Policies) at this level.


Changing the build intermediate paths for gitlab-runner I have a server pointed as ab12.xy24.com in which the projects are placed as:

 Path-1
 /home/ab12/public_html/project1/*
 /home/ab12/public_html/project2/*
 ................................
 /home/ab12/public_html/projectn/{project-files}

I am using gitlab-runner for CI and CD. The build is being built as:

 Path-2
 /home/ab12/build/{runner-token}/{commit-id}/{group-name}/project1/{project-files}
 /home/ab12/build/{runner-token}/{commit-id}/{group-name}/project1/{project-files}

I want the build path to be as in Path-1 style instead of path-2 style.

What have I tried?

I tried changing the build_path="path1" variable which only changed:

/home/ab12/build/{runner-token}/{commit-id}/{group name}/project1/{project-files}

to

/home/ab12/path1/{runner-token}/{commit-id}/{group-name}/project1/{project-files}

I cannot also use docker as due to its virtualization concept and takes much space, which is not desired.

Conceptually, this approach is not the way to go; the build directory is not a deployment directory, it's a temporary directory, to build or to deploy from, whereas on a shell executor this could be fixed.

So what you need is to deploy from that directory with a script as per gitlab-ci.yml below, to the correct directory of deployment.

stages:
- deploy

variables:
  TARGET_DIR: /home/ab12/public_html/$CI_PROJECT_NAME

deploy:
  stage: deploy
  script:
     mkdir -pv $TARGET_DIR
     rsync -r --delete ./ $TARGET_DIR
  tags:
    - myrunner

This will move your projectfiles in /home/ab12/public_html/

naming your projects as project1 .. projectn, all your projects could use this same .gitlab-ci.yml file.


Vendoring - what is it? In various tools there is the concept of "vendoring". i.e. Ruby's bundle, and Chef's Berkshelf.

"vendor" is not a verb and in documentation it's often assumed that I know what it means. Can someone help shed some light on what it is and the benefits of doing it.

Vendoring is the concept of downloading/installing a specific version of dependencies and making it available elsewhere (typically, within a local repo/folder of your supporting application)

Vendoring is usually done to prevent breakage occurring when dependences are no longer available. Vendoring also prevents your applications from breaking when dependencies are no longer available( for ex: the leftpad incident)

For example, let's consider you have your Chef cookbook that depends on another third party cookbook, say mysql2 chef gem. Now while mentioning in your metadata.rb, you didn't pin the version or vendor the cookbook. Lets consider the cookbook has had a version upgrade, breaking all previous APIs. The previous versions have also been taken down. In such a case, your cookbooks are suddenly broken as well and if you have a urgent deploy, your Chef run will fail.

Vendoring prevents this by making available the dependency on your local chef server, so even if the dependency has gone ahead, you can continue referring to the previous version and your cookbook will not break

(Also goes without saying, you must keep updating the dependencies and cookbooks, else you might get bit with some bad bugs)


Which Users Should I Create With Terraform And Which With Ansible? I am creating some Ubuntu VMs in Azure with Terraform.
I am then configuring the VMs with ansible.

When terraform creates the VM it creates an admin user

resource "azurerm_virtual_machine" "main" {
  ...
  ...
  os_profile {
    computer_name  = "hostname"
    admin_username = "testadmin"
    admin_password = "Password1234!"
  }

I will need a user to run the application and a user to run ansible.

Should the admin user that terraform creates be the ansible user and then get ansible to create the application user?

I'd say to do it as you last explained it: using the admin user created by tf with ansible, and then create any other user using ansible.

The downside is from the security pov, does ansible needs an admin user ? If yes then keep the tf created user, if not then i'd suggest creating a new non-admin user for ansible.

As ansible is a ssh based config mgmt tool, you should think about using a strong pki.

Using the std tf admin user may results in a unique admin created on your every ubuntu vm, with the very same ssh key and this is not a very good practice (although it is a widely spread practice from my personal experience), this is where a real and strong pki may help you.

Anyway, tf needs to create an (admin) user so keep it that way, then switch to you config mgmt to handle any other required users.


How can I retrieve the execution status of parallel triggered child jobs to a pipeline script I have a pipeline script that executes child jobs in parallel.

Say I have 5 data (a,b,c,d,e) that has to be executed on 3 jobs (J1, J2, J3)

My pipeline script is in the below format

for (int i = 0; i < size; i++) {
  def index = i
  branches["branch${i}"] = {
    build job: 'SampleJob', parameters: [
      string(name: 'param1', value:'${data}'),
      string(name:'dummy', value: "${index}")]
  }
}
parallel branches

My problem is, say the execution is happening on Job 1 with the data 1,2,3,4,5 and if the data 3 execution is failed on Job 1 then the data 3 execution should be stopped there itself and should not happen on the subsequent parallel execution on Jobs 2 and 3.

Is there any way that I can read the execution status of parallelly execution job status on the Pipeline script so that I can restrict data 3 execution to block in Jobs 2 and 3.

I am quite blocked here for a long time. Hoping for a solution from my community. Thanks a lot in advance.

Use the result property of object returned from the build step.

This was already asked and has an answer over on StackOverflow.  Additionally, this is hinted at in the official build step documentation in the description for the propagate parameter ("use the result property of the return value as needed").  I haven't tested it as I don't have a test Jenkins instance available right now, but based on those two pieces of information it looks like something like this should do the trick:

def b = build('my-job-name', wait: true)

def r = b.result


How to mitigate "[DEPRECATION NOTICE] registry v2 schema1 support will be removed in an upcoming release" in quay.io? [DEPRECATION NOTICE] registry v2 schema1 support will be removed in an upcoming release. Please contact admins of the quay.io registry NOW to avoid future disruption

This issue occurs if a docker pull is issued or the API is consulted.

You don't need to do anything; this warning is a mistake. See https://github.com/moby/moby/pull/39736


Getting a docker build ready for (Heroku) deployment? I've got a local application working using docker-compose up, this gets me the servers I need and they function just fine locally.

Now I've tried getting a build.sh started, but I can't figure out how to build a docker image using either docker-compose or just regular docker-cli with the final result of the build process. (The /dist/ folder).

Could somebody explain to me how a regular process like this would look like, and what steps a build process would walk through in docker terms?
Would I need a seperate command to package it for deployment? Is there a separate configuration required?

For building docker image using docker-compose you can add build: field to the service you want to build and assign to it the path of the Dockerfile for example:

version: '2'
services:
  web:
    build: .
    ports:
     - "8080:80"
    volumes:
     - /project:/var/www/html

This example define a web service which:

Uses an image that’s built from the Dockerfile in the current directory.
Forwards the exposed port 80 on the container to port 8080 on the
host machine
Mounts the project directory on the host to /project inside the
container, allowing you to modify the code without having to rebuild
the image.

To build the image, simply issue the build command via docker-compose, as such: docker-compose build.


Definition of change failure rate and mean time to recovery DORA metrics There has been a lot of talk in recent years about the Accelerate book and the DORA metrics.
As the DORA survey was looking at a wide variety of organisations, teams and cultures all with potentially very different current processes and technologies, the questions asked were fairly general and probably leave a fair bit of wriggle room for teams to interpret things their own way. Which is all fair enough.
However, I'm specifically trying to understand what change failure rate and mean time to recovery are trying to measure. Are they just trying to measure if the release process is good or if all of the changes in that release are good?
If I do 10 releases in a row and the release process works smoothly for that and I get no downtime, no manual release tweaks, no alerts, etc...then it looks like I've got a 0% failure rate. But what if there was a relatively subtle bug in that very first release and it doesn't get picked up for 3 days so there is a bug fix going into the next release to fix it. And of course the next release might also have some bugs and so on, so you could say I've got a 100% failure rate due to the bugs.
My instinct is that I should have a high failure rate in this scenario because although the process on the day of release looks good, it the software itself is not good. Just because you have not tested your release before or immediately after release, it doesn't feel like you should get away with claiming no failures.
So if I do get a bug that isn't fixed until the next release in say a week's time, then my mean time to recovery might be a week...unless I did a rollback of that initial release?
What are the common interpretations/implementations of these metrics?

You are asking the right questions and hitting on a very important aspect of these metrics. Individual metrics are not super helpful and measurements need to be defined. Having worked with these measurements across small and large organizations, let me walk you through my thought process.
Definitions:

Metrics - Some sort of calculation on one or more measures
Measure - An amount of something using a specific unit
Change Failure Rate (CFR) - The percentage of deployments causing a failure in production
Mean Time to Recover (MTTR) - How long it takes an organization to recover from a failure in production

This conversation could include Service-Level Indicators (SLI), Service-Level Objective (SLO), and Service-Level Agreement (SLA), however, for this purpose I will assume we have measurements (SLI) to ensure we are meeting our goals (SLO) and not breaking our promise to users (SLA).
Before getting into the details, the main question asked is:
What are we trying to measure?
At the end of the day, any organization exists to create value, these metrics are meant to track any interruption in the flow of value from the organization to its consumers / users.
As an example, if we wanted to measure how efficiently the organization adds new value to users,  Lead Time for changes is a great way. How long does it take new code or business ideas (depending on if you would like to include non technical business processes) to get from creation / ideation to the user? Lead Time includes CFR and MTTR, both of these can interrupt getting value to the consumer and increase Lead Time.
Using the value creation lens, let's look at CFR first.
What does the organization consider a failure in production?
I have typically used it as an event needing immediate fixing, on the call during the production deployment. The service we deployed needed to be rolled back, a hotfix needed to be pushed immediately to correct an issue discovered during the deployment, or something equally catastrophic. These are all interruptions to value reaching the user. A bug not presenting itself for a few days, would not create a Change Failure with this definition. However, if the bug began to trip indicators (SLI) (example, degraded speed or experience for the end user), exceeds our objectives (SLO), or even breaks or promise to users (SLA) then an incident is created and it will bring us to your next measurement, MTTR.
Assuming the bug mentioned creates an incident, MTTR measures how long it takes to fix it. If the fix is a roll back for the original release, or a hotfix of the release, it really does not matter. The measurement is how long it takes to correct.
Pulling this together.

Determine what the measure is.

For CFR, what does the organization consider a production failure? I provided a few examples, but all organizations are free to create their own measure. For MTTR, the measure is time, hopefully from issue creation to correction, sometimes we need to settle for incident discovery to correction.

With the Measure, track the Metrics.

How many failed production releases are there? How long, on average, does it take to correct production incidents?

With the Metrics, track and, hopefully, improve the flow of value from the organization to the consumer.

If a deployment is successful, it should not trip the measurement for Change Failure, however, if there are bugs, their interruption to value will be caught with Mean Time To Recovery.
These measurements and metrics (Lead Time To Change, Mean Time to Recovery, Deployment Frequency, and Change Failure Rate) are meant to tell a story in aggregate.
Assuming weekly deployment, if an organization has 0% CFR, but a 14 Month MTTR with incidents occurring every week, then the flow of value is constantly interrupted and the amazing CFR score means very little. Or, if the CFR is 50% and the MTTR is 2 Mins, then the high CFR looks horrible, but the actual interruption to users is only 2 Mins * bugs per week. It really takes both, or all four of the metrics, to create an accurate picture of the real state of the code and deployment process.
Hopefully this helps. Working through these measurements and metrics is a journey, probably with no conclusion, with a goal of improving the amount of value an organization can create for its consumers.


How to list all users with specific role in GCP gcloud projects get-iam-policy [PROJECT-ID] lists all users with their roles for specific project. There are different filters and formatters available but I can't seem to find the right way to just filter only by specific role.

$ gcloud projects get-iam-policy MY_PROJECT

bindings:
- members:
  - serviceAccount:12345678-compute@developer.gserviceaccount.com
  - user:alice@foobar.com
  role: roles/editor
- members:
  - user:you@yourdomain.com
  - user:someoneelse@yourdomain.com
  role: roles/owner
etag: ARBITRARY_ETAG_HERE
version: 1

What's the --filter= expression that I need to use to list only users with the role roles/owner?

That can be achieved using another gcloud command:

gcloud beta asset search-all-iam-policies --query policy:"roles/owner" --project $your_project_id --flatten="results[].policy[]" --format="csv(bindings.members[0])"


Docker calls to detect app crashing inside it Is there a way to know if an app has crashed inside docker using maybe bash script or API call from host container, am implementing CI/CD using Jenkins and will be deploying the built app via ssh to the main server and want to stop container when it crashes on auto-deploy.

Is there a way to know if an app has crashed inside docker

If your app is writing logs and is the main process in that container, then yes, just watch them

  am implementing CI/CD and want to stop container when it crashes on auto-deploy

What's your tools used for CI/CD? For example, when you are releasing an app with kubectl apply you can wait till automated deployment succeed: kubectl rollout status


What is the DevOps Best Practices for Hybrid environments? I want to learn best practices, and I hope you could help me with your expertise.

In a large organization that uses cloud services and taking advantage of Infrastructure as Code (IaC). My two questions are as follows:

1) What is the best way to patch a cloud IaaS or hybrid environment? I s there a way to do so without downtime at all? Please explain.

2) With the powerfulness of IaC and that we can provision environments in seconds. Is there even a point to patch? Why don’t we just keep provisioning whenever we want and decommission the older ones?

Any inputs will be greatly appreciated!

There may be a significant speed difference between provisioning an environment and bringing up that environment. Especially when it comes to what you mentioned: IaaS and hybrid environments - those are VMs and bare metal servers - it may take many seconds/minutes for some of them to come up.

So there may still be value in patching those environments, provided, of course, that the respective patches can be done without downtime. That depends on the patches - upgrading the kernel on a linux server, for example, would require a reboot to take effect. Upgrading Python won't.

The intended usage for particular IaaS/hybrid environments is also an important aspect in this discussion. The downtime could be completely insignificant for a distributed app designed to tolerate restarts of individual servers in the pool it's running on but probably a big deal for a centralized server app serving thousands of users.


Yaml linting failling when jinja is used I am using https://yamllint.readthedocs.io/en/stable/index.html to lint my yaml files (used in a saltstack context, and linted by jenkins jobs at every commit), but as reported and explained here, it is not possible to validate a yaml file in which is used jinja code.

How do you guys validate / lint your yaml files ?

Do you see a way to keep using yamllint without failing due to jinja code ?

Yaml structure error are the most occuring errors in my company, people from very different background come to edit pillar, that's why I really need a yaml validating process.

My bad, I was searching using the wrong keyword, I just found out that ignoring a file or dir is quite easy, you just need to add an gitignore style block to your yamllint config file, as :

ignore: |
  *globalParamName.sls
  *platform.sls

See the official doc here.


Can Terraform S3 remote state be kept in a different AWS account? I am currently using Terraform with S3 remote state. The IAM user that I'm using to run Terraform exists in the same AWS account as the S3 bucket holding the remote state files. This is all working fine.

However, I have IAM users in other AWS accounts (all part of the same AWS organization) and I want to grant them access to use terraform in the main account. It seems like I should be able to do this with IAM roles. I've created an IAM Role in the main account, and I've configured the assume_role attribute of the aws provider to assume a role in the main AWS account.

However, this approach fails because the user in the other AWS account doesn't have access to the remote state files in S3. I've tried granting access to the remote state files to the IAM role, but terraform apparently tries to fetch state before the aws provider is loaded and the new IAM role is assumed.

Is there any way to grant access to the remote state files in S3? Or more generally, is there any way to allow users across multiple AWS accounts to perform terraform operations across the same multiple AWS accounts, even though sometimes the S3 remote state will be located in a different account?

You can have two approaches:

Assume the role that has access to the AWS bucket. Assume this role in every account (don't forget to set up trust relationship). This is your approach. I think the issue might be some objects (state files) were created in a different way. Other culprit can be an account-specific KMS key that is used to create the objects. Don't forget, KMS keys are region-specific.
You grant access to other “users” via the S3 UI. (At the time of writing it is not supported by the Terraform AWS Provider.)

Go to the S3 bucket -> Permissions -> Access Control List -> Access for other AWS accounts -> Add account
Find the canonical ID for the account you want to give permission to.
Set the bucket-level and object-level permissions. Beware, the Principal (owner) of the object will be different, so the account that actually hosts the bucket might not have access to it.


What are best practices for sharing Docker compose files with ops team to use in Docker stack I'm looking for best practices to share the Docker compose files created by developers with the people that do the deployment.

We use 2 docker compose files: docker-compose.yml and docker-compose.prod.yml.

These files are included in version control (git in our case).

We have 2 options:

Include the compose files in the master branch, and checkout the entire master branch when the application needs to be deployed. Pro here is that it is easy to keep everything in sync, con that the entire projects needs to be cloned on the server when the application is deployed.
Include the compose files in a orphaned branch that can be checked out when the application needs to be deployed. The pro here is that only the compose files will be checked out, con that we have 2 branches that needs to be maintained.

Since we are fairly new to this, I'm wondering how other organizations approach this, and why one method should be preferred over the other?

Some thoughts:

You do not have to clone the entire project.  https://stackoverflow.com/questions/2466735/how-to-checkout-only-one-file-from-git-repository-sparse-checkout?utm_medium=organic&utm_source=google_rich_qa&utm_campaign=google_rich_qa discusses how to checkout a single file.  I have also had good results using curl to get single files off github.
You can containerize it.  Like this

FROM docker/compose:1.21.2
RUN mkdir /src
WORKDIR /src
COPY docker-compose.yml /src/docker-compose.yml
ENTRYPOINT ["docker-compose"]
CMD []

You can containerize docker-compose just like any other application.


SSH to VM inside Docker container I have a Docker image based on Ubuntu Xenial. Inside that container I try to start FreeBSD VM by using Vagrant and VirtualBox (Note: host OS for Docker is VM based on Vagrant and VirtualBox). VM is being provisioned fine, however, Vagrant can not ssh there. I also tried ssh with debug within container:

# ssh -vvv -p 2222 vagrant@127.0.0.1
OpenSSH_7.2p2 Ubuntu-4ubuntu2.2, OpenSSL 1.0.2g  1 Mar 2016
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug2: resolving "127.0.0.1" port 2222
debug2: ssh_connect_direct: needpriv 0
debug1: Connecting to 127.0.0.1 [127.0.0.1] port 2222.
debug1: Connection established.
debug1: permanently_set_uid: 0/0
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_rsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu2.2
ssh_exchange_identification: read: Connection reset by peer

The same from host OS:

# ssh -p 1111 vagrant@127.0.0.1
ssh_exchange_identification: read: Connection reset by peer

I run container with the next parameters:

docker run --privileged -v /dev:/dev -p 1111:2222 vm:latest

I tried to run lightweight image like it was described here and everything started successfully.

Any suggestions/ideas regarding it?

Note: I start a VM in a docker container
because I don't want to make garbage etc within my desktop OS and I don't have a hardware where I could test it. Therefore my approach is to use VM for this.

It looks like issue with ssh timeout is related to nested visualization restrictions. Thus, despite of enabling visualization in BIOS VM in container will still not be provisioned. I found similar answer here.

Once I started Docker container on host OS, issue has gone away and VM successfully provisioned.


Deploying software that depends on a virtual machine using Docker About

I am developing a piece of software which consists of a Node.js application which controls a virtual machine that runs Android-x86. Also there are some pieces of the Android SDK like ADB the software depends on.

My current approach

Currently there is no smart way of deployment whatsoever at all. Everything has to be installed manually. Depending on the system I also get issues regarding networking and so on from time to time.

To solve problems of these kinds I usually would use Docker. Therefore that was my first thought here as well. So such a Docker container would simply run the Node.js application as well as the virtual machine.

My problem

For me it became a bit confusing (and maybe even pointless) when I wanted to use a virtual machine inside of a Docker container. Due to there is obviously no way I can run an Android-x86 Docker container I cannot come up with any other solution than a virtual machine inside of Docker.

My question

Is it fine to do as I mentioned or are there any better ways or even best practices to use a virtual machine inside of a container?

Edits

I want to add that a possible solution does not have to be utilizing Docker. If there is any other technology that is able to create a container-ish construct which in further consequence is able to run a virtual machine I would be happy to get to know it.

Frankly, I've never seen running a VM inside Docker. I believe it would be very slow. Have you checked if https://github.com/budtmo/docker-android is useful?

This Video talks about the Docker and Android SDK as well - https://www.youtube.com/watch?v=YwBAqMDYFCU