How to use Hashicorp Vault with NodeJS application? I have Node JS app inside pods, which needs to read vault secrets.
So far I found 2 methods for doing that

Using init container to mount secrets as .txt files and read/parse them in my app
Using node-vault connect to vault server directly and read secrets, which requires initial token

For (1) I found this article, where the author is considering it as not secure and complex.

Secrets mounted as volumes are unwieldy—secrets can be stored as
environment variables or mounted as a volume. The former technique is
widely agreed to be less secure. If you opt for volumes, things
quickly get complex when you have a large number of keys. Kubernetes
creates one file per key, and you need to read all these files from
within the application. There are workarounds, but they can be equally
complex.

For the (2) I am not sure how to automatically unseal and obtain initial tokens without using Amazon secrets or other services. Of course I can't hard code initial token or roll_id/secret_id. What is the typical way of obtaining and passing initial token to the pods?

As the Linux philosophy says:

Make each program do one thing well

So it's worth it to use the vault agent, cause it's doing well what it would be done.
You can find an awesome example here


What is the utility of an inhouse DevOps if cloud providers already provide professional services? I've seen this in past roles when I was working a devops role as an employee.
What should be the purpose of an inhouse devops in a company that already pays for something like AWS Professional Services, in other words, the cloud provider has teams of engineers who are deployed on-prem to solve business needs, put together an infrastructure, and at the same time promote various AWS services to the client.
Where does the inhouse devops fit in this picture?
I remember some years ago I was in a remote meeting, solved a task, only to be told by my manager "okay, we'll give this to our consultant anyway". Not only was the remark awkward because the entire team was in the meeting, but I also don't see what the point of doing the task was in the first place if someone else would've been assigned to do it.
Does it make sense to take on a devops role if it's not for a cloud provider?

Some good reasons could be:

The salary of an inhouse devops is pretty sure less expensive as the consultants.
The consultants are only assigned during a limited period, to temporary add (wo)manpower, e.g. while putting together the infrastructure.
The consultants come with specific expertise that the inhouse devops do not (yet) have, whereas part of the job of the consultants is to perform knowledge transfer and/or on the job training for the inhouse devops.
The consultants are there to do the boring parts of the work, whereas the inhouse devops are assigned to the strategic / long term tasks (so that when the consultants leave, the know how doesn’t leave the company also.

BTW, IMO the best consultants are those who:

Say their jobdescription is like “My job is to make sure that you don’t need me anymore ...”.
Never touch the keyboard, and instead always coach / help inhouse devops to have them complete any task (it seems so easy when you see somebody else complete some task, but it is way more difficult to complete it yourself).


How to `terraform destroy` an `aws_instance` without also destroying its `aws_ebs_volume` (separate resource) $ terraform -version

Terraform v0.12.21
+ provider.aws v2.34.0

Given a terraform configuration for one set of single aws_instance, aws_ebs_volume and aws_volume_attachment resources, each named, say,foo;

How would one terraform destroy -target=aws_instance.foo without terraform also destroying the aws_ebs_volume.foo as the resulting plan insists that it will?

The aws console reports Delete on termination for the volume's block device to be false;
I've tried to terraform destroy -target=aws_volume_attachement.foo first;
... no-change.
I've tried removing the aws_volume_attachment from the configuration, terraform apply it and then terraform destroy -target=aws_instance.foo;
... didn't help either

Somehow terraform really wants to destroy the aws_ebs_volume resource together  with the aws_instance resource.

I, otoh, would very much like to preserve that aws_ebs_volume for later attachment to a re-created aws_instance...

Can it be done?

This may be what you are doing already (in which case this sounds like a bug), yet it seems just based on the limited information in the question that you may be forgetting a step: Run plan -destroy -target=... -target=... first.  The terraform destroy command will try to operate on changes listed in the .plan file.

The following example assumes a Terraform module named foo, with 3 EBS volumes and attachments named as: dev-sdb, dev-sdf, dev-sdg.  All resources are assumed to have count = local.enabled ? 1 : 0.  So this is the more advanced case you may encounter where each resource would have to be targeted with the array syntax [0].

First create a targeted plan file with terraform plan -destroy -target=....
Do this before trying to run terraform destroy

terraform plan -destroy \
               -target=module.foo.aws_volume_attachment.dev-sdb-attachments[0] \
               -target=module.foo.aws_volume_attachment.dev-sdf-attachments[0] \
               -target=module.foo.aws_volume_attachment.dev-sdg-attachments[0] \
               -target=module.foo.aws_instance.foo[0]  \
               -target=module.foo.aws_iam_instance_profile.foo[0] \
               -out=intermediate.plan

Run terraform destroy, optionally also using -target=... for safety precautions if you wish:

terraform destroy -target=module.foo.aws_volume_attachment.dev-sdb-attachments[0] \
                  -target=module.foo.aws_volume_attachment.dev-sdf-attachments[0] \
                  -target=module.foo.aws_volume_attachment.dev-sdg-attachments[0] \
                  -target=module.foo.aws_instance.foo[0]  \
                  -target=module.foo.aws_iam_instance_profile.foo[0] \
                  intermediate.plan

If you need this to be unattended for automation, add -auto-approve flag to the terraform destroy command.  Otherwise, it assumes an interactive terminal session and will prompt before destroying any resources.


How do you back up Jenkins jobs & master configs? I'd like to backup all Jenkins jobs and config files. What's the easiest way of doing it?

There are many ways to do this but the easiest way I can think is doing a backup of the Jenkins Home folder.

You can see where is your Jenkins home with:

echo $JENKINS_HOME

And for example, if you only want to backup the jobs you can go to:

cd $JENKINS_HOME/jobs

And make a backup for that folder.

All that configuration will be a bunch of XML files.

If you are using the official Jenkins docker image, the home will be on:

/var/jenkins_home


Jenkins slave not able to find dotnet When I check for dotnet version it is throwing an error

D:\jenkins\workspace\csharp-windows-test>dotnet --version
17:58:06 System.ArgumentNullException: Value cannot be null.
17:58:06 Parameter name: path1
17:58:06    at System.IO.Path.Combine(String path1, String path2)
17:58:06    at Microsoft.DotNet.Configurer.CliFolderPathCalculator.get_DotnetUserProfileFolderPath()
17:58:06    at Microsoft.DotNet.Configurer.FirstTimeUseNoticeSentinel..ctor(CliFolderPathCalculator cliFolderPathCalculator)
17:58:06    at Microsoft.DotNet.Cli.Program.ProcessArgs(String[] args, ITelemetry telemetryClient)
17:58:06    at Microsoft.DotNet.Cli.Program.Main(String[] args)
17:58:06

but when I search for 'where' it shows the location

17:58:06 D:\jenkins\workspace\csharp-windows-test>where dotnet
17:58:07 C:\Program Files\dotnet\dotnet.exe

This is causing build failures. Can someone suggest a best way to resolve this

Also I was able to execute the script on the machine directly without issues. This is only happening when called from Jenkins

When I did echo %USERPROFILE% from Jenkins it returned null. So I had to
manually set USERPROFILE value from Jenkins as environment variable. Also I had to set these

APPDATA=%USERPROFILE%\AppData\Roaming

LOCALAPPDATA=%USERPROFILE%\AppData\Local


Creating ec2 instances and installing mongoDB using automation I am trying to create a 3 node MongoDB replica set using automation (ansible or terraform) in AWS. I am able to do this in the management console of AWS, but I'm not sure how to approach it.

Essentially, the script must create the instances and then set up a 3 node MongoDB replica set on them. I want to add an EBS disk on it and then maybe secure it.

All of this via Ansible (or Terrafor) so I can automate it fully.

Is this possible?

Thanks!!

It's possible. I would use Terraform for creating the infrastructure on AWS and Ansible for provisioning. You will have to write .tf files and ansible-playbooks in order to do this.

Just to let you know, if all 3 nodes are equal you should do the installation and provisioning once and then create an AMI to set up the rest. Maybe an autoscaling group is a possibility too.

Keep in mind that AWS provides documentdb as document-oriented database as a service. It's not mongoDB but it's pretty similar (there are some functions mongo provides and docdb doesn't).


Reuse of docker swarm containers for batch jobs Docker compose allows for static stack description.
But what if I have a set of containers based on same image but different by subtle environment configuration? Moreover, I don't need them all at once, they can be launched and disposed of sequentially.

A quick shot would be to copy-paste the service, but a couple of dozens of them is a waste of resources.

Is there sort of elegant orchestration loop in this case?

UPD After some more googling, interesting fact came out that apparently what I do is batch job processing which has currently no conceptual (!) support in Docker compose because services are not batch jobs. While there is a proposed feature for that I will try out one of the community workarounds, Jobs As A Service (JaaS) but my question is still - maybe there is an elegant way to do the following:

version: '3.1'

services:

  task1:
    image: myjob
    environment:
      PARAM: foo

  task2:
    image: myjob
    environment:
      PARAM: bar

You may have asked yourself "why not just take a series of docker run/exec/rm", but the rest of the environment like database runs in the stack.

Correct me if I am wrong, but that is one of the reasons why docker orchestration tools exists like docker-swarm to be able to deploy multiple docker containers by using one command line.

  Create a service with 5 replica tasks (–replicas)

  Use the --replicas flag to set the number of replica tasks for a replicated service. The following command creates a redis service with
  5 replica tasks:

$ docker service create --name redis --replicas=5 redis:3.0.6

4cdgfyky7ozwh3htjfw0d12qv

If for example 1000 identical containers that use a different environment variable have to be deployed, multiple docker service create could be added to a script, instead of copy-paste multiple times in one docker-compose file.


What is the recommended way to monitor differences between multiple Kubernetes namespaces/clusters? Given a Kubernetes cluster with two namespaces; what is the recommended way to monitor and sync differences in configmaps, deployments, ingresses etc. between the namespaces?

Edit: The deploy script that is described in this answer is very much  looks like Helmfile  and we are now in the process of upgrading to Helmfile so that we can retire our scripts that run multiple Helm releases as that’s that Helmfile does well.

This is a bit of an alternative and indirect  answer. Use infrastructure-as-code best practices to avoid environments getting out of sync. I believe people will find it a helpful alternative suggestion so a valid answer.

Prevention is better than cure. Don't let the namespaces get out of sync by treating your configuration as code and manage it via git. Helm is a package manager for Kubernetes configuration. It can handle arbitrary configuration objects such as Openshift extensions. This let's you setup a deployment pipeline and a git workflow that ensures your environments stay in sync. (Update: use Helmfile to manage multiple Helm releases).

By way of an example in our setup we have a Helm chart that just installs configmaps and another that just installs secrets. The actual values to install are passed in as a parameter like my-app-secret-values.yaml. So we have different yaml values all under git and a deploy script that iterates over them applying the helm chart that loads the values and updates the corresponding configmap or secret. Helm reads, compares, and only writes when something has changed. So the deploy script will result in only changing a secret or config map that has changed in git. All the configuration is under git and managed like code. Syncing configuration between environments can then be managed just like promotion of code. As a bonus helm creates a backup every version of every object it changes stored inside of Kubernetes. If let's you list the prior versions and rollback to a previous configuration. Also as everything is deployed from git all the history of all settings in all namespaces can be seen in git.

We also install and run out main applications as traditional Helm charts that are promoted through environments. Those helm charts reference the configmaps and secrets that are separately deployed when we want to adjust the settings in a given environment.


How to not "fire and forget" when doing a remote trigger of a Jenkins build? I am attempting to trigger a parameterized Jenkins build from Bamboo, doing the below:

  POST -
  http://jenkins-url.com/job/jobname/buildWithParameters?ENVIRONMENT=dev&APPLICATION=hello-world

But I will get a 201 instantly which tells me that the build was created. How do I make this request wait and return the success status of the build instead of the fire and forget?

It is apparently possible as per Parameterized-Remote-Trigger-Plugin:

Edit:
created this in the end if you need it.
https://github.com/owenmorgan/jenkins-remote-builder

I've created the jenkins-remote-builder script which will follow your remote build to completion.
Some more details about it (from its README.md):

Example
jenkins=https://user:pass@jenkins.mydomain.com:8080
jenkins_job=MyApp-Deploy

environment=dev
application=myapp
revision=9fd71f63b351b8208264daf86d292ced580a2f60

./jenkins_remote_trigger.sh \
            -h ${jenkins} \
            -j ${jenkins_job} \
            -p "ENVIRONMENT=${environment}&APPLICATION=${application}&REVISION=${revision}"

Usage:
-h HOST     | --host=HOST                       Jenkins host
-j JOBNAME  | --jobname=test-build-job          The name of the jenkins job to trigger
-p JOBPARAM | --jobparam=environment=uat&test=1 Jenkins job paramiters
-q          | --quiet                           Don't output any status messages


Are all virtualization (OS/above-OS) utilities DevOps utilities? Are all virtualization (OS/above-OS) utilities commonly considered DevOps utilities?

These could be OS virtalization utilities like VirtualBox and Vagrant and maybe also WSL is considered as such and also, above OS virtualization tools like docker (AFAIK CMs like Ansible, Chef, Puppet and so forth aren't meant for "virtuallizing" anythin).

I'd say you have touched another, far more fundamental question - what makes a utility a "DevOps" one? That is, how do you measure the "DevOpsness" of a tool and formalize facts in this context?

Possibly, my wish list for DevOpsness of a tool, virtualization tools or other, if they fulfill further criteria down on this list:

Open and interoperable API, both for operational and intelligence needs
Optimized workflows, see "developer experience"
Machine readable configuration as code
User community
Short release cycles
Certificate of software quality which is nowadays yet very ambigous


Docker & Django - what directory does localhost point to? I'm completely new to Docker and Django and I'm having trouble getting my Docker app to serve inside Django. Currently, I'm running my docker container with this command:

docker run -it --name web -p 8000:8000 --link mysql:db -v ~/directory:/app django

where mysql and django refer to containers defined earlier.

Next, I'm using manage.py to make migrations and migrate, then serving with the following command:

mod_wsgi-express start-server --reload-on-changes --working-directory /app/voting_project /app/voting_project/voting_project/wsgi.py &

When I visit localhost:8000 in my browser, it successfully renders the HTML template, but the CSS and images which I see locally are missing.

The following errors are logged in the console:

I've played with the structure of my directories a bit, but I can't seem to get the browser to find anything in localhost:8000/static or localhost:8000/images.

I'm assuming 8000 is mapping to somewhere weird in my local directories, but I realize this could also be caused by a number of other things. Please let me know if I'm making any obvious mistakes.

Thanks so much!

This is most likely a problem with the django configuration better fitting to stackoverflow, and not so much a devops problem.

Check the settings.py for the following entries:

INSTALLED_APPS = [
    (...)
    'django.contrib.staticfiles',
]

At the end of this file

STATIC_ROOT='staticfiles'
STATIC_URL = '/static/'
STATICFILES_DIRS = [
     os.path.join(BASE_DIR, 'static'),
]

source Django manual: How to static-files


docker as slave for jenkins I have installed Jenkins and docker on the same machine Ubuntu.

Ubuntu 17.10
Jenkins ver. 2.73.3
Docker version 17.09.0-ce.

I am trying to set up jenkins docker containers as slaves for running my automation test suite.

I was able to correctly setup the docker plugin in Jenkins for spinning up docker containers and added a docker template with a image that I created for setting up docker environment.

The image has been built on docker hosted in Ubuntu.

Problem is now when I run a job from Jenkins. It gives an error message:

  (pending—Jenkins doesn’t have label docker-slave)

When I check the Jenkins logs in Ubuntu machine I see the following error message:

> com.github.dockerjava.api.exception.NotFoundException: {"message":"pull access
denied, "message":may require 'docker login'"}

In the ubuntu machine, I have already given done docker login.

The image that I am trying to build containers from is locally on ubuntu, not pushed to any repository, so why is trying to pull the image?

Also, what is the permission issue that I need to resolve? When build a job from jenkins, it's the jenkins user building the container. Do I need to add something else there?

Sounds like the docker image you are using doesn't have the correct name, or is stored in a private repository which requires login.

Please post more information


Docker Container always stopped and started I am using docker compose to build two containers, one is nodejs with pm2 and another is nginx.
The nginx one runs well but the nodejs with pm2 one keeps restarting: it runs for one second, then stops for ~10 seconds, then starts again and runs for one second, as you can see this is the output of docker ps -a:

I have used docker attach 'ID' but it still restarted.
Using docker logs shows nothing, but jenkins says that pm2 was started successfully:

This is the content of my Dockerfile:
FROM node:8
MAINTAINER test
RUN npm install -g pm2

WORKDIR /var/www/marketing

EXPOSE 80

RUN npm install
RUN pm2 start npm -- stage

If any of you could help me, I will update this question if you guys have any request.
Thank you

You should keep at least one service running in foreground to keep the container running, you can do that using ENTRYPOINT or CMD or both. For more info check Dockerfile: ENTRYPOINT vs CMD.

You can also use a docker image for pm2 from docker hub, that will save your time and you can check the Dockerfile for it to see how it is written.
Check the keymetrics/pm2-docker-alpine project, for example.


Docker swarm custom scheduler We have a 6 node docker swarm running and we often see bad scheduling (clean inbalance) happening.

I've researched this subject, and it seems docker swarm as standalone used to allow a way to choose different strategies to its scheduler, but since swarm was integrated into docker it only supports the spread strategy. In our case, since we are very memory intensive, we would benefit from only considering memory and not CPU/#containers as a metric.

How could we achieve this?

The scheduler itself is not swappable. However, you can set memory constraints and reservations on containers in your service:

version: '3'
services:
  redis:
    image: redis:alpine
    deploy:
      resources:
        limits:
          cpus: '0.50'
          memory: 50M
        reservations:
          cpus: '0.25'
          memory: 20M

The constraint will kill the container if it exceeds the limit even if the host has more memory available. The reservation will ensure that amount of memory on the host is available for the container.


Store data in Jenkins job description I have a job which I run with a boolean parameter called "BUILD_CLEAN=true", and it defaults to false. I can store this flag in the job ID currentBuild.displayName . Simple enough. But for my purposes I also want to save the commit hash(es) of that build as well, which starts becoming a silly place to store metadata. I'm probably not getting the terminology right, but here is my use case.

My Pipeline declarative job builds a repo, a subrepo and all using shell, and archives the binary artifacts. The code is of mixed types, so Artifactory is not an option for part of this solution either. I want to save just the hashes someplace, to make it easy for subsequent builds made in the same day (24 hours) to inspect/list them, and trigger a separate job with the full list for that day. Do I save these hashes/values into an XML file per build? What is the best way to make extracting small pieces of data easy? GOAL: I'm attempting to implement a kind of generalized incremental build job, but am unable to use any of the plugins since I'm building for 5 different targets, some of them across nodes. In my specially marked build (nightly) every 24 hours, I'm .bz2 zipping the objects so I can restore them for any incremental. So it's this archive that I'm wanting to find, check the hashes and retrieve into my workspace.

TAGS: Jenkins,pipeline,tags,incremental (I'm unable to define any tags due to the rep system.)

Not sure if this would work for you, but what I do is store all of the build metadata into a BuildInfo.json file. You could just save it as an XML file like you said, but I found it easier to parse info with Groovy if it was a JSON file. Along with the most recent commit hash, we store most of the build information in this file, including the result, duration, user who kicked off the build, etc.

One advantage of this approach is we're able to create other jobs that take action of already built products, such as for deployments. These jobs can just look for the presence of a BuildInfo.json file and take action based off the information contained within. Example:

import groovy.json.*

def build_info = [
    'Product': 'MyApp',
    'Build Type': 'Full',
    'Build Name': currentBuild.displayName,
    'Version': '1.2.3.4',
    'Source Branch' : params.Branch,
]

def json_output = readJSON text: groovy.json.JsonOutput.toJson(build_info)
writeJSON file: "BuildInfo.json", json: json_output, pretty: 4


Terraform - resource repeated multiple times I have a terraform plan which consists of multiple aws_sqs_queue resources which more of less share the same config. Before I remove any repeating configs and use variables I wanted to run terraform plan to see if it works. However I keep running into -

module root: 1 error(s) occurred:

* data.template_file.ep_match_result_queues: resource repeated multiple times

Ive tried googling but there is not much information available. Any help will be much appreciated. My plan looks like this: (ignore extra braces, typos as this is just a skeleton to give you a general idea how the plan is sturctured):

data "template_file" "ep_match_result_queues" {
  template = "${var.namespace}-sm-ep"
}

resource "aws_sns_topic" "sns_topic_name" {
  name            = "ep_sm_match_result_topic${var.environment}"
  display_name    = ""
  policy          = <<POLICY
{
 #policy
}

resource "aws_sqs_queue" "queue1" {
 #config
}

resource "aws_sqs_queue" "queue2" {
 #config

    redrive_policy             = <<POLICY
{
 #policy
}
POLICY
}

resource "aws_sqs_queue" "queue3" {
#config
}

resource "aws_sqs_queue" "queue4" {
 #config

    redrive_policy             = <<POLICY
{
 #policy
}
POLICY
}

resource "aws_sqs_queue" "queue5" {
 #config
}

resource "aws_sqs_queue" "queue6" {
  #config

    redrive_policy             = <<POLICY
{
 #policy
}
POLICY
}

resource "aws_sqs_queue" "queue7" {
 #config
}

resource "aws_sqs_queue" "queue8" {
  #config

    redrive_policy             = <<POLICY
{
 #policy
}
POLICY
}

resource "aws_sqs_queue_policy" "queue_policy" {
  queue_url = [ ... ]
  policy    = "${data.aws_iam_policy_document.match_result_queues_policy.json}"
}

data "aws_iam_policy_document" "match_result_queues_policy" {
 #policy
}

Terraform scans the configuration directory for all files whose name end in .tf, and loads them all as configuration.

A common cause of this problem is if you accidentally duplicate one of the .tf files under a new name, and do not update its contents. In that case, Terraform will see both files define the same resource and produce the error shown here.


What are best practices of structuring encrypting variables since Ansible 2.3.0? This document indicates that since Ansible 2.3 it should be possible to encrypt certain vars using a !vault | prefix instead of putting a variable and keys in a vault file and encrypt it completely.

notsecret: myvalue
mysecret: !vault |
          $ANSIBLE_VAULT;1.1;AES256
          66386439653236336462626566653063336164663966303231363934653561363964363833313662
          6431626536303530376336343832656537303632313433360a626438346336353331386135323734
          62656361653630373231613662633962316233633936396165386439616533353965373339616234
          3430613539666330390a313736323265656432366236633330313963326365653937323833366536
          34623731376664623134383463316265643436343438623266623965636363326136
other_plain_text: othervalue

At the moment the structure is as follows:

ansible/group_vars/testing/vars
ansible/group_vars/testing/vault

When an encrypted variable is moved from the vault to the vars directory and ansible-vault decrypt ansible/group_vars/testing/vars is run it returns:

ERROR! input is not vault encrypted data for ansible/group_vars/testing/vars

  This vaulted variable be decrypted with the supplied vault secret and
  used as a normal variable. The ansible-vault command line supports
  stdin and stdout for encrypting data on the fly, which can be used
  from your favorite editor to create these vaulted variables; you just
  have to be sure to add the !vault tag so both Ansible and YAML are
  aware of the need to decrypt. The | is also required, as vault
  encryption results in a multi-line string.

Questions

Should variables that need to be encrypted, be encrypted one by one
using the command line?
What are best practices to restructure the
old Ansible structure? E.g. remove the vault files and put all
encrypted vars in the vars file?

Having been working with ansible vault quite a bit recently (specifically regarding what the encrypt and how to encrypt those things without making the code unreadable) I've found very little incentive to change my habits in 2.3.

If I'm going to have a series of encrypted variables, or files, keep them separate (I prefix them all with vault_) and just leave it at that.

A few times I have used the !vault feature but am left distinctly unamazed as it seems easier to just be very explicit about what is and is not coming from a secrets file. That way no one editing my plays makes assumptions about what is and isn't sensitive data.


Ansible Playbook fails start Apache Cassandra I have been tried ran a Ansible Playboo, but after change the cassandra.yml when the service tries start I received this error bellow:
● cassandra.service - LSB: distributed storage system for structured data
   Loaded: loaded (/etc/rc.d/init.d/cassandra; bad; vendor preset: disabled)
   Active: failed (Result: exit-code) since sex 2021-04-02 01:06:05 UTC; 15min ago
     Docs: man:systemd-sysv-generator(8)
  Process: 5052 ExecStop=/etc/rc.d/init.d/cassandra stop (code=exited, status=1/FAILURE)
  Process: 4883 ExecStart=/etc/rc.d/init.d/cassandra start (code=exited, status=0/SUCCESS)
 Main PID: 4972 (code=exited, status=3)

abr 02 01:05:37 ip-10-5-10-195.ec2.internal cassandra[4883]: Starting Cassandra: OK
abr 02 01:05:37 ip-10-5-10-195.ec2.internal systemd[1]: Started LSB: distributed storage system for structured data.
abr 02 01:05:40 ip-10-5-10-195.ec2.internal systemd[1]: cassandra.service: main process exited, code=exited, status=3/NOTIMPLEMENTED
abr 02 01:05:40 ip-10-5-10-195.ec2.internal runuser[5063]: pam_unix(runuser:session): session opened for user cassandra by (uid=0)
abr 02 01:05:40 ip-10-5-10-195.ec2.internal cassandra[5052]: Shutdown Cassandra: kill: sending signal to 4972 failed: No such process
abr 02 01:05:40 ip-10-5-10-195.ec2.internal runuser[5063]: pam_unix(runuser:session): session closed for user cassandra
abr 02 01:06:05 ip-10-5-10-195.ec2.internal cassandra[5052]: ERROR: could not stop cassandra
abr 02 01:06:05 ip-10-5-10-195.ec2.internal systemd[1]: cassandra.service: control process exited, code=exited status=1
abr 02 01:06:05 ip-10-5-10-195.ec2.internal systemd[1]: Unit cassandra.service entered failed state.
abr 02 01:06:05 ip-10-5-10-195.ec2.internal systemd[1]: cassandra.service failed.

In my cassandra.yml I just change the mount point/path where Cassandra stores the data and sure that this works.
Looking at Cassandra system.log I found:
ERROR [main] 2021-04-05 16:57:34,074 CassandraDaemon.java:822 - Exception encountered during startup
java.lang.ExceptionInInitializerError: null
    at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:240)
    at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:676)
    at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:800)
Caused by: java.lang.IllegalStateException: Failed to mkdirs /hints; unable to start server
    at org.apache.cassandra.config.DatabaseDescriptor.createAllDirectories(DatabaseDescriptor.java:1395)
    at org.apache.cassandra.db.commitlog.CommitLog.<init>(CommitLog.java:95)
    at org.apache.cassandra.db.commitlog.CommitLog.construct(CommitLog.java:78)
    at org.apache.cassandra.db.commitlog.CommitLog.<clinit>(CommitLog.java:65)
    ... 3 common frames omitted

It's weird because I create the directory at new mount point with cassandra:cassandra permitions:
drwxr-xr-x  2 cassandra cassandra   6 abr  5 14:46 commitlog
drwxr-xr-x  2 cassandra cassandra   6 abr  5 14:46 data
drwxr-xr-x  2 cassandra cassandra   6 abr  5 16:53 hints
drwxr-xr-x  2 cassandra cassandra   6 abr  5 16:53 saved_caches

I have been spend long hours trying solve but without success, someone my can help me?

I found my fault.
It's necessary define a cassandra.service to Systemd manages Cassandra:
path: /etc/systemd/system/cassandra.service
And adjust like you installation
[Unit]
Description=Cassandra Cluster Node Daemon

[Service]
Type=forking
User=cassandra
ExecStartPre=/usr/bin/echo "Starting Cassandra Daemon"
ExecStart=/usr/sbin/cassandra
ExecStartPost=/usr/bin/echo "Cassandra Daemon Running"

ExecStopPost=/usr/bin/rm -rf /cassandra/saved_caches;/usr/bin/echo "Cassandra Daemon Stopped"

SuccessExitStatus=143

[Install]
WantedBy=default.target


How is DevOps related to Agile? Agile has been around for quite a while. And several offspring came out of it, such as Extreme Programming, Scrum, Lean Software Development and others.

Isn't DevOps just another offspring of Agile?
Is there any substantial difference between DevOps and Agile?

DevOps and agile aren't same thing but they are complimentary to each other. Agile development is development that adheres to the principles stated in The Agile Manifesto. In brief, agile is the word for an environment in which the priorities, according to the authors of the manifesto, are-

Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation and
Responding to change over following a plan.

The point of DevOps is to deliver technology to business units in a timely fashion and ensure the technology runs without interruption or disruption. To achieve it's goal DevOps mostly focus on-

deep communication between software development and IT operational groups
automated deployment processes

Below are list of relation/differences with agile and DevOps-

Agile software development is a methodology for developing software, but DevOps, on the other hand, is all about taking software which is ready for release and deploying it in the safest, most reliable manner possible.
Agile is all about development. Sometimes, it takes over the entire company. Even when it does, agile discipline doesn't inevitably lead to DevOps. The practice of DevOps involves a separate discipline and methodology from those of agile.
Agile practices, like Continuous Delivery practices, can be part of a DevOps activities.


What happens when a PAT expires In azure devops when you create a PAT you can set it's lifetime up to a year.  What happens when it expires?  Does it auto renew?  Do I need to crate a new one and for every service that uses it, I need to track down and update it's credentials?

I had expirienced the same problem, I needed to renew manually my PAT.

VS

MSDN


How to create slave jenkins given master jenkins? Below code snippet, taken from here creates master jenkins that runs in a container:

# Create Docker Group with GID
# Set default value of 497 if DOCKER_GID set to blank string by Docker Compose
RUN groupadd -g ${DOCKER_GID:-497} docker

# Used to control Docker and Docker Compose versions installed
# NOTE: As of February 2016, AWS Linux ECS only supports Docker 1.9.1
ARG DOCKER_ENGINE=1.10.2
ARG DOCKER_COMPOSE=1.6.2

# ......

# Install Docker Engine
RUN apt-key adv --keyserver hkp://pgp.mit.edu:80 --recv-keys 58118E89F3A912897C070ADBF76221572C52609D && \
    echo "deb https://apt.dockerproject.org/repo ubuntu-trusty main" | tee /etc/apt/sources.list.d/docker.list && \
    apt-get update -y && \
    apt-get purge lxc-docker* -y && \
    apt-get install docker-engine=${DOCKER_ENGINE:-1.10.2}-0~trusty -y && \
    usermod -aG docker jenkins && \
    usermod -aG users jenkins

Above container is dind that launches task(each container) on single EC2 host.

To regulate load across multiple EC2 host and launch slave jenkins instance(on each).

We would like to have master jenkins generate build_id for each build that can run on two slave jenkins instances.

To have master & slave jenkins run within each container, across EC2 hosts.

How to approach on making slave container and take build_id from master jenkins?

You could use the EC2 plugin or the ECS plugin if you want to provision agents dynamically. These would provide some environment variables to use when constructing build_id - e.g. :

BUILD_TAG:

  String of "jenkins-${JOB_NAME}-${BUILD_NUMBER}". All forward slashes ("/") in the JOB_NAME are replaced with dashes ("-"). Convenient to put into a resource file, a jar file, etc for easier identification.

NODE_NAME:

  Name of the agent if the build is on an agent, or "master" if run on master

You would have to configure the clouds in Jenkins, assign a label to the templates you want to use as agents, and then request those in your pipeline or job configuration.


Ansible Expect Module hung on attempting a simulated return keystroke I would like to use the expect module in an ansible playbook to telnet through a termserver to reach a NetApp filer. I tried the telnet module, but using the port argument after the ip address doesn't seem to work.
Manually I would type the following:

[sansible@scspr0415857001 ansible]$ telnet 10.233.82.7 2033
Trying 10.233.82.7...
Connected to 10.233.82.7.
Escape character is '^]'.
Connected to port 33.

login: admin
Password:

rtpsb-f6280-02a-1::>

~~~~~~~~~~~~~~~~~~~

I think my issue is that the playbook is getting caught up on simulating a return keystroke after "Connected to port 33."

I've also tried using echo -ne '\n'

Playbook:

---
- hosts: localhost
  connection: local
  gather_facts: false

  tasks:
    - name: Expect Attempt
      expect:
        echo: yes
        command: telnet 10.233.82.7 2033
        responses:
          Question:
            - Connected to port* "echo"
            - login: "admin"
            - Password: "password"

The full traceback is:

  File "/tmp/ansible_piMb3N/ansible_module_expect.py", line 190, in main
    encoding='utf-8')

fatal: [localhost]: FAILED! => {
    "changed": true,
    "cmd": "telnet 10.233.82.7 2033",
    "delta": "0:00:30.122472",
    "end": "2018-03-08 13:08:12.487220",
    "invocation": {
        "module_args": {
            "chdir": null,
            "command": "telnet 10.233.82.7 2033",
            "creates": null,
            "echo": true,
            "removes": null,
            "responses": {
                "Question": [
                    "Connected to port* \"echo\"",
                    {
                        "login": "admin"
                    },
                    {
                        "Password": "password"
                    }
                ]
            },
            "timeout": 30
        }
    },
    "msg": "non-zero return code",
    "rc": 1,
    "start": "2018-03-08 13:07:42.364748",
    "stdout": "Trying 10.233.82.7...\r\r\nConnected to 10.233.82.7.\r\r\nEscape   character is '^]'.\r\r\nConnected to port 33. ",
    "stdout_lines": [
        "Trying 10.233.82.7...",
        "",
        "Connected to 10.233.82.7.",
        "",
        "Escape character is '^]'.",
        "",
        "Connected to port 33. "
    ]
}
        to retry, use: --limit @/home/sansible/ansible/expect-r1.retry

PLAY RECAP   *********************************************************************************************************************
localhost                  : ok=0    changed=0    unreachable=0    failed=1

Any help would be greatly appreciated!

I figured out the issue. The "Question:" parameter needs to be omitted:
    ---
    - hosts: localhost
      connection: local
      gather_facts: false

  tasks:
    - name: Expect Attempt
      expect:
        echo: yes
        command: telnet 10.233.82.7 2033
        responses:
          Connected to port 33. :  "echo"
          login: "admin"
          Password: "password"


Running Two infinite commands on single pod/image I have a Dockerfile with a CMD, it calls a .sh file that runs a Django app
( makemigrations, migrate and uwsgi )
And that runs fine in a pod.

There's a django admin script that runs infinitely that I'd like to run in the same pod.

Per the docker docs I can't have a second command in the Dockerfile
Adding the script on the sh file wouldn't work because the both processes ( the app or command script ) are running and don't end so the second one never runs.

If I go in the pod and run exec -it bash to manually run the script i can't leave the pod without killing the process (Ctrl-C) or can  I?

Any suggestions?

Though it's not advised to have multiple processes inside a container, but you have 2 options:

Use a process manager like supervisord or wrapper script to use as CMD (meta code)

start_first_process_in_bg
if START_FAILED
  exit
start_second_process_in_bg
if START_FAILED
  exit

while sleep 5; do
  check_first_process
  check_second_process

  if ANY_PROCESS_FAILED; then
    exit


Ansible communication medium I am a little confused as to what is the protocol that Ansible uses for communication between  Controller node and Managed Remote nodes.

I believed that it was "SSH" protocol.

However, the official documentation mentions this
"On the managed nodes, you need a way to communicate, which is normally ssh. By default this uses sftp. If that’s not available, you can switch to scp in ansible.cfg"

I am a little confused as I used to believe SSH and SFTP and SCP are completely different protocol.
AFAIK,
SFTP is used to securely transfer files and SCP is for secure copy. This may be over SSH connection, but why would ansible use SFTP or SCP?

Is it about transferring some files/python modules using these protocols?
If yes, what files are getting transferred actually?

Kindly Excuse me if this question sounded silly, nevertheless any elaborate response would be much appreciated.

This is a great question as some of the terminology can be very confusing - particularly surrounding the (unfortunate) use of "SFTP" to describe multiple similar technologies.

Will the real SFTP please stand up

This comes mainly from the fact, that RFC 913 defines one SFTP protocol, the "Simple File Transfer Protocol" which offers no encryption and went largely unadopted, with users favoring TFTP and FTP depending on their needed application.

Since the FTP protocol, defined in RFC 1738 in 1985 is completely encrypted and packet sniffers are able to dumping plaintext passwords from this protocol, Some thought that it would be a good idea to secure this protocol by implementing FTP over an SSL connection. In 1997 a mechanism for doing that was released in RFC 2228 which often was referred to as SFTP.

Meanwhile, back at the ranch...

Meanwhile, a similar security issue was occurring around the telnet protocol - passwords and sensitive information were unencrypted and subject to sniffing. In response, SSH-1 was released in 1995 which provided an encrypted equivelent to telnet for use over network connections.

This protocol became very popular very quickly and before a formal specification allowing for FTP over SSH was released, users of the SSH protocol had implemented a version of BSDs rcp (remote copy) over SSH. This protocol was extremely limited however, which led to a revision in SSH - SSH2 which added an SSH File Transfer Protocol (not to be confused with FTP over SSH, another SFTP) because, according to wikipedia:

  Compared to the SCP protocol, which only allows file transfers, the SFTP protocol allows for a range of operations on remote files which make it more like a remote file system protocol. An SFTP client's extra capabilities include resuming interrupted transfers, directory listings, and remote file removal.

And In fact, the IETF set up a working group to codify a formal standard for SSH-2 (released as RFC 4251) but

  Eventually, development stalled as some committee members began to view SFTP as a file system protocol, not just a file access or file transfer protocol, which places it beyond the purview of the working group.

And eventually SSHFS was released as just that.

Conclusion

So the answer is a bit complicated.

SSH and SFTP and SCP could be regarded as completely different protocols depending on which SFTP you are referring to and which version of the SSH protocol you are referring to. Yet, the command for actually doing a file transfer remained "scp" even after the SFTP and SSH-2 protocol was developed. If one is referring to SSH-2 however, these could arguably all be considered the same protocol.

Side note: SSH was released by Tatu Ylönen of the Helsinki University of Technology in 1995. As Linus Torvalds released Linux in 1991 as a student at the University of Helsinki until 1996, it was kismet that SSH would be so popular and widely used in Linux.


How should GCP projects be organized? I have recently inherited the previous director of devops duties in managing a transition from Heroku/AWS to GCP. I'm wondering what is best practice for organizing projects in GCP.

Should the be organized by production, staging, and QA environments? Should they be organized by each individual self contained front-end project or middleware project? If so, where do databases live in that function? The middleware project?

You can see how many core questions can lead to adjacent questions.. does anyone have best practices here about how they organize their GCP projects in the enterprise?

Corollary question: how does your project organization affect cost or inter-networking?

Thanks!

Speaking primarily from the Google App Engine (GAE) perspective.

My preference is for environment-driven project split. This might be of interest: Advantages of implementing CI/CD environments at GAE project/app level vs service/module level?

I wouldn't go for architecturally-driven project split unless there are serious reasons for doing so (if you feel you have such reasons you should ask with details):

probably difficult (if not impossible) to avoid inter-project bandwidth charges
added configuration complexity for inter-operation between projects (GCP pretty much considers them completely isolated/standalone, except maybe for shared IAM permissions). Possibly relevant: Google app engine access cross project resources

Depending on the GCP products used there may be sufficient support for architectural splits inside the same project. At least GAE does IMHO.

My GAE apps are all standalone and do not use inter-networking at all, so no costs from this perspective. There are still networking (bandwidth) costs for accessing various GCP-provided infra/services, but those are irellevant/unavoidable from the project split perspective.


How to create Golang Linux binaries using a Windows host When go build is issued on a Windows host a .exe binary is created. How to ensure that a Linux binary is created on a Windows host?

The following was tried, but did not solve the issue:

GOOS=linux;go build hello-world.go

You can easily set the target operating system and processor architecture using the environment variables GOOS and GOARCH respectively. So, as you want to build it for linux operating system, following command with above environment variables will do,

$ GOOS=linux GOARCH=amd64 go build -o hello main.go

Here is the list of all the supported operating system with which you can easily do cross compilation using Go lang.


Chef resource to register gitlab runner automatically We would like to automate installation of gitlab runner using chef.  we are  successful with it. Now, the Issue is regarding registering gitlab runner with chef resource.
we have gone through chef resource. and official cookbook https://gitlab.com/chef-platform/gitlab-ci-runner but looks a bit complicated for us. we wonder if we have any pre-built chef resources to register gitlab runner. if yes, can we know how we can register with a example resource.
Additionally, I have tried looking at resource https://github.com/TYPO3-cookbooks/site-gitlabcirunnertypo3org/blob/master/recipes/default.rb

name = "#{node['fqdn']}-docker"
  gitlab_ci_runner name do
      options(
        registration_token: registration_token,
        url: node['site-gitlabcirunnertypo3org']['gitlab_url'],
        executor: 'docker',
        :'docker-image' => 'docker:1',
        :'tag-list' => ['docker']
      )
end

But I dont understand how it is registering the runner. Thanks for help!

Your second link just use the official cookbook. This official cookbook still uses the LWRP syntax, so you need to check in the provider directory for the resource action code -- which for register is basically just two execute commands (extract from the file linked above):

converge_by("Register runner #{new_resource.description}") do
  execute "#{env} gitlab-runner register --non-interactive #{options}"
  # giving time to gitlab-ci to manage the request
  execute "sleep #{new_resource.sleep}"
end

So mostly, your exemple ends up calling the gitlab-runner register command with the options defined in the options hash (with a little transformation of the hash into command parameters).

There's no 'core' chef resource to register gitlab runners out of using an execute resource to run the command.

A basic example of this resource is present in the cookbook Readme:

gitlab_ci_runner 'my runner' do
  options({
    registration_token: '1234567890',
    url: 'http://gitlab-ci.myinstance'
  })
end


How do I get Openshift to let Tomcat retain it's hosted site's changes I have a pod in Redhat Openshift running Tomcat and inside Tomcat I have Ozone Widget Framework, my site hosted in Tomcat. I have executed the oc command to use my existing configuration so that my pod won't disappear and lose my session and data. I ran

oc cluster up --host-data-dir=/path_to_data/openshiftdata --use-existing-config

But after making changes within OWF, when I take down the cluster and bring it back up my pod lives on, but my OWF instance is reverted back to it's default state.

How do I get Openshift to let Tomcat retain OWF's state?

You need to use a Persitent Volume.

Your /path_to_data/openshiftdata is probably on a disk that belongs to the pod and is being destroyed each time.


Is it possible to securely delete a file from a virtual disk in a cloud provider? Considering that virtual hard drives are generally formatted using something like ext3 that abstracts away from the underlying block device, which in and of itself is an abstraction over the top of a physical and network topology.

Is it possible to use something like wipe or shred against a single file or an entire device to mitigate the risk of data being recovered at a later date if that machine was compromised?

You can't really use those tools with ext3 anyway. Wipe manpage has a large note about it:

  No secure deletion program that does filesystem-level calls can sanitize files on such filesystems, because sensitive data and metadata can be written to the journal, which cannot be readily accessed. Per-file secure deletion is better implemented in the operating system.

Moreover, the fact the data will be written really in-place is not guaranteed and you may be writing aside the original data.

That said the reason to do this goes back to a point where a data was written on disk on consecutive sectors on the same physical medium, usual storage nowadays use striping technology to spread data on multiple physical medium, each write in the system toward this storage will have an impact on all the stripes this file was part of. Each provider will have its own custom storage backend, but I'm pretty confident the same principles are used.

When you get the picture even higher, you're writing a file on a virtual hard disk which is itself a file on a storage pool, probably streched on multiple lun composed of raid array of physical disks.
There's so many IOs running on all those strates of striping/aggregation that the probability someone could recover a file by magnetic reading of physical disks (what wipe and shred should protect against) is very very very low as the storage used for this file has probably been reused a bunch of time already.

Even on existing ext3/ext4 filesytem, tools like extundelete have a hard time recovering any file if the disk is not unmounted/mounted read only very quickly after the deletion.


Why does docker not restart on a stack trace? I'm running a container with the following: docker run bot --restart-policy always

However, the container is never actually restarted if it exits due to a stack trace, e.g.,:

Traceback (most recent call last):
  File "/jirabot/source/jbot.py", line 32, in <module>
    rtm = slack_client.rtm_read()
  File "/usr/local/lib/python2.7/site-packages/slackclient/client.py", line 235, in rtm_read
    json_data = self.server.websocket_safe_read()
  File "/usr/local/lib/python2.7/site-packages/slackclient/server.py", line 283, in websocket_safe_read
    data += "{0}\n".format(self.websocket.recv())
  File "/usr/local/lib/python2.7/site-packages/websocket/_core.py", line 310, in recv
    opcode, data = self.recv_data()
  File "/usr/local/lib/python2.7/site-packages/websocket/_core.py", line 327, in recv_data
    opcode, frame = self.recv_data_frame(control_frame)
  File "/usr/local/lib/python2.7/site-packages/websocket/_core.py", line 358, in recv_data_frame
    self.pong(frame.data)
  File "/usr/local/lib/python2.7/site-packages/websocket/_core.py", line 301, in pong
    self.send(payload, ABNF.OPCODE_PONG)
  File "/usr/local/lib/python2.7/site-packages/websocket/_core.py", line 250, in send
    return self.send_frame(frame)
  File "/usr/local/lib/python2.7/site-packages/websocket/_core.py", line 275, in send_frame
    l = self._send(data)
  File "/usr/local/lib/python2.7/site-packages/websocket/_core.py", line 445, in _send
    return send(self.sock, data)
  File "/usr/local/lib/python2.7/site-packages/websocket/_socket.py", line 117, in send
    return sock.send(data)
  File "/usr/local/lib/python2.7/ssl.py", line 707, in send
    v = self._sslobj.write(data)
socket.error: [Errno 32] Broken pipe

What am I missing about the purpose of the restart policy always?

The command line is order sensitive. This command:

docker run bot --restart-policy always

Runs the bot image as a container with the value of CMD set to --restart-policy always. To modify the restart policy of the container being run, you need to pass the option after run and before your image name:

docker run --restart-policy always bot


What is a "Walking Skeleton"? One of my agile teams has taken an interesting approach in the early stages of their project.  Instead of starting the project with a Sprint 0 where they setup code infrastructure and decide on the solution architecture, they have started building a "Walking Skeleton", which they describe as a DevOps practice.

What this seems to come down to is building something very small (in the case of an API a single endpoint that just returns 200-OK), getting this working in continuous integration, and building out the continuous delivery pipeline to deploy this through each of the environments:

  Development ► Test ► UAT ► Pre-production ► Production

In the process they have managed to tick off many of the non-functional requirements that could have been missed if deployments were left to the last minute.

My question is this: what is a "Walking Skeleton" and what benefit does it provide to a Agile team following DevOps practices?

A "Walking Skeleton" is a form of "proof of concept" of your basic architectural concept. Where a proof of concept typically focuses more on a single functionality, a "Walking Skeleton" is a minimalistic end-to-end implementation. A "Walking Skeleton" is not an outline of your concept (only a "skeleton") but is really executable and shippable (it can "walk" :O) and should be accompanied with tests.

Alistair Cockburn has described it (and is being quoted often):

  A Walking Skeleton is a tiny implementation of the system that performs a small end-to-end function. It need not use the final architecture, but it should link together the main architectural components. The architecture and the functionality can then evolve in parallel.

The advantage here for DevOps is that a "Walking Skeleton" should be developed early on in the project and results in working, shippable and testable code. This way DevOps can set up a full continuous integration chain early in the project, instead of being onboarded in the final phase of projects. This means that any issues that would arise are also being tackled in an early stage instead of rush work at the end.


Trigger PagerDuty alert after a number of incidents Is it possible to only have an alert triggered after a certain number of incidents from an integration?

For example, if my application reports a single non-critical failure of a kind, that's worth-while to troubleshoot, but probably not so bad as to wake up somebody in the middle of the night.

However, if the application is reporting the same failure over and over again, it's a symptom of a bigger problem that somebody should look at ASAP.

Any ideas?

The easiest way to build this would be to have some sort of metrics-based alerting system like Prometheus, datadog, etc.  Those systems allow you to have a counter that gets incremented and you can see how many incidents happened in a given period of time on a pretty graph.  Most metrics systems will tie nicely to PagerDuty for the alerting piece.  Having any of these metrics systems functioning will also give you a historic baseline for your systems that is often incredibly handy in figuring our what went wrong and when.

The threshold should probably be number of incidents over a given period of time.  If you just pick an arbitrary number, say 100, for this you will always eventually get to that limit.  If you see 10 incidents in an hour that could be bad for you.

The hardest part of this would be that I don't think PagerDuty makes it easy to access metrics about the number of incidents.  You can look at pretty graphs in their UI, but there's no easy way to feed that into your metrics system.  You would need to add a bit of code to the other places you have creating incidents and have them increment this counter.


AWS Cloudformation: Callback only for CREATE_COMPLETE or CREATE_FAILED of entire stack I am creating a CF stack;

I have been using the --notification-arns flag of aws cloudformaction create-stack to send notifications to an email topic; however this produces a huge amount of spam, given that it records every X_IN_PROGRESS and X_COMPLETE, where X=CREATE/DELETE for every resource on the stack;

I just want to get notifications for the stack CREATE/DELETE events.

Is this possible?

I believe, alternatively, you could use CloudTrail+CloudWatch rules, but then you will get CREATE_STACK events for all stacks
Here is some information about that:

https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/cfn-api-logging-cloudtrail.html
https://docs.aws.amazon.com/AmazonCloudWatch/latest/events/EventTypes.html

And, of course, there's the option to put a lambda which listens on the notification topic and filters the needed events before sending the email.


How to avoid a high number of requests for avoiding Amazon AWS EC2 instance AutoScalin? If I set autoscaling for an EC2 instance in Amazon Web Services, how can I set optimized autoscaling for large numbers of connections in order to avoid unusual autoscaling?

First of all, the number of connections is not a common metric to use for autoscaling. Much more common is CPU utilization or possibly the amount of network traffic.

If you are happy with the performance of your application at a certain utilization level, you can modify the corresponding alarms configured in the auto-scaling group, so that autoscaling happens only at higher levels of your metric.

Secondly, you can limit auto-scaling to a certain number of instances. When this number is reached, no further scaling will occur.

Finally, if your traffic has a significant proportion of GET requests, it may be easiest to ensure caching headers are set appropriately and traffic is routed through a CloudFront distribution before it reaches the load balancer. This will likely eliminate most of the GET requests from the load balancer altogether.


How do I add my IP address to the start of my terminal using the bashrc file? I want to customise my terminal on an Amazon linux device.
I have colours using $(tput setaf 33);, user name with /u but the IP address not there.

#blue=$(tput setaf 33);
PS1="\[${blue}]\u$(tput sgr0)"
PS1="${blue}[\u@\w $(tput sgr0)"
PS1+=""
export PS1;

Well... you would need to know what IP address you actually want. Is it a public IP address? Is it a private subnet address? Do you only have 1 IP address? A few things you would need to figure out.

But, if we just assume that we want to have whatever ip address your VM thinks it's address is then:

Add the following to your .bashrc

PS1="\[$(tput setaf 33)\][\u@$(ip -4 -o addr show scope global $(ip -4 route | grep default | head -n1 | awk '{print$5}') | awk '{gsub(/\/.*/, "",$4); print $4}' | head -n 1) \w ]\[$(tput sgr0)\] \[$(tput setaf 34)\]\\$\[$(tput sgr0)\] "

To throw an IP address into your prompt.

Now, because that's not really human friendly, let's break it down:

Make the prompt blue (honestly I can't remember why I always wrap with \[ \] I just remember someone telling me I needed to do that, so I guess see what works for you:

\[$(tput setaf 33)\]

This bit you already had figured out:

[\u@

This insanity to find the default IP address, it's a bit hard to break down without just reading through the ip man pages but basically we are searching for the interface marked as default then grabbing the IP address and stripping all the extra stuff out with awk/grep/head:

$(ip -4 -o addr show scope global $(ip -4 route | grep default | head -n1 | awk '{print$5}') | awk '{gsub(/\/.*/, "",$4); print $4}' | head -n 1)

I tend to want to know my ${PWD}:

\w ]

Reset our colour scheme just to be stafe (I've found weird stuff happens when you keep layering colours without resting:

\[$(tput sgr0)\]

Set some other colour:

\[$(tput setaf 34)\]

Set the $ or # prompt which is handy if you want it to be obvious if you are logged in as root:

\\$

Then reset colour again:

\[$(tput sgr0)\]


Best practices for external backups for S3 buckets? I would like to know how everyone is making external backups of S3 buckets and best practices/recommendations for this. Regardless of how good AWS is it is still a single point of failure.

Especially on a more corporate scale, vast amounts of buckets that are very large.

The S3 buckets functionality seems to be very limited.

My resolution was the following:

  The way I'm looking at doing this now is by downloading the data to an external server (bare-metal as EC2 instance is a lot more expensive) and then pushing that to my backups provider.

In terms of S3, for increased durability, consider cross region replication as an option if you want to increase the durability. AWS also provides their AWS Backup service you might investigate. I'm not certain of any benefit to transferring large amounts of data for durability to another provider. You'll pay a lot more for transfer, and it won't be integrated first class with the security IAM provides.

If the concern is about someone maliciously doing something then there are security best practices, access logging, cloudtrail, and other options to provide assurance. If it's more about the possibility (as low as it seems) for a bucket to fail, then the cross-region replication might be a great option for you with minimal configuration impact.


How can I build & maintain a personal, cross-platform 'toolkit' of commonly used applications? I've discovered a number of open source tools over the years which have become an essential part of my personal workflow & productivity in my current day job (project manager / business analyst). I can often move from company-to-company and even account/machine (and OS too) and want to get up and running with the minimum of fuss.
What I'm trying to combine:

apps/tools (which use a variety of frameworks and languages including python, flask, javascript, node, java etc)
config files
other misc. files (e.g. fonts)

However, by necessity, I am often behind corporate firewalls and restrictive IT policies which make it time-consuming to get internal license approvals for each tool, download each of the tools individually, keep them up-to-date, blocked external package repos, maintaining & building-upon the associated configuration files over time...
Docker is the obvious solution here, but...
... it is often blocked by these same IT policies, as are its alternatives.
What I've tried
Using a GitHub repository, I've since settled on building/installing everything locally, making it 'portable' (using batch scripts, symlinks and a number of config file tweaks) and then uploading.
Drawbacks of this approach include: the complexity, difficulty maintaining this cross-platform (currently windows-only), working around applications installed via git-clone, working around quirks of 'portable' versions of runtime dependencies like Python, github 50mb single file-size limits...
Another possible approach I considered
Using continuous integration (like Travis-CI and AppVeyor), my main concern here is that I've not actually 'built' some of the apps I use (e.g. emacs-w64, which I download the pre-built binaries for) so wondering if this will just add more complexity than my current solution ... and since Travis-CI doesn't support windows it adds an additional dependency, as well as the overhead of learning the continuous integration world & maintaining build scripts for linux, osx & windows..
I'm not looking to open up discussion here, but an answer that will meet my needs with the minimum of fuss (this is for personal use rather than devops for my company / others after all) from the experts ... even if the answer is simply that I need to put the effort into a CI solution...

For your own personal use, I think a GitHub repository with all of your bootstrap and installation scripts is a great idea. It serves as a portable way to download and distribute any scripts you need where firewalls and restrictions aren't an issue.

As for within your organization, one suggestion that your IT might be open to as a compromise is to create a machine (barebones, VM, etc.) that is solely dedicated to downloading and storing the artifacts that you need. The criteria for this machine would be:

It has access to download artifacts (files, apps, etc.) from a source (preferably either the open internet or from white-listed addresses).
It is able to store the artifacts in a central location.
It is not able to access internal infrastructure
It is has auditing to track any artifacts that it downloads/stores

With this setup, you could download something like the latest version of Python onto the server (via logging in with credentials, API, etc.) and have it store Python in a central location for your other servers to download and install. You could store your artifacts in a multitude of ways (artifact repository, cloud storage, file server, etc.), but only that server should be able to upload to your storage. Your IT team could even set alerts when a new artifact is downloaded on this server or uploaded to your storage so that they can inspect it.

I think this approach demonstrates a few things to your IT department:

These artifacts are important enough to you and your teams that you are willing to go out of your way to establish a safer and more traceable method to retrieve them.
You have given your IT department an automated way to track who and when an artifact was downloaded in the event an artifact is malicious and needs to be deleted and traced to a source (something they probably didn't have in place before).
The method you propose not only gives you more accessibility to the tools you need to be productive, but you are proposing ways to help your IT team be more productive by not having to manage all aspects of downloading artifacts.


Automate custom docker image deployment on AWS This question is related to my previous one - but a bit more descriptive.
I am trying to build automate the generation and deployment of a docker container. The docker container is created with packer.
After the creation of the docker image, I am using packer's post processing modules to upload it to the AWS ECR:

{
    "type": "docker-push",
    "ecr_login": true,
    "aws_access_key": "<snip>",
    "aws_secret_key": "<snipe>",
    "login_server": "https://<snip>.dkr.ecr.eu-west-1.amazonaws.com/"
}

This works just fine. The docker image got uploaded to my repository in the ECR.
Now, the next step is where I am stuck.
I want to run start the container on AWS.

I am thinking a little script that uses the aws-cli commands could be suited for that.
I do not want to use the web ui - and I am sure this can be done over the command line, especially when someone wants to integrate this into a pipeline of some sort.

However, I have a few questions, as I am not sure if I understand the big picture correctly. My docker image is available over the ECR. To run it, I need some base instance, right? According to a reply in the previous thread, the are various way of running docker containers, either by using the ECS or by spinning up a EC2 instance. Since this is related to some research project where docker and virtualization is analyzed I am questioning what would be the best option for my case. Probably EC2? It's important that I can automate it so that no user interaction is required.

I read the docs and checked several tutorials but I still am not entirely sure how to everything correctly in the right order. Especially part where the environment is set up and the image is started.

I know that I can list my images with:

aws ecr list-images --repository-name prototypes.

How would I fire up a new instance and run my image on it?

If anything is unclear, please let me know so that I can update my question to be as detailed as possible.

I will give it a try and explain all required steps with examples.

Please add comments with questions and improvement suggestions if anything is not well enough explained.

The OP (original poster) refers to awscli; while I provide corresponding examples, I also discuss limitations of this approach and give examples of doing same with Python.

EC2 deployment

The purpose of this answer is to demonstrate that as OP correctly assumed, no Web UI access neither human interaction is mandatory to deploy a container on an AWS EC2 instance.

Scope and limitations

In this use case, we want to start just one container manually for demo/research purposes, therefore we explicitly decide to do no orchestration.

For a complex and possibly more realistic deployment scenario, this example is therefore not reusable. For same simplicity reasons, we will use EC2 service and create there a single virtual machine.

According to a short documentation research, there seems to be no official AMI with pre-installed Docker daemon. Thus we need to deploy some default EC2 instance and install there Docker daemon, then start the daemon.

While using a private registry, a login there is required; for simplicity purposes, I give also an example to run a container from the public Docker Hub registry.

Also, I am not going to provide a copy-paste automation script but single steps which can be easily adopted towards individual needs. In any case, intermediate results require parsing and passing operations which is then a pure programming activity concern.

A boilerplate automation could be done either with bash using awscli or with Python, using the boto3 SDK for AWS.

Python is worth to consider because you can do better processing of JSON responses you get from AWS. You can also tell Python to execute SSH login and bash commands on the remote host.

Execution plan

A. Identify recent Linux AMI (Amazon Machine Image) ID
B. Create keypair and instance (further, also referenced as VM)
C. Connect to the instance
D. Install and configure Docker daemon
E. Run container

A. Identify recent Linux AMI image ID

Search for a recent and up-to-date Amazon Linux image following the official documentation:

$ aws ec2 describe-images --owners amazon --filters 'Name=name,Values=amzn2-ami-hvm-2.0.????????.?-x86_64-gp2' 'Name=state,Values=available' --query 'reverse(sort_by(Images, &CreationDate))[:1].ImageId' --output text
ami-01f14919ba412de34

B. Create keypair and VM

With awscli, you would just say:

$ aws ec2 create-key-pair --key-name ec2-docker-test

However, the API sends you your public key wrapped in a JSON data message. Therefore, using Python it is way easier to process this data message to a .pem file:

import boto3
ec2 = boto3.resource('ec2')

# create a file to store the key locally
outfile = open('ec2-keypair.pem','w')

# call the boto ec2 function to create a key pair
key_pair = ec2.create_key_pair(KeyName='ec2-docker-test')

# capture the key and store it in a file
KeyPairOut = str(key_pair.key_material)
print(KeyPairOut)
outfile.write(KeyPairOut)

With the AMI ID we have got previously we can now create the VM:

instances = self.ec2r.create_instances(
                 ImageId=AMI_ID,
                 MinCount=1,
                 MaxCount=1,
                 InstanceType=instance_type,
                 KeyName='ec2-docker-test'
             )

AWS API allows also lookup of instance state and IP. With little bit more  Python/boto3 stanzas you can acquire this data to know when you can proceed.

Note: Firewall consideration

To configure firewall access, please create a security group and link it to machines's VPC. For readability reasons, example is not included.

C. Test SSH connection

As soon as the instance is up and running, you can do everything you like by means of remote command execution.

In a Bash script, you would use ssh only or scp to upload a local bash script and run it remotely.

 ssh -t <REMOTE_IP> -i <KEY_FILE> <COMMAND>

With Python, you use further modules like paramiko to establish SSH connection and run commands remotely.

client = paramiko.SSHClient()
client.connect(ip, username=username, key_filename=key_filename, port=port,
                           timeout=timeout,
                           auth_timeout=timeout)

stdin, stdout, stderr = ssh.exec_command("whoami")

For readability purposes, I list further commands unwrapped. In your script, you might want to use elementary helper subroutines to have calls like ssh_call(command).

D. Install and configure Docker daemon

As said above, anything else from here is running remote commands.

That is, the concluding steps are, following an example in the official documentation; proceeding is same as you would do it locally, but you need to wrap it in your script and pass over intermediate results if required:

install and configure Docker daemon
login to your Docker registry if needed
pull image
run Docker container

    sudo yum update -y
    sudo amazon-linux-extras install docker
    sudo service docker start
    sudo usermod -a -G docker ec2-user
    # relogin or continue with sudo, which you shouldn't
    aws ecr get-login --no-include-email --region region

E. Run container

Pull custom image and run container from a private registry in AWS.

    docker pull aws_account_id.REGISTRY.ecr.REGION.amazonaws.com/REPOSITORY/IMAGE:TAG
    docker run --name MYSERVICENAME -d -p PORT_HOST:PORT_CONTAINER IMAGE:TAG

Pull an official public  image and run another container listening on port 8001 in a single step:

    docker run --name nginx -d -p 8001:80 nginx:latest

Alternative scenario with ECS

As described in these Python/boto3 examples here and here, it's possible to define an machine cluster with the AWS ECS service and run containers there.

Please note that ECS cluster offers you an abstraction either from AWS EC2 or AWS Fargate to deliver the actual virtual machines.

If we go for EC2, again we need to pick up a recent machine image, but this we need a machine optimized for ECS. In a nutshell, these machines will have an ECS agent; through its configuration, we assign a machine to our target cluster.

A small bonus is also that ECS machines have Docker daemon already preinstalled.

So, we create an ECS cluster and add a virtual machine to it.

ecs_client.create_cluster(clusterName=cluster_name)

ec2_client.run_instances(
        ImageId=AMI_ID,
        MinCount=1,
        MaxCount=1,
        InstanceType="t2.micro",
        IamInstanceProfile={
            "Name": "ecsInstanceRole"
        },
        UserData="#!/bin/bash \n echo ECS_CLUSTER=" + cluster_name + " >> /etc/ecs/ecs.config"
    )

In the cluster, we can create a task definition of the service which are both abstract of the container concept.

ecs_client.register_task_definition(
        containerDefinitions=[
        {
          "name": "<MY_SERVICE>",
          "image": "<MY_IMAGE>",
          "portMappings": [
            {
              "containerPort": 80,
              "hostPort": 80
            }
          ]
        }
        ],
        family="hello_world"
    )

Now, possibly you ask yourself, how would the Docker daemon on the instance know about your private registry to pull the specified image? For that, the machine's ECS agent needs additional configuration.

Now, you can launch the service, which will use the previously defined task to tell the previously authorized Docker daemon to pull the previously built and published image from ECR and run the container on the machine you've included in the ECS cluster in the data center house that Jeff built.

ecs_client.create_service(
        cluster=cluster_name,
        serviceName=service_name,
        taskDefinition=task_name,
        desiredCount=1,
        clientToken='request_identifier_string',
        deploymentConfiguration={
            'maximumPercent': 200,
            'minimumHealthyPercent': 50
        }
    )

Configuration of security groups in the ECS context has been highlighted on StackOverlow already. In short, these are further calls to the API parts related to security group management.

Conclusion

Both Bash and Python implementation options have their benefits and drawbacks.

Seen on long term, Python being a general purpose programming language with large community, allows complex string manipulation and abstractions from a cloud API or a system process.

Without knowing the real goal of the deployment, an end-to end of above steps might not be worth the effort in terms that for deploying a set of related containers which is more typical scenario, other approaches could be more pragmatic.

Before stepping into technical implementation, best is to make well elaborated decisions and choices addressing business goals and budget:

how many times will automation need to run and in which time period vs. time to implement it end-to-end from the beginning?
assess whether you need only EC2, or ECS with EC2 either Fargate for stateless tasks, or maybe you need Kubernetes?

Further reading: "The Benefits of Managed Kubernetes vs. Amazon ECS" (short argument from a cloud consultant blog as of August 2019).

  A 2018 CNCF survey cites 83% of organizations use Kubernetes as
  their container orchestration solution vs. 24% for ECS.


Staging - Test multiple microservice versions together I been trying to search for answer for this question but so far I didn't found anything useful.

Our problem is that we have 2 microservices with many staging branches, each of these microservices are on different servers and environments.
We want to test each microservices branch with each other in our staging environment so microservice1-branch20 with microservice2-branch10.
They need to communicate two-way so they both need to know that they only can contact with that version of the microservice.

Is there any good way to do these tests without putting them together in one docker container?

Yes, this can be done by Service Discovery. The product family you're looking for is of products such as Consul. You should have the container services report their identities and versions when they load. Report their version and ip (for example), and have the tests run against these values.

This will require some effort with installations and scripting, and has a learning curve - but this is the way to go.


Where does TFS 2017 actually store artifacts? We had been using a TFS build process that copies build output to some fileshare for later use. But with TFS 2017 we want to use that fancy "Publish Artifacts" build stage that has the TFS server provide a .zip file of the artifacts via the RESTful API.

But now I want to be careful about where all these build artifacts are going to be stored and what disk space to keep an eye on. Do these build artifacts just become massive binary blobs in the SQL database that TFS uses? Can we configure what location TFS uses to hang onto these artifacts? Most documentation I've found just describes the artifacts from an external perspective with the implementation abstracted away.

There are two options when publishing artifacts: Server and File Share.

If you publish to the server, they are stored in the TFS database. You can't configure the location, however you can configure retention policies to ensure that old builds aren't kept around forever.

If you publish to a file share, they're stored on, well, a file share.


How do you reuse a specific volume in Kubernetes? Let's say that you wanted to create a Jenkins Deployment. As Jenkins uses a local XML file for configuration and state, you would want to create a PersistentVolume so that your data could be saved across Pod evictions and Deployment deletions. I know that the Retain reclaimPolicy will result in the data persisting on the detached PersistentVolume, but the documentation says this is just so that you can manually reclaim the data on it later on, and seems to say nothing about the volume being automatically reused if its mounting Pods are ever brought back up.

It is difficult to articulate what I am even trying to ask, so forgive me if this seems like a nebulous question, but:

If you delete the Jenkins deployment, then later decide to recreate it where you left off, how do you get it to re-mount that exact PersistentVolume on which that specific XML configuration is still stored?
Is this a case where you would want to use a StatefulSet? It seems like, in this case, Jenkins would be considered "stateful."
Is the PersistentVolumeClaim the basis of a volume's "identity"? In other words, is the expectation for the PersistentVolumeClaim to be the stable identifier by which an application can bind to a specific volume with specific data on it?

Yes, the PersistentVolumeClaim is the workhorse here. You can evict pods, delete deployments, etc., to your heart's content and so long as you don't delete the PVC, your data will be there.

You don't need to use StatefulSet for this.  From the docs:

Unlike a Deployment, a StatefulSet maintains a sticky identity for each of their Pods. These pods are created from the same spec, but are not interchangeable: each has a persistent identifier that it maintains across any rescheduling.

https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/

You might want that anyway, but it does nothing for keeping your PV/PVCs around.


What is Titus and what issues does it solve? I read an article from Medium Titled "The Evolution of Container Usage at Netflix" and watched a couple of videos on the topic, but it is unclear to me why Titus (referenced in these) should be used. Is it different than Kubernetes or should it be used in concert with Kubernetes?

It's different to Kubernetes in that it seeks to leverage existing AWS services instead of bolting on top. Kubernetes supports a number of cloud providers and contains a number of components that exist in AWS already.

Although we're starting to see better kubernetes AWS integration and with AWS joining the cloud native foundation this is only going to get stronger.

Titus seems to be an alternative to Kubernetes but its not available yet. My bet is AWS will strive to better integrate K8 and likely Titus won't take off even when it is released.


Automatic production deployment for a Node.js app on RHEL 7? I am looking for some way to have a maximally simple(no Jenkins or anything like that) automatic production deployment for a Node.js application on a RHEL 7 system. The caveat is:

At the organization where I work, we have a very restricted IT environment, that prohibits access to any ports other than 80 and 443 from anywhere, and only allows 22(ssh) from within some rooms in the office building. To upload files a tunnel is needed which is slow and is also not available in some places. And SSHing to the server to run commands is a huge pain for multiple reasons too. And no - I can't do anything about it.

In light of this the 'trigger' has to initiate from the server itself. The server should somehow decide to pull changes from a certain branch, ideally when a special commit message is seen, and then run some pre-configured scripts to deploy it. The tests at that point are already all done, the app has been at the staging and beta server and so the only things needed is to copy directory, update dependencies, compile frontend assets and restart the systemd service that runs the app (all that I already have in the script that Jenkins executes on staging server after running all the tests).

I would imagine the sever would need to poll, e.g. every minute for changes to the repo, and yes, that can be done with a cron job, but I am not sure how, and if there are some tools that make it easier, without creating a resource hog that Jenkins does (which is what we use on our staging and beta servers), then I would like to know about it (I wasn't able to find anything yet, maybe I am not using the right search terms).

Any advice?

Update-1: I am aware of Puppet and Chef, but if at all possible, would prefer avoiding proprietary solutions if possible for now because a - our project is not big enough yet for the decision makers to put real resources in it, and b - even if those have free versions available, they are most likely cumbersome and do a lot of things, whereas we want as simple a solution as possible, because that solution might be part of the package if we sell the project elsewhere (naturally we don't want to also "re-sell" Puppet or Chef). Some kind of Open Source solution, or simply a method of doing it using simple shell scripts would be best.

Update-2: As Tensibai mentioned - Chef is open source and fits our purposes, but it's a huge platform with a lot of features, which is precisely what we are trying not to use in production server (otherwise we'd just use Jenkins like we do in staging).

Ok, let's assume you'll pull the latest version of branch prod from a git repo, here's what it would take with chef on a basic illustrative purpose:

git "/opt/my_application" do
  source "https://<git_host>/<you_repo_url>/<repo.git>"
  revision "production"
  action :sync
  notifies :run,"execute[app_config]"
end

execute "app_config" do
  command "npm install"
  cwd "/opt/my_application"
  action :nothing
end

For more advanced use cases you can have a look at the poise application cookbooks, it includes a javascript plugin for node.js application.

chef is fully open source, either client or server side (out of the fancy UI) and on all OSes including windows.


Which container will be targeted when replicas larger than 1 Below is a simplified version of my docker-compose.yml file.
I can understand or find any resources to tell me which container of the 5 django replicas the nginx container will target if i reference it.
For example, as you can see if my compose file I have one instance of Nginx and five of Django.
If my nginx config i have an upstream like this:
upstream docking_django {
    server web:3000;
}

Which web container will the upstream come from? Will it load balance between them?
I'm deploying to AWS ECS and when I try running it, it seems like it's load balancing somehow. But I would like to understand it further since it doesn't seem to behave the same way always.
version: '3.9'

services:
  web:
    image: .dkr.ecr.eu-west-3.amazonaws.com/tin-api-v2/django:${IMAGE_TAG}
    build:
      context: .
      dockerfile: ./Dockerfile
    networks:
      - demoapp
    deploy:
      replicas: 5

  nginx:
    image: .dkr.ecr.eu-west-3.amazonaws.com/tin-api-v2/nginx:${IMAGE_TAG}
    build:
      context: .
      dockerfile: ./nginx.Dockerfile
    ports:
      - 80:80
    depends_on:
      - web
    networks:
      - demoapp
    deploy:
      replicas: 1

networks:
  demoapp:

Yes Felix. Loosely speaking it "load balance" across your 5 web container instances. If you are using the ECS / Compose integration and you are bringing up this stack with a compose up this is roughly what happens:

it creates an ECS cluster
it creates task def for the web task and the nginx task
it creates a service for web (with 5 tasks in it)
it creates a service for nginx (with 1 task in it)
it exposes the nginx service via an ALB to the public (because you expose the port 80)
it creates an internal mechanism based on ECS Service Discovery to be able to resolve web inside the cluster (that's how the nginx is able to reach an entity called web).

ECS SD leverages CloudMap/Route53 which will round-robin across the 5 entries (web tasks). You can read more here.


How do you instruct AWS FARGATE to provision a t2.small via ECS CLI instead of the default micro? I am trialling AWS ECS for the first time. I've run through the tutorials and now I'm trying to deploy an existing Docker Compose suite which I've been using on Digital Ocean for some time.

I have four containers in my composition, an ExpressJS app, NGinx, MemcacheD, and a Python Flask API. These work OK on a Digital Ocean 1GB RAM host.

I've been following the guide and specs here: https://github.com/aws/amazon-ecs-cli#launching-an-aws-fargate-task and other general ECS CLI guides.

I've got ECS to make the cluster+service from the docker YAML (with a few tweaks), but when I try to UP the service, I get an error that the t2.micro EC2 which Fargate boots isn't big enough:

INFO[0016] (service CI) was unable to place a task because no container instance met all of its requirements.
The closest matching (container-instance ...) has insufficient memory available.
For more information, see the Troubleshooting section of the Amazon ECS Developer Guide.

So I'd like to force it to boot an t2.small (2GB RAM) or higher.

I see that you cannot directly tell Fargate to use a specific instant type from ECS - that setting is only for "EC2" direct, but I want to use Fargate to automate the instance management.

I have tried using the task_size memory parameter (which I believe to be the correct way of informing Fargate of your application's requirements?) but this doesn't affect the choice of EC2 instance type which Fargate boots - it still boots the default t2.micro.

I added this YML file with --ecs-params:

version: 1
task_definition:
  task_size:
    cpu_limit: 512
    mem_limit: 2GB

When I rebuild my cluster from scratch with this definition, I can see the 2GB is correctly configured under the taskDefinition page (console.aws.amazon.com/ecs/home#/taskDefinitions/...) - so the configuration is provided OK, but when I view the instance it's booted, it's still a t2.micro with 1GB RAM.

Am I misunderstanding the task_size.memory parameter?

EDIT:
I was mistaken in thinking the 2GB setting would influence the host instance size. This is not the case: ecs-cli up brings up the cluster and instance long before the docker-specified services are created by ecs-cli compose service up which reads those task params.

The task size parameter is per TASK which in Docker Compose terms are "services" (I think of them as containers plus exec params). So as I have four services, there will be four ECS tasks. To get the four tasks to fit inside a 1GB host therefore, task size should be 256MB (not 2GB!)

task_definition:
  task_size:
    cpu_limit: 512
    mem_limit: 0.5GB

This now gets further creating the services, without the memory error. I don't quite understand why (as 0.5GB x 4 is still too big).

So while this has avoided the error, I still want to know how to create a larger instance when performing ecs-cli up with Fargate.

The short answer is "you don't have to".

The long answer is I was doing it wrong. Here's what was wrong and what I learned.

After working through tutorials, RTFMing and re-building my ECS configuration I found that I actually wasn't launching via Fargate. I had made a mistake during the initial configure command, so the default config was using EC2 launch type. When making the cluster it created an Autoscaling Group template with t2.micro in it, and then it was using this to launch the initial instance. I didn't think this was unusual, as I read that Fargate managed your EC2's for you, which I presumed was what was happening. No, this was "traditional" EC2 creation via a scaling template.

I figured out the mistake by viewing the config file. I noticed ecs-cli doesn't have any "view config" commands - they only write to the config, so it feels a bit mysterious when you are issuing later commands, especially as you are encouraged to set lots of defaults (presumably to simplify the CLI commands which get long and cumbersome).

less ~/.ecs/config

version: v1
default: MY-TRIAL-CONFIG
clusters:
  MY-TRIAL-CONFIG:
    cluster: MY-TRIAL-CLUSTER
    region: ap-southeast-2
    default_launch_type: ""      <= this was the problem
  tutorial:
    cluster: tutorial
    region: ap-southeast-2
    default_launch_type: FARGATE

So I then tried the EC2-only parameter:

ecs-cli up --instance-type "t2.small" --capability-iam

and this changed the Autoscaling Group and indeed launched a t2.small. So I now know how to do it the EC2 way.

To do it properly as I had originally intended with Fargate, I needed default-launch-type FARGATE:

ecs-cli configure --cluster MY-TRIAL-CLUSTER --region ap-southeast-2 --default-launch-type FARGATE --config-name MY-TRIAL-CONFIG

Now the config looks like this:

less ~/.ecs/config

version: v1
default: MY-TRIAL-CONFIG
clusters:
  MY-TRIAL-CONFIG:
    cluster: MY-TRIAL-CLUSTER
    region: ap-southeast-2
    default_launch_type: FARGATE
  tutorial:
    cluster: tutorial
    region: ap-southeast-2
    default_launch_type: FARGATE

After doing this, ecs-cli compose service up works very differently from the observed ECS launch flow.

it doesn't create any Autoscaling stuff
it doesn't launch any EC2 instances (I thought Fargate did, but managed it for you, but you don't get to see the container hosts)
It created a Cloudformation Stack instead (interestingly with a t2.micro EcsInstanceType parameter)
It's very clearly labelled on the Cluster home page: there's a row for EC2 launch types below the FARGATE row.

e.g.

MY-TRIAL-CLUSTER >
FARGATE
1          0             1
Services   Running tasks Pending tasks

EC2
0        0             0                                              0
Services Running tasks Pending tasks CPUUtilization MemoryUtilization Container instances

It's very clear now I see it. There are no instances for Fargate services.

You can also watch Fargate re-start the task-containers if they fail, either in the control panel, or with ecs-cli compose service ps and ecs-cli logs --task-id XXX --follow, to get a feel for the magic and how the Docker Compose features are being performed.


Code coverage versus functional coverage We are working on a big Java project where we use JUnit tests. Our Jenkins helps us to monitor the code coverage and shows us the progress we make with this.

But there seems to be something lacking here. In our case, one of the apps results in a zip file with data files generated based on different source files.

All code to generate these data files is covered, but that doesn't tell us if the content of the generated files itself is OK. Is this just a matter of generating additional JUnit tests and rely on the knowledge of the team to know what needs to be tested? Or can this be controlled and monitored in some way?

The way I would approach this would be to have a second part of the junit test that reads the generated zip file and then verifies it is correct. You are right that if you are only ensuring you can generate the file, but the file is incorrect, the unit test is incomplete.


Is it possible to define a minimum code coverage using Pygradle? Pygradle is used to build and test a Python project. In gradle it is possible to define a minimum code coverage percentage:

jacocoTestCoverageVerification {
    violationRules {
        rule {
            limit {
                minimum = 0.5
            }
        }

        rule {
            enabled = false
            element = 'CLASS'
            includes = ['org.gradle.*']

            limit {
                counter = 'LINE'
                value = 'TOTALCOUNT'
                maximum = 0.3
            }
        }
    }
}

If this limit is not reached, build will fail. Is it possible to define a minimum code coverage using Pygradle?

When the above snippet is run, the build fails and the following is returned:

FAILURE: Build failed with an exception.

* Where:
Build file '/home/user/SomeProject/build.gradle' line: 14

* What went wrong:
A problem occurred evaluating root project 'SomeProject'.
> Could not find method jacocoTestCoverageVerification() for arguments [build_fsdafasdfsa] on root project 'SomeProject' of type org.gradle.api.Project.

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 2s

At the moment it looks like it is not possible to define a minimum code coverage as the ticket that was created is still open https://github.com/linkedin/pygradle/issues/184


What methods are available for connecting many-to-many containers? I have two containers (Docker):

TensorFlow container
NodeJS container used as a frontend to the TensorFlow container.

Both containers have TCP listeners on a port specified using environment variables.

If I want to scale the NodeJS and TensorFlow containers, for example 2 instances of NodeJS and 4 instances of TensorFlow, what are my options for connecting them to evenly spread the load across the 4 instances of TensorFlow?

Currently I'm using the TensorFlow container name inside the NodeJS container, but I'm pretty sure that would only take advantage of a single TensorFlow container... Or??

The classic answer to balancing load is, of course, a load balancer. There are many different methods of doing this, but we're interested in ones that work well with dynamic containers.

This is one of the things an orchestration layer will address. In Kubernetes, for instance, you would create a Tensor Flow service, and scale it to however many pods you need (or use autoscaling!). Kubernetes then provides an ip for the service that kube-proxy load balances across the pods.

There are also service mesh options that replace this layer and perform a similar function.


Azure DevOps deploy to Azure VM's without a public IP Is it possible to deploy from Azure DevOps to an Azure VM that only has a private IP address? I need to deploy to VM's that won't always have a public IP address.
I thought there might have been some internal Azure network that could be used but running the deployment agent script on my VM without a public IP fails (it works if the machine does have a public IP). Other than that can Azure DevOps connect to a Azure VPN (Virtual Network Gateway) to access the VM's on my Virtual Network?

Yes it is possible. I was working on something like that a few days back. You can use Self-Hosted agents directly on one or all of the VMs: https://docs.microsoft.com/en-us/azure/devops/pipelines/agents/v2-windows?view=azure-devops