Will Docker Swarm work if masters are spread over 3 continents? I need to deploy a highly available Docker Swarm on Azure. I created a VNet in 3 locations: Brazil South (my location), East US and West Europe. Then I enabled network peering so VMs in these locations can see each other as if they were in the same VNet. Cluster seems working fine. The problem is that turns out latency is high between these locations: ~190ms.

Questions:

Is this a recommended topology for HA? Is there a way to improve it?
Will this latency pose any (future) limitations?

There are various timeouts within swarm mode that do not handle high latency well and may start to assume nodes within the cluster are down, lose quorum, or have networking issues. For HA, the standard advice is to have multi-AZ but not multi-region for a single cluster. For use cases that need multi-region, you would deploy multiple swarm clusters, one per region. If you really need multi-region with a single swarm cluster, then consider placing all of the management nodes within the same region and only deploying worker nodes to the other regions.


Jenkins node (agent) on shared web server - what are the right permissions I have a shared web server with multiple sites and each site has a dedicated user used as the owner and with the group apache.

I want to install Jenkins node on the server, but how will it be able to change files and run commands like git pull, I think that using runuser will miss the point.

The script on the node will basically run git pull, drush commands (Drupal sites), rsync from remote server and more.
I will also need to run chmod and chown at the end, but maybe for those commands I will just use the sudoers file (?).

Thanks.

A not very devopsy, but quite simple and effective solution with no security implications (no credential exchange or authentication required) would be to establish a file-based "message" exchange scheme, in a "mailbox" - a well-known filesystem location (setup once and owned by root) with 2 directories:

one owned by the jenkins user which creates request files containing deployment request information, one for each site
one owned by the apache group where each site dedicated users create their own response files containing request handling information for deployment requests for their site

When jenkins processing reaches a deployment stage for a particular site it creates a corresponding request file with the necessary information in it.

Each site user periodically (cron-driven, for example) checks for request files pertaining to their site, handle the request as appropriate, following their own site policies and provide status updates in the respective response files, which the jenkins user checks periodically.

When a request handling is completed the jenkins user removes the request file, signalling that it received the "message" and then the site user periodical job can remove the corresponding response file.

The names of the request and response files can be used to encode the particular site and request identification, so that the periodical checks don't have to fumble through multiple files.

The scheme can easily work across machines (if, for example, some of the sites are migrated to other servers) simply by placing the "mailbox" on a shared filesystem accessible from all those machines.

OK, an example, as requested. Just a basic skeleton, in python, hopefully self-documenting.

Prerequisites:

sudo mkdir -p /var/message_box/requests
sudo chown jenkins /var/message_box/requests
sudo chmod go-w /var/message_box/requests
sudo mkdir /var/message_box/responses
sudo chgrp apache /var/message_box/responses
sudo chmod g+w /var/message_box/responses

The mailbox.py file:

#!/usr/bin/python2.7 -u

import logging, os, re, getpass, sys, time, yaml

class Mailbox(object):

    base_dir = '/var/message_box'
    request_filename_format = '%s.%s.yaml'  # username.id.yaml

    def __init__(self):
        pass

    @property
    def request_dir(self):
        return os.path.join(self.base_dir, 'requests')

    @property
    def response_dir(self):
        return os.path.join(self.base_dir, 'responses')

    def msg_filename(self, user, request_id):
        return self.request_filename_format % (user, request_id)

    def request_file(self, user, request_id):
        return os.path.join(self.request_dir, self.msg_filename(user, request_id))

    def response_file(self, user, request_id):
        return os.path.join(self.response_dir, self.msg_filename(user, request_id))

    def create_msg_file(self, user, request_id, data, is_response=False):
        assert user and request_id and data and isinstance(data, dict)
        msg_file = self.response_file(user, request_id) if is_response else \
                   self.request_file(user, request_id)
        with open(msg_file, 'w') as fd:
            fd.write(yaml.dump(data))

    def msg_file_data(self, user, request_id, is_response=False):
        msg_file = self.response_file(user, request_id) if is_response else \
                   self.request_file(user, request_id)
        if os.path.exists(msg_file):
            with open(msg_file) as fd:
                data = yaml.load(fd.read())
            if data and isinstance(data, dict):  # expected data format
                return data
        return None

    def create_request(self, user, request_id, data):
        self.create_msg_file(user, request_id, data)
        logging.info('created request %s for %s' % (request_id, user))

    def create_response(self, request_id, status, response_data=None):
        assert status
        user = getpass.getuser()
        self.create_msg_file(user, request_id, {'status': status, 'data': response_data}, is_response=True)
        logging.info('created response %s with status %s for %s' % (request_id, status, user))

    def handle_requests(self):
        user = getpass.getuser()
        while True:  # keep handling requests indefinitely
            time.sleep(1)  # new request polling rate, in seconds
            for filename in os.listdir(self.request_dir):
                m = re.match('(.*)\.(.*)\.yaml', filename)
                if not m:  # not a valid request filename
                    continue
                [username, request_id] = m.groups()
                if username != user:  # not a request for this user
                    continue
                if os.path.exists(self.response_file(user, request_id)):
                    # request handling already started
                    # you may add here recovery code for request handling interrupted for whatever reason
                    continue
                msg_data = self.msg_file_data(user, request_id)
                if not msg_data:  # unexpected data format
                    continue

                logging.info('received request %s: %s' % (request_id, msg_data))

                # mark the request handling start
                self.create_response(request_id, 'in_progress')

                time.sleep(5)  # mock-up, replace with whatever request handling means

                # mark the request handling done
                self.create_response(request_id, 'done')  # you can add response data to the dict if needed

                logging.info('handled request %s, waiting for confirmation' % request_id)

                while True:  # wait for confirmation receipt before cleaning up
                    time.sleep(1)  # confirmation receipt polling rate, in seconds
                    if not os.path.exists(self.request_file(user, request_id)):
                        # the deletion of the request file is the confirmation receipt
                        logging.info('confirmation for request %s received, cleaning up' % request_id)
                        os.unlink(self.response_file(user, request_id))  # cleanup response file
                        break

    def execute_deployment(self, deployment_id, deployment_user, deployment_data):

        self.create_request(deployment_user, deployment_id, deployment_data)
        started = False
        while True:  # wait until it's done
            time.sleep(1)  # polling rate, in seconds
            msg_data = self.msg_file_data(deployment_user, deployment_id, is_response=True)
            if msg_data:
                status = msg_data.get('status')
                if status:
                    if not started:
                        logging.info('request %s handling started' % deployment_id)
                        started = True
                    if status == 'done':  # job completed
                        logging.info('request %s handling completed, cleaning up' % deployment_id)
                        # cleanup request file, which is the confirmation receipt
                        os.unlink(self.request_file(deployment_user, deployment_id))
                        break

def usage(err_msg, option_parser):
    if err_msg:
        logging.error('%s\n\n%s\n' % (err_msg, option_parser.format_help()))
    sys.exit(-1)

if __name__ == "__main__":
    import optparse

    logging.basicConfig(level=logging.DEBUG, format="%(levelname)5s  %(asctime)s %(filename)s:%(lineno)d] %(message)s")
    os.umask(022)

    p = optparse.OptionParser()
    p.add_option('-c', '--command', action='store', dest='command', choices=['deploy', 'handler'],
                 help='command/mode: <deploy|handler>, mandatory', default=None)
    p.add_option('-i', '--ID', action='store', dest='id',
                 help='deployment ID, mandatory for deploy command', default=None)
    p.add_option('-u', '--user', action='store', dest='user',
                 help='deployment user, mandatory for deploy command', default=None)
    p.add_option('-a', '--artifact', action='store', dest='artifact',
                 help='deployment artifact, mandatory for deploy command', default=None)

    opts, _ = p.parse_args()

    if not opts.command:
        usage('command is mandatory', p)

    mailbox = Mailbox()

    if opts.command == 'deploy':
        if not opts.id or not opts.user or not opts.artifact:
            usage('ID and user must be specified for deploy command', p)
        data = {'artifact': opts.artifact}
        mailbox.execute_deployment(opts.id, opts.user, data)

    elif opts.command == 'handler':
        mailbox.handle_requests()

The jenkins user driving the deployments, deployment info hacked to a string for this example - 'artifact' :

$ ./mailbox.py -c deploy -i 20 -u dancorn -a artifact
 INFO  2017-10-17 13:32:34,663 mailbox.py:49] created request 20 for dancorn
 INFO  2017-10-17 13:32:35,666 mailbox.py:109] request 20 handling started
 INFO  2017-10-17 13:32:40,678 mailbox.py:112] request 20 handling completed, cleaning up
$ ./mailbox.py -c deploy -i 123 -u dancorn -a artifact
 INFO  2017-10-17 13:33:32,359 mailbox.py:49] created request 123 for dancorn
 INFO  2017-10-17 13:33:33,362 mailbox.py:109] request 123 handling started
 INFO  2017-10-17 13:33:38,375 mailbox.py:112] request 123 handling completed, cleaning up
$

The apache users would launch the handler which in this example remains running (could be converted to a daemon, or a cron-driven approach):

$ ./mailbox.py -c handler
 INFO  2017-10-17 13:32:34,819 mailbox.py:77] received request 20: {'artifact': 'artifact'}
 INFO  2017-10-17 13:32:34,821 mailbox.py:55] created response 20 with status in_progress for dancorn
 INFO  2017-10-17 13:32:39,827 mailbox.py:55] created response 20 with status done for dancorn
 INFO  2017-10-17 13:32:39,827 mailbox.py:87] handled request 20, waiting for confirmation
 INFO  2017-10-17 13:32:40,828 mailbox.py:93] confirmation for request 20 received, cleaning up
 INFO  2017-10-17 13:33:32,888 mailbox.py:77] received request 123: {'artifact': 'artifact'}
 INFO  2017-10-17 13:33:32,889 mailbox.py:55] created response 123 with status in_progress for dancorn
 INFO  2017-10-17 13:33:37,891 mailbox.py:55] created response 123 with status done for dancorn
 INFO  2017-10-17 13:33:37,891 mailbox.py:87] handled request 123, waiting for confirmation
 INFO  2017-10-17 13:33:38,893 mailbox.py:93] confirmation for request 123 received, cleaning up


I want to scale up/down as fast as AWS Lambda but also be able to allocate the vCPU (minimum 8) per task, any advice? Goal

So I'm looking for an AWS product or combination of products to accomplish the following:

Ability to do CPU heavy (non-parallel) calculations (min 8 vCPU's per node)
Minimum timeout limit of 1800 seconds (AWS Lambda has a limit of 900 seconds)
Ability to automatically scale up and down to 0/1
Ability to scale up really fast (<30 seconds)
Event-driven execution model (one task per node, node gets destroyed after the job is finished)
Assign the desired vCPU's per "type" (3 predefined types) event/task/job

I'll try to be as detailed as possible in my description of what the job is, the two setups I've tried and what I don't like about them.

Definitions

ROE: Route Optimisation Engine
VRP: Vehicle Routing Problem

A single JSON containing all the information regarding the stops, vehicles, time windows, start/end addresses) which needs to be optimised. VRP's can be classified into three different complexities [easy, medium, hard] based on:

number of stops
restrictions per stop (time window, capacity)
number of vehicles
restrictions per vehicle (time window, maximum range, capacity, breaks)

Easy classified VRP's will need less CPU resources and can be solved quicker than for example high classified VRP's.
Solution: The best solution for the VRP (the most efficient routes for the VRP)

Setup 1 -  AWS Lambda

Diagram

AWS SQS

Standard type AWS SQS queue for optimisation request messages. Every message that enters triggers the ROE (AWS Lambda).

AWS Lambda

The Lambda (3008 MB memory) is triggered by the AWS SQS queue for optimisation request messages and processes alles messages when they are added to the queue. The maximum timeout limit of all AWS Lambda's is 15 minutes.

Problems

AWS Lambda does not allow us to select more CPU resources and therefore optimisations of medium complexity VRP's take a long time.
AWS Lambda maximum timeout settings of 15 minutes makes the usage of a Lambda for doing optimising high complexity VRP impossible

Setup 2 - AWS Batch

Diagram

Steps

WA sends optimisation request to API
API creates a VRP and stores the VRP in S3
API evaluates the complexity of the VRP (low, medium or high)
API creates an AWS Batch Job (with the parameter problemId=123456789)
API allocates the correct number of vCPU's and Memory (predefined) to that job based on the VRP complexity
API adds the AWS Batch Job to the correct AWS Batch queue (the queue is defined by the VRP complexity)
API returns a 200 - OK response to the WA
AWS Batch Job is taken from the AWS Batch Queue for execution
AWS Batch Job fetches the VRP from S3 based on the problemId it received during step 4
AWS Batch Job solves the VRP
AWS Batch Job stores the solution in S3
WA requests the solution from the API
API gets the solution from S3 and returns it to WA

Problems

Scaling up takes unacceptably long
Going from 0 to 1 node takes too long (300+ seconds)
Going from 1 to X nodes takes too long (60/300+ seconds)

You can check out AWS Fargate, I've just used it for a similar use case.

Setup:

Jobs are posted to an SQS queue where a scheduler lambda picks them up and launches an AWS Fargate task.

Caveats:

Lambda handles the SQS messages so you can't hand over the messages to Fargate (Lambda will auto-delete the messages from the queue once the lambda returns successfully), instead you'll need to handle retries via a separate lambda that gets notified whenever a task ends.
Fargate only supports up to 4 vCPUs
You need to pass any data to the container either via environment variables or as command line arguments

Launch times are not as fast as lambda, but much better than Batch and EC2 - I typically get my containers within 30 seconds of the lambda launching them.

Dead-letter/retry handling is a bit of a nuisance, but it's not that hard to implement as long as your process is well-behaved. You can easily get the process exit status from the API once the task has finished and you can store the number of retries as a tag on the task since you'll be launching a new task for each retry.

The scheduler and retry handler should be fairly trivial to write and if you need to pass larger amounts of data you can store it on S3 and send the location as an argument to your container.

A similar option - if you don't mind paying for the extra, unused capacity is to use ECS instead of Fargate and have unused capacity in the cluster so you can spawn new containers more or less instantly. This removes the limit on vCPU but you'll instead need to manage the cluster size (can be done via Autoscaling) and pay for the unused capacity.


Bad edit to sudoers file on EC2 instance, how to fix it? While writing a script to edit the sudoers file without using visudo, the script did not check for issue, and I have introduced some bad edits into the sudoers file, and thus cannot use sudo at all anymore. (I know I did a very bad thing, and should probably have used something like this)

Question: Is there anyway I can recover from this mistake, or do I have to destroy the VM and start from scratch with a new one?

How about user-data ?

I believe adding the "#cloud-boothook" allow to force the user-data to run at every restart.

#cloud-boothook
#!/bin/bash
echo 'test' > /home/ec2-user/user-script-output.txt

If so, you could fix your sudo scripts hopefully... or install / add AWS run commands configuration https://aws.amazon.com/blogs/aws/new-ec2-run-command-remote-instance-management-at-scale/

For sure if it is an EBS Drive, you can detach & fix and then re-attach on the original system. A bit of a pain though.