How to handle AWS Offsite or cross-account RDS (mysql) backups? AWS Provides a great solution to provide point-in-time recovery backups within RDS. However, while we take every measure to protect the root account, I'm paranoid that if someone were to gain access they can wipe the entire AWS account including all backups and therefore take the whole company down. What are the best ways to handle this? Do you have 100% trust in your root account and policies to prevent someone malicious from wiping the database and all backups?

What are the best ways to handle an offsite backup to a different RDS account? Should we simply execute mysqldump periodically from a separate system, or is there a better way?

On top of protecting your root account as best as you can, you should be able to reduce the risk of such resource deletion if you have more than 1 resource to protect.

For example if all AWS resources you had to protect were 2 RDS database servers, you could create 2 other accounts that have access to only copy the snapshots of one of your database servers each - and no other access to any other resource. This setup would then require an intruder to hack both root accounts of your main and one backup account each, which should be significantly more difficult.

To copy snapshots from one account A (your current RDS account) to account B (your new account) you should need the following parts  (untested)

Create a new IAM role in Account A, for access from account B:

Create a new policy for reading RDS snapshots from account A

{
    "Version": "2012-10-17",
    "Statement": [
        {
        "Effect": "Allow",
        "Action":
            [
                "rds:CopyDbSnapshot",
                "rds:DescribeDbSnapshots"

            ],
        "Resource": "arn:aws:rds:::"
        }
   ]
}

and assign it to the IAM role

Check that the "trust policy" is enabled as described in this walk through for another cross account action: https://aws.amazon.com/blogs/database/setting-up-for-cross-account-native-backup-and-restore-in-amazon-rds-for-microsoft-sql-server/
Then in account B, you should be able to run copy-db-snapshot from account B: https://docs.aws.amazon.com/cli/latest/reference/rds/copy-db-snapshot.html
To automate, I prefer to use Lambda functions, which can simply call AWS API calls, you can trigger them with a Cloudwatch Event


How does one set up OpenStack Swift (object) storage container-specific API keys with Ansible? I've got the following OpenStack commands for setting up Swift object storage containers:

openstack container create $CONTAINER_NAME
openstack container set --property Full-Key=$CONTAINER_API_KEY $CONTAINER_NAME

I'd like to convert them to Ansible so they can be run as part of a role.

For the first command, I know I can use the os_object module for creating the container, but it doesn't appear to provide a parameter for setting a property, which in this case is for setting up key access to the container.  (It's based on the documentation at Container Specific API Keys.)

So how would I execute the second command within Ansible natively, without resorting to running the shell command directly?

After further research, it's evident that it's currently impossible to do this natively.  Therefore, we have to do this in the shell with the command module:

command: openstack container set --property Full-Key={{ KEY }} {{ CONTAINER }}


How long is it appropriate to keep old feature branches in git? I am on a large team with many developers. We have a large codebase that has many feature branches from our large group of developers. We are accumulating many feature branches.

We generally look to git-flow for best practices.

How long should feature branches last?

Is it safe enough and not costly or degrading to performance to keep all of them?

Are there considerations and maintenance commands that developers need to execute every so often to have working, efficient, and responsive git ?

Thanks.

Feature branches should stay until the feature is done, once done the branch must be closed/deleted. Establishing this kind of behavior should be a team goal and usually takes some time, it's just easier to forget about them in the hurry up I'm ready with my pull request mode.
Keeping all of them makes maintainability harder. You might think that each branch follows a naming convention with a map to a JIRA ticket or whatever kind of mapping you have and it's really straight forward to distinguish them, but that's not the case once you have a dozen of them and even more are constantly piling up. I highly doubt performance has anything to do, it's more like a confusion cost switch.
Regarding the commands, I do not think so.


What is the performance different running docker on Windows versus Linux? We are looking at containerizing some of our applications and servers, Redhat servers to be more specific and there is an idea to run Docker on windows instead of Linux so the end user has an easier time operating the system. First though I had was what is the performance difference between using Docker in Windows vs in Linux?

The performance difference between linux containers running on Windows, Linux, or Mac will depend entirely upon how much ram you allocate to the Docker Daemon.  The difference should not be noticeable.  Additionally, you will be able to take advantage of the Alpine base image (~5 Mb) for small, compact containers.

If you need to run IIS in your container, then you should look at windows containers.  The disadvantage is that the containers will be significantly larger in size, and you will lose cross-platform compatibility.  Docker windows containers.


How to transition from a complex branching reality to a single-branch model? In large organisations, using the waterfall methodology typically results in very complex branching structures (aka branch spagetti).

What branching strategies can be used to transition from a complex branching reality to a single-branch model like trunk-based development?

Update:

To clarify, the question is about the migration/transition itself, not about the before and the after methodologies, which are pretty clear.

It can't really be "at EOB today we're still waterfall with gazillion of branches but tomorrow 1st thing we'll switchover to trunk-based, single-branch CI".

Because you mention waterfall, I understand that the numerous branches you are alluding to are feature-branches rather than maintenance-branches.

In this setup, I also assume that these branches are created according to a waterfall plan that tries to minimise conflicts.  This implies that the goal of the development is to produce several distinct products.  When using a single-branch development model, it is important to also work on a single product. If several products are simultaneously developed in a single-branch development model, it effectively “glues” together versions of theses products, so that we might have in version a of the repository a healthy product X and a buggy product Y, while in version b the product X has a regression and Y a bug fix, but we have no version where both X and Y are healthy.  Such a situation would force us to consider X and Y as being developed in distinct repositories, which is a hint that they should be.

Therefore, the first steps should perform a repository split:

Arrange the repository so that it is easy to split it into several small repositories. For instance, rearrange the current repository so that each top-level directory corresponds to a repository you want to create in the future. Doing so, you can continue to use the branch-spaghetti discipline everybody is familiar with.
When step 1 is completed, refine the branch-spaghetti discipline by requiring that any single branch can only touch files in one top-level directory.
When each branch complies with step 2, perform the split. Developers can easily convert their pending changes to patch a single repository, just by removing the first-level of the path.

Now that the split has been performed, you can start working on the branch discipline itself.

Introduce programming techniques helping the development of short-lived branches. Branches being short-lived is a crucial aspect of all single-branch methodologies. One of their goals is to reduce the time spent on merging and debugging long-lived branches. A popular technique is the introduction of “feature-flags” where a “factory” uses a configuration flag to either produce the historic version of an object or the new, initially partially developed, version of that object.
By now, you have zillions of repositories with only a few branches in each, and you can turn the “we globally adopt the trunk-based development discipline” button, without seeing the original branch-spaghetti mountain collapsing on the trunk.

The actual split of repositories might be optional, but you would then have to adopt policies that cleanly delineate the allowed scope of each submitted patch (to limit the risk of conflicts when merging changes in the main branch). Reducing the overhead bound to conflicts is one of the goals of single-branch model methodologies, so I assume this is relevant in your context.


How do I run opendistro elasticsearch in docker on HTTP I'm trying to run the docker image of the AWS opendistro elasticsearch on my laptop for tesing
https://opendistro.github.io/for-elasticsearch-docs/docs/install/docker/
When I run it, it expects an HTTPS connection. How do I run it on HTTP?
I'm using this command to run:
docker run -d --name elasticsearch-od -p 9200:9200 -p 9600:9600 -e "discovery.type=single-node"

in your docker-compose.yml you need change this parametres:
environment:
ELASTICSEARCH_URL: https://odfe-node1:9200
ELASTICSEARCH_HOSTS: https://odfe-node1:9200
For this:
environment:
ELASTICSEARCH_URL: http://odfe-node1:9200
ELASTICSEARCH_HOSTS: http://odfe-node1:9200
And after in your file config for elasticsearch put this:
https-enabled = false
Try and enjoy ;)


What are the two project layouts offered by Rancher on installation? During login after initial setup, Rancher CLI asks the following:

Do you want to continue connecting (yes/no)? yes
NUMBER    CLUSTER NAME                          PROJECT ID        PROJECT NAME   PROJECT DESCRIPTION
1         xxx                                   c-5tcls:p-b4r55   System         System project created for the cluster
2         xxx                                   c-5tcls:p-tmx85   Default        Default project created for the cluster

What is the difference and is there a way to make this choice non-interactively in a headless environment?

Note. For some reason, 1 and 2 get randomly swapped from time to time we run this.

Project is a Rancher specific concept. A Project can have multiple Kubernetes namespaces.

The System Project has all the Kubernetes system components like kube-dns, ingress etc. Think of it like System Services running on a machine/laptop.

Default project is where you can launch your workloads to start with. Similar to a scratchpad.

You can create new projects too. For example, Web App, Database, etc. Or it could be based on different users using the cluster. Project-For-Bob, Project-For-John.


Can't access startfile http://localhost/ Having created a docker container as:
docker run -d --name=jetty  jetty

How do I now get a hello world page as?
Looking up localhost:8080
Making HTTP connection to localhost:8080
Alert!: Unable to connect to remote host.

lynx: Can't access startfile http://localhost:8080/

and:
Looking up localhost
Making HTTP connection to localhost
Alert!: Unable to connect to remote host.

lynx: Can't access startfile http://localhost/

containers:
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS               NAMES
5379d3f41d20        jetty               "/docker-entrypoint.…"   4 seconds ago       Up 2 seconds        8080/tcp            jetty

created with:
docker run -d -p 80:8080 -p 443:8443 jetty

gives a start page as:
Error 404 - Not Found.

   No context on this server matched or handled this request.

   Contexts known to this server are:

   Context Path Display Name Status LifeCycle
     __________________________________________________________________

   [1]icon  [2]Powered by Eclipse Jetty:// Server
     __________________________________________________________________

References

   1. http://eclipse.org/jetty
   2. http://eclipse.org/jetty


Jenkins and Kubernetes Design Involving GitHub I am new to DevOps and am currently designing a CI/CD pipeline that involves integrating GitHub webhooks with Jenkins in a Kubernetes environment, and I would like to see whether this is feasible or not.
Currently, I have a GitHub repository setup with some code. In Kubernetes, we have a volume setup where we plan to clone said repository into this volume. This repository will need to be up-to-date at all times, but we shouldn't manually update this repository every time a change is made. With Jenkins, we want to setup webhooks in GitHub so any time changes made in the code repository will be updated in the volume.
I am still new to the idea of webhooks and Jenkins will interact in a Kubernetes environment, specifically a volume, so any thoughts and comments would be appreciated.
Thanks in advance!

I would use the webhooks GitHub provides if you don't need something specific from Jenkins (Jenkins is in many cases a consumer of GitHub events/webhooks).
https://docs.github.com/en/developers/webhooks-and-events/webhook-events-and-payloads#push
Using this webhook you could inspect each payload for specific parameters (ref, repo, etc...) and do something (like pull) if those parameters are met. In either case, you will need a service that can process the webhook/payload and perform the work/automation.


How to programmatically enable and disable SQS triggering AWS Lambda? I have a requirement where I want to stop my lambda from receiving messages from SQS. And I want to implement this like an API which can toggle between the disabling and enabling of the AWS Lambda getting triggered by SQS. I tried looking for options but there were suggestions telling to invoke the permission which I don't find feasible.

Call Lambda's UpdateEventSourceMapping API with the {"Enabled":true/false} request body.
If, for some reason, you want your own API for this, you can create one in API Gateway as described below.
Before you begin, you need the UUID of the event source mapping you wish to enable/disable. Use the AWS CLI to get this (it isn’t shown on the UI):
aws lambda list-event-source-mappings --function-name my-func

{
    "EventSourceMappings": [
        {
            "UUID": "b962dcee-173c-47r1-9600-5c6d81catf85",
            "BatchSize": 1,
            "EventSourceArn": "arn:aws:sqs:us-east-1:123456789012:my-queue",
            "FunctionArn": "arn:aws:lambda:us-east-1:123456789012:function:my-func",
            "LastModified": 1595058588.141,
            "State": "Enabled",
            "StateTransitionReason": "USER_INITIATED"
        }
    ]
}

Next, either create a new API in API Gateway or add a resource & a GET method to your existing API. Configure it as shown below. The UUID you got above is highlighted below:

The execution role shown above must allow API Gateway to invoke Lambda APIs.
Scroll further down & provide this mapping template:
Content-Type: application/json

{
    "Enabled" : $input.params("Enabled")
}

Save & deploy to a stage. Now you can hit this API with the query param Enabled=true / Enabled=false to enable/disable your Lambda's SQS trigger. Here's a test done from the API Gateway console:

For a detailed screenshot-guided walkthrough of this entire process, see my blog post.


How to Allow AWS S3 Access From only Specific IPs Recently we were testing with AWS VPC, and a requirement for our project was that we needed to allow nodes within a VPC access to S3 buckets, but deny access from any other IP address.

Is there anyway to Allow AWS S3 Access From only Specific IPs?

please attach below policy to your AWS S3 Bucket.

  Allow Access to Specific IP Addresses

 <div class="code">
 {
   "Id": "S3PolicyId1",
   "Statement": [
     {
       "Sid": "IPDeny",
       "Effect": "Deny",
       "Principal": {
         "AWS": "*"
       },
       "Action": "s3:*",
       "Resource": "arn:aws:s3:::bucket/*",
       "Condition": {
         "IpAddress": {
           "aws:SourceIp": "54.240.143.188/32"
         }
       }
     }
   ]
 }
 </div>

  Deny Access to Specific IP Addresses

{
  "Version": "2012-10-17",
  "Id": "S3PolicyId1",
  "Statement": [
   {
    "Sid": "IPAllow",
    "Effect": "Allow",
    "Principal": "*",
    "Action": "s3:*",
    "Resource": "arn:aws:s3:::bucket/*",
    "Condition": {
      "NotIpAddress": {"aws:SourceIp": "54.240.143.188/32"}
    }
   }
  ]
 }


Auto build and upload to AWS new docker image after commit in repo I'm new at DevOps, and I looking for solution to my problem.
I have a private code repository where I have a some Flask apps in one of many folders. Example of structure:
- Project
-- Django Apps
--- django1_dir
--- django2_dir
--- django3_dir
-- Flask Apps
--- flaskApp_dir1 (dockerfile inside)
--- flaskApp_dir2 (dockerfile inside)
--- flaskApp_dir3 (dockerfile inside)

And I have a configured cluster on kubernetes on AWS instance. And a private docker repo. When I want to load new version of my flaskApp to cluster I need:

Someone modify flaskApp on repo
I need pull repo
Locally build a new image based on new dockerfile
Push app to repo
Manually push docker image with new APPVersion to AWS trough ecr
Then kubernetes change image to new version

My question is:
how can I automate this process? I have an script to push dockerimage to AWS through ecr.
It is possible to create some kind of trigger that would work in such a way:

Someone do a commit
Script building a new docker image and push it to AWS automaticly

I read something about it, but as I said I'm totaly new in DevOps so any hint will be awesome.
Now I have configured kubernetes, that aws instances and repo for docker images. I read about git hooks, flux and some other solutions but it's quite unclear for me so I looking for best practice for that problem. I wouldn't particularly want to install huge new tools either.
So maybe any hits or keywords I could get hooked on?

how can I automate this process? I have an script to push dockerimage to AWS through ecr.

You are on the right track. Yes, this should be an automated process from Git push or when someone merge a Pull Request.
An important aspect of this is that this "automated process" should be on a dedicated system - such that it is repeatable in a consistent way. Typically this is called a "Continuous Integration Server" or just "Build Server".
Some popular choices for this kind of process:

Jenkins - old but popular
Tekton - Kubernetes native - but still in development
GitHub Actions - probably most interesting if your Git repo is on GitHub
Bitbucket Pipelines - probably most interesting if your Git repo is Bitbucket
AWS CodeBuild - limited but may be interesting if you use AWS
GitLab CI - probably most interesting if your Git repo is on GitLab

What you need to do is to configure your Git repository system (e.g. GitHub, Bitbucket, Gitlab or AWS CodeCommit) to initiate Pipelines on the Build system (a product of those listed above)


How to analyze the file activities due to process? I need to analyze the file i/o operations caused due to a process.
So I would like to know, which all files are affected by this program. Is there a tool for this particular use case?.

One of our debugging for a slow URL processing got the culprit being the file being read each time a URL was processed by the system. So I would like to get recommendations for a tool which would monitor the file activities per process.
Any suggestions?

You can use lsof with the -p flag, like this : lsof -p $PID .


Should I clean up my local machine before installing Docker? This is a basic question for which I didn’t see anything on in the docs or forum searches.

I have a development machine which has all kinds of stuff installed to do machine learning, like anaconda, python, etc. I want to containerize all of my development using Docker. Would it be a good practice to uninstall everything of importance on the host machine, since everything is going to be running on containers? I guess the main question I have is whether my docker build can/will use any of these ML from my local host machine when building the container.

Ultimately, as long as you have the disk space and RAM, you can run both in parallel with very few issues.

The biggest problem you may run into is that the default ports of the applications you run in a container may already be in use.  For example SQL server install runs on port 1433 by default.  If you try to run a SQL server container and do not map a different port on the host machine to 1433 on the container you will run into errors.  This is easy enough do though:

docker run -e "ACCEPT_EULA=Y" -e "SA_PASSWORD=" -p 1431:1433 -d mcr.microsoft.com/mssql/server:2019-CU3-ubuntu-18.04

In the above example, MSSQL server will run on port 1431 and map to port 1433 in the container.  On your local machine you can access either your SQL server on port 1433 or the container SQL server on 1431.


How to increase the size of a disk on a running instance on Amazon EC2 I got a running instance of a EC 2 server setup with Ubunto. What's the best way to increase the disk size without any downtime and minimal risk?

Reading through the guides, one way would be to create a new disk, migrate data, turn off the instance swap disk and turn it back on. This approach sounds a bit risk will require some downtime. I wonder if there is a better approach?

Amazon AWS just released (on 13th February 2017) a new feature that allows to change a size of an EBS volume.

source: https://aws.amazon.com/blogs/aws/amazon-ebs-update-new-elastic-volumes-change-everything/

This allows to increase the size of an EBS volume on an existing instance, while it is running.

It is important to note that changing the volume size, does not change the size of the filesystem on the volume (for most filesystems). Additional steps might be required (depending on filesystem) in the operating system itself. For example a resizefs on ext4 filesystem used by most Linux OS today.

Full documentation from AWS describing the process can be found at http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-expand-volume.html

In some cases the block device has mapped partitions, and only then one (or more) of the partitions include a filesystem. In this case the partition will need to be resized first, and only then the filesystem. This process is described in the documentation as well.

The new "online" resize feature described in the blog only applies to current generation instances, and there are some other considerations and limitations that need to be checked before attempting a volume resize.


What is a Notes Day and can it be used in DevOps transformation? Lately I am hearing about more and more companies adopting the Pixar concept of a Notes Day and I was wondering what exactly is a Notes Day, what would be a good format for such event and mainly if it could be successfully used in a DevOps inspired transformation of a company?

I see some interesting parallels in this story and The Phoenix Project.  (Spoilers ahead for the book, if you haven't read it read this anyway :))

I take it the title for "Notes Day" comes from the term of art in Hollywood, where "notes" are constructive criticism about a work.  In this story, Catmull plays the part of Parts Unlimited CEO Steve Masters.  Catmull accepts that he can learn from his employees how to do a better job, as Masters does in Phoenix Project when he does his public mea culpa and apologizes to Bill Palmer.

In a sense, Catmull is also Brent- he's a major constraint.  If his time was so precious, it meant that Pixar was struggling to delegate authority.  It seems  Catmull was critical to every decision, rather than able to set large-scale goals and leave it to employees to figure out how to achieve them.

An event like this might work well at some companies, less well in others.  I think it's the key idea behind Notes Day that is most valuable: accountability and transparency from the highest levels of management to the last employee.  That value can be spread throughout a company in a variety of big and small ways.


Consul Helm chart: Connect Injector service not working Below is my config that I am applying over the latest version of the chart (0.4.0).

The consul client daemonset pods come up and successfully register with the cluster.
The connect injector webhook deployment comes up and shows as running successfully. This the is pod startup log:

  Listening on ":8080"... Updated certificate bundle received. Updating
  certs...

  10.42.0.1 - - [18/Dec/2018:20:36:26 +0000] "GET /health/ready HTTP/2.0" 204 0

From then on, I tried some sample deployment using the annotation: "consul.hashicorp.com/connect-inject": "true"
This did not spin up any sidecar containers. Checking the injector pod, there were no messages that the deployment was even contacted.

Initially I had issues with this in a cluster without RBAC. I removed the RBAC references from the chart and deployed; but had the issue above.
I then setup/enabled RBAC on another cluster and the chart installed correctly. This did not work either even with the correct RBAC settings in place.

I am using Vault as a certificate provider for Consul; checking both logs, I don't see any errors, and Consul is setup correctly as the provider.

Does anybody have any ideas on what to check?

This is my config:

global:
  enabled: false
  domain: consul
  image: "consul:1.4.0"
  imageK8S: "hashicorp/consul-k8s:0.3.0"
  datacenter: dc1

client:
  enabled: true
  join: ["provider=aws region=region tag_key=tag tag_value=tag"]
  grpc: true
  extraConfig: |
    {
      "enable_local_script_checks": true,
      "encrypt": "key"
    }

dns:
  enabled: true

syncCatalog:
  enabled: false

connectInject:
  enabled: true
  default: false
  imageEnvoy: "envoyproxy/envoy:v1.8.0"

Thanks.

I was reading the documentation to learn more about the admission controllers, when I read the prerequisites; the pertinent one being

  Ensure that MutatingAdmissionWebhook and ValidatingAdmissionWebhook
  admission controllers are enabled

Per the docs there is a recommended set of controllers to use:

  --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,Priority,ResourceQuota

I manage K8s clusters with Rancher in non-test environments. By default, Rancher only enables a few of these controllers:

  NamespaceLifecycle,LimitRanger,ServiceAccount,PersistentVolumeLabel,DefaultStorageClass,DefaultTolerationSeconds,ResourceQuota

When I upgraded the cluster, this did not get enabled as I suspected with just the version upgrade. After updating the Rancher K8s configuration and applying the changes, I successfully spun up the test client/server deployments which successfully injected the envoy containers and registered the services in Consul.

TL;DR: RTFM


Autorun ElasticSearch with Docker Compose All of the tutorials online say to manually start elasticsearch as a daemon with a docker command from the command line. Is there a way to auto run it like redis, postgres and other services do from the docker compose yaml file?

https://github.com/elastic/stack-docker/blob/master/docker-compose.yml

This is an official docker-compose for running Elasticsearch, and will make a good starting point for whatever you're trying to achieve. This would be the key part:

  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:${TAG}
    environment: ['http.host=0.0.0.0', 'transport.host=127.0.0.1', 'ELASTIC_PASSWORD=${ELASTIC_PASSWORD}']
    ports: ['127.0.0.1:9200:9200']
    networks: ['stack']

But I'd recommend cloning the entire repository locally and running docker-compose up to experiment.


What are the limitations of Puppet in comparison to Ansible? I would like to understand the differences between Puppet and Ansible, especially what kind of Puppet limitations has in comparison to Ansible.

Are there any things you cannot do it in Puppet, but you can in Ansible? In other words, why some people moving away from Puppet to Ansible?

There are of course several pro and cons for each of Puppet, Ansible, Chef and add your favorite tool here as well. So I'll try to stay away from opinion, and share what is great in Ansible as a matter of fact.

The main capability that puts Ansible above the others is not having to rely on some custom/additional agent running on the target nodes, instead being founded on ssh connections only. Yes, it still requires an ssh server, Python and a bunch of Python libraries on the nodes, and if your distro of choice (or, good luck, there are some windows nodes) does not ship with them, it will be a little bit painful to bootstrap. But that's unlikely, and may well even make you think again about your distro.

That will simplify monitoring, not eat additional resources, not force the system to run a daemon as root all the time, and in general feels better inside the UNIX philosophy. Chef has chef-solo, Puppet can be run master-less, but they both work "the other direction," by cloning and via hooks respectively. While with Ansible, a merge in the source repository can trigger the deployment in a fashion we all are comfortable with, be it in Jenkins, in the git master, or in some other tool like Rundeck for instance.


How do I handle multiple AWS accounts in Terraform? I've read up on Terraform and using AWS S3 and DynamoDB for sharing state and locking. My problem is that we have multiple AWS accounts, by which I mean multiple sign-in locations like:

https://account1.signin.aws.amazon.com/console
https://account2.signin.aws.amazon.com/console

Is it better to have separate S3 and DynamoDB instances in each account, or should we have a single location that's used across all of them?

The answer is highly dependent on what you put inside it, if the states are only account related I'd go with a separate S2/dynamodb per account, if the states are cross-account then you'd need a central place to record them.


What's the difference between checkin and checkout? When teaching SCM classes to students that are new to Software Configuration Management, it happens that a question comes up like "What's the difference between checkin and checkout?".

And a variation of it is that such students get confused about these SCM concepts (they understand them as the other way around).

So what kind of metaphor can you use to explain this crucial SCM concept to such audience?

To explain something to anybody, try to compare it to something they are (hopefully) familiar with already.
So that's why I just answer such question like so:

Think of it as arriving at a place to stay (a hotel, a resort, etc):

the very first thing you do (when you arrive) is to checkin.
the very last thing you do (when you leave) is to checkout.

A similar SCM concept applies when you want to apply changes to software components ...  except that it applies the other way around:

the very first thing you do (when you start) is to checkout (or think of it like borrowing it).
the very last thing you do (when you finish) is to checkin (or think of it like giving it back).

Note: this applies to centralized systems (such as the ones used in mainframe environments ...). In systems such as git the "checkout" concept has a completely different meaning (which IMO is also why in those systems there is hardly any confusion about both concepts).


How to properly automate CM based on log events? Let's say I have a number of tools like Docker bench security, some static security scanners, etc. Now I have let's say free Ansible edition (no expenses are considered).

I have lots of logs, but not much automated, actionable info that triggers scripts. What is a proper way to automate link between static scanners and CM tools like Ansible and Jenkins?

Should it go like:

Parse logs for set of errors and then automatically trigger "correcting scripts"?
Just run a playbook once in a while as it checks the state before doing anything?
Use tools/plugins for Ansible?
Something else?

There is a natural tension here between:

(a) fixing something that is wrong automatically: because for example, a CVE has been created with a high enough severity to need to do something about it.

(b) breaking the system: because the automation introduced a breaking change or a defect.

I have a tendency towards systems being under constant development, even if no features are being added to them they need to kept current to avoid exposure to vulnerabilities.   The rate of change to the 3rd party components of the majority of systems I cover is so high that's its actually worth rebuilding everything and redeploying it daily.

This introduces a couple of requirements:

It is highly automated,
99% of user journies are covered by end-to-end tests so each build has a high degree of confidence that it will work,
We don't roll out to all users, we employ canary release and a rollback mechanism,
We use version constraints in package manifests, etc... i.e. the version of React must be >= 16.7.1 and < 17.0.0
We manage by exception, i.e. if the end-to-end tests fail or the canary release fails humans get involved.

This, in turn, makes systems like trivy, OWASP ZAP and SonarQube more about generating metrics and stopping vulnerabilities from going live than triggering automated remediation.

We do then have a set of rules like:

The live application must not have an unmitigated CVE with a CVSS score of 7.0 or higher.
The list of CVEs with a score of 4.0 or higher will be reviewed on a weekly basis to assess if remediation should be undertaken (i.e. change to a different library, introduce a new safeguard, remove a defunct library, fix the CVE at source, etc.)

This then gives my SRE teams the operational boundaries to work within, they may in practice:

just re-run the deployment ad-hoc to fix it now rather than within 24h,
override a default, for example, if there is a known vulnerability deploy something with a less critical but still higher than CVSS 7.0 known vulnerability,
Introduce a new safeguard, i.e. a Web Application Firewall Rule to mitigate,
Swap out the dependency for something less vulnerable,
Disable that functionality entirely until a fix can be introduced.


Cheap alternative to Azure SQL database I am building a system that processes articles from news websites and pushes the content to an analytics engine. The majority of the Azure cost from an SQL database.
This database is used to store a list of the urls to different websites. Each row is the name of the news source and the matching url link. If one wishes to add a news source, a new row is added manually. So the number of records is very little and stay like so when productionising, i.e. in alpha phase now, we have about 5 records, long term we may have 100-200 records maximum.
The bulk of the database cost comes from the read-write operations that happen frequently (every 15 seconds, all the rows are updated).
We feel like there should be a cheaper alternative Azure resource to store this little amount of data, rather than an expensive SQL database. A no SQL database somewhere, a simple JSON file that can be hosted somewhere on Azure, etc...?
What are your thoughts?

You may be describing a key value store. It is unlikely that you need SQL or relational.
To avoid vendor-lock-in, and to control your costs. Avoid complex services from the cloud providers. Rent a machine or virtual machine from them, and put your own services on it. There are many Free and Open Source databases. Do some research: start with key-value stores (these are the simplest, fastest, but also low functionality, databases).


Cancel a Spot Request vs Terminate Spot Instance I just started getting familiar with EC2's Spot Requests and Spot Instances (1 day into it!).  I understand that I can save some money doing a Spot Request to create a Spot Instance as opposed to creating an on-demand Instance.

What I don't understand is the correct way to "shut things down".  If I want to make sure that I'm no longer charged, do I need to to cancel the Spot Request and terminate the Spot Instance?

I noticed that I could solely terminate the Spot Instance: what would this mean in terms of my Spot Request which exists without my instance?

You can definitely cancel your spot instance requests via the Management Console and via the CLI. There appears to be a command to terminate the instance also.

See the following AWS docs as a reference:

AWS Docs - Cancel Spot Instance

As for terminating just the spot instance, I'm not sure but I think the, still active, spot request would spin up the instance once the spot price matches the request again.

** Edit **

Worth noting that if you terminate the spot instance within an instance hour then you pay for the full hour, even if you only used a few minutes of it.


parameter-store values show '***' instead of actual value I am using CodeBuild to access the Systems Manager Parameter Store. In buildspec.yml, I have the parameter-store configured to access some values in Systems Manager (I certainly have the /mysite/dev/mongodb_uri created in parameter store):

version: 0.2

env:
  variables:
    MONGO_URI: 'mongodb+srv'
  parameter-store:
    MONGODB_URI: /mysite/dev/mongodb_uri

phases:
  pre_build:
    commands:
      - echo $MONGO_URI
      - echo $MONGODB_URI

Unfortunately, while CodeBuild will echo out the value for key in the variables object, it will not for the parameter-store object:

[Container] 2019/12/24 03:16:09 Running command echo $MONGO_URI
mongodb+srv

[Container] 2019/12/24 03:16:09 Running command echo $MONGODB_URI
***

It just gives me 3 stars. If I use aws cli, I get the right value:

$ aws ssm get-parameters --names "/mysite/dev/mongodb_uri"
{
    "Parameters": [
        {
            "Name": "/mysite/dev/mongodb_uri",
            "Type": "String",
            "Value": "mongodb+srv://myuser:mypassword@mysite-cluster-n1qub.mongodb.net/test?retryWrites=true&w=majority",
            "Version": 2
        }
    ],
    "InvalidParameters": []
}

My CodeBuild Service Role policies grant me access to Systems Manager:

{
        "Effect": "Allow",
        "Action": [
            "ssm:GetParameters",
            "ssm:DescribeParameters",
            "ssm:GetParameter",
            ...
        ],
        "Resource": [
            "arn:aws:logs:us-east-2:880783130023:log-group:/aws/codebuild/mysite-dev",
            "arn:aws:logs:us-east-2:880783130023:log-group:/aws/codebuild/mysite-dev:*",
            "arn:aws:s3:::codepipeline-us-east-2-*",
            "arn:aws:ssm:us-east-2:880783130023:parameter/mysite/dev/*",
            "arn:aws:codebuild:us-east-2:880783130023:report-group/mysite-dev-*",
            "arn:aws:ssm:us-east-2:880783130023:*"
        ]
    }

So why does it give me '***' instead of the actual value when I echo the parameter in buildspec?

The "***" is the masking of parameter store output in your log file. The value is available in the environment variable $MONGODB_URI and can be used to pass along to any command or script that needs this value. CodeBuild is merely doing a best-effort masking of log statement which may contain sensitive information.

Also, if you must see the value printed out, you may assign $MONGODB_URI to another variable within your buildspec and print that out.


One Checkout When Running Parallel Stages In Jenkins Introduction

We have a multibranch pipeline project. The build configuration is defined by a jenkinsfile. The system consists of one master and a couple of agents.

The stages are run in parallel, and for each one we do a checkout of the repository.

See attached file below.

The Problem

is that the size of the workspace accumulates greatly as the amount of branches increases.

The Solution #1

Periodic cleanup of the workspace. A custom script will be written.
Something like this one:

// Check if a slave has < 10 GB of free space, wipe out workspaces if it does

import hudson.model.*;
import hudson.util.*;
import jenkins.model.*;
import hudson.FilePath.FileCallable;
import hudson.slaves.OfflineCause;
import hudson.node_monitors.*;

for (node in Jenkins.instance.nodes) {
    computer = node.toComputer()
    if (computer.getChannel() == null) continue

    rootPath = node.getRootPath()
    size = DiskSpaceMonitor.DESCRIPTOR.get(computer).size
    roundedSize = size / (1024 * 1024 * 1024) as int

    println("node: " + node.getDisplayName() + ", free space: " + roundedSize + "GB")
    if (roundedSize < 10) {
        computer.setTemporarilyOffline(true, new hudson.slaves.OfflineCause.ByCLI("disk cleanup"))
        for (item in Jenkins.instance.items) {
            jobName = item.getFullDisplayName()

            if (item.isBuilding()) {
                println(".. job " + jobName + " is currently running, skipped")
                continue
            }

            println(".. wiping out workspaces of job " + jobName)

            workspacePath = node.getWorkspaceFor(item)
            if (workspacePath == null) {
                println(".... could not get workspace path")
                continue
            }

            println(".... workspace = " + workspacePath)

            customWorkspace = item.getCustomWorkspace()
            if (customWorkspace != null) {
                workspacePath = node.getRootPath().child(customWorkspace)
                println(".... custom workspace = " + workspacePath)
            }

            pathAsString = workspacePath.getRemote()
            if (workspacePath.exists()) {
                workspacePath.deleteRecursive()
                println(".... deleted from location " + pathAsString)
            } else {
                println(".... nothing to delete at " + pathAsString)
            }
        }

        computer.setTemporarilyOffline(false, null)
    }
}

## The Solution #2 ##
One checkout per job and node. Thus each parallel stage will execute one executable instance of the same repository. For me personaly this is the preferred method, although more complex than solution #1.

pseudocode:

    stage("Build") {
        try {
            parallel builds
        } catch (err) {
        }
    }

    def build(predef) {
        return {
            node() {
                checkout_repository(${env.NODE_NAME})
                Execute build
                ...
            }
        }
    }

    def checkout_repository(node){
        No checkout for this node
            Flag that we are checking out a repository on node (think ~semaphore)
            Cheking out ..
            Flag that repository exist on node
    }

The Question

What are the cons in solution #2
Does it follow the jenkins way of doing it. Or this to much freestyle?

The job:

Your difficulty stems from encouraging potentially long-lived branches, which is deviating from the CI methodology - which is supposedly why you'd be using jenkins - a CI tool - in the first place.

Normally you shouldn't be running the tests for those branches in the Jenkins pipeline for the trunk - you want to measure the quality of the integrated work, not that of the work done in the silos.

I do understand the desire to run such tests before merging into trunk, so maybe it's OK to use jenkins to do it if you want to, but with separate jobs/pipelines, not tied into the trunk pipeline. And keep in mind that the results of such tests aren't necessarily conclusive: just because they pass over there doesn't mean the same tests will also pass in the trunk, you'll need very strict conditions to make that always true.

Personally I'd go for a fully automated patcing/merging solution based on pre-commit verifications which can enforce the strict conditions necessary to prevent post-merge trunk quality regressions (but I'm biased, I built such solutions). See for example How to ensure that git subtrees are kept up to date?

Anyways, back to the question. Since the code for the various pipeline segments that you seek to perform in parallel comes from different branches you won't be able to use the same workspace, you have to pull separate workspaces from the respective, different branches. Which makes the question kinda moot.

Final note: assuming the same workspace could be used in multiple builds, your #2 approach requires a pristine build structure. It's not uncommon for less-than-perfect builds running in parallel in the same workspace to interfere with each-other: race conditions creating/deleting directories, overwriting artifacts, too high peaks of resource usage, etc - typically hard to repro intermittent failures. These could easily happen if, for example, the builds would be for the same product but for different CPU architectures, or builds including code shared across multiple products. Just be aware of the possibility.


Best Practices for a multi environment deployment using Jenkins I have 3 environments, each on their own Virtual Network with their own configurations. Do I need to have 3 separate instances of Jenkins if I am to do Continuous Deployments on each of the environments? What are some best practices regrading deploying on a multi environment architecture?

This is a very good question as it is a common anti-pattern to couple ci/cd tools with environments.

Jenkins is a build factory. As such, it should be totally agnostic of the notion of environment, or even delivery.

The best pratice is to have some sort of staging process and / or interface (if you can afford it: a dedicated delivery software).

Staging process

You should try to create some sort of staging jobs for each environment, in which you'll define the configuration of the environment and bundle the final package with it.

Next you'll have delivery jobs for each environment.

Delivery tools

In order to do create the staging and delivery jobs, you may use very basic scripting tools, like shell, but there are more adapted scripting tools, like Ansible, puppet, chef...

If you can afford the expense, you may consider investing in a deployment software (I'll mention some that I worked with in the comments section).

Notice that those softwares manage some kind of environment staging process.

Obviously, it makes a lot of sens to combine deploy softwares and scripting.

Beyond segregating tools and environment

Since you asked for best practices, it appears to me that it is worth mentioning another commonly found anti-pattern: coupling SCM tools and delivery process.

It is a very good practice to store environment configuration (not passwords or confidential informations, of course) and go live scripts in SCM tools (like svn or git...). It is however a very bad practice to check-out environment configuration and go live scripts DURING the go live. It might simply be unavailable when you need it.

This check-out phase should be part of the staging process that I mentioned before.

Idem potence

Another best practice is that your scripts should be idem-potent: this means that you should be able to play your scripts once to perform the staging and delivery. Then you should be able to play them again and again, and they would not change the state of your system unless something chacked in the configuration.

Scaling

As a final best practice I shall share from direct experice is scalling Jenkins: different teams have different needs and uses of Jenkins. When too many teams share a common Jenkins, there might be issues with resources. The worst case is when a team requires to restart the Jenkins master while another needs to deliver.

The ideal would be to have one Jenkins per team or per group of teams that share the same goal and delivery planning.


Multiple gradle builds with docker I have a multi-branch setup on Jenkins for my Gradle project. I would like to be efficient in terms of performance and cost as well (money!). Meaning, if would like to build my project many times, for different branches, commits, in parallel. For that I'm thinking about having everything inside of containers, but I'm not sure if my containers would share the libraries resolved by gradle. Moreover, I don't want to resolve dependencies for every time I bring them up, it would be time consuming and network usage as well.

Or I'm thinking about deploying a simple vm with Ansible and run my builds from there, making this whole process simpler.

So I wonder which approach would is better? Anyone faced a similar issue? Does someone has any benchmark to share?

There is currently an issue open to share Gradle cache between containers:

https://github.com/gradle/gradle/issues/851

I think the best solution at the moment is to bake as many dependencies in to the Docker image as possible, and then use a volume to share between sequential builds, limiting it to one concurrent build per host. You can mount the cache on to the host with a docker-compose.yml like so:

version: '3'
services:
  gradle:
    image: gradle:4.0.1-jdk8-alpine  # I'd suggest using this as a base to build your own Gradle image with dependencies baked in
    working_dir: /usr/src/app
    volumes:
      - ./:/usr/src/app:Z
      - ./.gradle:/home/gradle/.gradle:Z

Alternatively, another solution would be to look at network-level caching e.g. Squid.


What is the optimal size of a Jenkins Pipeline Stage? Should I create single checkout stage or multiple stages like checkout scripts, checkout code, checkout something else?

How should I decide on what piece of logic to put into separate stage?
Or are stages just a pipeline visualization tool?

Single responsibility principle (SRP)

  Martin defines a responsibility as a reason to change, and concludes
  that a class or module should have one, and only one, reason to be
  changed (i.e. rewritten). As an example, consider a module that
  compiles and prints a report. Imagine such a module can be changed for
  two reasons. First, the content of the report could change. Second,
  the format of the report could change. These two things change for
  very different causes; one substantive, and one cosmetic. The single
  responsibility principle says that these two aspects of the problem
  are really two separate responsibilities, and should therefore be in
  separate classes or modules. It would be a bad design to couple two
  things that change for different reasons at different times.

  The reason it is important to keep a class focused on a single concern
  is that it makes the class more robust. Continuing with the foregoing
  example, if there is a change to the report compilation process, there
  is greater danger that the printing code will break if it is part of
  the same class.

5-15 lines

  Functions should normally be short, between 5-15 lines is my personal
  "rule of thumb" when coding in Java or C#. This is a good size for
  several reasons:

  It fits easily on your screen without scrolling
  It's about the conceptual size that you can hold in your head
  It's meaningful enough to require a function in its own right (as a standalone, meaningful chunk of logic)
  A function smaller than 5 lines is a hint that you are perhaps breaking the code up too much (which makes it harder to read /
  understand if you need to navigate between functions). Either that or
  your're forgetting your special cases / error handling!

Personal view

I have the SRP in mind when creating Jenkins stages. This implies that I create a separate stage for checkout, build and publish for example. If a stage becomes too large, i.e. longer than 15 lines, then I create a script and run the script in this stage. One of the benefits is readability, possible to apply unit testing on the script and CI agnostic, i.e. migration to another CI would be possible as well.

Example

pipeline {
    agent any

    stages {
        stage('Build') {
            steps {
                echo 'Building..'
            }
        }
        stage('Test') {
            steps {
                echo 'Testing..'
            }
        }
        stage('Deploy') {
            steps {
                echo 'Deploying....'
            }
        }
    }
}


What is a minimal viable number of hosts for an experimental Kubernetes cluster? For example, to play with Docker Swarm in a way which makes sense you need 2 or better 3 hosts, right?

What is that number for K8S?

I'd say the minimal viable configuration to test and get a good overview of pod placement would be 5 machines, 2 masters and 3 nodes.
This allows you to play with a master failure, placement of multiple pods "stretched" across the cluster, etc.


iptables rules to permit GitHub Actions from breaking (chains default to DROP) Assuming my iptables rules default to DROP on the INPUT and OUTPUT chains, what is the bare minimum set of rules that I must add to my chains to prevent a script running in GitHub Actions from stalling indefinitely?
I'm using (free) GitHub Actions for my open-source application's CI/CD infrastructure. When I push changes to github.com, it automatically spins up an Ubuntu 18.04 linux server in Microsoft's cloud that checks-out my repo and executes a BASH script to build my application.
For security reasons, early on in my build script I install and setup some very restrictive iptables rules that default to DROP on the INPUT and OUTPUT chains. I poke a hole in the firewall for 127.0.0.1, RELATED/ESTABLISHED on INPUT, and only permit the _apt user to send traffic through OUTPUT.
This works great when I run the build script in a docker container on my local system. But--as I just learned--when it runs with GitHub Actions, it stalls indefinitely. Clearly, the instance itself needs to be able to communicate out to GitHub's servers in order to finish. And I appear to have broken that.
So the question is: what -j ACCEPT rules should I add to my iptables INPUT and OUTPUT chains to only permit the bare necessities for GitHub Actions executions to proceed as usual?
For reference, here's the snippet from my build script that sets-up my firewall:
##################
# SETUP IPTABLES #
##################

# We setup iptables so that only the apt user (and therefore the apt command)
# can access the internet. We don't want insecure tools like `pip` to download
# unsafe code from the internet.

${SUDO} iptables-save > /tmp/iptables-save.`date "+%Y%m%d_%H%M%S"`
${SUDO} iptables -A INPUT -i lo -j ACCEPT
${SUDO} iptables -A INPUT -s 127.0.0.1/32 -j DROP
${SUDO} iptables -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
${SUDO} iptables -A INPUT -j DROP
${SUDO} iptables -A OUTPUT -s 127.0.0.1/32 -d 127.0.0.1/32 -j ACCEPT
${SUDO} iptables -A OUTPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
${SUDO} iptables -A OUTPUT -m owner --uid-owner 100 -j ACCEPT # apt uid = 100
${SUDO} iptables -A OUTPUT -j DROP

${SUDO} ip6tables-save > /tmp/ip6tables-save.`date "+%Y%m%d_%H%M%S"`
${SUDO} ip6tables -A INPUT -i lo -j ACCEPT
${SUDO} ip6tables -A INPUT -s ::1/128 -j DROP
${SUDO} ip6tables -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
${SUDO} ip6tables -A INPUT -j DROP
${SUDO} ip6tables -A OUTPUT -s ::1/128 -d ::1/128 -j ACCEPT
${SUDO} ip6tables -A OUTPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
${SUDO} ip6tables -A OUTPUT -m owner --uid-owner 100 -j ACCEPT
${SUDO} ip6tables -A OUTPUT -j DROP

# attempt to access the internet as root. If it works, exit 1
curl -s 1.1.1.1
if [ $? -eq 0 ]; then
        echo "ERROR: iptables isn't blocking internet access to unsafe tools. You may need to run this as root (and you should do it inside a VM)"
        exit 1
fi

This can be achieved by running your build script in a docker container and applying your iptables rules inside that container, which won't affect the host runner's connectivity.
For example, if the below script is executed in a GitHub Actions job (in the Ubuntu 18.04 GitHub shared runner), it will run the build script (docker_script.sh) in a debian docker container that has no internet connectivity, except from the _apt user.
#!/bin/bash
set -x

###################
# INSTALL DEPENDS #
###################

apt-get -y install docker.io

##################
# DOWNLOAD IMAGE #
##################

# At the time of writing, Docker Content Trust is 100% security theater without
# explicitly adding the root public keys to the $HOME/.docker/trust/ directory
#
#  * https://github.com/BusKill/buskill-app/issues/6#issuecomment-700050760
#  * https://security.stackexchange.com/questions/238529/how-to-list-all-of-the-known-root-keys-in-docker-docker-content-trust
#  * https://github.com/docker/cli/issues/2752

docker -D pull debian:stable-slim

#################
# CREATE SCRIPT #
#################

tmpDir=`mktemp -d`
pushd "${tmpDir}"

cat << EOF > docker_script.sh
#!/bin/bash
set -x

# SETTINGS #
SUDO=/usr/bin/sudo

# DEPENDS #
${SUDO} apt-get update
${SUDO} apt-get install iptables curl

# IPTABLES #

# We setup iptables so that only the apt user (and therefore the apt command)
# can access the internet. We don't want insecure tools like `pip` to download
# unsafe code from the internet.

${SUDO} iptables-save > /tmp/iptables-save.`date "+%Y%m%d_%H%M%S"`
${SUDO} iptables -A INPUT -i lo -j ACCEPT
${SUDO} iptables -A INPUT -s 127.0.0.1/32 -j DROP
${SUDO} iptables -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
${SUDO} iptables -A INPUT -j DROP
${SUDO} iptables -A OUTPUT -s 127.0.0.1/32 -d 127.0.0.1/32 -j ACCEPT
${SUDO} iptables -A OUTPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
${SUDO} iptables -A OUTPUT -m owner --uid-owner 100 -j ACCEPT # apt uid = 100
${SUDO} iptables -A OUTPUT -j DROP

${SUDO} ip6tables-save > /tmp/ip6tables-save.`date "+%Y%m%d_%H%M%S"`
${SUDO} ip6tables -A INPUT -i lo -j ACCEPT
${SUDO} ip6tables -A INPUT -s ::1/128 -j DROP
${SUDO} ip6tables -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
${SUDO} ip6tables -A INPUT -j DROP
${SUDO} ip6tables -A OUTPUT -s ::1/128 -d ::1/128 -j ACCEPT
${SUDO} ip6tables -A OUTPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
${SUDO} ip6tables -A OUTPUT -m owner --uid-owner 100 -j ACCEPT
${SUDO} ip6tables -A OUTPUT -j DROP

# attempt to access the internet as root. If it works, exit 1
curl 1.1.1.1
if [ $? -eq 0 ]; then
        echo "ERROR: iptables isn't blocking internet access to unsafe tools. You may need to run this as root (and you should do it inside a VM)"
        exit 1
fi

# BUILD #

# ...
# <DO BUILD HERE>
# ...

exit 0
EOF
chmod +x docker_script.sh

##############
# DOCKER RUN #
##############

docker run --rm --cap-add "NET_ADMIN" -v "${tmpDir}:/root/shared_volume" debian:stable-slim /bin/bash -c "cd /root/shared_volume && docker_script.sh"

# exit cleanly
exit 0

Note that:

You have to execute the docker run command manually rather than just specify the container: in the GitHub Actions yaml file in order to add the NET_ADMIN capability. See Also https://stackoverflow.com/questions/63959783/how-to-run-script-in-docker-container-with-additional-capabilities-docker-exec/64045054#64045054

This is a security risk unless you pin the root signing keys before calling docker pull. See also https://security.stackexchange.com/questions/238529/how-to-list-all-of-the-known-root-keys-in-docker-docker-content-trust

The above script should be executed as root. For example, prepend sudo to it in the run: key for the step in the GitHub Actions workflow.


How to keep polling GitHub repository and get commits by it's messages on a regular interval using Jenkins? I need to know how to write a Jenkins job which polls a GitHub repository continuously and send an email of commits made to that repo.

Considering you have proper credentials in Jenkins and admin right for Git repo, Create a webhook between Jenkins and your Git repo. Webhook will help you push (reverse poll)the commits. You can use jenkins DSL (groovy) to operate on that Git repo by using Git commands. You can use specific Git commands to list Authors of last commits (or any other requirement) and Use email-ext (or any other) plugin in jenkins to be able to send email through DSL.

One line reference to list Author (yeah..it works as it is in any Git repo)-

git log -1 --pretty=format:'%an'

My reference of email to culprits if something goes wrong my jenkins build

step([$class: 'Mailer', notifyEveryUnstableBuild: true, recipients: emailextrecipients([[$class: 'CulpritsRecipientProvider'], [$class: 'RequesterRecipientProvider']])])


How to avoid branchageddon with large organisations? How do you avoid a branchageddon situation when working with large organisations?

We work with a number of large financial organisations whose approach is to not take updates to software, but instead only high/critical security patches and bespoke functionality.  These organisations will only take patches and custom release in between major updates. Major updates can be years apart and carry high costs. This approach causes us (the software house) to have a branch of our code per major customer, which carries all the costs and inefficiencies of long term branching.

My questions to the community are:

Have you experienced similar update acceptance approaches from your
customers?
What suggestions do you have to help work with this
approach?
What suggestions do you have to help change organisations
approaches to taking software updates?

As Michael mentioned, offer a standard solution based on release versions/numbers, with a reasonably long lifespan for your industry (maybe interleaved with one or more shorter lifespan intermediate versions, if it makes sense for your typical customers).

Give your customers the option to embark on this standard release track, maybe with a decent migration deadline in place.

If they insist on a full-custom branch support strategy just charge them accordingly, to properly cover all your extra costs for offering such full-custom support - it only makes business sense. Some customers will migrate to reduce their costs (which will help you reduce the number of custom branches), some won't.

Variable support billing, growing progressively with the age of the release versions from which the custom branches originate can also be an incentive for customers to migrate to newer branches faster, helping with faster closing of the older custom branches. This can help reduce the number of custom branches per customer - if you have customers that simultaneouly run multiple versions of your software.

Make sure you don't fall into the trap of doing full branch merges from/to any of the release branches (both standard and custom), all changes to them should be either individually developed or cherry-picked merged fixes.

Because each of these branches will gradually diverge from each-other, the number of hotfixes requiring customisation/individual development will grow exponentially (plain cherry-pick merging will fail). You need to take the development cost for these into account.

With no (significant) branch merges in the picture you can (and should, I cannot stress its importance enough) build fully automated CI/CD pipelines for these branches, accompanied with a good hotfix tracking/management system in place, making the hotfix delivery just routine (or almost).


Dockerize CI installation on master node VM Carrying on from this question, would you want to dockerise your CI (Gitlab (especially if Gitlab also contains all your repositories) or Jenkins) installation on your master node virtual machine?

What are the advantages and disadvantages to this, than just installing normally onto the VM without docker?

Pros

possible to run the CI on a container orchestration platform

Cons

docker in docker in docker


Key attributes of Continuous Integration (CI) software I work in the IT department of automation/dev company and we are just about to implement/add CI into our set of tools. And we are having difficulties to choose the one.

At the moment we've been thinking about these systems:

Jenkins
CircleCI
TravisCI

Question: What are key attributes of CI's soft which the systems should have and based on which we should compare them?

EDIT: We are expecting from CI the standard things: From building the applications, running tests (Unit/Integration/Performance/..), saving statistics and providing information either via email/on the page (reports).

Issue there is that we don't know which if there are more functionality and if, what is it? This is what I would be looking for under words "key attributes". The names above are just for reference, I'm looking for answer "why use this or that" over simplified "use this or that".

We are developing in .NET (C#) environment with some libraries written in C++/VBA.

In general, there are five main differences between different CI software solutions.

Cost: Is the software open source and free or proprietary? Does your DevOps team already have a budget for software or are you expected to compare free options?
Maintenance: Is the software something you need to host on your own and maintain, or is it being offered as a service? If you need to host it on your own, do you have the team members available who can run the maintenance? If it is being provided as a service, where does your code reside?
Security: Are there any security or compliance concerns about where the build machines are hosted? Are the permissions required by the CI software into your SCM well understood?
Community: Does the CI software have a good community around it with the plugins and support for the technologies you use? Is it easy to get support or troubleshoot when things go wrong?
UX: How easy is the software to use? Does the CI software support scripts stored within your SCM for the tasks you need to do? Is the reporting easy to understand or will it require some training?


Ansible URI module and skip SSL validation I have a playbook which looks like
- name: CURL post example
  uri:
    url: "https://dev-box.local/api/"
    method: POST
    body: '{
           "max_conns": 0,
           "max_fails": 2,
           "fail_timeout": "3s"
           }'
    body_format: json
    validate_certs: no
    return_content: yes
    force_basic_auth: yes
    headers:
      Content-Type: "application/json"

but during the execution I'm getting
 FAILED! => {"changed": false, "connection": "close", "content": "{\"status\":403,\"message\":\"Forbidden\"}\n", "content_length": "37", "content_type": "application/json", "date": "Thu, 03 Dec 2020 21:03:13 GMT", "elapsed": 0, "json": {"message": "Forbidden", "status": 403}, "msg": "Status code was 403 and not [200]: HTTP Error 403: Forbidden",

when I will do the same call as the curl -k from the command line on the same box as ansible it works fine.
Any thoughts on that?

You seem to be passing force_basic_auth: yes but no user, nor password.
Example from the Ansible docs:
- name: Create a JIRA issue
  uri:
    url: https://your.jira.example.com/rest/api/2/issue/
    method: POST
    user: your_username
    password: your_pass
    body: "{{ lookup('file','issue.json') }}"
    force_basic_auth: yes
    status_code: 201
    body_format: json

The 403 is usually Basic Auth failing. If it's not, it could be the API server you're running requiring some sort of authentication.
If the curl -k command includes something like https://user:password@dev-box.local/api/ or the --user <user:password>, it should work. Or maybe it's using data from .netrc?
I hope this helps.


How to investigate a main process that has died in a docker container? Sometimes you have to investigate a container, which is stopped, or a container which after starting up dies very quickly and stops.

docker exec -ti <id> bash only works on running containers, once it finishes, the bash prompt terminates as well.

With docker start you cannot supply a different command, and if the container dies abruptly again you won't have enough time to get into the container and do your investigations.

We can do docker commit, then docker run on the new image with a different command, but I'm wondering if there are any other alternatives.

Note: docker logs just returns whatever the apps printed to stdout/stderr. That might not be enough to figure out what the problem was.

General ways to track why a process in Linux failed are good. One such way is to run a process using strace which will tell you the system calls process did and usually point to the reason for a failure.

You can create a Dockerfile that looks something like this:

FROM original_image

RUN apt-get -y update && apt-get install -y strace

# build with `docker build -t debug_version`

Then run your new image using docker run debug_version strace original_cmd.

For processes that fork children (and then die) you want to run strace with the -ff option. You can also map some file using Docker data volumes and use the -o option from strace to write to it. But in general strace will leave output on stdout, which is readable using docker log.

Related Q: Linux process terminates mysteriously


Does Kubernetes internal networking keep traffic to co-located Pods where possible? In Kubernetes, if I run a Service Foo on every Node, and also Service Bar on every Node, i.e. with DaemonSets. If a Pod from Service Foo needs to make a request to Service Bar, does Kubernetes networking magic optimise things so communication will be done within the same Node, i.e. without a external network call? Or will a random Pod Bar be picked meaning usually Bar will be on a different Node, so there will be a network hop?

Context to question: I'm thinking to run CoreDNS as a Daemonset, because with the cache plugin, I'm hoping most DNS queries will not need to use the network as there will always be a CoreDNS Pod co-located within the same Node and most requests will be cached. Currently we only run 3 CoreDNS pods and we have 12 Nodes and we see some DNS failures at times of network traffic surges - particularly during a deployment when lot of logging is done, which have to be shipped across the network to ElasticSearch.

In case specifics matter:

The CoreDNS service runs as clusterIP: 172.20.0.10
The services that use CoreDNS run as type: NodePort
The Kubernetes cluster is AWS EKS, v1.14.9. Networking is done with the AWS VPC CNI.

This looks to be answered on offcail docs here

NodeLocal DNSCache improves Cluster DNS performance by running a dns caching agent on cluster nodes as a DaemonSet.
With this architecture, Pods will reach out to the dns caching agent running on the same node, thereby avoiding iptables DNAT rules and connection tracking.
Having a local cache will help improve the latency.

Image Reference : https://kubernetes.io/docs/tasks/administer-cluster/nodelocaldns/#architecture-diagram


Sonarqube taking massive amounts of memory I am running Sonarqube on a monolithic application on our CI server and the scanner (not the Sonarqube server) is taking over 60GB of memory.
Is this normal?
What can I do so the scanner will not use so much memory during the Sonarqube scan?

Please refer to the following article on the SonarQube - SonarScanner setup best practices. It will be good if the following hardware recommendations  are considered for the SonarQube installations. Following are some of the important considerations.

SonarQube server should be on a real DB not on H2.
The analysis should occur very close to the SonarQube DB.


Force role session name in AWS IAM I want to enforce users to use their IAM username as the role session name when assuming a role in AWS. I've tried the following condition in IAM policies:
{
  "Effect": "Allow",
  "Principal": {
    "AWS": "arn:aws:iam::<redacted>:root"
  },
  "Action": "sts:AssumeRole",
  "Condition": {
    "StringLike": {
      "sts:RoleSessionName": "${aws:username}"
    }
  }
}

While this works fine when assuming a role as a user it does not work when assuming a role from a role with administrator privileges (Allow * on *). The only way to block this would be an explicit Deny when a role tries to assume a given role and does not have session name set up. Any ideas how to write a policy doing this? A simple Deny like below does not work because aws:username is not present when Assumed Role is the principal (see docs).
{
  "Effect": "Deny",
  "Principal": {
    "AWS": "arn:aws:iam::<redacted>:root"
  },
  "Action": "sts:AssumeRole",
  "Condition": {
    "StringNotLike": {
      "sts:RoleSessionName": "${aws:username}"
    }
  }
}

I have not tried it but I think if you change your deny to a list of elements it will work.
{
  "Effect": "Deny",
  "Principal": {
    "AWS": "arn:aws:iam::<redacted>:root"
  },
  "Action": "sts:AssumeRole",
  "Condition": {
    "StringNotLike": {
      "sts:RoleSessionName": [
        "${aws:username}",
        "AnotherAllowedName"
      ]
    }
  }
}

This will allow an IAM user to assume the role if they use their username as the session name and it will also allow other Principals to assume the role if they use "AnotherAllowedName" as the session name but all other session names would be declined. I found this AWS blog post helpful when researching this answer


How to produce a map from the cartesian product of two lists to use with for_each on a resource with Terraform I need to create a datadog synthetics monitors with the resource datadog_synthetics_test
I need to monitor multiples cluster, with multiple point of view :
# Clusters
variable "datadog_gke_clusters" {
  default = [
    # NPRD
    {
      name        = "cluster01"
      environment = "nprd"
      url = {
        private = "https://private.domain-priv.com"
        public  = "https://public.domain.com"
      }
    },
    # PROD
    {
      name        = "cluster02"
      environment = "nprd"
      url = {
        private = "https://private.domain-priv.com"
        public  = "https://public.domain.com"
      }
    }
  ]
}

# Point of views
variable "gke_slo" {
  description = "Response time threshold in ms to respond to a request."
  default = [
    {
      zone              = private
      monitor_threshold = 50
    },
    {
      zone              = public
      monitor_threshold = 100
    },
  ]
}

for_each need a map or a list of strings
resource "datadog_synthetics_test" "gke-monitoring" {
  for_each = ... # cartesian product of lists

  type    = "api"
  subtype = "http"
  ....
}

There is a function [setproduct(sets...)][1] which produce a list which combine values of each list, but not a map.
I would need to produce a map which look like this :
  {
    "cluster01-private" = {
      name        = "cluster01"
      environment = "nprd"
      url = {
        private = "https://private.cluster01.domain-priv.com"
        public  = "https://public.cluster01.domain.com"
      }
      zone              = private
      monitor_threshold = 50
    },
    "cluster01-public" = {
      name        = "cluster01"
      environment = "nprd"
      url = {
        private = "https://private.cluster01.domain-priv.com"
        public  = "https://public.cluster01.domain.com"
      }
      zone              = public
      monitor_threshold = 100
    },
    # PROD
    .... etc
  }

How could I do that ?

In 2 steps, you can first create your cartesian product in a variable:
locals {
  datadog_gke_clusters_by_pov = flatten([
    for cluster in var.datadog_gke_clusters : [
      for pov in var.gke_slo_ : {
        cluster    = cluster
        pov        = pov
      }
    ]
  ])
}

And then use this new variable in your resource block:
resource "datadog_synthetics_test" "gke-monitoring" {
  for_each = {
    for valuue in local.datadog_gke_clusters_by_pov : "${valuue.cluster.name}-${valuue.pov.zone_name}" => valuue
  }
...
# here you can use each.value.cluster.* and each.value.pov.*
}

Have a good day!


What are alternatives for Docker in Docker (dind) usage for Jenkins slave running in container? When using Jenkins slave running in container, what are alternatives or "safe" (in terms of security) implementations for Docker in Docker.

Architecture example:

Jenkins master running as Docker container
Jenkins slave running as Docker container
Test task executed in Docker container

Some known alternatives:

DOOD (Docker-Outside-Of-Docker)
Jenkins Plugin using Docker API

Constraints clarification: the idea is to use Jenkins as an orchestrator and Docker for clean environment, the rest of architecture is a subject to change.

To avoid relying on dind (out of building new images from a dockerfile which should be ok) I'd look around Jenkins's kubernetes plugin.

It obviously needs to bring to life a kubernetes cluster but allow to have a one time jenkins agent to run the test in a clean isolated environment, and then tear it off. The maintenance costs of the kubernetes cluster should not be greater than maintaining your Jenkins slaves.

The bonus point is that you can version you slaves with specific environment and don't have to tweak around for multiples version of ruby/java/python and let the task choose the proper "environment" by specifying the agent image to use and have a ready to test environment without any tweak pre-test.

I'd head toward kurbenetes as container scheduler because it is relatively lightweight to configure and maintain.


Jenkins declarative pipeline Warning: JENKINS-41339 probably bogus PATH I am struggling with this issue inside my declarative pipeline. I am trying to use python's docker image as my agent and run some scripts and I am receiving warning:
Warning: JENKINS-41339 probably bogus PATH=$PATH:$ANDROID_HOME/tools:$ANDROID_HOME/platform-tools:/opt/android-sdk-linux/tools:/opt/android-sdk-linux/platform-tools; perhaps you meant to use ‘PATH+EXTRA=/something/bin’

I do not have access to Jenkins server global variables configuration. I am using one of the Debian slaves.
I have read other issues about this problem stackoverflow/43987005 and stackoverflow/44378221. Also discussion on Jenkins forum and Jenkins official documentation.
This is code I am executing:
//  https://www.jenkins.io/doc/book/pipeline/syntax/
pipeline {
  environment {
    PATH = "/bin/sh:$PATH"
  }
  agent {
    docker {
      image "python:3.9"
    }
  }
  stages {
    stage('Test') {
      steps {
        withEnv(["PATH+EXTRA=$PATH"]) {
          sh "PATH is: $PATH"
        }
      }
      post {
        cleanup {
          cleanWs()
        }
      }
    }
    stage('Build') {
      steps {
        echo "'Let's build"
      }
    }
    stage('Deploy') {
      steps {
        echo "'Let's deploy"
      }
    }
  }
}

Problem appeared after I wanted to implement docker image as agent and by changing environment section inside my pipeline still haven't fixed my problem. I was trying also these alternatives:
PATH = "/usr/local/bin:$PATH"
PATH = "/usr/local/bin:${env.PATH}"

I also went to Configure, section Prepare an environment for the run and filled up Properties Content input but I still didn't receive $PATH content into my job. I am still receiving global value of $PATH. I also unchecked the Keep Jenkins Environment Variables checkbox.
The server I am using has Environment Injector Plugin plugin installed.

This post explains that/why environment variables don't propagate to Docker agent.
If you need a custom container, you could define path and whatever else as layers on top of base image in a Dockerfile. Then using that image in pipeline would be what I think is usual way of doing it. You can build that image outside of Jenkins, but it's also possible to build container from within pipeline too if you want (get Dockerfile from SCM or some other use case).
In the context of your question, I think you can still use base image, and if you know full path you'd like to set, do that at container (agent) startup with args. But $PATH variable won't be available at host to be resolved within container at startup. If you must dynamically set path in container, could adjust it in shells.
pipeline {
    agent {
        docker {
            image 'python:3.9'
            // cannot resolve $PATH var in container (becasue HOST does not know what it is).
            args '-e PATH=/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/i/know/my/full/path/ahead/of/time'
        }
    }
    stages {
        stage('Test') {
            steps {

                // as set by args above. Applies to all shells
                sh 'echo $PATH'

                // You can also adjust it within container in each shell
                sh '''
                    export PATH=/even/more/path:$PATH
                    echo $PATH
                   '''

                // But this is back to what's in args in a new shell instance
                sh 'echo $PATH'
            }
        }
    }
}


Is it possible to load balance TLS/SSL based logstash communication to elasticsearch? if i have filebeat -> logstash -> elasticsearch pipeline with TLS or HTTPS encryption , is it possible to load balance between 2 destination elasticsearch clusters in active-standby ?
what i mean is, in case one of the clusters goes down i would manually or automatically switch all the data streams to the other cluster.

im looking at various proxy setups but im not sure i understand the terms correctly, and TLS would not solve the problem.

You can do this a couple of ways, but I have never seen anyone do this with a CNAME.

One option, if you have control over your IP addressing is to use BGP to fail-over (or load balance) an IP to your cloud hosting providers.

A second option is to invest in a DNS-based geographic load balancing solution, such as the GTM from F5, the GSLB solution from A10, or the GSLB solution from Kemp (F5 is the most commonly used in the industry). With these solutions you will set your name server (NS1 and NS2) for your domain to be the GSLB solution of your choice. You will set a very low TTL for your DNS records and configure health monitors to your ELK stack from the GSLB solution. These can be as complex as logging into ELK and scraping for data or as simple as making sure you can establish a TCP connection and/or ping your server. When this health monitor fails, the GSLB solution will failover traffic to the alternate data center automagically. You will be able to do this as an HA pair with one GSLB solution at once DC as NS1 and the other at your secondary site as NS2. The GSLB will then forward lookup to your previous nameservers which will publish IP addresses for both sites. These will be rewritten to eliminate the IP that the GSLB does not want to direct traffic to at the time (again, this can be used for active-active sites - you can even use geolocation to send users in Europe to a Eurpoean cloud provider  and US based users to a US based provider, or do a simple active/standby)

A final way to do this might be with local load balancing solution, such as F5's LTM, A10's SLB, or Kemp's solution. You may also be able to use nginx for this for a cheap solution. In this scenario, you might have this solution onsite in an office with the URL to the load balancer served up by your local DNS server to a local IP (a Virtual IP) on your load balancer. You would then configure the load balance to direct users to the desired cloud provider by configuring a health monitor (of whatever complexity). You would then need to configure this load balancer to bridge your SSL connection. You would add your certificate and private key to the load balancing solution. Users would establish an encrypted connection to the load balancer. This encryption would be stripped off by the load balancer and a new encrypted connection established to the cloud provider by the load balancer, and the data forwarded along over the new SSL tunnel.


Why is it necessary to have virtual machines to run multiple server instances? I'm trying to learn about virtualization and containers. In an educational video it is said that the introduction of virtual machines was a big deal because it made companies available to run multiple server instances on the same machine, so they did not have to necessarily buy a new machine just to run a new instance.
But what if I have a Node.js server application? I could run multiple of those on the same machine without any kind of virtualization. Why is even running multiple instances necessary? Why were virtual machines needed? My best guess is that Node.js is designed to support running multiple instances (the instances doesn't "know" about each other), therefore they are not needed, but back then the servers that they used would have "known" about each other if they weren't separated by virtual servers.
Do people do the same using containers? If I have a microservices architecture, and let's say that I only have one computer, am I using containers only to check whether they run correctly in different environments, or do I create multiple of even the same microservice?
Please excuse my sloppy wording, I mainly try to get what is it that I'm actually missing / misunderstanding, and I try to convey my abstract, high-level understanding of the issue.
Link to video: https://youtu.be/JSLpG_spOBM?t=77

Multiple apps on the same server

But what if I have a Node.js server application? I could run multiple of those on the same machine without any kind of virtualization.

Yes, you can run multiple apps or multiple instances of the same app on a server. The problem is that these apps will interfere.
Some examples of resources, that might interfere:

They would all like to listen to the same port e.g. 8080.
If they need more memory, they will eventually fight about the same memory.
If the apps is busy, they will fight about the same CPU resource.
They all would like to write logs to the same files on the file system.
They all might want to save data to the same place on the file system.

This needs to be managed in some way, and we don't want to do it manually.
With Virtual Machines on the same server, or Containers on the same server, each instance get its own IP-address, each instance can listen to what port it wants, e.g. all can listen to 8080. You can bound how much memory an instance has, e.g. 1GB - so if an app leaks memory, it does not affect the others. You can also bound how much CPU each instance can use, so they only use their fair share. Each instance has its own view of the filesystem, so all instances can write to the same path (as they see it) - even though it is technically different places.
Multiple instances - Availability

Why is even running multiple instances necessary?

What would you do when you need to patch the server and need to restart it? How will you upgrade your app to a new version? What happens if the hardware on your server crash? What happens when your single instance can not handle all the traffic?
The answer to all those questions, typically is: run more than one instance of your app, and let a load balancer forward the traffic to your instances that are alive.
Why virtualization?

Why were virtual machines needed?

They are not really needed. This is more about economics, a way to use your expensive hardware more efficiently.
If you start a new project, but a server - the server is very powerful and expensive. If you only start a Node.js app on this server, it will probably take long time before this Node.js app is so popular that it need to use the full server. It would be better if you could rent only a small part of the server in the beginning - that is how the Cloud with AWS and Google Cloud Platform works.


AWS Redshift UNLOAD not firing S3 PUT event I am trying to use Redshift to UNLOAD data to S3 and then pick that data up with a Lambda function and process it. All of the constituent parts (the UNLOAD, the Lambda function) now work, however the Lambda event trigger (S3:ObjectPUT) is never firing. If I upload manually via the console, it all works. If I run the UNLOAD the data appears in the bucket, but the trigger event doesn't run.

I have tried creating / recreating the function (as per this forum thread - https://forums.aws.amazon.com/thread.jspa?messageID=768407&tstart=0), but to no avail.

Has anyone else had this - is there something I'm missing?

You did not tell how big data amounts you tested with, but if you test ok with little data block successfully and larger real situation fails, there may occur same as occurred here:

  The larger files are almost certainly being uploaded via S3 Multipart Upload instead of a regular Put operation.

  ..

  It sounds like the function is only subscribed to s3:ObjectCreated:Put events currently, and you need to add s3:ObjectCreated:CompleteMultipartUpload to the configuration.


Isolate service networks with Docker Compose I have two containerised services, A and B, with which I wish to isolate network communications with docker compose. There is also a shared service C which is not isolated from A or B and is free to communicate with either.

I have tried to achieve this with the following docker-compose.yaml:
version: "3.3"
services:
  C:
    build: C
    depends_on:
      - A
      - B
    expose:
      - 8080
    networks:
      - A
      - B
  A:
    build: A
    expose:
      - 8080
    networks:
      - A
  B:
    build: B
    expose:
      - 8080
    networks:
      - B

networks:
  A:
    internal: true
  B:
    internal: true

With this, C is able to connect to both A and B on the hosts A:8080 and B:8080 respectively and  A and B are isolated as required. However, neither A nor B is able to connect to C on C:8080. Removing the network options from the compose file, all services are able to talk to each other. What am I doing wrong?

I don't experience same issue with similar setup to what you describe. 'A' & 'B' can ping & access services on 'C' just fine. Few things to look at:

Port aside, can you ping 'C' from either of the other containers?
You probably shouldn't (can't?) name containers capital 'A', 'B', 'C' if you're going to build them in docker-compose. If that's what's actually used, try more meaningful names.
Does name resolution work if you do nslookup c from either a & b? It should resolve via nameserver 127.0.0.11
firewall on 'C' or service not up on c:8080?


How to integrate firebase functions into devops pipeline I have a project which has firebase functions

From my local I can run firebase deploy --only functions and it will build and deploy firebase functions to firebase.

Now I want to enable CICD via Azure Devops.
So I need to build the functions in CI and get the build artifact in CD to deploy it to different firebase project (as different environments).

The problem is building and deploying is done in the same time via firebase deploy

I googled a lot not able to find a solution

I found the solution.

As build artifact you need to publish, firebase.json, .firebaserc and whole functions folder including package.json (without node_modules)

Then in release using sudo firebase deploy --token $(firebase-token) --project $(firebase-project) --only functions --force


Curl Error in HTTP2 framing layer when trying to install Jenkins plugin I am trying to create a custom Jenkins docker image. For that I need to install certain plugins in Jenkins. I am doing so by adding the below lines to the Dockerfile
COPY plugins.txt /usr/share/jenkins/ref/

ENV CURL_CONNECTION_TIMEOUT=60
RUN /usr/local/bin/install-plugins.sh < /usr/share/jenkins/ref/plugins.txt

My plugins.txt looks like this -
configuration-as-code

I had more plugins in the list but seems like I am unable to install even if there is only one.
The full error message that I receive is -
Creating initial locks...
Analyzing war /usr/share/jenkins/jenkins.war...
Registering preinstalled plugins...
Using version-specific update center: https://updates.jenkins.io/2.235...
Downloading plugins...
Downloading plugin: configuration-as-code from https://updates.jenkins.io/2.235/latest/configuration-as-code.hpi
curl: (16) Error in the HTTP2 framing layer
15:34:43 Failure (16) Retrying in 1 seconds...
 > configuration-as-code depends on snakeyaml-api:1.26.3
Downloading plugin: snakeyaml-api from https://updates.jenkins.io/2.235/latest/snakeyaml-api.hpi
curl: (16) Error in the HTTP2 framing layer
15:35:39 Failure (16) Retrying in 1 seconds...
curl: (16) Error in the HTTP2 framing layer
15:35:52 Failure (16) Retrying in 1 seconds...
curl: (16) Error in the HTTP2 framing layer
15:36:05 Failure (16) Retrying in 1 seconds...
15:36:06 Failed in the last attempt (curl -sSfL --connect-timeout 60 --retry 3 --retry-delay 0 --retry-max-time 60 https://updates.jenkins.io/2.235/latest/snakeyaml-api.hpi -o /usr/share/jenkins/ref/plugins/snakeyaml-api.jpi)
Downloading plugin: snakeyaml-api-plugin from https://updates.jenkins.io/2.235/latest/snakeyaml-api-plugin.hpi
curl: (22) The requested URL returned error: 404 Not Found
15:36:08 Failure (22) Retrying in 1 seconds...
curl: (22) The requested URL returned error: 404 Not Found
15:36:11 Failure (22) Retrying in 1 seconds...
curl: (22) The requested URL returned error: 404 Not Found
15:36:14 Failure (22) Retrying in 1 seconds...
15:36:15 Failed in the last attempt (curl -sSfL --connect-timeout 60 --retry 3 --retry-delay 0 --retry-max-time 60 https://updates.jenkins.io/2.235/latest/snakeyaml-api-plugin.hpi -o /usr/share/jenkins/ref/plugins/snakeyaml-api-plugin.jpi)
Failed to download plugin: snakeyaml-api or snakeyaml-api-plugin

WAR bundled plugins:

Installed plugins:
configuration-as-code:1.43
snakeyaml-api:1.26.4
Some plugins failed to download! Not downloaded: snakeyaml-api
The command '/bin/sh -c /usr/local/bin/install-plugins.sh < /usr/share/jenkins/ref/plugins.txt' returned a non-zero code: 1

Curl tries to download snakeyaml-api-plugin when snakeyaml-api fails. The latter is not being downloaded due to the error curl: (16) Error in the HTTP2 framing layer
For multiple plugins, I get the HTTP2 framing layer error multiple times. Many a times, the installation gets stuck at a point without any progress
You can look into the shell script install-plugins.sh here
Tested on Windows 10 as well as MacOS Mojave
Docker version -  19.03.12

If HTTP/2 is not working try forcing HTTP 1.1:
COPY plugins.txt /usr/share/jenkins/ref/

ENV CURL_OPTIONS -sSfL --http1.1
ENV CURL_CONNECTION_TIMEOUT 60
RUN /usr/local/bin/install-plugins.sh < /usr/share/jenkins/ref/plugins.txt


Kubernetes create multiple jobs based on template and list We are currently migrating many of our production data ingress/processing scripts/processes into a kubernetes cluster. Each "customer" or company that we interface with has a bunch of data and we want to process all of this data on a per customer basis. I have created a container with a script where I pass a single customerID into. E.G. One container will run and process one customers data. This will need to happen for one or more customers on a schedule.

I have been using the following definition to work specifically for customer 756, and it is working as expected.

apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: image-puller-756
spec:
  schedule: "30 * * * *"
  concurrencyPolicy: Forbid
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: image-puller-756
            image: index.docker.io/my_company/image_puller:latest
            imagePullPolicy: Always
            env:
              - name: ALLOW_TTY
                value: "true"
              - name: PYTHONUNBUFFERED
                value: "0"
              - name: prodDbUser
                valueFrom:
                  secretKeyRef:
                    name: image-creds
                    key: username
              - name: prodDbPass
                valueFrom:
                  secretKeyRef:
                    name: image-creds
                    key: password
            command: ["/bin/bash"]
            args: ["-c", "/usr/bin/python3 -u /go/bin/main.py -c 756"]
          restartPolicy: Never
          imagePullSecrets:
          - name: myregistrykey

Once scheduled it will run hourly which is perfect, however I now need a way to duplicate this definition for ~500 different customers and apply it to the cluster. Currently I am using a template file and doing a Find and replace with a list of the customers I'm testing, but I'm looking for a better way to manage it. The customers will only have certain processes ran for them based on some logic, and I would like to be able to run another process that will act as a schedule and dynamically add or remove jobs for the customers.

I thought that there might be a way using yaml or k8s to pass a list and a template and have it create them. The approach just seems very antiquated and I'm struggling to think that this is the correct one.

Have you checked the documentation for cron-jobs? You can use the run command.

$ kubectl run hello --schedule="*/1 * * * *" --restart=OnFailure --image=busybox -- /bin/sh -c "date; echo Hello from the Kubernetes cluster"
cronjob "hello" created

This way you can loop through your customers in a bash script to deploy your containers.

Not knowing the big picture its hard to answer completely, but I also think ansible templates would work for this. You could turn your above config file into a jinja2 template. You could pass your customers in as a JSON dict and use with_items to loop through them or treat your customers as an inventory. This would give you an easy way to deal with certain "snowflake" customers.

EDIT: as far as doing this with Vanilla Kubernetes, I don't think it's possible. You could add your own custom controller and extend the kubernetes API. This would allow you to creat an API object that defines a customer, and then have a custom controller pick that up and create your cron job. Check out https://github.com/coreos/prometheus-operator as they have implemented such a solution to allow multiple teams to run there own Prometheus solution on the same cluster.


Ideal methods of handling server logs on AWS We currently have some ruby applications hosted on EC2 servers on AWS, and every few weeks we see the disk space get completely consumed by the app log as well as the worker logs.

What are the best practices regarding log storage for a cloud hosted environment such as ours? We've considered log rotation, pushing the logs to CloudWatch or simply truncating log files.

I like to use ELK, Elasticsearch, Logstash and Kibana with Beats.

Filebeat, that belongs to Beats series will forward the logs that read from a file to logstash.

In logstash you could tag, filter, parse and modify the log entries that are stored in Elasticsearch.

For visualizing you can use Kibana. You can create a great dashboards based on the logs with all the information that really gives value to your business

If you don't want to install agents, you can forward logs using rsyslog directly but with al beats series you could get more in deep information rather than simply logs.

This ELK suite could give you much control over your application but you must maintain it