Terraform | The "count" object can be used only in "resource" and "data" blocks, I'm trying to upgrade out Terraform seeing as 0.12 (stable) came out recently. I've ran into some issues with a certain pair of resource blocks. I'm getting the below error message when trying to do terraform plan, if I remove the count index it gives me a different error and says that I need the count index but that gives this error:

The "count" object can be used only in "resource" and "data" blocks, and only
when the "count" argument is set.

Here is the terraform stripped down to only the relevant parts.

resource "azurerm_network_interface" "network_interface" {
    count                             = "${var.vm_server_count}"
    location                          = "${var.location}"
    }
}
Problematic resource block

resource "azurerm_network_interface_backend_address_pool_association" "network_interface" {
  network_interface_id    = "${azurerm_network_interface.network_interface[count.index]}"
  ip_configuration_name   = "example"
  backend_address_pool_id = "${azurerm_lb_backend_address_pool.network_interface.id}"
}

What am I missing here? This is Terraform 0.12 stable using Azure as the provider.

The resource "azurerm_network_interface_backend_address_pool_association" "network_interface" block doesn't have the count argument set, so count.index is not meaningful in that block.

Given the specific pair of resource types you're using here, I expect your intent was to create one association for each network interface, in which case you'd add the following count argument to the second block:

  count = length(azurerm_network_interface.network_interface)

Although this didn't return an error on Terraform 0.11, it did not behave as I expect you intended: it would've created only one azurerm_network_interface_backend_address_pool_association for the first network interface, and left the others unassociated.

If associating only the first one was your intent instead, then replace the count.index in the second block with 0 to be explicit that it's using only the first one:

  network_interface_id    = azurerm_network_interface.network_interface[0]


How to unseal a Vault in practice? It is possible to force that multiple keys have to be entered in order to unseal the vault. How does this work in practice? If someone has entered a key and there is another key required to unseal the vault should one ask a colleague to enter another key and how should this be done to prevent that colleague A or someone else sees that key?

I assume your vault would be being accessed over a network, so you would ask a colleague to use their own command line on their own workstation to enter their key. Mine requires 3 keys to unseal so I just have 3 users independently enter the vault unseal command with their key.


Referencing current variable in Ansible conditional statement? I'm using the willshersystems.users role from ansible galaxy for creating users on ubuntu servers. Now that I've got it working consistently, I'm trying to set the state attribute based on whether the user should have access to a specific server given a list.

Is something like the below possible? My google fu seems to be failing me at the moment, any advice would be very appreciated!

vars:
  user_servers:
    user1:
      - testing1

  users:

    - name: user1
      groups: '{{ user_groups }}'
      state: "{{ ( inventory_hostname_short in user_servers[self.name] ) | ternary('present','absent') }}"

roles:
  - role: willshersystems.users

How about something like this:

hosts: all

vars:
  user_servers:
    user1:
      - testing1

  users:
    -name: user1
    -groups: '{{ user_groups }}'
    -state: present

roles:
  - name: Add filtered users
    tasks:
      - name: Filter users
        debug: User {{ item }}
        with_items: "{{ users | default([]) }}"
        when: {{ inventory_hostname_short }} in user_servers[item]
        register: filtered_users

      - name: Add filtered users
        include_role: wilshersystems.users
        with_items:
        - {{ filtered_users }}
        loop_control:
        - loop_var: users


Is it possible to locate the temp keypair generated by Packer? I`m creating a new image and everything is working. I would like to debug via ssh during the instance creation. A temp keypair is created and attached to the temp instance. My question is, can I get this keypair somewhere in order to debug it?

==> amazon-ebs: Prevalidating AMI Name...
    amazon-ebs: Found Image ID: ami-0866798422f5d546b
==> amazon-ebs: Creating temporary keypair: packer_5cc6c77d-494a-f185-b5b3-f9b59e62fd4e

Best,

Armando

First, make sure that packer runs in debug mode (packer build -debug ...), otherwise the temporary key will not be written to disk but will only be kept in memory.

packer will store the key in its working directory. In other words, look in the same directory from which you started the packer process.

If you use AWS, the file will be called ec2_amazon-ebs.pem. Assuming you are building an Ubuntu image, you should be able to login to the instance with this command:

ssh -i ec2_amazon-ebs.pem ubuntu@<ip>

(where <ip> can be found in the packer output or through the AWS console)


Nginx ingress controller redirects the url to /login I am trying to access a url example.com/service/test1 but it redircts me to example.com/login.

Below is my YAMl file:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: test-ing
  namespace: test
  annotations:
     nginx.ingress.kubernetes.io/ssl-redirect: "false"
     nginx-ingress.kubernetes.io/use-regex: true

spec:
  rules:
  - host: example.com
    http:
      paths:
      - backend:
          serviceName: svc-test
          servicePort: 8080
        path: /service/test
  tls:
  - secretName: tls-secret

It just works

Your configuration is correct.

Provided YAML file generates nginx config like this:

...
server {
 listen       80;
 server_name  example.com;
 ...
 location ~* "^/service/test" {
   root    /var/www/example.com/htdocs;
 }
}

Testing this config via this cool tester shows that location filter works:

http://example.com/service/test1 -> Location: ^/service/test
http://example.com/service/test -> Location: ^/service/test
http://example.com/login -> Errors: No locations matched

For more details, see this explanation of Ingress Path Matching and examples from NGINX Docs:

  Examples show how to use advanced NGINX features in Ingress resources with annotations.
  Examples of Custom Resources show how to use VirtualServer and VirtualServerResources for a few use cases.

Check your app

Check your app, maybe it forward non-authorized requests to /login url.
You may use curl -v or wget, e.g:

curl -v -L http://example.com/service/test1 | egrep "^> (Host:|GET)"

wget  http://example.com/service/test1 2>&1 | grep Location

Howto debug nginx ingress

There is an Nginx debug mode for debugging nginx locations and redirects.
To enable it, use --v=5 in kubectl edit deploy.

See: k8s debug logging:

  Using the flag --v=XX it is possible to increase the level of logging. This is performed by editing the deployment.

$ kubectl get deploy -n <namespace-of-ingress-controller>
NAME                       DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
default-http-backend       1         1         1            1           35m
nginx-ingress-controller   1         1         1            1           35m

$ kubectl edit deploy -n <namespace-of-ingress-controller> nginx-ingress-controller
# Add --v=X to "- args", where X is an integer

    --v=2 shows details using diff about the changes in the configuration in nginx
    --v=3 shows details about the service, Ingress rule, endpoint changes and it dumps the nginx configuration in JSON format
    --v=5 configures NGINX in debug mode


Are there any drawbacks of using a deb package as if it was a container to deploy an application? My team is currently trying to decide if we should deploy our Nodejs app as a deb package instead of trying to run it in a container such as Docker.

I got this idea from reading this blog here which makes some good arguments for using a deb package for a pre-existing python application. The main point from this blog which is appealing to us is the issue of maintaining the Docker ecosystem (port sharing,  permissions, hosting of Docker Images, etc.)

It seems like "dep-packages as the original containers" makes a lot of sense for small services where there is no concern of port conflicts and where all the dependencies are maintained within a virtual environment.

My gut, however, is telling me that if deb packages were a good fit, it would be more common and docker would be advertised as a more language specific solution.  Are there any drawbacks of using something like deb packages to deploy our services, instead of using a full system such as docker?

First, while Docker is sometimes seen and used as a ad hoc packaging system, it actually solves a totally different problem: Docker is about running programs. The Docker system allows to describe services, that can be scaled at will and to control swarms of containers. Debian packages are for installing programs and they are able to handle dependencies between software versions.  Docker certainly don't qualify as a descent packaging system: each “package” can only have one dependency, the system has no “recursive build” option and does not support complex version constraints!

A possible answer would be that, if you are willing to write a Debian package for your application, you can also use Docker to deploy your application.  This can be achieved with a configuration script apt_setup.sh which would look like

apt-key add - <<EOF
-----BEGIN PGP PUBLIC KEY BLOCK-----
<YOUR RELEASE OFFICER PGP KEY GOES HERE>
EOF

cat >> /etc/apt/sources.list <<EOF
deb https://my.organisation.org/repo debian-jessie main
apt-get update -y
apt-get upgrade -y
EOF

and a Dockerfile along the lines of

ADD apt_setup.sh /root
RUN sh -ex /root/apt_setup.sh && rm /root/apt_setup.sh
RUN apt-get install -y my-node-js-package

(In your specific situation, the apt_setup.sh would be more complicated, adding the nodesource repositories and some helper packages such as apt-transport-https.)

It is therefore really possible to use Debian packages and Docker simultaneously, however …

  My gut […] is telling me that if deb packages were a good fit, it would be more common

This is a correct hitch which leads us to ask ourselves why Docker proves to be popular as a ad hoc packaging system, while it is not intended to be one. (See above.)

The “official” packaging system from a given distribution is just a possibility among many others to install software in some computing environment. There are many other sources available, like community-specific package managers such as npm or opam, port trees like pkgsrc and plain source code distribution.  From this perspective, it is easy to understand the success of Docker as an ad hoc packaging system:

Docker specifications are very close from a shell script and whatever source it comes from, we install software using the shell.
Docker has a “built-in” (paying) service for hosting artefacts it produces, the Docker Hub.

Now what are the strength of Debian packages over Docker images as a package system? The tight control over dependencies at installation. (The possibility to upgrade and downgrade also exists but has no practical importance if we are implementing the immutable-server pattern.) This leads to the

Conclusion

If you only have a single product deployed in a single version (which is typical for SaaS), your version management needs are very simple and using Docker as a ad hoc package manager should not have any hard drawbacks.  As soon as you work with several versions of a single product or several products, the complexity of the version constraints problem you need to solve increases and you need an appropriate tool for this, which might be Debian packages or some configuration management system if you are mixing software from different origins.


Update production server (apache, wordpress, mysql) on docker containers, from staging server For our web servers running on Docker containers (Apache, mysql, wordpress, etc) we have a production and a staging virtual machines.

DEVs and content team work on staging and then I tar ball the files from the wordpress container and dump the mysql DB and scp them to production server.
Once in production I untar the files to the wordpress container and "restore" the DB.
(in fact we use a server in the middle, but I'm skipping that here to keep it short)

I was thinking on an internal docker registry, but that's only for images... so I will need to start/restart the images to container.

Or some kind of internal "GIT"? where I can push/pull versions?

what will be a better way to update production server?

My thoughts based on your described scenario are:

Your tarball strategy, as described, might sound safe but is prone to risk in my opinion. A better approach is to version-control the WP files you will modify, while keeping the core intact and within the Docker container. Here's a reference I have used in the past that talks about version-control strategies for WP. You would of course need to spend some time and understand which folders/files to version-control for your specific site.
The version-controlled files would be checked out on the host machine (which runs Docker) and be volume-mounted within your Docker container.
Everytime your changes are validated in Stage they get pushed to Git. Getting these changes into Prod would essentially be a git pull.
If you wish you might also use a tool like composer as part of your deployment workflow. A dependency manager should be part of any deployment worklow.

I would also recommend tying all this together with a tool like Ansible, i.e. use Ansible to

update local git repo
run the composer bits
ensure docker container is running

Disclaimer: Although I've worked on DevOps automation for Wordpress sites, I am by no means an expert on Wordpress, so suggestions and clarifications welcome.

Hope this helps.


Do I need header files once I install the php extention on Dockerfile? Usually when I build docker images for php I use this approach:

FROM  php:7.2-apache

RUN apt-get update && \
    apt-get install -y libghc-postgresql-simple-dev &&\
    docker-php-ext-configure pgsql -with-pgsql=/usr/local/pgsql &&\
    docker-php-ext-install pdo pgsql pdo_pgsql &&\
    // Install some other extentions there
    apt-get autoremove -y &&\
    apt-get autoclean &&\
    rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/* cache/* /var/lib/log/*

But this can result an image of 1.2GB and I need to shave some space. AFAIK header files such as the libghc-postgresql-simple-dev take a lot of space in my image so after building and installing the module I though of removing it and use the non-dev variant libghc-postgresql-simple or removing it completely.

So I want to ask:
 1. Instead of dynamically linking it how I can statically link it on my build once i run docker-php-ext-install in order to remove the libghc-postgresql-simple-dev package?
 2. Would once I build the php module removing the libghc-postgresql-simple-dev and using the libghc-postgresql-simple would make the php psql extention to work as supposed to?

The idea behind it is to optimize the build in order to make my image as small as possible.

You shouldn't need to statically link things to remove the *-dev packages and have everything still work. If you do, then that's a bug in the base package and the -dev package and should be reported upstream. So yes, you should in theory be able to, after installing your extensions and such, do the following and shrink the image significantly:

apt-get install -y libghc-postgresql-simple && apt-get remove libghc-postgresql-simple-dev

There are two other things you can do to improve your image sizes:

Use the --no-install-recommends switch for all of your apt-get install invocations, and explicitly list every package you depend on. Recommended dependencies are a type of optional dependency in Debian packages that are installed by default. They can often take up a lot of space, but you may not actually need most (or any) of them, so you can save space by not auto-installing them and only pulling in what you actually need. This can easily cut down image sizes multiple hundreds of MB.
Use the --purge option whenever removing packages. This will ensure that they leave nothing behind, instead of potentially leaving some config files and other similar stuff, which can also help cut down on your space usage.


Multiple Ingress controllers in Kubernetes We currently have at our company a Kubernetes cluster with an ingress controller that manages the traffic of the services and their endpoints.

We want to change our URL gradually, first add the new URL to point to the same services that are currently being used by the old URL, and once we see that everything works fine, we'll just make the old URL redirect to the new one.

That's the plan in a gist, however, I have no idea where to start. Do I add another ingress controller that is just copied from the one we currently have and update the SSL certificate to match the new URL?

Thanks!

I assume you've created your nginx ingress controller in it's own namespace? By default it's called ingress-nginx.

You can create your second Nginx Ingress controller in a seperate namespace so that it's logically isolated from your first. You'll need to give it a different name, but I see no reason why you can't do that.

Doing so will spin up a seperate AWS ELB, which you can then map to your new DNS record and then you can slowly migrate your services over.


Tool to produce nice output about desired vs. current state I've been trying to stick to the "infrastructure as code" principle as much as possible in my work.
But after the devops project got more complicated I started loosing the centralized perspective because the scope of used technologies and complexity are very rich: terraform, terraformer, terragrunt, pscore, bash, Azure CLI, Azure DevOps CLI, AzureAD module, msbuild, Java, various REST API calls...
So I started writting scripts and code that calculate "desired vs current" state just for validation and overview, if everything is in the right state. Most of the time this information is everything I need, I don't need to "go to the desired state" because sometimes that actually means manual intervention (sometimes even from someone else).
The output of these scripts is simple, currently they write into stdout something like this:
Name of thing         Desired state      Current state     Result
----------------------------------------------------------------------
THING A               This               That              NOT GOOD
THING B               This               This              GOOD
THING C               This               ?                 Timeout after 5 minutes
...

Result:
- 2 things needs attention

My question is if there is some tool or standard targeted at this specific use case?

I have scripts, I could call them "diff calculators" that can output whether some "thing" or "subthing of thing" is in desired state and what is the current state
The tool produces nice table based UI output

I found Pester which is just a powershell unit testing that can save Nunit XML results which can be published inside the pipeline.
It's not "desired"/"current" state I had hoped for but it can in the end provide a nice overview of all the "things" whose states I need to observe.


Is it possible to have multiple gitlab-ci files in a single repo? I have a single repo handling integration tests for 4 different software projects. As those projects share common resources, I'd like to create a generic file in which all those common features are set properly for preparing the CI/CD job.
The project organization looks like the following:

project_root:
  > folder_1
  > folder_2
  > folder_3
  > folder_4

From GitLab's documentation on the include keyword, I understand that it is possible to include a file that would handle the common features of all those projects so I don't have to write those command lines in the 4 differents gitlab-ci.yml. So I thought I could do the following:

project_root:
 common-features-handler.yml
  > folder_1
   gitlab-ci.yml
  > folder_2
   gitlab-ci.yml
  > folder_3
   gitlab-ci.yml
  > folder_4
   gitlab-ci.yml

The thing is, the documentation is unclear on whether it is possible to have multiple gitlab-ci.yml files in the same repo, regardless of them being in separate folders. It just says that this file should be "placed at the root of [the] repository", so I guess this, in itself, excludes the possibility of having several gitlab-ci.yml files in the same repo. I tried including the common-features-handler.yml file in the gitlab-ci.yml from folder_1 but the job didn't even launched. My guess is that, as it was not at the root of the project, it simply wasn't seen by the CI pipeline. This makes me think that it's not possible for one repo to handle several gitlab-ci files, but I couldn't find anything about it in the doc or online to confirm or infirm this hypothesis.

So is it possible to have multiple gitlab-ci files in a single repo in one way or another? If not, is there a way for avoiding writing the same setup command lines four times?

No, you can't have multiple gitlab-ci files per repository.
You can see more information in the following links :

https://gitlab.com/gitlab-org/gitlab-foss/issues/18157
https://gitlab.com/gitlab-org/gitlab-foss/issues/28592

Now, there are some work in progress about this, but it's not available yet.
For example, you can see here ( https://gitlab.com/gitlab-org/gitlab-foss/issues/56214 ) that they are working on pipeline which can include other (complete) pipelines.

The include keyword is more about reducing the size of your main gitlab-ci file.


How can I enable active tracing by default for an AWS Lambda function using Terraform? On the console, there’s the option to enable active tracking for X-ray tracking but I can see this in the Terraform aws_lambda_function resource the aws_xray_sampling_rule resources?

I'm currently manually enabling it in the console to enable me to test it but I'd like to enable it through Terraform when I build the Lambda?

This looks like it was added to Terraform back in May 2017, you enable it with the tracing_config block:

tracing_config {
  mode = "Active"
}

You will also need to make sure that Terraform has a minimum of Write access to X-ray to enable the above to work:

data "aws_iam_policy" "aws_xray_write_only_access" {
  arn = "arn:aws:iam::aws:policy/AWSXrayWriteOnlyAccess"
}


TLS handshake failure We have an ODBC Driver trying to connect through TLS , but fails.
We tested using openssl with the following command and worked fine:

openssl s_client -connect xxx.com:31020

the server accepts TLS 1.0 to TLS 1.3 (downgraded just for debugging)
While Trying to connect , here is the wireshark handshake capture flow:

and also while expanding the client hello TLS, this is what we have :

Any idea what's causing the handshake not to start properly?

Since we are using ISTIO as Ingress Gateway with Kubernetes, the issue is related to ISTIO gateway config.
We simply had to replace the matching host to '*' in ISTIO gateway config


How to do blue-green deployment with PL/SQL code? Is it possible to do blue-green deployment when your software is mostly a set of PL/SQL packages?

I don't really want to duplicate the database, the financial risks in case of any desyncs are too big. I understand that step 1 is to decouple schema changes from package changes, but how to proceed further?

Is splitting packages into a separate DB that I can spawn multiple copies of the only option?

Yes, it is possible, but it takes a bit of work. I assume that the concern with duplicating the database is 1) time and 2) size/storage, otherwise it seems like it would be trivial to spawn databases for testing against. Based on the fact that you are targeting PL/SQL i'm guessing you are working with Oracle.
You will either need: 1) a smaller-scale dataset that can act as a representative sample of you data which is sufficient for testing all of your corner cases or 2) to clone the whole production database. Honestly, I recommend option #2. There are numerous cases where things operate correctly at a smaller scale, but run into issues once scaled up - especially with database data. Ideally, you could store and replay a sample of queries for load testing using a database proxy like ScaleArc (if Oracle won't let you store your queries).
Saying you need to clone your whole dataset may make me sound like a kook and like it will take forever, but you may find this answer in which I detail a methodology for cloning multi-terabyte databases in seconds without significantly increasing your storage costs due to duplication and delta-snapshotting useful. This may be harder if you are using Oracle's proprietary storage system. Be forewarned, the tech isn't cheap, but if you wanted cheap, you wouldn't be using Oracle. The best way to sell that to your upper management is to prove that the tech is cheaper than the outage/data mangling it would prevent. It's your insurance policy against outages - so how much insurance are you willing to buy?


AWS lambda: works locally but fails in deplyment I have a node.js lambda function that adds a database entry and then calls another lambda function. The code works fine when I test locally via

serverless invoke local --function create --path mocks/create-event.json

but it fails when I invoke the deployment

aws lambda invoke  --function-name create --payload file://mocks/create-event.json out.txt

here is the relevant lambda code:

try {
    await dynamoDbLib.call("put", params);  //OK locally and in deployment
    var lambda = new AWS.Lambda();
    const req = {
      FunctionName: 'eqtor-api-dev-mysim',
      InvocationType: 'RequestResponse',
      LogType: 'Tail',
      Payload: JSON.stringify({ name: 'me' })
    };
    const result = await lambda.invoke(req).promise(); //OK locally but not in deployment
    console.log(result);
    return success(params.Item);
  } catch (e) {
    return failure({ status: false });
  }

The command shows

{
    "ExecutedVersion": "$LATEST",
    "StatusCode": 200
}

and in out.txt, I get

{"statusCode":500,"headers":{"Access-Control-Allow-Origin":"*","Access-Control-Allow-Credentials":true},"body":"{\"status\":false}"}

Any tips on how to debug this?

I've resolved the issue now.

First of all, console.log(e) should be used to log the result. Then the error will appear in CloudWatch.

The problem was that the role that was used by the lambda did not permit invoking other lambdas. This can be fixed by granting the missing InvokeFunction permission for the role.


Jenkins parameters default to env variable? I am trying to crate parameters field which is default to current build no.

parameters {
    string(
        name: 'DEPLOY_BUILD_NUMBER',
        defaultValue: '${BUILD_NUMBER}',
        description: 'Fresh Build and Deploy OR Deploy Previous Build Number'
    )
}

But it looks like the assignment of BUILD_NUMBER env var to parameter DEPLOY_BUILD_NUMBER is not happening.

even following one didn't help

steps {
        script {
            if (params.DEPLOY_BUILD_NUMBER == null){
                DEPLOY_BUILD_NUMBER = env.BUILD_NUMBER
            }
        }
}

Is there any other way to do this ?

You need to use double quotes instead of single quotes.  Single-quoted strings don't perform string interpolation.

For instance:

parameters {
    string(
        name: 'DEPLOY_BUILD_NUMBER',
        defaultValue: "${BUILD_NUMBER}",
        description: 'Fresh Build and Deploy OR Deploy Previous Build Number'
    )
}


PTC with Jenkins on Deployment based on the Build request which raised on PTC I've build which is checked-in on the PTC Integrity and I've raised the CR or SR ticket(Build & Deploy Request) on the PTC life manager by developer for deploy his build on some XXXX server and the version of the build is xxxx.

As of now as a deployment team we are following the below process

If we get a ticket we assigned it to us.
And then collect all those details which is on the ticket by manually and doing deployment also by manually.

Instead of this now we are planning to do this on automation.

How do I get the following details on the Jenkins to do my deployment. I've some bat script to do the deployment. To kick off that bat file, it needs the following details:

Version of the build
Which server the Build needs to deploy
Where the build file is located

I need to collect all those above details from the PTC ticket which I had created earlier
Is there any possiblity from Jenkins can I able to get those values from the ticket number.

I'm very new to PTC.
I came to know from google there is a plugin available for PTC with Jenkins

Could someone please help on this

You can pass your data using environment variables or through job parameters.

Here's an example of setting environment variables in a pipeline:

pipeline {
    agent any
    environment {
        FILE = "Makefile"
        SERVER = "example.com"
        VERSION = "v1.1.0"
    }
    stages {
        stage ('build') {
            steps {
                echo "VERSION is: $VERSION"
                bat "script.exe ${version} ${server} ${file}"
            }
        }
    }
}

If you want to use parameters instead you could do so in the following pipeline example:

pipeline {
  agent any
  parameters {
    string(description: 'file', name: 'file')
    string(description: 'server', name: 'server')
    string(description: 'version', name: 'version')
  }
  stages {
    stage("foo") {
      steps {
        echo "VERSION is: ${params.version}"
        bat "script.exe ${params.version} ${params.server} ${params.file}"
      }
    }
  }
}

See the following documentation for building a parameterized pipeline:
https://github.com/jenkinsci/pipeline-model-definition-plugin/wiki/Parametrized-pipelines

If you're using a freestyle job it's slightly different but you can find detailed instructions here:
https://wiki.jenkins.io/display/JENKINS/Parameterized+Build


Running Protractor tests in docker through teamcity I have a project for which I am using Teamcity to automate deployment, and I would also like to automate testing.

It is an angular frontend, and we have some protractor tests.

I would like to run these tests as one of the steps in our build configuration.

I would like to have these tests run inside a docker container instead of the teamcity agent.

I am able to run other steps using docker containers, so I have teamcity and docker working as expected.

My main problem here is being able to run this within the docker container. I would like to have a container that would be able to run without having to rely on external containers, however it seems that the containers I have found and tried using rely on an external selenium instance. Either I am misunderstanding how this should run, or I would like a self contained container to run the tests and drop dead afterwards.

Bear in mind that these tests can be run locally in our development workstations, with selenium and protractor running locally. I would like to achieve the same through a docker container, without that container having to rely on an external selenium server instance.

I have tried the following docker containers:

https://hub.docker.com/r/caltha/protractor/
https://hub.docker.com/r/webnicer/protractor-headless/

When I try to run the tests, they both complain selenium server could not be found.

This is my protractor configuration file:

// conf.js
var spec_files = "tests/*.specs.js";
exports.config = {
  framework: 'jasmine',
  seleniumAddress: 'http://localhost:4444/wd/hub',
  specs: [spec_files],
  multiCapabilities: [{
    browserName: 'chrome'
  }],
  baseUrl: 'test',
  restartBrowserBetweenTests: true
}

The baseUrl parameter is fed into protractor with the following command:

protractor conf.js --baseUrl="http://[Replaced with Environment domain name]:8080"

What am I missing here?

I have been able to figure out how to get everything to work.

Let's start from the beginning:

Changes to config file

Remove the selenium server settings (the docker container starts one on every run, and uses that automatically)
Add 'useAllAngular2AppRoots' otherwise the tests will all fail

The resulting config file will be something like this (ignore the params section, that is just an object with additional parameters you can inject into your tests):

// conf.js
var spec_files = "tests/*.specs.js";
exports.config = {
  framework: 'jasmine',
  specs: [spec_files],
  useAllAngular2AppRoots: true,
  multiCapabilities: [{
    browserName: 'chrome'
  }],
  restartBrowserBetweenTests: true,
  params: {
    sleepTime: 0
  }
}

Setting up the build step in teamcity

Usually when I want to run a build step I do it in a docker container, and execute command line steps inside that container.

To get this to run I could not do this, as we need to inject a few parameters at the end of the run command, so you need to create a docker command, looking like this:


How to change PagerDuty incident name? I have AWS Cloudwatch hooked up with PagerDuty. Whenever an alert goes out, it goes to PagerDuty, which then sends us a message in Slack. The problem is, this is what it shows in the chat:

  Triggered #3362: Maximum CPUUtilization of 53.6 GreaterThanOrEqualToThreshold 50.0 for InstanceId i-xxxxxxxxx

That's about as useful as a rock. There's no way I could interpret the instance ID, nor should I have to read all that stuff at a glance and figure out what's wrong.

The question is, how do I change the incident name that gets generated on PagerDuty, or is this something that can be done in Cloudwatch, or am I hosed?

Within the CloudWatch integration in PagerDuty, you can change the "Derive name from" field to something like "Alarm Description".


Does Jenkins job create Jenkinsfile? On execution On creation of a new job(as free style project), and running it, I see that Jenkinsfile is generated in job's  workspace, after running the job.

But I see that all the pipeline code is commented. This is declarative syntax pipeline code.

So,  apart from scriptlet generator option,

1) Does Jenkins(by default) create Jenkinsfile with DSL syntax in its workspace? For its job...

2)
If yes, Can Jenkins generate scripted pipeline in it's workspace? Groovy syntax...

No, Jenkins file is not created at execution.
Actually Jenkins file is a suite of all steps or activity which you want to build as a job or a groovy code.

Jenkins file come from code repository, if you added file in your project.

Below is a Jenkins file example:

pipeline {
    agent any

    stages {
        stage('Build') {
            steps {
                echo 'Building..'
            }
        }
        stage('Test') {
            steps {
                echo 'Testing..'
            }
        }
        stage('Deploy') {
            steps {
                echo 'Deploying....'
            }
        }
    }
}

for more reference watch this video: https://www.youtube.com/watch?v=KZY4Rwsewro


How to implement DTAP in k8s? Dev, Tst, Acc, Prd (DTAP) is about isolating multiple environments to ensure that data will not be mixed, e.g. production data in development or lost, e.g. database removal removes all production data. DTAP is also about environments that are identical, e.g. if a database in prd, then there should be a database in dev as well, but the data should not be mixed. It is about prediction. The more the environments are identical the bigger the change that if a certain change works in dev, then it will work in tst, acc and prd.

Discussion

There are at least two options, i.e. one k8s cluster or multiple.

one k8s cluster

Lower costs
Dangerous as pods from production environment could be removed, while the intention was to removed dev pods.

multiple k8s clusters

Higher costs
Lower change that data will be removed accidentally due to isolation

Kubernetes supports namespaces.

  multiple virtual clusters backed by the same physical cluster

These namespaces can allow you to use one cluster for all your environments. You can also do fancy things in your CI/CD pipeline so that each branch or each commit gets its own namespace. Those pods in prod namespace can still speak to pods in dev namespace unless you segment with NACL's. Just make sure you CNI supports that.

That being said most large companies I have worked for are running a cluster per environment.

I don't think there are right or wrong answers here. Do what you think is best for your company/team/workload/environment etc.


Download an artifact from Nexus to Jenkins I use ssh username and private key auth to download sources from Git, build the artifact and deploy it to my nginx server. Now I need to download an artifact as a zip file to Jenkins agent, prepare it and deploy to my nginx server. I can only access Nexus using username/password pair.

Is there a way to run scp mynexus.com/my_artifact.zip ~/my_artifact.zip and pass username with password from Jenkins credentials?

There are a few ways to do this, but by far the easiest is to put your credentials (either username/password or ssh keypair) into the Jenkins built-in credentials store and then use the sshagent step in your Pipeline script:

sshagent(credentials: ['my-credentials']) {
  sh('scp mynexus.com/my_artifact.zip ~/my_artifact.zip')
}

You may need to install the SSH Agent plugin to be able to use this step; I'm not sure if it's part of the default set of Pipeline plugins or not.


Jenkins Groovy to parse console output and mark build failure What would be a Groovy or pipeline step to parse Jenkins console output, identify a string and mark the build as failure if the string is matched?

This question seems similar to another on StackOverflow, but seems to be a close, but not exact duplicate of
https://stackoverflow.com/questions/37018509/jenkinsfile-build-log

To quote those :

  if you just want to check, that your log contains string myTestString you can just call manager.logContains('.myTestString.')

  If you want to get some information from the first matching line you can use manager.getLogMatcher(regexp)

You could use the Groovy flow control to break the build - you just need a step to execute the test:

Following the example on the link :

node {
    stage('CheckLog') {
      steps {
        if (manager.logContains('.*myTestString.*')) {
          error("Build failed because of this and that..")
        }
      }
    }


One Repository for Kubernetes configuration or one configuration per application repository We are currently migrating our Services to a Kubernetes environment.
My question is if the configuration yaml file for each application

should be in the repository where the application source itself is stored,
or if there should be a centralized repository just for all the configuration files.

Currently I'm drifting more towards a centralized repository holding all configs, as dependencies on different versions of applications / services can be managed there. Another thing is that a change in the configuration should not trigger a rebuild of the application itself and this is also easier if the configuration and the application repositories are not the same.

If you're just migrating I'd also recommend using helm.
When using helm create the chart definition within the source code repository of the application, https://helm.sh/docs/helm/helm_create/.
With an increasing number of services it can become harder and harder to manage so people have started to embrace a "monorepo" that contains a definition of the all the helm charts they currently have deployed and their respective versions. This way you can easily track when a version of service A was bumped and when a version of service B was bumped. https://www.infracloud.io/blogs/monorepo-ci-cd-helm-kubernetes/


How can I calculate costs of deploying 1000 instances of Drupal? What are the things that should I investigate for making an estimate of the cost of deploying 1000 instances of Drupal, for example?

I was thinking in:

Database space
Database usage (in ops/second? how can I test it?)
Application uploads space
Application processing time (again, how to measure?)
Bandwidth usage (for medium/low traffic sites)
DevOps / Technical Support time

This is all about bringing a SaaS platform. Is there anything else I should take in consideration?

How can I know how many "instances" would be able to run in a single host? Much like a Shared Hosting

Would Amazon EC2 / Google / Azure be cheaper than having dedicated servers?

In reply to 'Would Amazon EC2 / Google / Azure be cheaper than having dedicated servers?'

I've done a lot of investigating into this and in every case they are NOT.

80 Linux servers running 24h/365 - annual cost:
        Total cost per year
Google  £37,515.46
AWS     £29,196.53

These costs do NOT include data transfer.

You can buy a dedicated cloud appliance for less than £50k, fully supported, that will run more servers and last you 3 years (min).


How to identify the physical IP of where Kubernetes API server runs Question

In a Kubernetes cluster, there is a Docker network IP which looks to be the API server. However, not sure how to identify the physical IP and location of the box in which it is working.

If there is a guidance how to proceed, it would be appreciated. I have been handed over the system but no information.

$ curl https://172.30.0.1
{
  "paths": [
    "/api",
    "/api/v1",
    "/apis",
    "/apis/apps",
    "/apis/apps/v1beta1",
    ...
  ]
}

$ kubectl get pods -o wide
NAME                    READY     STATUS    RESTARTS   AGE       IP
logging-fluentd-2vc4v   1/1       Running   0          1d        10.1.13.155
logging-fluentd-bvdtz   1/1       Running   0          1d        10.1.3.3
logging-fluentd-xvxrk   1/1       Running   0          1d        10.1.10.71

$ sudo iptables -t nat -L -n
Chain KUBE-SERVICES (2 references)
target                     prot opt source               destination
....
KUBE-SVC-NPX46M4PTMTKRN6Y  tcp  --  0.0.0.0/0            172.30.0.1           /* default/kubernetes:https cluster IP */ tcp dpt:443
KUBE-SVC-BA6I5HTZKAAAJT56  tcp  --  0.0.0.0/0            172.30.0.1           /* default/kubernetes:dns-tcp cluster IP */ tcp dpt:53
....

$ ip route
default via 10.0.4.1 dev eth0 proto static metric 100
10.0.4.0/22 dev eth0 proto kernel scope link src 10.0.4.21 metric 100
10.1.0.0/16 dev tun0 proto kernel scope link
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1
172.30.0.0/16 dev tun0 scope link

The apiserver runs on master nodes. To get a list of all the master node IP's, this should help-

kubectl -n kube-system get po | grep -i apiserver | cut -f1 -d" " | xargs kubectl -n kube-system get po $1 -o=jsonpath="{.items[*].status.hostIP}"

Based on the version of Kubernetes you are running, you might need to update the location of the hostIP in the jsonPath parameter.


Modelling economic value of risk-reduction DevOPs efforts A significant part of the activity of software Engineers working on automated testing, continuous delivery, service integration and monitoring, and building SDLC infrastructure strives to reduce operational risks lying on the system they are working on.  To estimate the economic value of these efforts require a risk model. As a first approach, it seems reasonable to throw an ad hoc model where each risk occurrence is governed by a Bernoulli variable (roughly speaking a coin that can be weighted to fall more or less on a given side).  But how to find sensible parameters for this model.

To take a concrete example, if I develop a disaster recovery procedure and run drills each month to prove it working, I would need to estimate the odds of my company going bankrupt because it is unable to recover quickly enough from a disaster, and I was unable to find any sensible data on this.

Is there any sensible risk model, together with its parameters, that can be used to estimate the value of efforts invested in risk-reduction?

You are probably unable to find data on this for two main reasons. Firstly, because it varies from system to system. Does an outage cost you money due to contractual obligations? Or only in lost reputation and in user perception? How much does that cost? This varies from organization to organization. But I think that might be what you are getting at with the Bernoulli variable...

...Which brings me to the second thing. In order for you to be able to model risk, you need research in which someone has already statistically proved or measured the efficacy of DevOps methodologies (which means you have to define exactly what DevOps is first, a notoriously tricky issue.)

You also need to be able to compare that to old methods - meaning that someone must also have measured data for the Dev/Ops segmented model. I'm not sure that research exists or is settled. Probably because the efficacy and cost of Dev/Ops will also vary widely from organization to organization. In that respect it may provide an opportunity to measure the old methodology and then show the average improvement of changing methodologies if you can find enough research subjects.

In short, it is doubtful that such a formula for risk modeling exists. If it does, it likely exists as paid marketing research to a consulting firm and is unlikely to be available to the public and I would personally be very dubious of the accuracy of it's results due to the subjectivity of the input variables measured.


Does Immutable Servers imply no source code updates on the servers? So once an Immutable Server is in place, nothing should change at all. Or is it only configuration.

Are source code updates acceptable on Immutable Servers?

No.

Immutable is exactly what it means, immutable, no change on configuration or code running or system library or whatever, if a change has to be made, create a new image and deploy it, never change it while running.

Source code updates are the least thing to change on a running server, this should not happen on a running server, immutable or not.

You should create a versioned release of your code to deploy on servers, once you have this level of fixed versioning you can step ahead to versioning your server image which include this code release.

Any change, being a code patch or a system patch or a configuration change should trigger a new release to be deployed.


Let docker containers work with each other I’m new to docker and I want to dockerize a aleardy existing application.

It consists of a react application (I already created a container for that one successfully), a REST-api based on node-express (that one kinda works) and a mysql-database (not workin so far).

While developing I put a proxy into the package.json file of my React-Application. This enabled me to forward all the api-requests to a different port. And I want to also have this feature between my docker containers. So that my react application on port 3000 for example forwards all the unknown requests to the rest api on port 5000 via a proxy. Port 3000 and 5000 are in different containers.

Furhtermore I need the REST-container to have access to port 3306 of another mysql-container. This other mysql container still needs to be created though. It should use hypriot/rpi-mysql or something similar and within the dockerfile it should import a given .sql file into the database and then run in the background.

So my questions are: What would be the best way to share ports among the containers?
How can I set up a Dockerfile for a MySql container with a .sql file as setup?

What would be the best way to share ports among the containers?

You don't really "share" ports between containers. Instead, you want to create network and attach each container to that network.  Each container gets a network alias (essentially a hostname) that you can use to hit the service.

$ docker network create foo
$ docker run --network=foo --network-alias=mysql -p 3306:3306 my-mysql:latest
$ docker run --network=foo --network-alias=restapi -p 5000:5000 --link mysql my-restapi:latest
$ docker run --network=foo --network-alias=react -p 3000:3000 --link restapi my-reactapp:latest

So the myrestapi container can access the mysql container using the hostname mysql and port 3306 and your react app can hit the myrestapi container using the hostname restapi.  The containers will also be accessible from your localhost so you can hit your react app with a browser without additional setup

In docker-compose.yml (which I don't use a ton so someone may have a better solution), it would look something like this:

networks:
  foo:
services:
  mariadb:
    container_name: mysql # this works for your alias so long as you don't have another container called mysql
    image: mariadb:latest
    networks: ["foo"]
    ports:
    - 3306:3306

The other services would follow suit.

As for "priming" the database, that's a little trickier. If the DB has to be running, you would need to replace the docker container's entrypoint to start the server, run the mysql command to import the data, then your script would just spin forever.

Alternately, you could script your container start-up and after it starts, import the data that way.


How to find the constraint in a software development shop? Let us define a constraint as anything that prevents the system from achieving its goal and then look at the goal of DevOps in an organization.

A good definition is "DevOps enables a fast flow of features from development to IT operations to the customers." Improving DevOps can be achieved by removing impediments to flow.

First, we must find these constraints in the flow. And maybe there are multiple obstacles, how can we decide on which one to remove first?

A famous quote from Dr. Eliyahu M. Goldratt is

  Any improvement not made at the constraint is an illusion

This constraint, once found, is an important focusing mechanism for management of the system. So, how can you identify the constraint in an IT organization? And what to change about it to get the benefit of better flow?

To identify a constraint in any process is a relatively simple task. Work moves from a person to person throughout the organization and it will simply pile up in front of a constraint. You can look for people or teams with the highest number of blocking unresolved issues for example. It depends on how you track the flow of work, but if you do track it, then identifying constraint is usually not too difficult. It helps to know the value chains for your organization and simply follow along them.

Constraints can be of three types:

People: A particular skill or knowledge unique to an individual.
Policy: A written or unwritten policy or process, which limits the work either by increasing latency or reducing throughput.
Equipment: Particular computer system can limit the speed at which you can pass work through it or the number of people that can use it at the same time.

In regards to software, the Software Delivery Process is often a constraint and thus Continuous Integration / Continuous Delivery is such a powerful way to elevate this constraint and subordinating the organization to this constraint is often needed to increase the throughput of work. By automation in Software Delivery you remove slow people and often costly policy obstacles and approvals. Equipment in form of Computer System upgrades is a relatively small cost comparatively to the other two.

QA is another example of possible constraint in the process. Manual quality controls are work intensive and even automated testing can be constraint by equipment, in case the software is to run on expensive hardware. But the most common constraint in QA is policy, which tends to snowball with ever problem. The latency of the QA process and the small throughput can lead to larger units of work being put through, which can push back the cost into code integration.


How can I list all environment variables configured in Terraform? I configured AWS provider in Terraform by providing an access key and secret as environment variables according to the documentation:
$ export AWS_ACCESS_KEY_ID="anaccesskey"
$ export AWS_SECRET_ACCESS_KEY="asecretkey"
I now want to view those variables on the console as well as other environment variables configured on my system but I can't find the right command.
Update: I found out that I didn't configure the AWS provider like I thought. The AWS CLI was already configured with my credentials stored in AWS credentials file: ~/.aws/credentials.

On your command line, run printenv
Reference: https://man7.org/linux/man-pages/man1/printenv.1.html


Is scrum or kanban really useful for SRE teams? Agile practices like scrum and kanban were primarily designed for software development.
Interrupt and unplanned work is a significant component of what most SRE (Site Reliability Engineering) or DevOps teams do.  While it is always useful to use a tracking system like Jira to manage work, do sprint or kanban really work for SRE teams?
The constraints I see are:

The work is very dynamic in nature, with priorities changing on a daily basis.  Because of this, the sprint duration of two weeks seems very aggressive and it adds unnecessary overhead.
People being on call adds another dimension to the problem. Sometimes, more than one team member might get involved in on call / post-mortem tasks.
The team doesn't have a single "product" and hence it doesn't yield itself to a common planning process
Daily standup meetings may not make much sense because of the lack of overlap among tasks
The team might be working on tasks related to more than one partner teams and hence spanning multiple Jira projects.  Since a sprint or kanban board allows only one Jira project, it may not be able to fit in all the work.

From what I hear from many SREs that I have spoken to, sprint planning hasn't worked for them at all.  I would like to hear from the community here what their experience with sprint and kanban is.
I asked this question on scrum.org as well:
Can scrum be used effectively by SRE teams?
And here is a blog post that raises concerns about Agile and SRE in general:
Agile is Lava

We don't use Agile for the DevOps group ourselves, but we do integrate with the normal Scrum Teams. When something is needed by the team from DevOps, such as optimizing the build server, the related team puts a PBI in their backlog with a 'DevOps' label. Our lead has a custom dashboard in Jira with all issues labeled 'DevOps'. They work with the Scrum Master to get prioritization and then one of the DevOps Engineers is tasked with being an ad-hoc member of that Scrum Team for the life of that issue. This helps us prioritize work based on our "customer's" priorities, ties our effort to the sprint, and allows us to get "credit" for the work we're doing.

Prior to this, we used Kanban on Jira just for a todo list. Unfortunately, we would sometimes have to stop working on something we had planned to work on because one of the teams needed something. Now, unless it's an emergency, they basically request DevOps resources (people/time) via their backlog and the Scrum Master communicating that need to our Lead.


Ansible ad-hoc command to reboot servers in serial with a pause How do I go through all hosts in an Ansible inventory in serial with an Ansible ad-hoc and reboot servers with a specified number of seconds in between each reboot?

I am looking for something similar to:
ansible -i inventory nodes -a "/sbin/reboot"

It should have a pause and run in serial.

@Argyle explained the --fork 1 trick to achieve a serial run.

I suggest you have a look at the reboot module that might be better suited than running a command directly.

A quick example that will reboot all your nodes one by one waiting for each to be fully booted before moving to the next one.

ansible -i inventory nodes --forks 1 -m reboot

Pass needed parameters to the module depending on your requirements with the -a option.


Cannot access service using docker-swarm, but without Currently I try to setup docker-swarm using two nodes and deploy portainer on them.
Both nodes are virtual machines on the same host and are on the same network (172.16.255.255). Addtionally there is a firewall in these local network providing a VPN server (255.255.0.0). The VPN routes requests to 172.16.255.255 in the local network (e. g. 10.0.0.3 can access 172.16.0.80). No ports are blocked.
I have no trouble accessing the virtual machines (e. g. by ssh).
I followed the install installation instructions on portainer.io.
Compose File:
version: '3.2'

services:
  agent:
    image: portainer/agent
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - /var/lib/docker/volumes:/var/lib/docker/volumes
    networks:
      - agent_network
    deploy:
      mode: global
      placement:
        constraints: [node.platform.os == linux]

  portainer:
    image: portainer/portainer-ce
    command: -H tcp://tasks.agent:9001 --tlsskipverify
    ports:
      - 9000:9000
      - 8000:8000
    volumes:
      - portainer_data:/data
    networks:
      - agent_network
    deploy:
      mode: replicated
      replicas: 1
      placement:
        constraints: [node.role == manager]

networks:
  agent_network:
    driver: overlay
    attachable: true

volumes:
  portainer_data:

Run  command:
docker stack deploy --compose-file=portainer-agent-stack.yml portainer

The stack is successfully deployed and I can access the portainer UI as long as I'm in the local network. The UI is even accessible from other virtual machines, which are not part of the swarm.
But when I try to access the UI from my VPN, I get a timeout (SYN, but no ACK).
This happen for other container as well (e. g. I tried to test it with jwilder/whoami for different ports).
When I setup a container, with identical settings, without swarm, the service is accessiable via VPN.

The problem was that the vpn network has the subnet 10.0.0.0/8 and the ingress network of docker swarm has the subnet 10.0.0.0/24.
So there were a conflict.
Fix:
Changing either the vpn subnet or the ingress network subnet.


Compound calculation of GCP GCE SLAs based on number of instances? GCP publishes a 99.99% uptime SLA for GCE.

How would I calculate an estimated SLA if my application used services deployed on 1,000 GCE instances? Assuming that the services were dependent on each other, would the estimated downtime per year of my application be 1,000 * 0.0001 = 0.10 or 10% per year?

99.99% SLA uptime roughly translates to 52.5 minutes a year of downtime (calculator here).  Generally if the service is down, it is down for everything in that geographical area.  So if you had 1000 servers in the same geographical area and that area goes out of service, your downtime would be tied together.

However, most of the "big" cloud providers (e.g. Google) provide their services across different regions and zones.  By spreading your infrastructure out across different areas, you can make your network more resilient to cloud provider outages.


How should we automatically rebuild immutable infrastructure when new packages are available? We're going to be using Terraform to automate our infrastructure deployment and Packer to create the machine images deployed by Terraform. By following immutable infrastructure design principles, we will implement patching by creating a new image with the patch applied and then redeploy our infrastructure.

With this setup, are there any additional tools we can use to automatically detect when a package or the OS itself in our base image needs updating and trigger the build pipeline?

Chef Automate seems close to what I'm looking for, however, it seems to scan running nodes for compliance rather than analyze the image manifest itself.

Thanks!

Part of adopting the Immutable Infrastructure Pattern is decomposing your system into small manageable pieces that can move through CI/CD Pipeline very quickly, this means that OS patches can be done quickly and in a controlled manner.  I often see clients ending up with a halfway house where infrastructure is mostly immutable.

However, there are a few approaches to this which I have used in large-scale deployments of cloud architecture, typically I implement more than one as part of a Defense in Depth strategy:

Security Information and Event Management (SIEM): These products, for example, LogRhythm Security Intelligence Platform and more generic products such as ElasticStack take feeds from the operating system, which includes the output from a frequent check for updates.  The trick here is to get the information about what to patch quickly and automatically to inform decisions about when to roll those through your pipeline.
Vulnerability Management systems are more tailored than SIEM in that they are only focusing on vulnerabilities across the system so could catch issues with libraries installed as part of the software deployed to the system but not managed by the Operating System.  This might highlight vulnerabilities for which there is no patch (yet... hopefully).
Dependency Check tools form part of your pipeline and can be configured to fail the build if check-ins add vulnerabilities, this also works if a new vulnerability is added to the tool since the last check-in.


Defined start sequence of containers AFAIK you cannot influence the order how Docker Swarm starts containers for services.

Is that possible with:

Mesos?
K8s (Kubernetes)?
Rancher?
Other?

This question is not about specific orchestration of dependent services but the problem that if you run 10 services each with one JVM this creates a load peak. The system slows down and you need complex loops to wait for other services.
Are there any mechanisms known in other orchestration concepts to manage the startup somehow?

For example, in completely other domain, in JMeter you can tell a timeframe in which a thread group will start. So could I say "try to start this service with some offset"?

https://forums.rancher.com/t/start-order-of-stack-containers/3106/9

  We do not support depends_on, and neither does Docker in Swarm mode.
  It is not a real solution to the problem anyway and leaves you with
  unhandled pointy-edge cases when failures occur and containers are
  being replaced. Your services should know how to either wait for their
  dependencies or exit and be rescheduled to retry.

I agree with this comment. Services should always be able to start. Imagine that one service goes down then another service will not work anymore.


Different execution contexts for same POM - use properties files? Imagine that a Maven POM configures a set of integration tests for web apps (could be a functional test as well).

in developer context, you would clearly use Chrome and configure that accordingly
in the CI, you want a headless test with another browser driver

Same would be if you decide to configure all fixed dependencies' versions in on place, and for a specific stage/release.

For such cases, what is the best strategy to have environment-specific POM properties files for Maven, and is it a good approach?

I'm not an expert on this, but maintaining separate pom files seems like a recipe for things to get out of sync.

It would make more sense to define specific dependencies within the same pom.xml and execute them depending on specific goals or targets.  I.e. -Pdev.

You could also then define a list of shared dependencies used by all targets (i.e. chrome driver or what not), and specific dependencies for each target.


Does a slot swap cause an application to restart when using deployment slot settings? My web application in Azure has two deployment slots - staging and production.
The purpose of the application is to draw a shape. The version currently deployed to the production slot draws a square. The version deployed to the staging slot draws a circle.
Both versions of the application have an application setting called Colour which controls the colour of the shape. Importantly, this is a deployment slot setting - see the example capture below:

The value of this setting is blue in the staging slot and green in the production slot.
When I view the application in the staging slot, a blue circle is drawn. When I view the application in the production slot, a green square is drawn.
When I perform a slot swap from staging into production and view the application in the production slot, a green circle is drawn.
I am struggling to get my head around this. A big advantage of swapping deployment slots is that I am able to swap my pre-warmed staging code into the production slot and prevent users from suffering from downtime or cold-start performance issues. However, by swapping the staging application into the production slot which has different values in the deployment slot setting, haven't I made changes to the application configuration which will trigger a restart? If I was to modify web.config manually or update the deployment slot setting in Azure, the application in the slot would restart. How is slot swapping getting around this?

if both are running 'warmed up' and in 'Always On' mode then the swap would not cause a restart of any kind. But go to this URL and read the point #1 and if any of these things are done then it will restart.
https://docs.microsoft.com/en-us/azure/app-service/deploy-staging-slots#what-happens-during-a-swap


What Ansible does in cases there's a conflict between OS version and some software Ansible manages? Say I use an IaaS cloud on which I installed Ubuntu 18.04 manually via the cloud company's OS installation tool and I ought to establish an all-ansible LAMP environment that will include some more stuff besides Apache, MySQL, and PHP (like `php-cli, php-curl php-mbstring, php-mcrypt, php-gd, Certbot, ssmtp and so forth) -- all installed with Ansible as well, of course.

So far so good, but what if after 4 or 8 or 12 years the particular Ubuntu rlease (18.04) becomes too old for a newer version of whatever software I installed with Ansible and also and constantly being upgraded by Ansible?

How Ansible itself deals with such a rare (?) conflict, does it usually do some "rollback" to a "previous state" (just like an autosave and autoload for video games) and also stops to work (and maybe also prompts me) until I myself decide if I want to remove this particular software (or softwares) that cause trouble and then start the Ansible process again?

Ansible is at its core a programming language - a meta programming language written in python compiling itself into python and shell scripts mostly. It will do whatever you program it to do. It all depends on how your roles and playbooks are written. It certainly can be done - almost anything can be done - your install roles could track dependencies on package versions or ranges, obviously those might need to be updated over the years, but you can be safe if you put the work into it.

It is probably unreasonable to plan further than the End of Life of the LTS distribution though, as you will fail to receive new security updates and that instantly invalidates the system as production ready.


Can a 5 US $ VPSE support VAD stack with 2 relatively-small CMSs? I'm new to DevOps and quite troubled financially but I desire to put websites that I now host on a shared server environment (CetnOS 12.x.x) on some cloud like DigitalOcean or Lindoe (but without using their droplets with ready OSs - I want to install the OS myself with Vagrant).

To have full automation of every upgrade possible (even OS itself say Ubuntu 18.04 to 20.04 or Apache 2.4 to Apache 3.4, PHP 7.4 to PHP 8.4 and so forth) I desire to use the VAD stack: Vagrant-Ansible-Docker.

DigitalOcean's cheapest plan for example is this 5$ plan:

25 GB SSD
1 virtual CPU
1 GB RAM
1 TB monthly data-transfer

I use the Drupal CMS (>=8) for generally all my websites (which are generally small-scale perceptional websites for small business of free-professions).

Each VPSE will contain up to 2 separate (non multi-stie) Drupal apps and each such app will usually take no more than 256 MB RAM, if I'm not exaggerating).

My question

Can I base these such a X2 drupals installment on such 5$ VPSE with a VAD stack in decent performance? Or there's no way I could handle both VAD and X2 such Drupals on one such small VPSE?

In this scenario, the cpu cycles required to read from the database and render to PHP will all be handled by a single core, ie they will be handled one after the other. This will be about the slowest way to deliver website traffic out of your server.

Each time a request is made, a certain amount of memory is occupied by PHP. You can test the actual amount, but you can only ever be sure it is not greater than memory_limit. Based on a use of no more than 256MB per Drupal site request, you can divide your 1GB of memory by that number (minus some memory for apache, MySQL and the OS). The result are 3 requests fit into memory only. You certainly do not want to allow more, because either your system runs out of Ram or it will use swap disk space.

The use of swap disk space is even worse, as it will slow down requests even more and consumes cpu itself.

3 requests can be used up quickly by some Ajax requests in the page or your Admin backend, either one likely bringing the server to its knees. It would be best to check in Chrome inspector how many PHP requests the site makes on frontend and backend.

Your situation will improve somewhat if you can dial down your PHP memory limit to say 64mb, but you still have the single core as the bottleneck.

So the key to keep this stable is to reduce the number of requests allowed either in apache or php-fpm to a number multiplied by PHP memory limit that stays smaller than your 1gb memory.

Your best option would be to copy out your Drupal pages to a static website every time a change is made. A transparent way for this is to put a cdn in front of your website. However you need to be sure.

Another non technical practical solution is to quadruple your hosting budget and find a sponsor or side gig to pay the charges.


Stress testing for docker container I want my container to die under high pressure to check monitoring and alerting as well as compose autorestarts. What is best way to simulate heavy load within docker container. I've tried so far

root@docker-vm:/# stress --vm 10 --vm-bytes 2048M --vm-hang 20
stress: info: [238] dispatching hogs: 0 cpu, 0 io, 10 vm, 0 hdd
stress: FAIL: [238] (415) <-- worker 243 got signal 9
stress: WARN: [238] (417) now reaping child worker processes
stress: FAIL: [238] (451) failed run completed in 101s

for memory pressure (hoping OOM kill for host machine which has 4Gb of ram) and stress --cpu 100 which is obviously not really harmful.

Is there some tool to achieve that for docker or preferred approach I should use?

Just a quick follow-up on pumba. Installation (for Ubuntu):

$ curl -SL https://github.com/alexei-led/pumba/releases/download/0.7.2/pumba_linux_amd64 -O
$ sudo mv pumba_linux_amd64 /usr/bin/pumba
$ sudo chmod +x /usr/bin/pumba
$ pumba --version

It uses stress-ng under the hood, so same commands apply. Examples:

$ pumba stress -d 1m container_name

-d 1m - duration of a stress test (1 minute)

container_name - your target container (from docker ps output)
$ pumba stress -d 30s --stressors "--vm 10 --vm-bytes 512M --vm-hang 20" container_name

-d 30 - duration 30 seconds

-stressors - parameters passed to stress-ng app

In this particular example we create 10 workers spinning on malloc()/free() allocating 512MB each and sleep 20 seconds before freeing the memory.

Default stressor as in first example is --cpu 4 --timeout 60s


Is there one website that contains all relevant DevOps news and gets updated on a daily basis? I am looking for some DevOps news channel like https://nos.nl/teletekst#101 that shows the headlines of the Dutch news. I am aware of https://tweakers.net/ and github.com, but this does not contain all interesting news on one page.

Basically, I would like to receive updates regarding security, features, bugs about several products, e.g. k8s, docker, docker-compose, ubuntu, cloud, AWS, GCP, Azure, centos.

There is a protocol for this: RSS, Rich Site Summary. Technically, there are 3 protocols RSS, RSS2 and Atom, but most readers support all of them, and many servers mix RSS and RSS2 features.

First, choose your RSS reader, from:

browser add-ons like Feedbro,
centralized aggregators like Feedly and FeedReader,
stand-alone applications like RSSOwl
mail clients like Thunderbird.

Second, choose the sources you care about and subscribe to them - you're the curator and most news websites publish RSS feeds. Many don't even know about it, as CMSs like Wordpress sometimes publish an RSS feed by default. I wouldn't be surprised if it had something to do with SEO... Soon, you'll have an OPML list of your favourites.

From your question:

https://kubernetes.io/feed.xml
https://www.docker.com/blog/feed/
https://ubuntu.com/blog/feed
https://aws.amazon.com/blogs/devops/feed/
https://devblogs.microsoft.com/devops/feed/
https://cloudblog.withgoogle.com/products/devops-sre/rss
https://blog.centos.org/feed/
https://github.blog/feed/

And a few personal recommendations:

https://www.theregister.co.uk/devops/headlines.atom
http://feeds.arstechnica.com/arstechnica/security
https://stackoverflow.blog//feed
https://xkcd.com/rss.xml
https://www.theregister.co.uk/data_centre/bofh/headlines.atom

Many readers will have an option to search for a feeds on a website, which simplifies the search.


Why isn't my S3 Cross Region Replication working for some files? I've followed the documentation to set up cross-region replication to another AWS account.  It looked like it was working correctly until I noticed an entire prefix was missing.

Here is my destination bucket policy:

{
    "Version": "2008-10-17",
    "Id": "S3-Console-Replication-Policy",
    "Statement": [
        {
            "Sid": "S3ReplicationPolicyStmt1",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::<awsaccount>:root"
            },
            "Action": [
                "s3:GetBucketVersioning",
                "s3:PutBucketVersioning",
                "s3:ReplicateObject",
                "s3:ReplicateDelete",
                "s3:ObjectOwnerOverrideToBucketOwner"
            ],
            "Resource": [
                "arn:aws:s3:::bucket-backup",
                "arn:aws:s3:::bucket-backup/*"
            ]
        }
    ]
}

Looking further into it, I have also realized this particular prefix is where we store our public-access data, so maybe that's related somehow?

So I verified it was definitely public access causing the issue somehow; uploading an identical file but with the public read acl on totally broke it.

I worked with AWS support and they noticed I had "Block public access" enabled in the destination bucket.  Apparently rather than replicating the object and then making it private, it just fails replication.

tl;dr if you want to replicate objects that are public read, don't block public read at the destination.


You can add an entry in your ini file to set the sort_order for the callback plugin to none.

[callback_profile_tasks]
sort_order = none

By default it is set to descending, which is why the tasks are displayed in decreasing order of time taken.

Refer: https://docs.ansible.com/ansible/latest/plugins/callback/profile_tasks.html#parameter-sort_order


How to let build fail when docker image size exceeds certain threshold? docker images returns:

REPOSITORY            TAG            IMAGE ID       CREATED             SIZE
<none>                <none>         fcc125757e66   About an hour ago   270 MB

Attempts

docker images | awk '{print $6}' returns:

SIZE
ago
hour
ago
ago
ago

Expected outcome

270 so this could be used in an if statement:

if [ VAR > 250 ] then
   echo "Image too large"
   exit 1
fi

The output of docker images is really intended for humans, and it's not very parse-friendly. Instead, most Docker commands support a --format flag using Go templates. As you can see in the images documentation, you only really care about the .Size value, so you really want this:

docker images --format "{{.Size}}"

This just gives an output like this:

90.6 MB

If you have multiple images, they will each be on a separate line. I'm going to assume you will only ever have one line, because it vastly simplifies the next steps.

To get rid of the MB, we can simply use sed:

docker images --format "{{.Size}}" | sed "s/MB//g"
# => 90.6

But, you probably don't want this for your bash script. Most shells don't support floating point comparisons, so inputting 90.6 would just fail. Instead, let's use bc to truncate the number first:

docker images --format "{{.Size}}" | sed "s/MB/\/1/g" | bc
# => 90

Note that I've replaced the MB with a division by 1, which truncates in bc.

You can then pipe it into a shell script. I tested with this script:

VAR=$(cat)

if [ $VAR -gt "250" ]
    then
    echo "Image too large"
    exit 1
fi

Then run with:

docker images --format "{{.Size}}" | sed "s/MB/\/1/g" | bc | ./check_size.sh
# Exits with 0 (for my example)


Jenkins declarative pipeline: use branch if it exist I'm currently trying to adopt my declarative pipeline in the following way:

I have multiple stages, some apply for all branches, some only for specific names.

What I'm now searching for is a way to get a when condition for the existance of a branch like this:

if branch b exist
   checkout branch b
else
   checkout branch a

Sadly I haven't found a solution for it. I'm pretty sure there is a way using the git plugin though.

There's a StackOverflow solution for scripted pipelines, but I can't translate them though.

Can someone help me with this or point me at the right direction?

There's a resolveScm step implemented by the Pipeline: Multibranch plugin:
checkout resolveScm(source: [$class: 'GitSCMSource',
                             credentialsId: '<credentialsId>',
                             id: '_',
                             remote: 'https://your.git.remote/repository.git',
                             traits: [[$class: 'jenkins.plugins.git.traits.BranchDiscoveryTrait']]],
                    targets: [BRANCH_NAME, 'master'])

Above procedure will look whether $BRANCH_NAME exists, check it out if yes and check out master otherwise.


Jenkins: Access job/plugin configuration values inside pipeline I am trying the access the values set on a job's configuration page from within my pipeline.  These values are not made available as params, nor are they injected as envvars.
Setup

Jenkins, v2.263.1
GitLab Branch Source plugin, v1.5.3 (link)
Multibranch pipeline job which is pointed to a Gitlab repo
Remote Jenkinsfile Provider, v1.13 (link)

Problem
Ordinarily, one would have a Jenkinsfile in the root of the repo and therefore the scm would be associated with the repo we want to checkout and build.  However, in my case the code I want to build is in a different repo to the Jenkinsfile (hence the Remote Jenkinsfile Provider plugin).
This means that I need to checkout the code I wish to build as an explicit step in the pipeline, and to do that I need to know the repo.  This repo is, however, already defined in the job config.
The Branch Source plugin does export things like the branch name or merge request number/branch/target into appropriate envvars, but NOT the actual repo.
As this is a multibranch pipeline, I cannot use something like envInject either (multibranch jobs do not provide the option to 'Prepare an environment for the run' as with other jobs)
Goal
I would like to be able to access the server, owner and project fields set in the job config page. Ultimately I could manage with just the project's ssh/http address even.
Is there some clever way of accessing a job's config from within the pipeline?
Thanks for any suggestions!
Reference images

This is possible, but you'll have to use scripted Pipelines, not declarative, disable the Groovy sandbox for pipelines, and dig deep into the Jenkins API.
For instance, I wrote a Pipeline job that edits the parameters of a second job which has a full name of "SCA/deploy_to_environment/master".  Here's a snippet of that code - I'm eliding how to create the "Param" objects in the first line for brevity, this snippet is just to give you an idea of what a similar use case looks like:
def paramProp = new ParametersDefinitionProperty(envParam, pgParam, infoParam)

def job = Jenkins.instance.getItemByFullName('SCA/deploy_to_environment/master')

// There is no "setProperty" - we need to remove and replace.
// removeProperty is safe to run even if no matching properties are set on the job.
// However, this method only removes one property per method call.
// Theoretically, this means the job could accumulate multiple conflicting
// ParametersDefinitionProperty properties.
// TODO: run removeProperty in a while loop to eliminate all matching properties for extra safety.
job.removeProperty(ParametersDefinitionProperty)

job.addProperty(paramProp)

So to fetch the values instead of simply editing them, you would need to instead call getProperty (API doc here), passing the class name of the GitLab Project parameter classes. I suspect the class name you'll need is available somewhere in the source code or the API docs for that plugin, but I'm not sure.

However, I have to ask, why do you want to do this?  What's the big picture here?
It sounds kind of like you want to share a single Jenkinsfile or build pipeline among multiple repositories.  I also have this use case at my job - we need to have one Jenkinsfile shared across nearly 200 repositories.  If this is similar to your need, then consider using Jenkins Shared Libraries.  You can put all of your build pipeline code into a single shared library, then call the shared library in a single line from the Jenkinsfile in your individual repositories.


How to kill all Docker containers based on a given image? How to kill all Docker containers based on a given image?

As an example:

sudo docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                    NAMES
58251f04892a        mariadb:10.4        "docker-entrypoint.s…"   53 seconds ago      Up 52 seconds       3306/tcp                 optimistic_ramanujan
2a08ee7d9afe        mariadb:10.3        "docker-entrypoint.s…"   2 hours ago         Up 2 hours          3306/tcp                 thirsty_taussig
65abd7a3b3e1        mariadb:10.3        "docker-entrypoint.s…"   2 hours ago         Up 2 hours          3306/tcp                 angry_lederberg
32076aa4745e        mariadb:10.3        "docker-entrypoint.s…"   2 hours ago         Up 2 hours          3306/tcp                 distracted_kapitsa

I would like to kill all containers based directly on the mariadb image (ideally, both 10.4 and 10.3).

The following solution does not work since it will kill all containers having mariadb:10.3 as an ancestor--and not only those directly based on that image:

sudo docker kill $( sudo docker ps --filter ancestor=mariadb:10.3 )

This will do want I want: killing all containers directly based on mariadb regardless of their tag--and without killing containers having mariadb somewhere higher in their ancestor list.

sudo docker kill $(
  sudo docker ps --format '{{.ID}} {{.Image}}' | awk '$2 ~ /^mariadb:/ { print $1}'
)


what do you think of making fast decisions? I am trying to convince my fellow workers and management as part of DevOps transformation to adopt "fail fast" in almost everything. Amazon has two principles "bias for action" & "are right a lot" which I belive are quite similar. my dilemma is espically when adopting a new solution. as an Enterprise, they tend to evaluate several products first, then run many proofs of concepts, then an open bid. which is a waste of time and resources. my take is that decide on one solution quickly based on technical and financial merits, start using it if you fail, you can select another technology, reversing a decision on technology is not a big deal, isn't DevOps about being Agile and having quick feedback loops? I would like to hear your input.

start using it if you fail, you can select another technology

This approach has two major drawbacks:

Technical debt

While it may be tempting to apply "fail fast" mentality to architecture and design, skipping or abbreviating the design phase would be a mistake. fast-fail should be used in conjunction with good design practices, not in lieu of them. Making hasty decisions can often result in poorly scaled solutions. While using MySQL seems like a good idea when you are working with a 100mb database to generate reports, it scales poorly into the terabyt range for doing your archival reports.
Using DNSMasq to run an overlapping DNS Zone might seem like a good political decision at the time, migrating off of that and resolving the disparity to between your DNS Zone and the overlapping upstream zone a few years later turns into a nightmare.

Expense

Technology can be expensive. Buying a solution only to discard it later cares with it hugely wastful expenses. But even more expensive is the time expended by personnel. Salaries are every companies' largest expense. Wasting time developing a solution only to discard it and develop a replacement is wildly inefficient.

This however should not be construed to mean that decision paralysis or analysis paralysis isn't a problem. It is! Fail-fast just isn't the best way to combat it. There are other, much more tried and true ways to combat this without comprimising good design principles.


How to handle resource creation? We're moving from a .NET monolithic application to a microservices architecture hosted on Azure.

Until now, we only had one build definition on TFS and a release definition for each environment (development, staging and production).
All the Azure resources needed to deploy to each environment were created manually. We want to automate this part.

How's the recommended approach to handle new resources setup? Do you just put steps in the release pipeline that will create the needed resources when they don't exist? Or do you have a separate pipeline that's solely responsible for creating all the resources for a new environment?

You can use Terraform for azure to handle the dynamic creation of azure resources in the release pipeline in Azure DevOps. Terraform maintains a state file for all the created resources after a run.

If the script is run multiple times, it ignores already created resources unless there are any properties updated and created new resources.

FYI, follow below link for a quick example
Terraform example


Gitlab: Always having the possibility to launch a manual job (even if something failed before) I'm using gitlab-ci. I have a pipeline that has the following steps :

code-verification

some tests

deploy (manual)

some more tests

rollback (manual)

Both the deploy and the rollback job have to be launch manually which means that I'm using when: manual in my gitlab-ci.yml file.
However, I also want the rollback to be possible even if a step/job has fail before (for example, a faulty deploy or a failing in the "some more tests" stage).
I was thinking of using when: always but since I'm already using the when: manual, it doesn't work (I tried).
For now, I just put the "some more tests" and the "rollback" in the same stage so that the failure of one doesn't impact the other. But is there another way to do this?
I really wish to have everything in the same pipeline (so no trigger) and to always have the possibility to launch the rollback job. No matter what happened before.

You can use when: on_failure, in your case you can create a new job for rollback when your some more tests have failed.
Example:
cleanup_build_job:
  stage: cleanup_build
  script:
    - cleanup build when failed
  when: on_failure

For more example: https://docs.gitlab.com/ee/ci/yaml/#when


Why many companies prefer Jenkins than Aws CodePipeline? I have searched few job sites. Many companies are searching engineers who can handle Jenkins as CI platform.
I think the Aws CodePipeline is easier than Jenkins.
Anyone can advise me, please.
Thank you in advance.

Jenkins has been around for longer so it has a larger installed base.  It doesn't matter if AWS has something that is cheaper and better.  Many places don't want to put the time and energy into upgrading Jenkins.  Migrating to a new CI pipeline is even "scarier" -- regardless of whether it would be a wise strategic decision.

Jenkins 1.x made this far worse because of needing extensions for basic functionality.  Different departments wanting different plugins that wouldn't play well together led to multiple Jenkins instances.  Even if the technical challenge of merging these were relatively minor, the chance of getting departments that could never agree on anything before to suddenly decide that they can migrate to some new thing -- that is missing some feature from a plugin they can't live without -- seems pretty slim.


How to get CPU and Memory usage of nodes in a specific Kubernetes instance group? I have a couple separate instance groups, and kubectl top nodes will display the CPU and Memory usage of all nodes. I'd like to be able to just ask for the information for a specific instance group if possible.

Do your instance groups have labels identifying the groups they belong to? If so, you can filter to them using the -l/--selector option, like so:
kubectl top nodes --selector instance-group=group-1

This assumes that the label name is instance-group and the label value for the group you want to query is group-1.


How can I model this schema in MySQL? I want to create a schema that reflects the following relationships, and I'm trying to do this in MySQL Workbench. (I'm no DB expert)

I have 3 environments - sandbox, staging, and production
I release a product version in each environment, e.g., sb-1.0.0 for sandbox, st-1.0.0 for staging, and pr-1.0.0 for production. I can have multiple releases per environment, but there can only be ONE version that's currently released in each environment.
Each product version in each environment contains a set of services, same set of services in all the environments, but differing in their version numbers. So sb-1.0.0 = [svc1-2.0.0, svc2-2.0.0], st-1.0.0 = [svc1-1.0.0, svc2-2.0.0], and pr-1.0.0 = [svc1-1.0.0, svc2-1.0.0]. As you can see, I have the same 2 services in all environments, but the service versions can be different (or the same) in each  environment.

I want to accomplish multiple things.

I want to be able to create new releases in each environment, with each release containing the set of services and corresponding versions. So I want to be able to create a pr-2.0.0 = [svc1-2.0.0, svc2-1.0.0] version.
I want to be able to query my DB for version pr-1.0.0 and know all the services and their respective versions in this particular release.
I want to have the notion of a "current version" and a "desired version" for each service in each environment.
I want to be be able to then run a script (Kubernetes?) to release each service and their respective version based on the query.
I want to be able to extend it to more environments, e.g., add a UAT environment.

So how do I model my DB?

Have you thought about what happen with stale/old data??

Each new version creates a ton of new entries.
Management becomes a nightmare
the DB becomes a SPOF(single point of failure).

Better Solution: Service discovery
The better solution would be to use a Service discovery System and annote each Service with their version and the environment as a Metadata tag. This keeps the System more manageble.
More Help
IMHO, sticking Linux Code into a Docker Image and run multiple copies of it, is the best way of getting a immutable infrastructure up and running.


Is it possible to read secured keys from aws-secrets-manager without using aws access and secret key? I've created a local development environment for new developers coming to work in the company I work for.

One of the microservices running as part of this local development environment requires access to some AWS s3 bucket.

Up until today, I required each user who used this solution to get AWS access and secret keys by himself but lately the company has implemented Okta SSO solution and now there's a new motivation to use one single dedicated AWS access key through Okta which has read-only access to the required resources in AWS.

I want to avoid hard coding this AWS access key in my code and instead, I want to store the access and secret keys in AWS Secrets Manager and then using an assume role to read the keys from there but the problem is that I have to authenticate somehow against the company's AWS account in order to be able to assume the role and read the keys from AWS SM.

Is there any way to do this?

create a service which authorizes against SSO and the provides temporary credentials via AWS STS Service. That way you don't have to store anything anywhere and once the user loses access to SSO they will also lose access to AWS. You need a role with the access to your dev environment and the developers will assume that role.


How do I list pod name/version across multiple clusters in Kubernetes I have multiple clusters that I want to query and in each cluster I want to find and list the versions of the pods running in there.

I've written a shell script that sets the namespace and then changes context to each cluster and then runs kubectl describe pods | grep "/version=" -B1 in each cluster.

This seems to be a very hacky and clumsy approach to this problem. Is there a succinct command in the Kubernetes realm that I can issue that will take a list of clusters and return the name/version of each pod?

I feel that there are 2 problems described here and help with either or both would be appreciated:

How do I issue commands like kubectl describe pods to multiple clusters at the same time and aggregate those results?
What is best way to list names with versions of each pod in a cluster?

Update: I built an assumption into the questions that the answer would be a scripted kubectl command but it does not need to be. The objective is to find all the name/version combos of all pods running in listed clusters.

The best I have found so far for part 2 is to use the JSON output of get pods:

kubectl get pods -o json | jq '.items[] | select(.metadata.labels."app.kubernetes.io/name") | { name: .metadata.labels."app.kubernetes.io/name", version: .metadata.labels."app.kubernetes.io/version" }'

produces

{
  "name": "db",
  "version": "0.0.1"
}
{
  "name": "app",
  "version": "0.0.1"
}


kubernetes cluster node with 4cpu can't hold two pods with 2cpu I have a Kubernetes Cluster on GCE (Google Cloud Engine) (Also called GKE - Google Kubernetes Engine).

The Cluster is configured to have 4cpu per node. The pods I am using request 2cpu. I expect to be able to have 2 pods per node. But instead each time I launch a new pod, it also provisions a new node (am only getting 1 pod per node).

What is happening, How can I more efficiently use my resources? (I don't want to pay for nodes that are under-utilized)

Node:

  Machine type
  n1-highcpu-4 (4 vCPUs, 3.6 GB memory)

Pod (Yaml):

resources:
  limits:
    cpu: "2"
    memory: 1000Mi
  requests:
    cpu: "2"
    memory: 1000Mi

EDIT:

As mentioned below in the comments; it seems there are some 'default' pods that run on the node which utilize some small amount of CPU. For me there were two service pods taking up 100mcpu each. Additionally it seems there is a small overhead regardless of those small pods when viewing my node resources I see:

Resource type   Capacity   Allocatable
CPU             4 CPU      3.92 CPU

This means that each node in my cluster will only have 3.72 CPU that I can distribute to my pods. (3.92 allocatable, - 100m CPU per service pod (x2) = 3.72)

Knowing this, I can understand why my desired deployment will not work.

However, now this raises new questions; How can I most efficiently allocate pods that need 2cpu. Since GCE only lets me provision nodes with even numbers of cpu (2,4,6,etc) I will always have almost 2 wasted CPU. Is there a better solution?

EDIT2:

Settings my pods to use 1.8 CPU allowed me to deploy 2 pods per node. (and for now is good enough)
It would be desirable to be able to anticipate the overhead on nodes. For example, in my use-case above, allocating 4CPU per node, I should be able to know ahead of time that after the service pods and allocatable CPU are considered I will only be able to use 3.72 for my own pods. But if I now want to provision a new Node with 8CPU I will need to check these numbers all over again.

It seems like you found your answer but let me add a little color since we went through the same thing.

Your yaml for the pods can have both a request and limit for both CPU and memory.  Since you are discussing CPU here I'll stick with that.  What you can do in order to utilize your resources efficiently is set your resource request to the minimum requirement to get scheduled. Say 1.8.  But also set the limit to a higher limit, say 2 or more.  What this will do is make your pod be schedulable on the machine and if CPU is not completely consumed, your pod will utilize the available CPU up to 2 or the limit you set.  This is the way kubernetes can make good use of the resources by "sharing" CPU to the application that needs it.

See here for requests and limits:
https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/


Jenkins Pipeline GitLab API Credentials Management I have a Jenkins pipeline that executes a build whenever a change is pushed to the appropriate branch. Jenkins uses an API token to authenticate with the GitLab server and checkout the repository. Besides creating a "Jenkins" user account in GitLab and using that user's API token, is there any other way I can manage GitLab authentication for an automated CI/CD pipeline? Each user of Jenkins currently has their own API token.

Are you doing the building using web hooks or just polling? I think it's inevitable that you'll have to create a Jenkins account to manage API access. This Gitlab link explains how to use the Jenkins Gitlab plugin to manage building when changes are pushed. You'll see they use a Jenkins account as well. This is common practice in my organization to remove single points of failure, such as a departing employee.