Azure Dev Ops delete releases not going to production I have several release pipelines that build to a UAT environment on a continuous integration.  They await deployment to Production on an approval.  A new release cancels the previous Production deployment.
Is there a way I can set retention so these deployments that will never be pushed to Production go away?  I'd like to still retain the previous production deployments for roll-back purposes.

The interface is a little vague, but if I go to Pipeline > Edit then click "Retention" at the top, I can set values there that work.
Note: The updates/clean up only happen at... some time???  I'm not 100% sure when, but builds aren't immediately cleaned up if the policy changes.
The Days to Retain a Release box and the Minimum Releases to Keep influence each other, and the system seems to keep the build if either meets the criteria.
So, I set the UAT stage to 1 day / 1 release, so it always keeps AT LEAST 1 build, and always keeps all builds for AT LEAST 1 day.
I set the Prod stage to 3 day / 3 release, so it always keeps AT LEAST 3 builds, and always keeps all builds for AT LEAST 3 days.
Now it's much cleaner:

I hope this helps someone in the future! :)


Jenkins view is getting duplicate every container restart I'm using Jenkins container and facing an issue that each container restart, the RunningJobs getting duplicate:

Any idea what is the root cause and how can I avoid it?

The solution was to remove the directory init.groovy.d from Jenkins home folder (inside the container)


Ansible: When 'directory/file exists' in a loop I have written some Ansible code to put users from the users.yaml file in the variable 'nexus_local_users' to be used by a role. When doing this a random password is created and written to the 'credentials/{{ user }}/password.txt' file.

I'm looking for a condition that can make sure that once the user is created it won't be added to the variable again, to make sure the user isn't added twice. My first thought to achieve this was to check if a password has been generated already.

Is there any way to make sure a user is only added to this variable once and not on consecutive runs?

Here is a snapshot of the code I am using:

    - name: Include users.yaml and setup all vars for nexus
      include_vars:
        file: users.yaml
        name: users
    - set_fact:
       tmp_user:
        - username: "{{ item.username }}"
          first_name: "{{ item.first_name }}"
          last_name: "{{ item.last_name }}"
          email: "{{ item.email }}"
          password: "{{ lookup('password', 'credentials/' + item.username + '/password.txt length=15 chars=ascii_letters') }}"
          roles: "{{ item.roles }}"
      loop: "{{ users.nexus_local_users }}"
      register: tmp_users
    - set_fact:
        tmp_user: "{{ item.item | combine(item.ansible_facts.tmp_user) }}"
      with_items: "{{ tmp_users.results }}"
      register: tmp_users
    - set_fact:
        nexus_local_users: "{{ tmp_users.results | map(attribute='ansible_facts.tmp_user') | list }}"

I'm looking for something like this but can't figure out how to implement it (the when condition):

    - set_fact:
       tmp_user:
        - username: "{{ item.username }}"
          first_name: "{{ item.first_name }}"
          last_name: "{{ item.last_name }}"
          email: "{{ item.email }}"
          password: "{{ lookup('password', 'credentials/' + item.username + '/password.txt length=15 chars=ascii_letters') }}"
          roles: "{{ item.roles }}"
      loop: "{{ users.nexus_local_users }}"
      when: **directory/file credentials/{{ user }}/password.txt exists**
      register: tmp_users

Thanks in advance!

Gijs

You should first stat for the password file, and then set the fact depending on whether the file.stat.exists or not. And both those tasks need to be performed for each user.

As of Ansible 2.4 you can dynamically include_tasks and perform a loop on it. Meaning that you can group a few tasks in a file and loop those tasks for every value in your loop-variable.

I think this should be able to solve your problem by creating a separate tasks file containing these tasks that should be run for each user:

Stat for the password-file:

   - name: Check if password-file exists for user
     stat:
        path: "credentials/{{ item.username }}/password.txt"
     register: password_file

Next you can define your tmp_user var when the password.txt-file exists:

   - name: Set fact for user
     set_fact:
       tmp_user:
        - username: "{{ item.username }}"
          first_name: "{{ item.first_name }}"
          last_name: "{{ item.last_name }}"
          email: "{{ item.email }}"
          password: "{{ lookup('password', 'credentials/' + item.username + '/password.txt length=15 chars=ascii_letters') }}"
          roles: "{{ item.roles }}"
      when: password_file.stat.exists
      register: set_tmp_user

Then, if previous task defined a new fact tmp_user, that fact should be added to the tmp_users list.

   - name: Add user to list
     set_fact:
       tmp_users: {{ tmp_users + [ tmp_user ] }}
     when: not set_tmp_user is skipped

(not sure by head if you can use when: set_tmp_user is changed but not is skipped should do the job)

Then in you main tasks-file, you create an empty tmp_users list and call an include_tasks in a loop to loop through the above tasks:

   - name: Include users.yaml and setup all vars for nexus
     include_vars:
       file: users.yaml
       name: users

   - name: Initialize tmp_users
     set_fact:
        tmp_users: []

   - name: Set user facts
     include_tasks: set_user_facts.yml
     loop: "{{ users.nexus_local_users }}"

Disclaimer: I have not been able to test this out, or check its syntax, but I assume the general idea of which I try to accomplish here should be clear?

Also be aware that you currently can't perform a loop inside the included tasks file as that loop will by default overwrite the item-variable defined by the tasks-loop. See defining inner and outer variable names with loop var to solve this.

And an additional tip: add a debug task as first task to your included tasks file to output the current item.username in the loop.


Is this an appropriate architecture, or can improvements be made? Due to a combination of business/enterprise requirements and our architect's preferences we have arrived at a particular architecture that seems a bit off to me, but I have very limited architectural knowledge and even less cloud knowledge, so I would love a sanity check to see if there's improvement that can be made here:

Background: We are developing a replacement for an existing system that is a complete rewrite from the ground up.  This requires us to source data from an SAP instance through BAPI/SOAP Web Services, as well as use some databases of our own for data not in SAP.  Currently all of the data that we will be managing exists in local DBs on a distributed application, or in a MySQL database that will need to be migrated off of.  We will need to create a handful of web applications that replicate the functionality of the existing distributed app, as well as providing admin related functionality over the data we control.

Business/Enterprise Requirements:

Any databases we control must be implemented in MS SQL Server
Minimize the number of databases created
Phase 1 will have us deploy our applications to Azure, but we need the ability to bring these applications on-prem in the future
Our Ops team wants us to dockerize everything as they feel it will make their management of the code a lot simpler.
Minimze/eliminate replication of data
The coding stack is going to be .NET Core for microservices and Admin apps, but Angular 5 for the main front-end application.

From these requirements our architect came up with this design:

Our front-ends will feed from a series of microservices (I use that term lightly as they are 'Domain' level and rather large), which will be broken into Read Services and Write Services in each domain.  Both will be scalable and load balanced through Kubernetes.  Each will also have a read-only copy of their database attached to them within their container, with a single master instance of the db available for writes that will push out updates to those read only copies.

(Sorry for the poor quality images, I'm redoing them from memory since, naturally, there's no actual documentation for this stuff except in the architect's head)

Service to Service communication will happen through a message queue that each service will listen to, and process any relevant messages.  The primary use of this will be for email generation, as there isn't anything else we've identified that requires service to service communication for information yet.  Anything "business logic" related that would require multiple services being involved would likely flow from the front-ends, where the front-ends would call each service individually and deal with atomicity.

From my perspective, the thing that rubs me the wrong way is the read-only db instances spinning up inside the docker containers for the services.  The service itself and the db would have drastically different demands in terms of load, so it would make a lot more sense if we could load balance them separately.  I believe MYSQL has a way of doing that with master/slave configurations, where new slaves can be spun up whenever load gets high.  Especially while we have our system in the cloud and are paying for each instance, spinning up a new instance of the whole service when we only need another db instance seems wasteful (as does the opposite, spinning up a new db copy when we really just need a web service instance).  However, I don't know the limitations of MS SQL Server for this.

My largest concern is around the MS SQL Server implementation.  Coupling the read only instances so tightly to the services feels wrong.  Is there a better way to do this?

NOTE: I asked this over on software engineering and they pointed me here.  Sorry if this is not the appropriate SE.

Also there's no MS SQL Server tag

My largest concern is around the MS SQL Server implementation. Coupling the read only instances so tightly to the services feels wrong. Is there a better way to do this?

Essentially what you have designed is a caching system - the service containers have a local copy of the data presumably so that for reads they don't have to make an extra network trip.

As you've pointed out, a more standard approach is to have a cluster of read replicas that all of the containers can read from.  This allows you to scale them separately from the application servers, which is good, because they generally need different things (do you really want to allocate large amounts of RAM to every application container?).  This will add network calls for database reads, but until that's proven to be an issue I wouldn't complicate the architecture to solve it.

If it does become an issue, a much more lightweight way of handling the problem is to run an actual cache locally, like memcache or redis.  You can tune the TTLs on individual objects to be appropriate, and it will automatically drop off rarely-requested data to keep the application server light.


How to organize infrastructure as code projects? I have completed an Infrastructure-as-Code project using Terraform and Ansible. Currently, I organize my project as follows:

Project1
    Terraform
      ec2.tf
      rds.tf
      …
      variables.tf
      scripts\<some script files>
      templetes\<some tfl files as templates>
    Ansible
      Playbooks<some playbook yml files>
         playbook1.yml
         playbook2.yml
         ……
      ansible_hosts
      vars\<some variable files used by vars_file>
      files\<files to be copied to remote hosts>
      templates\<jinja2 templated files>
      roles\<some roles such as webserver, dbserver, etc.>

Note I did not layout Ansible code using the following links yet:

https://leucos.github.io/ansible-files-layout
http://docs.ansible.com/ansible/latest/playbooks_best_practices.html

e.g. I did not put vars/, templates/, files/ to individual role folder since Most of my roles did not have vars and templates. Besides, I want to manage vars, templates, files in one file instead of multiple roles, so I moved vars/, templates/ and files/ out of roles/.

For my next project, I may build an Infrastructure-as-code project using CloudFormation/Terraform and Ansible/Chef. Next, I may work on Jenkins, code pipeline, and so on.

In general, should I organize my DevOps projects based on projects, in this way:

Project1:
  Terraform
  Ansible
Project2:
  Terraform
  Ansible
  ......

Or should I organize my DevOps projects based on tools, in this way:

Terraform
  Project1
  Project2
Ansible
  Project1
  Project2
Cloudformation
  Project3
  Project4

I believe separate with projects makes more sense. but then how do we reuse code? for example, what if project1 and project2 use similar terraform code and some ansible roles. and I want to reuse the code of project1 to project2?

or is there any typical samples online I can refer?

The layout of your repository depends in many ways upon the context you are developing the automation in.  If, for example, you are building out the infrastructure for a product as part of a product team, then it would make sense to tie the infrastructure to the product - i.e. keep the infrastructure in the same repository as the software source code.

If however, you are building out common infrastructure components to stand-up basic infrastructure for DNS, File Storage, Email, etc. then it probably makes more sense to have a single common-infrastructure repository.  Some of this will be tied to the tool you are using, Terraform for example leads you down the path of having a single repository for all of your environments to support the terraform.tfstate construct.

Reusability

You quite rightly called out reusability and code sharing as a problem you are going to encounter sooner rather than later.  This is an important principle in Software Development called Don't Repeat Yourse - or DRY for short.  Most all DevOps tools allow you to modularize your code in such a way that you don't need to copy/paste code:

Terraform Modules
Ansible Roles
Chef Libraries
Jenkins Plugins

Conceptual Consistency

I advise that you don't use too many tools, try and standardise on a single tool rather than changing tools for each project, typically you see sets of tools being used across an organization:

Jenkins, Terraform and Ansible
TeamCity, Octopus Deploy and Desired State Configuration

This makes it easier to share code across projects, document what has been done so far and handover the code to other or future developers.


Using Azure Devops with Git Workflow I have used Azure Pipeline for CI/CD on various small projects that I have worked and it has been a great experience. I am working now on a decent size project that has a web app, mobile(xamarin) app and API. We plan to use Gitflow along with Azure Devops for the delivery process. But I have got a few questions that is really bothering me, and would like to hear other opinion.

I understand that feature branches are created to work on a user story, but when multiple people are working on a user story, do we create a feature branch, and then each developer branch of it and then work on the individual sub-tasks? And when they complete their work, they merge back to the feature branch. Is this a standard practice? Or do developers merge back to develop branch as they finish their sub-tasks?
Is it a good practice to do a build in Azure Devops for every PR merge to develop branch?
When a feature branch is merged with develop branch, do we immediately sent out a release or do we group and send out as nightly builds to QA?
I understand that both build and release pipelines offer a great flexibility. But generally does the build pipeline generates the artifacts or is it the release pipeline that does that?

I appreciate if someone can help me with these questions. Or atleast point me in the right direction

Most answers to these questions are going be subjective and influenced by your particular organization's workflows and culture. Little of this response if more than just my own opinion and experience and much is not really specific to Azure DevOps but can be applied to any build and release orchestration tooling.

Azure DevOps itself has very little to do with your chosen workflow apart from it being the common remote for the team and the source for pull requests etc. What source workflow you use is highly dependent on your individual situation. The number of people on your team doing simultaneous work, how much possible conflict there is between feature development efforts, your rate of release, complexity of code/projects, and philosophies about release management to name a few. Much has been written about various source management workflows. In the realm of Git, of course there's the over 10 year old GitFlow approach (note the recent update) and GitHubFlow. I won't regurgitate what they or the numerous other writings suggest about what flow to use. Your team needs to research the options and evaluate what is most applicable.

It certainly is not a bad practice to do a build for every PR. If developers are following good practices where feature branches originate from the merge target, there is a high probability that any testing/compiling they did in code from those branches will essentially match what will eventually be merged. However, that's never a guarantee so having the ability to do a pre merge PR build is a very helpful feature of ADO. And ADO can be configured to required a passing PR build before merge. You can also deploy PR builds if you want to support initial manual testing and/or more complex integration of a PR prior to merge.

This again will be a choice driven by your team and situation. Do you want every feature merge to result in a build? If you are practicing Agile, you may have work items that require manual testing as part of the definition of done. In that case, getting builds out immediately after code is complete could be imperative to forward progress. I follow a continuous integration approach and have builds trigger immediately upon commits/merges to the "typical" branch of work in progress (often that will be the development branch). The sooner you discover a problem, whether that is a compile error of the merged code, a unit test failure, or a manual integration test bug the better. Every minute you wait to fix an issue is compounded into more time resolving it later as the work fades from peoples' minds.

I have typically followed the philosophy that "what I deploy to production should be what we tested". While theoretically, if you do 2 builds of the same exact source code you should get the same result, there's no guarantee of that. Particularly now with the use of public packages/modules and the ability to allow flexibility with what versions are consumed, it's not an unreasonable possibility that a later build will result in different artifacts, for example, but pulling in a newly updated version of a dependency. For some very real perspective on this, see this article about how the "internet broke" some years ago. I work in the healthcare industry which has very strict rules that drive our implementation approach. We need to deploy just what was tested and approved to the production environment. If we did a fresh build for production release that introduced a changed library with an unknown security vulnerability, we could get into serious hot water.  I follow the practice of having a build pipeline do the actual build of final artifacts which become the singular source for deployments for testing and final promotion to production. ADO makes this very simple by publishing the artifacts into its own internal storage and providing those as an artifact source for a release pipeline. You can then deploy that (and additional) artifacts to each release stage of your pipeline. One key factor to consider is environment configuration management. Depending on your deployment target, you'll need to take into account how to influence the runtime configuration. My organization still deploys .NET applications to traditional IIS web servers using file copy. This requires an explicit step to apply config transformations for the application based on the target environment. This approach can be enhanced by using a separate artifact source for the transformation to better implement the good DevOps practice of separation of concerns (environment agnostic build artifacts vs. env specific configuration; env configuration, including secrets, can be retrieved from a different, more protected source so they are not in the application code repository). If you are deploying to a docker image, your docker runtime environment can drive environment specific settings for the container instances. Deploying to a cloud application host service? Use its instance configuration tooling, such as the app service/slot specific configuration settings in Azure. Applications like React, however, take a different approach that typically dictates that you do a build for the target environment since environment driven values are bundled into the deployed artifacts. I'm not a fan of this as it breaks my core philosophy of "build once, run anywhere". But there are some solutions to help make such builds environment agnostic.


How do you monitor status of multiple docker containers? My home server has about 15 distinct services running, each represented by a Docker container. Depending on how the developers implemented the Dockerfile, some services have health checks, but most do not.

At the moment, I do not have a way to make sure these services remain healthy and running. I do have the restart: always setting set, but worst case this just means Docker endlessly restarts a server if it fails to start or crashes on startup for whatever reason. Or maybe a docker pull caused an issue during an update. There could be many reasons why these services do not start, or stop running during the day.

What I'd like is to know when they go down. Maybe an email that notifies me, so I can go fix the issue, instead of only finding out about the issue when a family member notifies me or when I try to use that service.

Is there a reasonable way to monitor multiple Docker containers in this way? Do I have to hand-write a bunch of scripts that do ps to check for process up time or something? Preferably an out-of-the-box solution would be great. Note I use Docker Compose for everything, and I only have 1 physical machine where I run these services, so I don't need to worry about complex setups.

Note: answer based only on docs, I didn't play with docker much.

If you choose scripting you could pick a higher layer than just ps, based on either:

the docker-compose events CLI:

Usage: events [options] [SERVICE...]

Options:
    --json      Output events as a stream of json objects

  Stream container events for every container in the project.

the docker events CLI:

  Description

  Get real time events from the server

You'd be looking for the die event of the container(from Extended description):

  Use docker events to get real-time events from the server. These
  events differ per Docker object type.

  Object types

  CONTAINERS

  die

The event appears to be generated regardless of the restart policy (from Docker Events Explained):

But you're right, there's a ton of monitoring tools out there, I also suspect some already do what you're seeking.

Just aimlessly wandering through those links I got to Implementing Docker event monitoring from scratch:

  What else? I can create alerts on them. The next time any of my
  containers decide to puke and die, I can get an SMS message about
  their doomed state.


DNS Issues with Cloud formation template I am trying to automate the creation of an EC2 instance with CloudFormation but am running into issues with DNS.
Once the instance is created it can't seem to resolve any hostnames. I try to ping a hostname, such as google.com, and it can't be resolved. However, it can ping IP addresses.
My template is below and I understand that there are some unused parameters which I intend to use later on:
{  "Description": "Create an EC2 instance by AWS CloudFormation",  "Parameters": {
"KeyName": {
  "MinLength": 1,
  "Type": "AWS::EC2::KeyPair::KeyName"
},
"BaseAmiId": {
  "Default": "ami-bd3ba0aa",
  "Type": "String"
},
"DomainAdminUser": {
  "Description": "User name for the account that will be added as Domain Administrator. This is separate from the default \"Administrator\" account",
  "Type": "String",
  "Default": "installer"
},
"DomainUserPassword": {
  "NoEcho": "true",
  "Description": "The Windows administrator account password",
  "Type": "String",
  "MinLength": "8",
  "MaxLength": "41"
},
"DomainDNSName": {
  "Description": "Fully qualified domain name (FQDN) of the forest root domain",
  "Type": "String",
  "Default": "test.com"
},
"DomainNetBiosName": {
  "Description": "Netbios name for the domain",
  "Type": "String",
  "Default": "test"
}  },  "Resources": {
"SecurityGroupDemoSvrTraffic": {
  "Type": "AWS::EC2::SecurityGroup",
  "Properties": {
    "GroupName": "sgDemoSvr",
    "SecurityGroupIngress": [
      {
        "IpProtocol": "tcp",
        "FromPort": 3389,
        "ToPort": 3389,
        "CidrIp": "0.0.0.0/0",
        "Description": "For traffic from Internet"
      }
    ],
    "GroupDescription": "Security Group for demo server",
    "VpcId": "vpc-05eabde4dbadsf09"
  }
},
"EC2Instance": {
  "Type": "AWS::EC2::Instance",
  "Properties": {
    "AvailabilityZone": "eu-west-2b",
    "BlockDeviceMappings": [
      {
        "DeviceName": "/dev/sda1",
        "Ebs": {
          "DeleteOnTermination": "true",
          "VolumeSize": "32",
          "VolumeType": "gp2"
        }
      }
    ],
    "ImageId": "ami-029c4f36a959979a7",
    "InstanceType": "t2.micro",
    "KeyName": {
        "Ref": "KeyName"
    },
    "NetworkInterfaces": [
      {
        "Description": "Primary network interface",
        "DeviceIndex": "0",
        "SubnetId": "subnet-0b3f36d19eb3sdfda",
        "AssociatePublicIpAddress": true,
        "GroupSet": [
          {
            "Ref": "SecurityGroupDemoSvrTraffic"
          }
        ]
      }
    ]
  }
}  }}

Which flavor of Linux are you using? (eg Ubuntu/Debian, RHEL/CentOS etc)
Also which version? (eg Ubuntu started using the "netplan" configuration by default in 20.04)
At first glance, it looks like your DNS nameservers are hosed somehow.
And can you post the results for doing a: netstat -nutpl?
Look at which PID/Program Name is listening on port 53 (systemd-resolve? dnsmasq?) and check its nameserver configuration.
Maybe check here: sudo cat /etc/resolv.conf  and make sure the nameserver(s) listed there are what you think you should have them.
or check the DNS-related services are even running with something like:
Ubuntu:  sudo service systemd-resolved status (NOTE BOTH of the 'd' endings)
CentOS/RHEL: sudo service dnsmasq status
(or whatever is using port 53, which is the default DNS port on most OSes)


Tools used in enterprise level to transfer build artifacts from CI server to live server I recently learnt some basic devops and set my server on Digitalocean droplet with travis CI. As there is no travis deploy provider supports for Digitalocean, I have to write my own deploy script. However I have more than 50000 files i.e near about 1 GB of data. I tried scp, ftp, rsync but I feel it takes a lot of time.

What are the transfer tools that is used in enterprise level ? Do they also use RSYNC or SCP. Or there are other tools that is better than these which are being used by medium level enterprises which I am not aware of.

Apart from it I have another question is there any deployment script/tool that can parallely transfer my files over ssh ?

Try LFTP that has commands to mirror or parallel copy. LFTP can run over many protocols.

The likely problem you have is that you are transfering a lot of data from a build service to a single vm over the Internet. Enterprises use the same tools and protocols but they pay for more bandwidth else ensure that their builds and deploy happens within the same cloud provider. They then deploy to many servers in parallel using standard protocols like SCP with tools like Ansible. The networks within a single cloud provider (Azure, AWS, GCP) are prioritised. This means of you want fat deployment bandwidth, and to run on Digital Ocean, you should build on Digital Ocean.

It may be the case that a very large amount of the 50k of files dont change on each build. So what you really want is a tool that looks at what files are already deployed. We can expect such tools to use file name, timestamps, file sizes and checksums to check whether a file needs uploading. If your build is stateless and creates 1G of fresh files each time their timestamps are going to be newer that the identical files already deployed. LFTP has an "ignore time stamp" option to avoid transfers of newer but identical files. See this question and answer https://serverfault.com/a/311670


How to set CloudWatch alarms for non-compliancy in AWS Config rules? I have a set of rules in my AWS Config dashboard. And, I want to set a AWS CloudWatch alarm to be triggered whenever Config detects non-compliant resource(s).

[The plan is to link that alarm to an SNS topic for sending out emails to the team for any non-compliant event/resource]

Is there a straightforward way to do that? Or is there a workaround for the same?

I went through the available metrics in the Cloudwatch dashboard, but haven't found anything related to AWS Config.

From what I know of AWS Config there's no way at the moment to directly publish a metric from which you can base an alarm.

If the notification has to come from a Cloudwatch alarm then AWS Config will let you execute Lambda based on rules set up in AWS Config - link.

If you want a simpler option AWS Config also lets you publish directly to an SNS topic when your rule is evaluated - link

Failing this you can also watch for CloudTrail events and then publish a CloudWatch metric which you can base an alarm on. This is fairly in depth though and if the important part is getting the emails to be looked at by a person then your best bet is probably to have AWS Config publish directly to SNS itself.


Why are valid directives in my Dockerfile apparently being ignored? Using the following Dockerfile:

FROM ubuntu:latest

USER root

RUN apt-get -y update \
    && apt-get -y upgrade \
    && apt-get install -y \
        python-pip \
        nmap \
        vim-tiny

FROM python:2.7

USER root

RUN pip install --no-cache-dir cassandra-driver

FROM cassandra:3.11.4

WORKDIR /cassandra

COPY ./cassandra.yaml /etc/cassandra/cassandra.yaml

EXPOSE 7000 7001 7199 9042 9171

CMD ["cassandra", "-f"]

and then running the commands:

docker system prune -a -f

docker-compose -f docker-compose.yml -f dev-docker-compose.yml up --build

when I then enter the cassandra container via the command:

docker exec -ti cassandra_app /bin/bash

None of the things I asked for has been installed!

root@2f1776a10084:/cassandra# nmap
bash: nmap: command not found

root@2f1776a10084:/cassandra# vi
bash: vi: command not found

root@2f1776a10084:/cassandra# python --version
Python 2.7.13

root@2f1776a10084:/cassandra# python -c "import cassandra"
Traceback (most recent call last):
File "<string>", line 1, in <module>
ImportError: No module named cassandra

I also tried running the build with the "USER root" lines removed, but no difference.

I need a Cassandra container that includes "vi" and the python cassandra driver, and this wretched Docker is wasting hours and hours of my time trundling through its builds (which don't have any relevant error messages that I can see).

It has gradually become clear that Docker caches things when it can, hence the "-a -f" arguments to the "docker prune" command - I want to be sure I'm doing a clean build each time.

In case it is relevant, following are the Docker YAML files for the above build command:

docker_compose.yml :

version: '3.3'

services:

  application:
#    image: appweb/app:python3
    container_name: django_app
    build:
      context: ./application
    ports:
      - "8000"
     depends_on:
      - postgres
      - cassandra
      - cassandra-node
    networks:
      - appnet
    environment:
      - DOCKERISED=1
      - CASS_HOST1=cassandra:9042
      - CASS_HOST2=cassandra-node:9042
    extra_hosts:
      - "34.253.98.48:34.253.98.47"

  postgres:
    image: postgres:9.6.10
    container_name: postgres_app
    networks:
      - appnet

  cassandra:
    build:
      dockerfile: Dockerfile
      context: ./cassandra
    container_name: cassandra_app
    environment:
      - "CASSANDRA_SEEDS=cassandra"
      - "CASSANDRA_CLUSTER_NAME=App2 Cluster"
    networks:
      - appnet

  cassandra-node:
    build:
      dockerfile: Dockerfile
      context: ./cassandra
    container_name: cassandra_app2
    #command: /bin/bash -c "echo 'Waiting for seed node' && sleep 30 && /docker-entrypoint.sh cassandra -f"
    environment:
      - "CASSANDRA_SEEDS=cassandra"
      - "CASSANDRA_CLUSTER_NAME=APp2 Cluster"
    depends_on:
      - "cassandra"
    networks:
      - appnet

networks:   appnet:

dev-docker-compose.yml :

version: '3.3'

services:

  application:
    environment:
      - CASS_USER
      - CASS_PASS
      - POSTGRES_USER
      - POSTGRES_PASSWORD
      - TESTMODE=1
      - CQLENG_ALLOW_SCHEMA_MANAGEMENT="1"
    volumes:
      - ./application/source:/application/source
      - ./application/source/private_settings.py:/application/source/app2-config/app2/config/private_settings.py

  postgres:
    volumes:
      - postgres_data:/var/lib/postgresql

  cassandra:
    volumes:
       - cassandra_data:/var/lib/cassandra

  cassandra-node:
    volumes:
       - cassandra_data2:/var/lib/cassandra

volumes:   postgres_data:   cassandra_data:   cassandra_data2:

Any ideas? Thanks in anticipation

It looks like your Dockerfile is attempting a multistage build:

FROM ubuntu:latest
...
FROM python:2.7
...

This is a feature that enables you to build parts of your image in a separate build container and then import what you need. Unless you import stuff into your end container, it will be lost in the build process; this is exactly what is happening with your RUN apt-get... statement. To fix it, try removing the extra stages of the build, (FROM ubuntu:latest and FROM python:2.7) and moving your commands to after FROM cassandra:3.11.4 like so:

FROM cassandra:3.11.4

USER root

RUN apt-get -y update \
    && apt-get -y upgrade \
    && apt-get install -y \
        python-pip \
        nmap \
        vim-tiny

RUN pip install --no-cache-dir cassandra-driver
...

I'm sorry you're having a hard time with Docker. Hopefully this helps and it starts making more sense. Good luck, friend!


Automating the creation of an Apache virtual-host file specifically When I create new web applications on my up-2-date Ubuntu-LAMP with Apache 2.4.x environment, I create their virtual-host files this way:

s_a="/etc/apache2/sites-available/"
s_e="/etc/apache2/sites-enabled/"

read -p "Have you created db credentials already?" yn
case $yn in
    [Yy]* ) break;;
    [Nn]* ) exit;;
    * ) echo "Please create db credentials in then comeback;";;
esac

read -p "Please enter the domain of your web application:" domain_1 && echo
read -p "Please enter the domain of your web application again:" domain_2 && echo
if [ "$domain_1" != "$domain_2" ]; then echo "Values unmatched. Please try again." && exit 2; fi

read -sp "Please enter the app DB root password:" dbrootp_1 && echo
read -sp "Please enter the app DB root password again:" dbrootp_2 && echo
if [ "$dbrootp_1" != "$dbrootp_2" ]; then echo "Values unmatched. Please try again." && exit 2; fi

read -sp "Please enter the app DB user password:" dbuserp_1 && echo
read -sp "Please enter the app DB user password again:" dbuserp_2 && echo
if [ "$dbuserp_1" != "$dbuserp_2" ]; then echo "Values unmatched. Please try again." && exit 2; fi

cat <<-EOF > "$s_a/$domain_2.conf"
    <VirtualHost *:80>
        ServerAdmin admin@"$domain_2"
        ServerName ${domain_2}
        ServerAlias www.${domain_2}
        DocumentRoot /var/www/html/${domain_2}
        ErrorLog ${APACHE_LOG_DIR}/error.log
        CustomLog ${APACHE_LOG_DIR}/access.log combined
    </VirtualHost>
EOF

ln -sf "$s_a"/"$domain_2".conf "$s_e"

My question

Is there a way to automate that / shorten the amount of lines code (the actual script-file is much bigger in principle), maybe even from some built-in behavior in Apache itself and without involving further software?

If other software might be needed than from what I've read so far it should be Docker and I should look for an Apache-virtual host docker script that will be based on my variable declarations (in dockerhub) which I will then run with a docker command, but maybe Apache itself will have automation functionality for this or there is some third solution the answer's my need?

This has already been covered a bit in the comments, but I figured I'd lay out a more extensive answer.

  Is there a way to automate that / shorten the amount of lines code (the actual script-file is much bigger in principle), maybe even from some built-in behavior in Apache itself and without involving further software?

You can use a2ensite as a replacement for your very last line, the ln; all it does is set up symlinks into /etc/apache2/sites-enabled, which is where Apache httpd looks by default for configuration.  Some details may vary by distribution.

However, this only changes your script a tiny bit.  There are much larger problems you have to address.

First, I'll mention that no, Docker is almost certainly not the solution you want.  Docker works best when used in a container orchestration layer like Kubernetes, and while I can easily imagine an architecture with a single Docker image that takes in parameters to create different vhosts, it's a much more complicated system than your presumably one-host system and requires substantial changes to how everything works.  Docker is a hip new thing, but what many people do not realize is that it solves a set of well-understood problems and creates a set of new, poorly-understood ones, and it's likely that you don't have the problems that such an architecture would solve.  So let's set that aside for now.

The first problem you have is that you're writing Apache configuration (primarily) by hand, with templating in bash and limited validation.  In reality in your example this isn't a big deal since it's a small and simple config, but it's easy to start this way and end up with a large block of sed and ed commands (this I know from experience).

The larger problem is in how ephemeral your changes are.  You configure all these vhosts, great.  Then your server dies.  What happens?  You're probably going to spend a lot of time trying to rebuild it.  Hopefully you have a recent backup to help, but the point here is that the last few decades we've settled into this idea of treating configuration the same way as code - rather than setting things up by hand, you write all of your configuration into text files and a program translates that into configuration on the server.  This has a multitude of benefits, including backups, code review, and a history of changes that you can look back on.  Most configuration management tools also provide convenient abstraction layers.  Here is an example that lays out how to translate an Apache vhost file into an Ansible template, and then you only need to add new entries to the variable list and rerun the configuration.  When it comes to things like your database passwords, those too can be stored and version-controlled in Ansible Vault or a similar product.


Condition to skip a test when running in a container I'm writing an Inspec profile which has a specific control that I want to skip if the control is running inside a docker container. This looks like:

control 'ssh daemon' do
  impact 'critical'
  only_if('not in docker') do
    condition_expression
  end
  describe service('sshd') do
    it { should be_installed }
    it { should be_enabled }
    it { should be_running }
  end
end

I need the condition_expression to return false when in a container. Is there an elegant way to do this with ruby or inspec without shelling out?

In Ruby you can use this function:

# return true if we are inside a docker container
def in_container?
  return File.file?('/.dockerenv')
end


What is the definition of "cattle not pets"? The term "treat your servers like cattle not pets" has proliferated in recent years, particularly when applied to Docker containers and Virtual Machines

What does it actually mean?

Randy Bias chronicles the history of the term stating that it probably originated in 2011 or 2012 when Bill Baker used the analogy when describing "scale-up" vs. "scale-out" architectural strategies. Bias adopted this into his presentations about cloud architectural patterns:

  In the old way of doing things, we treat our servers like pets, for example Bob the mail server. If Bob goes down, it’s all hands on deck. The CEO can’t get his email and it’s the end of the world. In the new way, servers are numbered, like cattle in a herd. For example, www001 to www100. When one server goes down, it’s taken out back, shot, and replaced on the line.

Bias continues to define Pets as

  Servers or server pairs that are treated as indispensable or unique systems that can never be down. Typically they are manually built, managed, and “hand fed”. Examples include mainframes, solitary servers, HA loadbalancers/firewalls (active/active or active/passive), database systems designed as master/slave (active/passive), and so on.

and cattle as

  Arrays of more than two servers, that are built using automated tools, and are designed for failure, where no one, two, or even three servers are irreplaceable. Typically, during failure events no human intervention is required as the array exhibits attributes of “routing around failures” by restarting failed servers or replicating data through strategies like triple replication or erasure coding. Examples include web server arrays, multi-master datastores such as Cassandra clusters, multiple racks of gear put together in clusters, and just about anything that is load-balanced and multi-master.

Fundamentally, what Bias and Baker are trying to convey is there has to be a transition from how we treat servers from being "Unique Snowflakes" with names and emotional attachments, to a model whereby if we have a problem with the server we create a replacement and destroy the problematic server.

Finally, it is probably worth mentioning that in regulated environments taking a server out the back and shooting it may not be optimal.  In these cases it is often advantageous to "freeze" the server, for example using docker pause to freeze a container.  This can then be used to perform a Root Cause Analysis as part of the Incident or Problem Management Process.


How to assign saltstack pillar values from Jenkins comma separated input parameters? I have a Jenkins job with string parameter as target "server1,server2". Jenkins pipeline script is expected to execute the job on server1 and server2 respectively.
There is a common salt sls file that updates directory permissions and groups on both servers. The expectation is when the job is executed on server1, a group by name xx-server1-yy must be created and updated in server1, similarly a group xx-server2-yy be created in server2 only.
One sls file for Jenkins job on multiple servers with groups created specific to each server. What is the best approach?

You didn't say how the sls file is executed.

One way would be to pass pillars on command line:

salt '*' state.apply ftpsync pillar='{"ftpusername": "test", "ftppassword": "0ydyfww3giq8"}'

Or you can set the group name as a grain on the minion in advance.
Yet another way would be to set the group name as a jinja variable in the sls formula, depending on the hostname grain. Assuming your host names are set and standardized.


Automated Testing Jobs We have multiple environments close to 7 for an application that has UI and also integration server that accepts SOAP xmls.

Its hard to maintain the environment integrity when many people working, developing, testing on these servers. Almost every 2nd day one or the other environment has issue and lot of effort and time goes in fixing that.

What we are looking for is an automated testing tool that has a capability to run/schedule test cases both UI based (like macros) and XML based. Lets say this tool runs the test cases every morning on each of the environment and sends out an email report of the output/results.

Based on your experience which tool fits best to our requirement. Please suggest.

Regards,
Viv

I see a mismatch between the problem you describe and the solution you're asking for.  Your problem seems to be managing your environments.  You seem to think automated testing is the answer.

But why not automate managing your environments to prevent the manual effort that goes into fixing them?

It's very important to reset test environments to a known good state before running automated tests.  Without a reset, it takes effort to figure out if a test failure is due to code changes or environment changes.  It's about the same effort as manually fixing the environment in my experience.

There are many approaches for creating or resetting test environments.  Probably it's a too broad question and a matter of opinion.

Once you have a repeatable way to provide a known good test environment, there are many tools for automating regular test runs and emailing results.  Probably it's a too broad question and a matter of opinion.


Can we use Ansible for monitoring configuration? I understand how Ansible works and what it is used for.
However, I was wondering if there is a way or feature (in open-source or enterprise edition) in Ansible using which, we can make sure that the configuration of a VM instance or server is always in sync with Ansible?
For example:
I used Ansible to install a few things (like MySQL server, spin up some docker containers etc.) on a Google Compute Instance. Now after some time I manually SSH into this VM and stop the docker container created by Ansible. Can Ansible be notified of this in order to create the container again?
So if Ansible sets configuration of a VM in state A and another user manually changes the state to state B, is there a way that Ansible can be notified of this change and runs the playbooks again to set the configuration back to state A?

The simple answer is "yes" you can use Ansible for monitoring configuration, but you will have to do some extra work. As Vasily stated in their answer, Ansible does not have a built in triggering mechanism, so you need something to trigger the convergence of state. This can be done in several ways, the easiest being probably ansible-pull. As the docs state, ansible-pull

... inverts the default push architecture of ansible into a pull architecture, which has near-limitless scaling potential.
The setup playbook can be tuned to change the cron frequency, logging locations, and parameters to ansible-pull. This is useful both for extreme scale-out as well as periodic remediation.

emphasis added.
Ansible is idempotent, so it does know about the state of target machines, contrary to what Vasily's answer states - it just doesn't persist it in the same way as Terraform does.
Your real problem does not sound like the question you are posing ("Can Ansible be used to ensure the state of a machine?"), but rather "how can I detect configuration drift and trigger a remediation?"
If you think of this as a monitoring problem, you can reframe the question as "how should I monitor my services in production?". Monitoring implies alarms on changes in state, which trigger actions to remediate. In your case, you would define an alarm on the health check of the service, when that is triggered, execute the playbook which converges to the desired state.
Since you have already invested time in writing the Ansible playbook, it would make sense to re-use it. The simplest would be to use that same playbook to invoke a cron which runs ansible-pull every 5 minutes or so, depending on your particular case.


How to SSH from Jenkins node to Jenkins node safely? We have a Jenkins Job that builds something on node A and then needs to SCP it to node B.
You can just put the SSH keys for node B on Node A, and SCP with an execute shell build step, however I suspect that keeping SSH keys on a Jenkins node is a bad practice, and would like to avoid it.
What is the best practice regarding this use case? Is there a plugin or a feature for this? Should we just install Hashicorp Vault on the node and configure the keys there separately?

Yes, storing your ssh keys directly on a build node is a bad practice. Nodes can be replicated, deleted, or given access to from other systems, and you don't want to lose track of what systems have access to your secrets.

You should also not pass them directly into the build job, either as a parameter or as an environment variable. This can cause a huge headache with logging and tracking to make sure your keys aren't inadvertently output to places they shouldn't be. Instead, you should:

Use a centralized secret store (such as Hashicorp Vault) to retrieve the secrets at build run-time.
Use a plugin (such as the Credentials or SSH Credentials Plugin) to reference the keys during the build.


What is the difference between continuous deployment to continuous delivery? I understand continuous integration as "orchestrating and continuously installing and/or upgrading an environment, or part of an environment, based on communally-maintained code".

- my own definition.

But what is the difference between continuous deployment to continuous delivery, if at all there is one and both are not just parallels.

Long story short:

Continuous delivery is the capability to deploy the software to any given environment at a given point of time.  The objective is to perform push-button deployments of any version of the software to any environment on demand.

Continuous Delivery is often conflicted with Continuous Deployment.  Any change the developer makes gets deployed all the way up to Production is Continuous Deployment

The referenced article will give some info:
DevOps in nutshell


any way to limit "docker logs" output by default? Context*
I forgot to pass the "--tail" options to the "docker logs" command when inspecting a running container. The affected container dumps tens of millions of logs.
The command line was something like:
  $ docker logs contianer01 | vim -

When executing such line my server becomes completely unresponsive for minutes due to exhaution of free RAM memory with peak loads beyond 60x100%. I loose control of the terminal and ssh connection fails.
Question:
Is it any tunning in docker to "fail fast" and forbid such big log outputs in first place (maybe tunning ulimits for the docker daemon in some config file)
NOTE: Probably "ulimit", "resource-control" and "linux" tags must be added to this question but I have not enough "reputation".

One way to limit docker logs is to specify limits in docker daemon's config file.
You can put this in /etc/docker/daemon.json
{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3"
  }
}

and then restart your docker daemon.


How do I run Ansible Azure playbooks while avoiding storing credentials in files? Background

We use Ansible to provision and manage Azure infrastructure. At the moment we run Ansible "manually" i.e. we manually execute playbooks for various automated tasks. No CI infrastructure.
Probably not relevant but we manage our inventory using dynamic script azure_rm.py.
We are encouraged to be as secure as possible i.e.

Don't store Vault passwords in ~/.vault_pass or in any local file
Don't store Azure secrets in ~/.azure/credentials
Don't store anything secure in .bashrc.

In such a scenario, I am having trouble coming up with a coherent strategy to ensure that my playbooks can access Azure secrets, while following the guidelines above.

Question

How can I avoid storing Ansible Vault and Azure credentials on files, while still ensuring my playbooks can access them?

What I've tried

So far I have come up with a wrapper script that

asks the user for Vault password
Uses that to decrypt a Vaulted Shell script
Evaluates the script, which loads Azure environment variables into the environment;
Runs the playbook on the environment that has been thus set.

Any better (more elegant, less complicated, more "Ansible") solutions out there?

Vault password

First of all, you should get familiar with the fact that vault password file can be executable script. In this case Ansible executes it and expects to receive password as its output.

For example you can use gpg-agent or keychain to store your actual password and unlock it when required. Read more in this blog post: https://benincosa.com/?p=3235

If you are a bit paranoid, you can add notification when your password script is called, like this:

#!/bin/bash
PARENT_PROCESS=$(ps -p $PPID -o args | tail -n 1)
osascript -e "display notification \"Vault password used by ${PARENT_PROCESS}\" with title \"Ansible\" sound name \"default\""
gpg --batch --use-agent --no-tty --decrypt key.gpg 2>/dev/null

This vault password script uses key.gpg as actual vault key and also shows popup notification (for MacOS) with parent process name when script is used. Gpg-agent caches unlock password for some time, so there is no need to enter password every time you start playbook.

Just set vault_password_file = ./vault_pass.sh in your ansible.cfg.

Environment

You said that you use azure_rm.py as dynamic inventory script. This means that you have to set credentials into your environment variables before you start ansible-playbook for it to be able to use them.

You can make two files:

secure_env (encrypted with vault):

export AZURE_SECRET=xxx;export AZURE_SUBSCRIPTION_ID=xxx;

set_env (plain text):

echo -n "Setting secure vars... "
eval $(ansible-vault view secure_env)
echo "done."

When you open new terminal to execute your automation tasks, you have to run:

source set_env

At this moment, bash evaluates set_env and secure_env (decrypted via ansible-vault). After this command you have Azure credentials defined for the current shell, so you can execute playbooks as usual:

ansible-playbook provision-my-azure-instances.yml

So using this two approaches, you can store key.gpg and secure_env in your repository; then in the new terminal call source set_env once, enter gpg password once (to unlock future use of key.gpg); then call ansible-playbook as many times as you like without any passwords.


Can I upgrade a docker image/container from buntu 18.04 to ubuntu 20.04? I have a host ubuntu 20.04 running my software on docker with a container derivated from ubuntu 18.04. I would like to use ubuntu 20.04 for several reasons (like the better software on 20.04).
If I run do-release-upgrade on the container terminal I receive the following:
"do-release-upgrade
Checking for a new Ubuntu release
You have not rebooted after updating a package which requires a reboot. Please reboot before upgrading."
Rebooting the container or rebooting the host machine does not solve the problem.

Docker doesn't recommend running an apt-get upgrade and dist-upgrade on a docker image as many of the "essential" packages from the partent image cannot upgrade inside an unprevileged container. take a look at the Docker best practices


Pass string to Jenkins job from Slack In an ideal world our less tech-knowledgeable employees could just use a Slack command to run a job and get the results slacked back to them, negating the use of the Jenkins GUI. I found Start a build in Jenkins using a Slack Command on GitHub, but it's got a few limitations including that it seems to require a 1:1 ratio Heroku instance to Jenkins job, which isn't great for our purposes.

My thought is to have people use a singular command like /jenkins, and use parameters after it that will be parsed by one job, and used to kick off others downstream. That keeps us to one slack command / one instance, but still gives us all the flexibility we need.

The issue falls in getting the parameters to Jenkins. I'm picturing something like the following workflow;

User types /Jenkins job2 param1 param2 in Slack. This is calling an umbrella job that will parse the parameters to find out what to run.
Command is somehow sent to the umbrella jenkins job. Heroku instance, direct integration, w/e.
Umbrella job receives job2 param1 param2 as a string. (Perhaps as the value of a single parameter, perhaps pre-split into multiple fields)
Umbrella job splits the string if it hasn't already been done, and figures out that the user wants job2 run, using parameters param1 and param2.
Umbrella job does a buildjob to run the downstream job as called, which will Slack back to the user their results.

It's steps 2-3 that are tripping me up. How can I pass a string (with spaces!) from Slack to my Jenkins job?

If my entire premise is wrong I'm also open to having my frame challenged.

The plugin you found doesn't have a 1:1 ratio between Heroku instance with build jobs since it takes the job name as parameter, so you should be able to run any number of jobs from a single Heroku instance.

As for passing a space in a job parameter, the word split is made on spaces so quoting won't change anything. You could possibly try to use the HTML spacing character %20 and see if it works?

Personally I would try to use the plugin to run the correct job with the associated parameters directly instead of going through filtering jobs.


How to create pipeline with jenkins? I followed this tutorial and when i get to 29:32 the option "Build Pipeline View" is not displayed in the list i only get list view and my view.What can be the cause of this ? ( i followed every step carefully)

This is a plugin that you can install.
Plugin Name: Build Pipeline
Plugin Web page: https://plugins.jenkins.io/build-pipeline-plugin/


Bad practice having a single SSH gateway with access to all environments? I have started at a new company and am in the process of locking down SSH access to be through a single SSH gateway server.

My question is, should I create a separate SSH gateway server for each environment? (QA, Dev, Stage, Prod)

We are fully in AWS so my current plan is to create an SSH Gateway server on a DevOps VPC and create peer connections to the QA/Dev/Stage/Prod VPCs.

Then I would manage access to environments via SSH keys.

My concern is that this single SSH gateway server has access to all environments, which is a little terrifying. But that being said, it is easier to manage and monitor access.

I assume your users connect to the gateway with ssh (possibly with additional credentials like a hardware-generated one-time-password), and then jump to the actual servers with a second ssh connection.

In itself, this is fine (and the usual thing that's done regularly by companies I've worked with). The secondary ssh connection (still) takes care of only letting those people in who are authorized. The job of the gateway is primarily to avoid the need for the ssh port of the servers to be open to the internet on the network level.

Obviously, you want to avoid people opening ssh tunnels, using proper settings in your global ssh/sshd configuration, and you will want to tighten down the gateway itself (for example, by having read-only home directories for your incoming users; and having no tools installed that can help in somehow bypassing any of your rules - i.e., no telnet, netcat and so on; ideally a chroot environment in which nothing except the ssh executable is available).


What approach should I take to store environment variables for a repo? I am setting up a new infrastructure as code repository for a small to medium size project. The problem I am currently trying to solve is how to store my environment variables such that they can be read and used by all services and scripts within the repo. Some of the tools being used are Ansible, Powershell, Bash...

My current solution is a simple text file, manually created per environment. Each line has a variable that the services and scripts can look up. However, this seems clunky, as I have to create the logic to read this file in every script that is made.

So my question is, what approach should I be taking to provide all my services with environment variables? Am I on the right track, or should I be using something entirely different for this?

I'm not sure what I can use to provide this to all the different tools.
For details sake, I am not talking about secrets management. I mean variables such as: DNS address of the environment's DC, log server address, etc...
There are significant constraints on addition of new software/tools due to the sensitivity of the environment. For example, I was considering using JSON format for the environment file, but Bash does not parse that natively. Meaning I would have to install a third party tool like 'jq' to use.

Any feedback welcome. Thank you!

General Notes

I don't really understand this one:

  However, this seems clunky, as I have to create the logic to read this
  file in  every script that is made.

You can source all variables from a bash file in one line (source file-name.sh).  If they are "exported", you can pick them up in applications in any language you like (Java, python, or whatever), and most have built in syntax for doing it (like spring boot's SPEL syntax which lets you define your config to directly populate from environment variables automatically on start-up).

You can also make this export part of your ~/.bashrc file or whatever so it's auto sourced in your shell contexts, etc.

Repositories + CI/CD

This may depend on your repository and CI implementation more than the installed code.  For example, GitLab CI and GitHub Actions both support environment variables, as does Jenkins, etc.

In GitLab you can even go to your group level (above a bunch of related projects), and set per-environment environment variables there which are pushed into your deployment so you can use them in your installed code.  Then, you don't even need to store them in any extra files.

References:

https://docs.gitlab.com/ee/ci/variables/
https://help.github.com/en/actions/configuring-and-managing-workflows/using-environment-variables

Docker

Whatever your code is, you can probably bundle it into docker and pass the variables to the container at run time.


What OS should a master node CI virtual machine be? Say you are developing a new CI/CD system, deploy builds across Linux, Windows and Mac, and have now reached the point of wanting to deploy a master node which hosts your chosen CI tool. How do you choose what OS and version your master node VM should be?

For example with Gitlab, I can have the VM with Ubuntu, Debian or CentOS, or with Jenkins, the VM could be Mac, Windows, Ubuntu, Debian etc...

What's the major deciding factor when choosing an OS and version for the master node of the CI system?

I think the major deciding factor is the expertise of you and the rest of your company in the chosen OS. If you are a Windows shop, and your company is willing to leverage the cost of the node, it's probably best choice to host it on a Windows VM. For choosing a Linux distro, I would see what is most common across your company. In my opinion, it is not worth the benefit of creating a vm with a one off distro just because of a few particular features, unless those features of the distro are absolutely necessary.


How to keep many servers (5000+) up-to-date? Initially asked here: https://stackoverflow.com/questions/60674502/how-to-keep-many-servers-5000-up-to-date-with-git-and-its-rate-limits

The initial post:

We're making a php service that will run on many servers, think 5000+. We host our code on git (bitbucket). We wondered what the best way to keep the servers up-to-date would be.

We figured either post-commit hooks (but what would happen then if a few servers didn't receive the update notification?) or git fetching every minute with cron. We want to go with the cron way of doing things, since it isn't possible it would fail, even if a server would be offline (either turned off or disconnected from the network), it would still resolve itself eventually.

We're doing a fetch every minute, and then compare to see if it needs to pull, if so it pulls and runs the migration code.

We would like to run this every minute so that the servers will be synchronized with each other as soon as possible.

Now we wonder, what about rate limits? We're using bitbucket, and the rate limits are 60 000 requests per hour (so a 1000 per minute), which would limit us to 1000 servers max before we will get problems then?

But it also says, if we make a public repo, we can make unauthenticated calls, the limits of which go by IP rather than per user, so we won't run into any limits then no matter how many servers we will have. Downside is, we will have to encrypt the repo then, then on pull, decrypt it and copy over the decrypted files.

Is this the best way of handling this? It seems very unconventional. What is the standard or recommended way of handling this (if there is any)?

After having read the answers, we now think that having a group of servers that pull in any changes (from git). For readability, we're calling these the "gitpull-servers", and the 5000+ servers the "webservers".

The plan is to make the gitpull-servers fetch (and possibly pull) from git.

The webservers periodically query the gitpull-servers to see if there is an update, and if there is, the webservers will pull the update in either by downloading a tar/zip (requires more network) or by using rsync (requires more iops on the gitpull-servers).

We think this is a good solution, but none of us have any experience with deployments at this scale, so we aren't sure if we've overseen anything or if this could be done in any other (potentially better) way.

It would be really helpful to get feedback on this, as well as to know about the conventional or the recommended way of handling a problem like this at this scale.

I don't think the proposed plan here makes much sense.  Tools like Ansible are free and designed to do this with good, central management and logging.

Put code in BitBucket.
Make a Jenkins server (open source / free / low on resources
/ well documented and easy to use).
Make BitBucket call Jenkins with a hook on change.

Or, if you feel like it, Jenkins can poll bit-bucket, but this is less efficient.

Have Jenkins call ansible (also open source / free / low on resources
/ well documented and easy to use) in its pipeline.
Ansible can target all 5000 servers with a dynamic inventory based on querying a database/file/API/whatever you have.
This way, bit-bucket has almost no load, ansible does everything from one location, runs are recorded in Jenkins with info on each host.

You won't have any API / rate limits here and you'll easily be able to target more hosts, with no effort on your end, with the dynamic inventory.

This is industry standard tooling.  You can also use Ansible to keep the servers up to date patch wise and all kinds of other stuff while you're at it, so it's a re-usable solution.


AWS Inspector "High Risk" : port open to the internet with IP whitelisting We use AWS Inspector to identify risks with our AWS assets. Today I received a finding, with a "high" severity. The finding said that the FTP port was open to the internet. However, after reviewing the security group associated with the instance, I saw that the FTP port is only open to a set of whitelisted IP addresses that are trusted.
Does this really constitute a high severity in your opinion? Is it problematic to open ports to the internet with a specific set of whitelisted IPs?

If it is FTP (and not FTPS) then yes, it is a high severity risk. You are sending unencrypted traffic over the open internet. Even if you are whitelisting specific IP addresses, the traffic is still unencrypted between those specific instances.
In addition, FTP is a very old (created in 1971) and non-secure protocol. It is not meant to be a secure method of transferring files. I recommend you research some of the security concerns with FTP and switch to either FTPS or more preferably, use a more native AWS approach for transferring files between instances. If your organization is under any regulation (FEDRAMP, HIPPA, etc.), using FTP across the open internet will not meet the compliance requirements.


How to add timeout to a helm chart? I have a helm chart which pulls a lot of images from various registries and deploys a lot of pods. It runs a lot of k8s jobs before getting the pods up.
Overall, helm install command takes huge time, so usually my helm install will also have --timeout 3600.
Like: helm install --name will also have --timeout 3600
Can I embed this --timeout 1800 to the helm chat itself? (rather than providing a command line parameter)

Since the timeout is a Helm option and not a Kubernetes object attribute, it can't be embedded in the chart itself.
The --timeout option is the amount of time the Helm utility will wait for Kubernetes commands to complete before marking the release as FAILED.
More Info on Helm CLI options


Why screen folder in /var/run disappears in Docker container after running Ansible playbook? Files

I've got the following Dockerfile:

FROM ansible/ansible:ubuntu1604py3
ENV DEBIAN_FRONTEND noninteractive
RUN pip3 install ansible
RUN printf '[local]\r\nlocalhost ansible_python_interpreter="env python3"\r\n' > /etc/ansible/hosts
ADD . /opt
RUN ansible-playbook /opt/install_screen.yml -c local -v
RUN useradd -d /home/ubuntu -ms /bin/bash -g root -G sudo -p ubuntu ubuntu
USER ubuntu
WORKDIR /home/ubuntu
CMD ["/usr/bin/env", "screen"]

and the following install_screen.yml playbook file:

- hosts: localhost
  tasks:
    - name: Install screen (a full-screen window manager)
      apt: name=screen state=present
      register: screen_installed
    - name: Configure screen
      shell: bash -x /etc/init.d/screen-cleanup start
      register: screen_configured
      when: screen_installed is success
    - shell: ls -la /var/run/

Build

Now, I build the Docker container in a shell by:

$ docker build -t screen .

and the Configure screen task shows the following output:

  changed: [localhost] => {"changed": true, "cmd": "bash -x /etc/init.d/screen-cleanup start", "delta": "0:00:00.009613", "end": "2018-05-01 23:09:49.159514", "rc": 0, "start": "2018-05-01 23:09:49.149901", "stderr": "+ set -e\n+ test -f /usr/bin/screen\n+ SCREENDIR=/var/run/screen\n+ case \"$1\" in\n+ test -L /var/run/screen\n+ test -d /var/run/screen\n+ find /var/run/screen -type p -delete\n++ stat -c%a /usr/bin/screen\n+ BINARYPERM=2755\n+ '[' 2755 -ge 4000 ']'\n+ '[' 2755 -ge 2000 ']'\n+ chmod 0775 /var/run/screen\n+ exit 0", "stderr_lines": ["+ set -e", "+ test -f /usr/bin/screen", "+ SCREENDIR=/var/run/screen", "+ case \"$1\" in", "+ test -L /var/run/screen", "+ test -d /var/run/screen", "+ find /var/run/screen -type p -delete", "++ stat -c%a /usr/bin/screen", "+ BINARYPERM=2755", "+ '[' 2755 -ge 4000 ']'", "+ '[' 2755 -ge 2000 ']'", "+ chmod 0775 /var/run/screen", "+ exit 0"], "stdout": "", "stdout_lines": []}

then my ls -la /var/run/ shell command shows:

  changed: [localhost] => {"changed": true, "cmd": "ls -la /var/run/", "delta": "0:00:00.004997", "end": "2018-05-01 23:09:49.442670", "rc": 0, "start": "2018-05-01 23:09:49.437673", "stderr": "", "stderr_lines": [], "stdout": "total 24\ndrwxr-xr-x 6 root root 4096 May  1 23:09 .\ndrwxr-xr-x 1 root root 4096 May  1 23:09 ..\n-rw-r--r-- 1 root root    0 Mar 28 19:06 init.upgraded\ndrwxrwxrwt 2 root root 4096 Feb 28 19:13 lock\ndrwxr-xr-x 2 root root 4096 May  1 23:09 mount\ndrwxrwxr-x 2 root utmp 4096 May  1 23:09 screen\ndrwxr-xr-x 2 root root 4096 May  1 23:09 systemd\n-rw-rw-r-- 1 root utmp    0 Feb 28 19:13 utmp", "stdout_lines": ["total 24", "drwxr-xr-x 6 root root 4096 May  1 23:09 .", "drwxr-xr-x 1 root root 4096 May  1 23:09 ..", "-rw-r--r-- 1 root root    0 Mar 28 19:06 init.upgraded", "drwxrwxrwt 2 root root 4096 Feb 28 19:13 lock", "drwxr-xr-x 2 root root 4096 May  1 23:09 mount", "drwxrwxr-x 2 root utmp 4096 May  1 23:09 screen", "drwxr-xr-x 2 root root 4096 May  1 23:09 systemd", "-rw-rw-r-- 1 root utmp    0 Feb 28 19:13 utmp"]}

Problem

It seems after installing screen and running /etc/init.d/screen-cleanup start during provisioning, the container have the problem running screen command as below:

$ docker run -it screen
Cannot make directory '/var/run/screen': Permission denied
$ docker run -it screen ls -la /var/run/screen
ls: cannot access '/var/run/screen': No such file or directory

In above Ansible's output I can see that /var/run/screen folder exists as per the following line (during ls -la /var/run/ run):

  drwxrwxr-x 2 root utmp 4096 May  1 23:09 screen

But it disappears after Ansible playbook command is finished.

Workaround

The following one-time workaround works to recreate the folder:

$ docker run -it -u root screen bash
ubuntu# bash -x /etc/init.d/screen-cleanup start
...
+ mkdir /var/run/screen
...

Rather than solving the immediate problem you're having, let me suggest an alternative solution to your actual problem:

  I need screen, so I can use multiple interactive shells in one instance of Docker. If I run multiple Docker instances, each takes 1G of RAM, so 10+ instances will kill my RAM.

Run screen (or tmux, but screen if you insist) locally on your workstation.  Start up your docker container, and in the windows where you want to do something on the container, use docker exec -it <container-id> bash to get a new interactive shell on the container.  This will be much simpler than running screen inside the container, and allows you to also have windows running commands on your workstation.

You may also want to look into using Vagrant with the Docker provider to simplify the process of doing development on a Docker container.


Does every docker container has a a OS? Does every container have an OS?

For example, the base docker python image
has a default OS in it? I couldn't find any info on Google.

I am in doubt because there are some images made with Python that use the Alpine OS.

Your docker image could be without an OS, that's called distroless image.

https://docs.docker.com/develop/develop-images/baseimages/

https://github.com/GoogleContainerTools/distroless


Where do you start with hosting the infrastructure for your infrastructure? So I'm trying to get started with the HashiCorp stack, but the question I keep asking myself, when looking at any of this DevOps infrastructure stuff, is what infrastructure should you be using to host the tools that will be managing - well - your infrastructure?

Like, for example, should I just be spinning up individual VMs to host each part of the HashiCorp stack? Should I be doing that instead using containers? Should I be managing those containers? With the same tools I'm going to be using to manage the containers managed by those containers?

Not a stupid question, but it's lacking details. If by stack you mean Terraform, Vault, Consul etc, then you would go about provisioning machines/VMs/containers for those services likely using some form of ansible or equivalent.

It would be prudent to store those scripts in git or any other source control system.

  Like, for example, should I just be spinning up individual VMs to host each part of the HashiCorp stack? Should I be doing that instead using containers?

Personally, I find containers more flexible especially when upgrading/ testing. The only downside is you have to remember to mount a data volume into the container to persist any configuration data which would normally live beyond the lifecycle of the container.


How can I validate GoCD configuration before applying it We have a GoCD installation where we like to keep the pipeline configuration XML files (cruise-config.xml) in a separate git repository, and install it (preferably via a pipeline and automatically) to the GoCD server whenever it is changed.

However we couldn't find an easy way to validate the XML config file before sending it off to GoCD, and obviously we do't want to install invalid configs, as that would break our pipelines. Are there any tools, or utilities that help us with validating the config file?

The best solution we found so far, is using the MagicalGoConfigXmlLoader  interface from the go source code, as it calls an internal validateCruiseConfig command, that does all validations, or die if the file is invalid:

MagicalGoConfigXmlLoader loader = new MagicalGoConfigXmlLoader(new ConfigCache());
loader.loadConfigHolder(configAsString);

The main problem with this solution that it essentially requires the whole GoCD server source code checked out, in a working state to work properly. Also as it's an internal API it is subject to changes between GoCD versions, making GoCD upgrades harder.


Two identical Azure deployments; one always throws 404 - how to find out what the difference is? I am deploying a C# ASP.NET Core web service to Azure using Pulumi. I can deploy it in 3 ways:

Run it locally from Visual Studio, i.e., not using Azure at all.
Deploy it to Azure from my local developer computer.
Deploy it to Azure from Jenkins (whicn runs on a different computer).

I have this problem:

When I run it locally, I can call the service fine, e.g. from Postman or from a C# application. The web service returns what I expect.
When I deploy it to Azure from my local machine, I can also call it fine. The web service returns what I expect.
When I deploy it to Azure from Jenkins and then try to call the webservice, it returns "NotFound" to all calls no matter what I do. (This presumably means HTTP 404.)

The deployments in 2 and 3 should be exactly the same. My question is: How can I find out what the difference is between these two deployments in Azure?
The Jenkins-deployed webservice exhibits the following curious behaviour:

It does not log any exceptions (even when I wait several minutes for them to show up).
If I go to my resource group -> Application Insights -> Logs and search for "requests", it does list requests. Curiously, it says that it returned HTTP 200 to all the requests, even though what I get when calling them is 404.
The above is true even for web service calls that should never return 200 (they should return 201).
The above is true even for web service calls to methods that shouldn't even exist (i.e., when I deliberately corrupt the method URI before calling the service).

During deployment I authenticate with Azure using a service principal. My Jenkinsfile looks like this:
withVaultSecrets([
    "path/to/secret/in/vault": [
        "sp_name", "application_id", "object_id", "sp_secret"
    ]
]){
    script {
        env.PULUMI_CONFIG_PASSPHRASE = 'jenkinspassphrase'
        env.ARM_CLIENT_ID = "${application_id}"
        env.ARM_CLIENT_SECRET = "${sp_secret}"
        env.ARM_TENANT_ID = "${azure_dev_tenant_id}"
        env.ARM_SUBSCRIPTION_ID = "${azure_dev_subscription_id}"
        env.AZURE_CLIENT_ID = "${application_id}"
        env.AZURE_CLIENT_SECRET = "${sp_secret}"
        env.AZURE_TENANT_ID = "${azure_dev_tenant_id}"
    }//script
    dir("./src/deploy/KmsStack"){
        powershell "pulumi login --local";
        powershell "pulumi stack init jenkinsfunctionaltest --secrets-provider=passphrase"
        powershell "pulumi up --yes"
    }//dir
}//withVaultSecrets

The script which I use to deploy locally looks like this, with the same service principal credentials:
cd $PSScriptRoot
cd webapi
dotnet publish /p:DisableGitVersionTask=true
cd ../deploy/KmsStack
$env:PULUMI_CONFIG_PASSPHRASE = 'jenkinspassphrase'
$env:ARM_CLIENT_ID = ...
$env:ARM_CLIENT_SECRET = ...
$env:ARM_TENANT_ID = ...
$env:ARM_SUBSCRIPTION_ID = ...
$env:AZURE_CLIENT_ID = ...
$env:AZURE_CLIENT_SECRET = ...
$env:AZURE_TENANT_ID = ...
pulumi logout
pulumi login --local
pulumi stack rm jenkinsfunctionaltest -y
pulumi stack init jenkinsfunctionaltest --secrets-provider=passphrase
pulumi stack select jenkinsfunctionaltest
pulumi up --yes

How can I find out why these two deployed services behave differently? The Azure portal GUI is rich and has lots of sections. Can you recommend me where to look? Might there be some security settings that differ? How can I find them?
Thanks in advance!

We found out what was wrong. It was not an Azure issue. The problem was that we were deploying a bad ZIP file. The ZIP file was missing web.config, which meant that the web application could not start up.
We were zipping our published web application by having this in the CSPROJ file:
<Target Name="ZipOutputPath" AfterTargets="Publish">
  <ZipDirectory SourceDirectory="$(OutputPath)\publish" DestinationFile="$(MSBuildProjectDirectory)\kmswebapp.zip" Overwrite="true" />
</Target>

This turned out not to work because the compiler does things in a different order than we expected. At the time when it generated the ZIP file, web.config was not generated yet, so web.config never got packed into the ZIP file. Hence Azure could not start the application.
When we deployed from our local machines, it worked because we didn't clean the publish directory before each run, so there would be a web.config left over from the previous run, and this old (but unchanged) web.config would get packed into the ZIP file and deployed to Azure, so Azure would know how to start the application.
We solved it by removing the above from our CSPROJ file and doing (roughly) this in our Jenkinsfile:
powershell "dotnet publish ./src/webapi/WebAPI.csproj"
powershell "if (!(Test-Path('${publishDirectoryPath}/web.config'))){throw 'We need web.config to exist in the publish directory'}"
powershell "Compress-Archive -Path '${publishDirectoryPath}/*' -DestinationPath './src/webapi/kmswebapp.zip' -Force"

This generates a proper ZIP file including web.config, and Azure can now start our application so it can respond properly to requests.


ingrex-nginx in kubernetes with express node.js does not return req.currentUser This is currentUser middleware:
    export const currentUser = (
      req: Request,
      res: Response,
      next: NextFunction
    ) => {

      if (!req.session?.jwt) {
        return next();
      }
      try {
        const payload = jwt.verify(
          req.session.jwt,
          process.env.JWT_KEY!
        ) as UserPayload;
        console.log("payload in curretnUser middleware", payload);
        req.currentUser = payload;
      } catch (err) {
        console.log(
          "error in currentUser middleware in common module",
          err.message
        );
      }
      next();
    };

this is payload object that logged:
      payload in curretnUser middleware {
      id: '609b356f99a625001a76e65e',
      email: 'test@test.com',
      iat: 1620784495
   }

this is the router in "auth" service:
router.get(
  "/api/users/currentuser",
  currentUser,
  (req: Request, res: Response) => {
    // if req.currentUser is not defined it will send undefined
    console.log("request to api/users/currentuser", req.currentUser);
    res.send({ currentUser: req.currentUser || null });
  }
);

When request hits this route, this gets logged:
request to api/users/currentuser {
     currentUser: {
     id: '609b356f99a625001a76e65e',
     email: 'test@test.com',
     iat: 1620784495
   }
 }

How ever when I send POST request through postman I get currentUser:null. Somehow kubernetes or ingress-nginx is not attaching "currentUser" to request object and I could not figure out why. This is the nginx-ingress config:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-srv
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.org/client-max-body-size: "4m"
    nginx.ingress.kubernetes.io/use-regex: "true"
    nginx.ingress.kubernetes.io/enable-cors: "true"
    nginx.ingress.kubernetes.io/auth-method: POST
spec:
  defaultBackend:
    service:
      name:
        nginx-ingress-default-backend
        # name: default-http-backend:80
      port:
        number: 80

  rules:
    - host: ticketing.dev
      http:
        paths:
          - path: /api/users/?(.*)
            pathType: Exact
            backend:
              service:
                name: auth-srv
                port:
                  number: 3000

Try using this:- use-forwarded-headers
apiVersion: v1
kind: ConfigMap
data:
  use-forwarded-headers: "true"
metadata:
  name: nginx-configuration
  namespace: default


Combine Lists of Objects in Ansible I'm really trying to merge lists of objects along with defaults so I can loop over them and create resources in kubernetes. I've gotten it down to a simple(ish) playbook and will post it here.

- hosts: localhost
  vars:
    default_thing_1: 123
    default_thing_2: 456
    default_transport: train
    list_of_things:
      - name: foo
        common_thing_1: "sam"
      - name: bar
        common_thing_1: "sam"
        common_thing_2: "ham"
      - name: biz
        common_thing_1: "eggs"
  tasks:
    - name: Set the Things Facts
      set_fact:
        thing:
          i_am: "{{ item.common_thing_1 | default(default_thing_1) }}"
          eggs_ham: "{{ item.common_thing_2 | default(default_thing_2) }}"
      loop: "{{ list_of_things }}"
      register: things_fact_results
    - name: Set Optional Things Facts
      set_fact:
        thing:
          transport: "{{ item.transport | default(default_transport) }}"
      when: "item.common_thing_2 is defined and item.common_thing_2 == 'ham'"
      loop: "{{ list_of_things }}"
      register: transport_fact_results
    - name: Debug the Facts
      debug:
        msg: "{{ item.0 | combine(item.1|default({})) }}"
      loop: "{{
        things_fact_results.results |
        map(attribute='ansible_facts.thing') | list |
        zip(
          transport_fact_results.results |
          map(attribute='ansible_facts.thing') | list
        ) | list
      }}"

The output is as follows...

PLAY [localhost] ***********************************************************************************************************************************************************************************

TASK [Gathering Facts] *****************************************************************************************************************************************************************************
ok: [localhost]

TASK [Set the Things Facts] ************************************************************************************************************************************************************************
ok: [localhost] => (item={'name': 'foo', 'common_thing_1': 'sam'})
ok: [localhost] => (item={'name': 'bar', 'common_thing_1': 'sam', 'common_thing_2': 'ham'})
ok: [localhost] => (item={'name': 'biz', 'common_thing_1': 'eggs'})

TASK [Set Optional Things Facts] *******************************************************************************************************************************************************************
skipping: [localhost] => (item={'name': 'foo', 'common_thing_1': 'sam'})
ok: [localhost] => (item={'name': 'bar', 'common_thing_1': 'sam', 'common_thing_2': 'ham'})
skipping: [localhost] => (item={'name': 'biz', 'common_thing_1': 'eggs'})

TASK [Debug the Facts] *****************************************************************************************************************************************************************************
fatal: [localhost]: FAILED! => {"msg": "Invalid data passed to 'loop', it requires a list, got this instead: [({'i_am': 'sam', 'eggs_ham': '456'}, AnsibleUndefined), ({'i_am': 'sam', 'eggs_ham': 'ham'}, {'transport': 'train'}), ({'i_am': 'eggs', 'eggs_ham': '456'}, AnsibleUndefined)]. Hint: If you passed a list/dict of just one element, try adding wantlist=True to your lookup invocation or use q/query instead of lookup."}

PLAY RECAP *****************************************************************************************************************************************************************************************
localhost                  : ok=3    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0

I think if I'm reading the loop item that's what I'm looking for. Though I'm not sure why ansible is refusing to perform the loop.

Any help about how to merge lists of objects together with optional attributes that could or could not be set because k8s is pretty picky about what's being set and not when passing in objects.

Thanks.

loop requires a valid list. But we got here is,

[({'i_am': 'sam', 'eggs_ham': '456'}, AnsibleUndefined),
 ({'i_am': 'sam', 'eggs_ham': 'ham'}, {'transport': 'train'}),
 ({'i_am': 'eggs', 'eggs_ham': '456'}, AnsibleUndefined)]

Since no value is assigned to transport_fact_results when common_thing_2 is not defined.

if you remove the when condition it will work.

CODE:

- hosts: localhost
  vars:
    default_thing_1: 123
    default_thing_2: 456
    default_transport: train
    list_of_things:
      - name: foo
        common_thing_1: "sam"
      - name: bar
        common_thing_1: "sam"
        common_thing_2: "ham"
      - name: biz
        common_thing_1: "eggs"
  tasks:
    - name: Set the Things Facts
      set_fact:
        thing:
          i_am: "{{ item.common_thing_1 | default(default_thing_1) }}"
          eggs_ham: "{{ item.common_thing_2 | default(default_thing_2) }}"
      loop: "{{ list_of_things }}"
      register: things_fact_results
    - name: Set Optional Things Facts
      set_fact:
        thing:
          transport: "{{ item.transport | default(default_transport) }}"
      when: "item.common_thing_2 is defined and item.common_thing_2 == 'ham'"
      loop: "{{ list_of_things }}"
      register: transport_fact_results
    - name: Debug the Facts
      debug:
        msg: "{{ item.0 | combine(item.1|default({})) }}"
      loop: "{{
        things_fact_results.results |
        map(attribute='ansible_facts.thing') | list |
        zip(
          transport_fact_results.results |
          map(attribute='ansible_facts.thing') | list
        ) | list
      }}"

OUTPUT:

PLAY [localhost] ************************************************************************************************************************************************************************************************

TASK [Gathering Facts] ******************************************************************************************************************************************************************************************
ok: [localhost]

TASK [Set the Things Facts] *************************************************************************************************************************************************************************************
ok: [localhost] => (item={u'common_thing_1': u'sam', u'name': u'foo'})
ok: [localhost] => (item={u'common_thing_1': u'sam', u'common_thing_2': u'ham', u'name': u'bar'})
ok: [localhost] => (item={u'common_thing_1': u'eggs', u'name': u'biz'})

TASK [Set Optional Things Facts] ********************************************************************************************************************************************************************************
ok: [localhost] => (item={u'common_thing_1': u'sam', u'name': u'foo'})
ok: [localhost] => (item={u'common_thing_1': u'sam', u'common_thing_2': u'ham', u'name': u'bar'})
ok: [localhost] => (item={u'common_thing_1': u'eggs', u'name': u'biz'})

TASK [Debug the Facts] ******************************************************************************************************************************************************************************************
ok: [localhost] => (item=[{u'eggs_ham': u'456', u'i_am': u'sam'}, {u'transport': u'train'}]) => {
    "msg": {
        "eggs_ham": "456",
        "i_am": "sam",
        "transport": "train"
    }
}
ok: [localhost] => (item=[{u'eggs_ham': u'ham', u'i_am': u'sam'}, {u'transport': u'train'}]) => {
    "msg": {
        "eggs_ham": "ham",
        "i_am": "sam",
        "transport": "train"
    }
}
ok: [localhost] => (item=[{u'eggs_ham': u'456', u'i_am': u'eggs'}, {u'transport': u'train'}]) => {
    "msg": {
        "eggs_ham": "456",
        "i_am": "eggs",
        "transport": "train"
    }
}

PLAY RECAP ******************************************************************************************************************************************************************************************************
localhost                  : ok=4    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0


How to ensure Nodes are killed when Job is finished using Cluster Autoscaler in Kubernates I have large Kubernates jobs which take the whole resources of one node / droplet in Digitalocean (it's not running on AWS).
Using resources in a deploy.yaml, the cluster autoscaler works to scale up, but sometimes the nodes continue to run instead of killing / scaling down when the job is done.
How do I make sure the nodes get killed when the job is done? They are expensive!

When a Job completes, no more Pods are created, but the Pods are not deleted either.
https://kubernetes.io/docs/concepts/workloads/controllers/job/#job-termination-and-cleanup

clear it by another cronjob.
use operator to wrapper it and add remove mechanism in the operator.


SqlPackage.exe: Error SQL0: Required contributor with id 'id' could not be loaded while trying to build dacpac using sqlpackage.exe, we are getting the following error

Microsoft.SqlServer.Dac.DacServicesException: An error occurred during deployment plan generation. Deployment cannot continue.
Error SQL0: Required contributor with id 'BcDevKit.DeploymentContributors.DacMacroSqlDeploymentContributor' could not be loaded.

The above error can be fixed if I copy the contributors from
    C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Common7\IDE\Extensions\Microsoft\SQLDB\DAC\150\Extensions

to C:\Program Files (x86)\Microsoft SQL Server\150\DAC\bin\Extensions

I want sqlpackage to search for contributors only in Visual Studio directory. Can anyone help?

I have figured the answer on my own and wanted to help someone like me.

So, the problem is I was referencing sqlpackage.exe from C:\Program Files (x86)\Microsoft SQL Server\150\DAC\bin directory, that is why it was looking for the contributor in it's sub-directory \Extensions.

I was looking around in Visual Studio installation directory and found the sqlpackage.exe in C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Common7\IDE\Extensions\Microsoft\SQLDB\DAC\150 directory, updated the path in environment variables and it worked.


Site monitoring browser-only tool running in frontend only? Infrastructure monitoring tools like Nagios or much more "low-level" curl are well known in the DevOps community.

But, given I would like just to track some APIs in similar manner just in frontend, what are  known more or less mature popular tools or components to support this functionality? i.e. like Nagios but running completely inside the browser (loading static configuration file from server together with the JS libs would be ok).

You have a few options for this:

If you can pay for this, use commercial API monitoring solutions like Runscope or APImetrics. They're easy to setup, have no maintenance overheads, and show you detailed data and alerting etc. The problem with this approach is that they cost money, and if you have a miniscule ops budget or this is a hobby project, it can be hard to justify spending on this.

...which brings me to the alternative: roll your own. If you have your own system monitoring and metrics setup (Nagios/Icinga/Sensu, and Grafana etc), you can write check scripts for whatever data you need (availability, response time or latency etc). Some of these tools integrate with time series databases like graphite/influxDB, so you can push the check data to that and use a visualization tool like Grafana to show graphs and charts based on this data (here's a blog post describing Icinga-Grafana integration)


Jenkins Declarative pipeline: Execute stage conditional on execution of previous stage I have a Jenkins declarative pipeline with several stages. Some of these stages need to be executed based on whether a previous stage has been executed. In pseudocode:

pipeline{
  stages{
    stage('1: Always do this thing') {
      steps {
        sh 'do_thing.sh'
      }
    }
    stage('2: Maybe do this thing') {
      when{ condition }
      steps {
        sh 'sometimes_do_this_thing.sh'
      }
    }
    stage('3: Do this thing if 2 was executed'){
      when { pipeline.stages.2.was_executed }
      steps {
        sh 'conditional_path.sh'
      }
    }
  }
}

I accept that the best way to do this might be in the scripted pipeline, but I have declarative pipelines. Including some scripted bits in them is also acceptable, if kept to a minimum.

So: What is the most elegant way to execute a stage in a pipeline conditional on the execution status of the previous one?

A pretty simple solution is to declare a global variable on top of your pipeline, and check it's value in your last conditional stage:

def flag = false;

pipeline {
    stages {
        stage('1 - Always') {
            steps {
                sh './always.sh'
            }
        }

        stage('2 - Maybe') {
            when { condition }

            steps {
                sh './maybe.sh'
                script { flag = true }
            }
        }

        stage('3 - If Maybe was executed') {
            when { expression { flag == true } }

            steps {
                sh './conditional.sh'
            }
        }
    }
}


Jenkins in private subnet For development environment, to be used by multiple developers, we have a pipeline to be created with Git-Jenkins-Tomcat(dev)
Do I need to place jenkins in private subnet and access thru bastion host?

If you want to make it private either you need to setup a vpn so that you can access jenkins gui using http://private-subnet-ip:jenkins_port or you can move jenkins behind a load balancer and use load-balancer-dns to access jenkins gui. If it is in public subnet then use NACL/Security Groups to restrict access only to trusted sources. For more security, do not assign public ip to jenkins machine, assign dns hostname(even to load balancer you should do that) and implement ssl over http so that it can be accessed over https only.