In Jenkins when to use SSH site and when to use a node? What are the advantages of each option? What are the disadvantages? Is there a best practice in this? I fail to see any critical differences between them.

In my current place of employment we use nodes exclusively, however they are all in our internal network.

I have a task that needs to be delegated to an external server. It does not need access to our GIT. Essentially, it will upload data to a file server, and does not need to access our servers, network, data, etc.

If I need to build, run checks, tests, ... I use a node.

If I want to open a connection to a server to deliver the built code to production for example, I use the sshCommand step from SSH Step plugin. When the code is delivered the ssh connection is closed.

Having a node always connected to your jenkins can be unpleasant. Let's say the server you connect to belongs to a partner and he wants to audit the times you connect to put the files on the server. In that case he cannot say. Having a node always connected also means you need to maintain the connection, imagine the server is configured to disconnect you after a while or the server reboots and the connection is not relaunched.


Multiple git profiles especialy with Atom I have configured Git to use a specific CA certificate, but how can opt-out for it other than using the switch -c http.sslVerify=false? (or enforce other certificate which seems better in terms of security).

Okay, a workaround was to create a shell script for a shortcut and use it like

gitf clone

That is okay for a quick test in a secured and trusted environment but it is a potential security breach. Do not try this at home.

The real gotcha comes then in if I would like to use the Atom editor. Somehow I cannot find an option for the git executable.

You'd better add this CA to a bundle of valid CA and keep using it for all cases with

git config --add --global http.sslcainfo=/path/to/cabundle
git config --add --global http.sslbackend=openssl

Using https://curl.haxx.se/docs/caextract.html as starting CA bundle sounds a good idea in my opinion.


How to connect GCP Load balancer to Kubernetes cluster(single node cluster) I'm creating a single node(name: node1) k8 cluster for demos. I have setup a nginx-ingress controller.

service/volted-marmot-nginx-ingress-controller        LoadBalancer   10.100.132.29   <pending>     80:30787/TCP,443:30043/TCP   3h48m

I can access it by node1_ip:30043 for e.g. How do I put a load balancer in front of it? What I want is to have loadbalancer_ip:443 point to node1_ip:30043. If that's the wrong way to think of it, please let me know. I can't do it anyway, as the target pool does not have any option to choose port(e.g. 30043) anyway.

If I set the loadbalancer port to 300043 then loadbalancer_ip:300043 works but I was hoping to hide that.

You can use the GCP HTTP(S) Load balancer. Since you are running a kubernetes cluster (Assuming you are using GKE), GKE uses Google compute VM instance groups which can be put behind a load balancer as target group.

You can follow the below steps to setup a load balancer and put your kubernetes node behind a load balancer.

Login to GCP console and go to NETWORKING > Network Services > Load balancing
Click on Create Load Balancer at the top of the page.
Click on Start Configuration for HTTP(s) Load balancing
Select From Internet to my VMs if you want to access it from public internet and click continue
Give a Name to your load balancer
Click on Backend Configuration and Select Backend services (from drop down) > Create a backend service
Give a name to backend service and select the other parameters appropriately. Select Instance groups in Backend type.
In the New backend section, select the kubernetes instance group name from the drop down. After selecting the proper instance group, you can enter the ports which will accept the requests. You can modify other things if required. At the end, click Create.
Next you can setup Host and Path rules if you need any.
Next, you have to configure Frontend configuration which will be your frontend to accept the requests on the port and send it to the backend target group which we configured in the step 8.
Once all the configurations are done, click on Create to create your load balancer.

If everything was setup correctly, you should be able to access the contents from the load balancer endpoint.


docker micro-service cannot connect to bitcoin docker I have a micro-service watchblockchain that makes requests on bitcoin blockchain. When I use both services, everything works just fine.
Then I want to put those services in docker containers.
watchblockchain cannot communicate with the bitcoin docker.
I use docker image: kylemanna/bitcoind/

Here is my bitcoin.conf file:

disablewallet=1
printtoconsole=1
rpcauth=jfjobidon:123...xyz
testnet=1
txindex=1
server=1
rpcport=8332

I can open a console in the bitcoin container and verify that the blocks are downloading and I can make requests on the blockchain
ex: bitcoin-cli getblockcount

In watchblockchain, I have this connection config:

var clientBTC = new bitcoin.Client({
    host: 'bitcoin',
    port:  8332,
    user: 'jfjobidon',
    pass: 'Lw.....='
});

And I mapped bitcoin to 127.0.0.1 in my hosts file

My docker-compose.yml:

version: "3"
services:
  watchblockchain:
    build: ./watchBlockchain
    depends_on:
      - redis
    deploy:
      replicas: 1
      restart_policy:
        condition: on-failure
  redis:
    image: redis:4.0
    command: redis-server --appendonly yes
    ports:
      - "6379:6379"
    deploy:
      placement:
        constraints: [node.role == manager]
    command: redis-server --appendonly yes

When I launch: docker-compose up, I have the following error:

  Error: connect ECONNREFUSED 127.0.0.1:8332

I think it is a network communication problem.

How can I get my watchblockchain service to communication with the bitcoin docker ?

the containers hosts file will differ from your hosts hosts file.

see it kind of like a VM.

Also 127.0.0.1 is the wrong address, you might need to configure the actual ip/hostname your host has and where the service is listening on inside the watchblockchain container.

So Problems:

Your docker container knows nothing about the host bitcoin. an entry in your docker hosts /etc/hosts won't change that.
Maybe your bitcoin service only binds to localhost 127.0.0.1 - this will also keep any docker container from connecting to it. Maybe you can configure it so that it binds to the docker bridge IP. So only all the docker containers are able to connect to it.
It would be easier if your bitcoind would also be a docker container and was configured inside the docker-compose.yml. That way you could resolve the host using the container name. But still your bitcoind should be configured to bind on every interface.

Hope this helps ;)


How to resolve merge conflicts in Azure DevOps current UI I have followed the instructions on command line:
https://docs.microsoft.com/en-us/azure/devops/repos/git/merging?view=azure-devops&tabs=visual-studio

all I want to do is resolve this merge conflict its complaining about.

To be clear, I have done a git pull, it says Already up to date.

I also tried:

git pull origin feature/ENGA-2514
From ssh.dev.azure.com:v3/nfib/NfibEnterprise/engage.mobile
 * branch                feature/ENGA-2514 -> FETCH_HEAD
Already up to date.

I have also tried:

git log --merge
fatal: --merge without MERGE_HEAD?

Where is this elusive merge conflict?

Try this solution: Pull Request Merge Conflict Extension. It helps to resolve conflicts inside the pull request page.


In a Database Per Service model - should i include the database in the service container? I am implementing a simple setup with two micro services and would like each of them to have their own DB - each service is running in it's own container.
Should i include the DB's in each service container (so each container will contain the service application + DB with a total of two containers), or should i create a new container for each DB (with a total or four containers)?.
Given the fact that each DB is used only by one service, it seems cleaner to encapsulate the Service+DB in a single container which will be the overall service. On the other hand - deploying a DB as a container is much simpler and faster than combining it with the application container.

Keep the DB containers separate. The concept of microservices and containers is to allow each part to be independently updated without introducing changes and outages to the rest of the application stack. Containers should also be running a single app per container to allow error detection and log gathering from that application. Each container is simply an isolated process, so unlike a VM, there's little overhead on the host to split each of these out.


Which organization types are targeted by DevOps transformation? Due to my lacking vocabulary in this context I am not sure how to designate the two types of organizations I would like to talk about before I can put my question. Anybody capable of better wording feel free to improve this.
Probably there are mixed types but I suppose I have to polarize a little bit to make the point.

OrgType1: "digitalized product organizatons" either B2B or B2C, they have their products, as they become more and more digital, they have some own IT product teams and IT operations. They have found out that they need (more than before) product diversification quite recently.

OrgType2: "digital consulting service organizations" (not necesarily just IT consulting but they are typical example here) these guys are B2B only, if they have sustainable products and related operations then it is rather a side effect.
Rather, their software product lifecycle is limited to just one or several projects, and that diversification has always been their core business. Actually, the real product is here much more every single member of this organization who depending on the role might, but by far not always does, develop software.

OK then, so OrgType1 is clearly what we already know that DevOps is of a great help, because there are established delivery processes, they get disrupted, need to be optimized, and so on and so forth. Well, bigger organizations consist of many parts, regional business units, business lines, departments, and every one has its combination between operations and their-own-breed-of Devops. Okay - anyway there can be some evolution.

OrgType2 can care about DevOps only so much as it is important for their customer (or they even sell DevOps consulting ;) but then if it comes to onsite software development, you somehow start from zero in almost every project, and this is magnified as with OrgType1 through all departments, business units and so on but it is worth because every "DevOps entity" starts just with 1-2 people who need to discover what they are up to. Seems like OrgType2 is "siloed by design" in a much more dense way which makes it also somehow scalable and persistent.

This all considered and taken into account, the question from my current perspective is: how do DevOps practices apply for the OrgType2, maybe there are resources where this has been explored?

DevOps applies to Organization Type 2, the Business to Business entity as much if not more than the first Organization type. There are two possible scenarios here:

Scenario 1

In the first they are selling a product or service to businesses. In terms of DevOps, this makes them virtually indistinguishable from Organization Type 1. Though their customer is a business/group of individuals, they are providing a product or service to a customer. This customer just happens to be a business instead of a private individual, this is is irrelevant to execution of DevOps. Contracted with specific software goals, clearly defined requirement or otherwise, the best value and customer service will be provided by a DevOps model for reasons that do not need to be restated here.

Scenario 2

This scenario happens to be a bit more exceptional as in this case, Organization Type 2 is providing consulting services to a business. Presumably and usually, a business hires a consultant for business assistance within the consultant's domain of expertise. Businesses have technical support for assistance with specific technologies and can contract en engineer for migrations, but when issues such as risk management, overall architectural design, process, flow and high-level questions arise, this is when a consultant is hired for their business advice.

This is why it is important to remember that DevOps is not a role. I would argue that DevOps represents a business or organizational management processes - or even a philosophy. It is more or less a specific way of implementing an agile methodology that has been extended well beyond just development to encompass many disciplines and departments. As such, DevOps is a package deal - in order to be successful at DevOps, you have to be committed to a specific business philosophy of organizational management.

In fact, in order to illustrate this point, In "The Phoenix Project", Board member Erik Reid repeatedly drags the Book's main character, Bill Palmer, to a parts manufacturing plant in order to illustrate and teach Bill things about technology and software development by drawing business lessons from the manufacturing sector.

This is authors Gene Kim, Kevin Behr and George Spafford's way of driving home the point that DevOps is more than just way of doing things, or specific technologies, but is a philosophy of business simply known by various nom de plumes.

So, in order to provide the best value to the company to which a consultant is consulting, it is incumbent upon the consultant to demonstrate and explain the value of DevOps and assist a company in successfully executing on this business philosophy and guide the company to specific technologies and ways of doing that. In short, it is the consulting companies job to act as an "Erik Reid" to whatever business has contracted their consulting services.

In short, DevOps should be very important to Organization Type 2 because it is their job as a consultant to explain to their customer why it is important - they should always be selling DevOps consulting. Consultants working with a company who refuses to consider DevOps should consider politely suggesting that perhaps they aren't the right fit. their customer can hire a generic code monkey from anywhere if they want to continue to (unsuccessfully) continue to do projects the way they have always done them.


Kubernetes configuration with yaml anchors I have a deployment.yaml file and want to reuse the environment for all my deployments like this:
apiVersion: apps/v1
kind: Deployment
metadata:
  name: beat
spec:
  selector:
    matchLabels:
      app: beat
  template:
    metadata:
      labels:
        app: beat
    spec:
      containers:
      - name: beat
        image: myimage
        command: ["celery", "-A", "wsgi.celery", "beat"]
        env: &default
        - name: FIRST_ENV
          value: my-value
        - name: SECOND_ENV
          value: another-value
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: flower
spec:
  selector:
    matchLabels:
      app: flower
  template:
    metadata:
      labels:
        app: flower
    spec:
      containers:
      - name: flower
        image: myimage
        command: ["celery", "flower", "-A", "wsgi.celery"]
        env: *defaultenv

But it seems like kubectl apply -f deployment.yaml won't work with YAML anchors.
error: error parsing deployment.yaml: error converting YAML to JSON: yaml: unknown anchor 'defaultenv' referenced

Is it possible to use YAML anchors or is there another preferred approach of how to reuse repeating blocks for k8s configuration?

YAML anchors are supported, but only for the same YAML file. You can't create the anchor (&) on a deployment file and reference (*) the value on another one.
If you want to share ENVs values across multiple Deployments, you can create a ConfigMap with the ENVs and use the envFrom spec. Example:
apiVersion: v1
kind: ConfigMap
metadata:
  name: my-configmap
data:
  FIRST_ENV: my-value
  SECOND_ENV: another-value
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: beat
spec:
  selector:
    matchLabels:
      app: beat
  template:
    metadata:
      labels:
        app: beat
    spec:
      containers:
      - name: beat
        image: myimage
        command: ["celery", "-A", "wsgi.celery", "beat"]
        envFrom:
        - configMapRef:
            name: my-configmap
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: flower
spec:
  selector:
    matchLabels:
      app: flower
  template:
    metadata:
      labels:
        app: flower
    spec:
      containers:
      - name: flower
        image: myimage
        command: ["celery", "flower", "-A", "wsgi.celery"]
        envFrom:
        - configMapRef:
            name: my-configmap


SaltStack Reactor fails with exception I have the next conf files for salt reactor:

reactor.conf

  'salt/netapi/hook/cd':

  /srv/reactor/dev.sls

dev.sls

# vim: sts=2 ts=2 sw=2 et ai

{% set payload = data.get('post') %}

{%      set master = [grains.fqdn] %}

core|api|{{ master }}|jenkins-cd:
  runner.state.orchestrate:
    - tgt: {{ master }}
    - mods: myreactor.deploy
    - pillar:
        temp_var_transmit: {{ payload }}

deploy.sls

{%      set payload = salt.pillar.get('temp_var_transmit') %}
{%      set master = [grains.fqdn] %}
my_state:
  salt.function:
    - name: cmd.run
    - tgt: {{ master }}
    - arg:
      - "salt-run state.orch my-reactor.cd_state pillar='{\"temp_var_transmit\":{\"host\":\"{{payload['host']}}\",\"artifact_id\":\"{{payload['artifact_id']}}\",\"artifact_version\":\"{{payload['artifact_version']}}\"}}'"
{% endfor %}

  4. cd_state.sls

{% set data = salt.pillar.get('temp_var_transmit') %}
my_deploy:
  salt.state:
    - tgt: {{ member }}
    - expr_form: compound
    - sls: '{{ data.artifact_id }}_{{ data.artifact_version }}'
    - queue: true
    - test: {{ dry_run_mode }}

When I send API request and run salt-runner.events, I get the next error:

jid": "20180119065728472630",
    "return": "Exception occurred in runner state.orchestrate: Traceback (most recent call last):\n  File \"/usr/lib/python2.7/dist-packages/salt/client/mixins.py\", line 395, in _low\n    data['return'] = self.functions[fun](*args, **kwargs)\nTypeError: orchestrate() argument after ** must be a mapping, not list\n",
    "success": false,
    "user": "Reactor"

Any thoughts?

I was able to find the root of issue. The problem was in the configuration of salt master. After every change of reactor config, I must run salt-call state.highstate command on master to sync reactor source code changes with it. I missed this step and thus my reactor config differed from my original source code in IDE. So once I ran mentioned command, everything became in sync. Thanks everybody for help!

P.S. regarding mentioned error, it occurred because I used kwarg in dev.sls for pillar and represented pillar as list:

>     kwarg:
>      - pillar:
>          temp_var_transmit: {{ payload }}

Kwarg can not contain a list in this case. It may contain key:value format only. So the correct way is either use without kwarg as I did or remove hyphen near pillar.


Elasticsearch on Docker - file permission issues I am trying to test LDAP in elasticsearch/kibana by running it in Docker on Windows 10.
I pulled the image down:
docker pull docker.elastic.co/elasticsearch/elasticsearch:7.10.0

And ran it:
docker run -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" docker.elastic.co/elasticsearch/elasticsearch:7.10.0

I did the same with Kibana and everything came up as expected.  I then upgraded the elastic licence and started the 30 day trial to get the LDAP features in X-Pack.
I opened a shell to the container and updated the elasticsearch.yml file with my LDAP information and set the path to the role_mapping.yml file.  I also updated role_mapping.yml file with the LDAP informaiton. I followed the instructions here: https://www.elastic.co/guide/en/elasticsearch/reference/7.10/ldap-realm.html
Now when I restart the elasticsearch container I get the following error then the container halts:
uncaught exception in thread [main]
java.lang.IllegalStateException: security initialization failed
Likely root cause: java.security.AccessControlException: access denied ("java.io.FilePermission" "/role_mapping.yml" "read")
at java.base/java.security.AccessControlContext.checkPermission(AccessControlContext.java:472)
at java.base/java.security.AccessController.checkPermission(AccessController.java:1036)
at java.base/java.lang.SecurityManager.checkPermission(SecurityManager.java:408)
at java.base/java.lang.SecurityManager.checkRead(SecurityManager.java:747)
at java.base/sun.nio.fs.UnixPath.checkRead(UnixPath.java:810)
at java.base/sun.nio.fs.UnixFileSystemProvider.exists(UnixFileSystemProvider.java:524)
at java.base/java.nio.file.Files.exists(Files.java:2514)
at org.elasticsearch.xpack.core.XPackPlugin.resolveConfigFile(XPackPlugin.java:361)
at org.elasticsearch.xpack.security.authc.support.DnRoleMapper.resolveFile(DnRoleMapper.java:86)
at org.elasticsearch.xpack.security.authc.support.RoleMappingFileBootstrapCheck.create(RoleMappingFileBootstrapCheck.java:47)

I overwrote the elasticsearch.yml file by copying a version with the LDAP config commented out so I could start the container.  When I did this I checked the file permissions and both elasticsearch.yml and role_mapping.yml have the same:
-rw-rw---- 1 elasticsearch root  199 Dec  2 15:51 elasticsearch.keystore
-rwxr-xr-x 1 root          root  568 Dec  2 18:36 elasticsearch.yml
-rw-rw---- 1 elasticsearch root 2301 Nov  9 21:28 jvm.options
drwxrwxr-x 2 elasticsearch root 4096 Nov  9 21:32 jvm.options.d
-rw-rw---- 1 elasticsearch root 8451 Nov  9 21:35 log4j2.properties
-rwxr-xr-x 1 elasticsearch root  502 Dec  2 20:15 role_mapping.yml
-rw-rw---- 1 elasticsearch root  197 Dec  2 20:10 roles.yml
-rw-rw---- 1 elasticsearch root    0 Nov  9 21:32 users
-rw-rw---- 1 elasticsearch root    0 Nov  9 21:32 users_roles

I even tried CHMODing the role_mapping.yml file to 777 but I still get the same result.
I feel like I'm missing something obvious...

I have discovered the issue, the path to the role_mapping.yml file was incorrect in elaticsearch.yml. I had to use the full path /usr/share/elasticsearch/config/role_mapping.yml


Inspec tests conditional on OS version What is the idiomatic way to tell Inspec to run test X on OS version A and test Y on OS version B? I'm aware of the technique of dumping the Chef node object to a JSON file in /tmp but that isn't doing it for me as that runs within the Test Kitchen VM whereas Inspec executes on the real host and therefore can't see it.

What I would suggest, like comments under your post, is to take this to the #inspec channel on the Chef Community Slack to ask there. That said, here's how I might attack this:

control 'some-control-slug' do
  title 'My control'
  impact 1.0
  desc 'Example code for Stack Exchange'

  if os.windows? && ::Gem::Version.new(os.release) < ::Gem::Version.new('6.1')
    # Only executed on Windows machines older than 2008r2
    describe directory('C:/Users') do
      it { should exist }
    end

  elsif os.windows? && ::Gem::Version.new(os.release) >= ::Gem::Version.new('10')
    # Only executed on Windows Server 2016 or newer
    describe directory('D:/MyFolder') do
      it { should_not exist }
    end

  end

  # Executed on all host types
  describe sys_info do
    its('hostname') { should match(/SomeAwesomeHost/i) }
  end
end

Additionally, you can limit the entire inspec control with Chef-like hooks, such as only_if. The above example code, as written, may execute on a Linux target. We could limit all of them by placing the following up near the desc or title:

only_if { os.windows? }


Azure Devops, multiple on-prem environments, variables per environment I've configured first deployment pipeline to an on-prem DEV environment.
The project consists of 3 identical environments, DEV, UAT and PROD
While their setup is exactly alike, their config (database credentials etc.) is obviously not.

I'm using the DevOps variables to alter web.config and appsettings.json configuration files while deploying.
Therefore I'd like to apply the same deployment pipeline, targeting 3 different environments with 3 different variables sets.

This is the way I've seen in Octopus. Is it possible in the DevOps?

You can create a Release Pipeline with a stage for every environment.
On each stage you can copy files, etc. and run tests against the environment.
If the test/stage succeeds, the pipeline goes ahead to the next stage.
Between the stages, you can configure that an authorized person has to approve the next step.
The screenshot below is a release pipeline of one of my little side projects.


hello world with flask and redis using docker networking The sample from Docker, a "hello world" with redis and flask, seems to build fine, but, as an exercise can it be broken into two components?

The redis service uses a public Redis image pulled from the Docker Hub
registry.

and

Compose pulls a Redis image, builds an image for your code, and starts
the services you defined. In this case, the code is statically copied
into the image at build time.

What I have in mind is to use the default redis image as its own container, and then modify the flask app so that it connects to the redis image.  This is reasonable?

Yes, it's absolutely reasonable and this is how real apps work. This is the microservice approach which allows to independently scale instances of a service and a DB.
What you have to do is to run your Flask and Redis services in a way so the first one can achieve the second one. If they are on the same machine, you just have to open Redis port used by your Flask app (probably 6379):
docker run -d -p 192.168.10.20:6379:6379 your_redis_image

And specify IP address and port of the Redis container in your Flask app. This is better to be done with a config file like the following yaml:
redis:
  host: '172.16.10.20'
  pw: 'YourPassWord'
  port: 6379
  db: 0

And use it in the code:
import yaml, redis

cfg_file open('config.yaml', 'r')
cfg = yaml.load(cfg_file)
redcfg = cfg['redis']

r = redis.Redis(
    host=redcfg['host'], port=redcfg['port'],
    db=redcfg['db'], password=redcfg['pw'],
)

Don't forget to copy this file into container to be accessible from the inside.


How to use tags in Amazon Web Services and Azure I am currently building a solution on Azure (though the question should relate to AWS equally) and I am using a UniqueId tag to locate and use objects.

For instance, we have a storage account that is used for caching applications' blobs. The storage account is created with a random name but is found and presented to the application through its UniqueId=AppCache tag.

At the moment I am taking the approach that as more tags become useful I'll start to add them.

Ultimately we're looking for a hands off 'ChatOps' style solution. So everything created should be automated.

Should I be looking at a structured approach to tagging (and what does that look like) or is the organic approach more sensible?

I would set up a base taxonomy to start with, you can use automation across your estate to ensure that all resources follow the taxonomy.  You have a choice between simply deleting resources that don't comply or reporting on it to a central team who are responsible for chasing down and educating the owners.

I have followed the Best Practice set out by Microsoft for both Azure and Amazon:

Owner - normally this is a team distribution list although sometimes it is an individual if it is a "10% time" project.  To make sure that the e-mail address entered is valid we run a hourly scan of our estate to detect if there are new e-mail addresses in this field then send an e-mail with an activation link, failing to activate results in deletion after three days.
CostCenter - the numeric cost centre, we use this to feed into the end of month reconciliation.
SDLCPhase - one of Development, Test, SIT, UAT, PerfTest, PreProduction, Production.  We use this to define different automated behavour:

Development and Test: deleted overnight and at weekends to save on costs.  Teams are responsible for reprovisioning using the automation in the morning.
SIT, UAT and PerfTest: deleted when the cost center in our finance system is closed down.
PreProduction: scaled down overnight and when the cost center is closed down.
Production: we validate that all resources deployed to the production subscription have this tag, if they don't we send out e-mails extensively.

DropDeadDate - for us, this is a required field for everything except Production.  We run a nightly scan and send e-mails to the owner 7 days and 1 day before the DropDeadDate then delete the resource after it has passed.

In terms of the automation around this I have a tendency to use Janitor Monkey and some custom bots written is Azure Functions that sit in Slack Channels and do my bidding.


What's better: 2 pods (on the same server) or 1 standalone application? For a client we're going to roll out a certain type of software installation.
(I can give more details in the comments if you like.)
The question is: would you go for 2 pods (on the same server)?
Or would a single installation be more stable?
The software is stable as such but my concern is pods, by nature, are meant to be replaceable...
There is also the option of a single pod on the machine, but I have deep concerns about this approach...
My guess is that once 1 pod gets into to much trouble (memory pressure to name a thing)
Kubernetes is going to restart it, as pods are meant to be replaceable.
(Bringing down the clients' application during the process.)
So I guess 2 pods is a solution here.
1 pod should be able to handle the load (his load + the extra load of the restarting pod).
What I don't know is whether 2 pods would be as stable as running the software on 1 node,
without Kubernetes.
In the same process we're also going to roll out 2 pods on 2 virtual machines, with other software,
so the choice whether we do or don't use Kubernetes doesn't depend on this part of the implementation alone.

Is the app stateless

The software is stable as such but my concern is pods, by nature, are meant to be replaceable

This is your primary concern. Is the app stateless (e.g. can the requests be served by any of multiple instances without problem?). If not - be very careful to run it on Kubernetes (e.g. don't use rolling-deployment) and don't run the app in the cloud :)
Apps running on Kubernetes or cloud should follow The Twelve Factor principles
Number of Servers
It would be good to be able to use two or more servers. This reduce the Single-Point-Of-Failure, and you can e.g. take down one server for maintenance but still serve your users.
App directly on Server or in Container

pods (on the same server) or 1 standalone application?

An application runs as a process on the Operating System, whether you run it on a container or as non-container process. There is almost no runtime overhead, what you get with containers is repeatability and maintainability.
A container is designed to be hermetically sealed, so it is not sensitive to your servers configuration e.g. what language-settings or character encoding. This was a bigger problem before containers was used. With containers the app runs in an identical environment every time, except the kernel. In addition it is easier to maintain and manage processes when run as containers, you can e.g. set limits for CPU usage, Memory limit and they get its own network e.g. multiple containers on the same server can listen to port 8080 (from the applications standpoint).
If your apps run as containers on a server cluster, you can also easyli scale out or in by adding or removing servers.
Control of resources

My guess is that once 1 pod gets into to much trouble (memory pressure to name a thing) Kubernetes is going to restart it, as pods are meant to be replaceable.

Yeah, but this is a good thing. Kubernetes is mostly self-healing. If you don't have such "control-loops", you will need manual internvention - since it is the same app that e.g. may have memory leaks.
Run app on Kubernetes or not

whether we do or don't use Kubernetes doesn't depend on this part of the implementation alone

The first thing to check: Is the app stateless?
and also, you should use two or more servers otherwise you get little advantage of running it in Kubernetes.
There is a cost to maintain Kubernetes. You need skills and you need to maintain an addition system. Make sure it adds enough value for you.


Baking Immutable Images I am working on a project and we are trying to bake immutable images on containers as IaC. These images can have J2EE, .NET or Python apps on it. All OS patches should be applied to this image frequently and app updates should be also updated on this ready to run images.
Having said that, an option we discuss includes Terraform for provisioning, Packer for building images. Many internet resources deploy additionally Ansible, Chef or Puppet into this mix. My question why Packer is not enough to cover baking images why would I need to consider Ansible/Chef/Puppet additionally? What can't Packer do and others can for these requirements?
Thanks for your time and help...

An idea key to your question is that of immutable infrastructure, which is the idea that an image is built once, deployed many times, and never changed at runtime. If the contents of the image need to change, an entirely new image is built from scratch and new instances replace the ones running the old image.

This is the opposite of what you might now call mutable infrastructure, where often a machine (or container) is booted from a generic OS image and then its contents managed by a configuration management tool on an ongoing basis. In this model, instances generally live longer and are changed in-place to respond to new requirements.

Packer is intended as a building block for immutable infrastructure. It is intended to orchestrate the creation of customized images that boot immediately into the desired state. If immutable infrastructure is the right fit for your use-case (which is not necessarily the case!) then indeed tools such as Ansible, Chef and Puppet may be unnecessary.

However, running these tools in a "one-shot" mode (e.g. Chef Solo) with Packer can often be useful for non-trivial images: these tools generally provide more powerful abstractions than Packer alone provides, making it easier to do things such as configure the init system to run certain services, or create user accounts. In this case, the configuration management tool would run as a provisioner in Packer, creating the necessary configuration within the image, but is not involved in the ongoing upkeep of instances booted from the image.

Most practical systems include a mixture of mutable and immutable infrastructure, since e.g. immutable infrastructure is inappropriate for stateful, centralized systems. In that case, using a traditional configuration management system can be useful to create a common base of functionality between your immutable and mutable systems, applying configuration in a one-shot way for immutable images and in an ongoing, stateful way to long-running, mutable machines.


Terraform get list index on for_each Terraform newbie here. I'd like to iterate a list using for_each, but it seems like the key and value are the same:

provider "aws" {
  profile = "default"
  region  = "us-east-1"
}

variable "vpc_cidrs" {
  default = ["10.0.0.0/16", "10.1.0.0/16"]
}

resource "aws_vpc" "vpc" {
  for_each             = toset(var.vpc_cidrs)
  cidr_block           = each.value
  enable_dns_hostnames = true
  tags                 = { Name = "Company0${each.key}" }
}

I'd like the tag Name to be "Name" = "Company01" and "Name" = "Company02" but according to terraform apply,

I get: "Name" = "Company010.0.0.0/16" and "Name" = "Company010.1.0.0/16"

What am I missing?

Found an easy solution using the index function:

tags = { Name = "Company0${index(var.vpc_cidrs, each.value) + 1}" }


Docker and shared memory optimization? Imagine you would run a bunch of services say based on the same Tomcat image.

Would Docker optimize memory usage for identical memory segments and reuse these patterns?

This question is discussed in some detail in issue 7950  The conclusion seems to be that:

Depending on your storage configuration, the kernel page cache may be used if multiple containers are accessing the same file (e.g., see OverlayFS and Docker Performance).
You may be able to enable Kernel same-page merging through use of of the ksm_preload tool.


Jenkins - restrict which nodes can be selected by user when creating jobs I have a situation where I have a Jenkins installation that is used by multiple teams. Each team gets their own Jenkins node, which has access to certain parts of the organization that the team is responsible of.
I want to allow the teams to create their own Jenkins jobs, but they should only be allowed to set those jobs to run on their respective Jenkins node. So I need a way to allow users to create jobs, but restrict the nodes they can choose.
The only thing we found is Job Restrictions plugin, which is no longer maintained at this time and up to adoption. And we had some trouble with it.
What are my options?

You could use Project-based Matrix Authorization Strategy for user authorisation in Jenkins.
Works well with Folders plugin, then you have each project in a separate folder and permissions on each folder are set to a team that looks after the project.
Each node/slave can have access permissions specified. For example, for node A you will only list Team A as users/team who can have Build and Configure permissions. Team B for node B etc. It is a pain if you have dozens of nodes though.
If integrated with AD, then groups as they are defined in AD can be used to restrict/give access without having to list all members of the team who needs access to which node/project.
Authorisation strategy is defined in
"Manage Jenkins" -> “Configure Global Security Option" -> "Authorisation" -> "Project-based Matrix Authorization Strategy"
This article describes it to a degree: https://www.thegeekstuff.com/2016/06/jenkins-security/
You could do the same with just Matrix Authorization Strategy, only you won't have projects. You might not need them at all (depends on how many project/teams you have) and then you will only restrict node permissions.


does boto3 have a way to get resource limits from a region? Right now I am looking for ec2 instances type limits for a given region.  I would like to put in place an audit script that tracks our usage vs our limits for a given set of things.  Also open to other tools that work well with Python

thanks

I'm sure it does, otherwise it wouldn't be the module in use in:

https://awslimitchecker.readthedocs.io/en/latest/

From their setup.py installation script:

requires = [
    'boto3>=1.4.6',
    'botocore>=1.6.0',
    'termcolor>=1.1.0',
    'python-dateutil>=2.4.2',
    'versionfinder>=0.1.1',
    'pytz'
]

Thing is, maybe what you're trying to build is already built in some way or another. Try searching for libraries that has more or less the description of the tool you're trying to build, before starting.

Happy coding!


What should I use for configuration management on AWS? I am trying to find a solution for configuration management using AWS OpsWorks. What I can see is AWS offers three services for OpsWorks

Chef Automate
Puppet
AWS stacks

I have read basics of all three of them but unable to compare between three of them. I am unable to understand when to use which solution.

I want to implemnet a solution for my multiple EC2 instances, using which I can deliver updates to all my instances from a central repository(github). And, rollback changes if needed.

So following are my queries:

Which of the three solutions is best for this use case?
What should I use if my instances are in different regions?

I am unable to find anything useful on these topics so that I can make my decision. It would be great if I can get links to some useful articles as well.

Thanks in advance.

A good way to find comparison information about things like this is googling for "X vs Y", e.g. "Chef vs AWS stacks", "Chef vs Puppet" or something like that. That does turn out subjective information, and while it is nothing like having hands-on experience, you still get a few nice nuggets here or there.

For example, Chef gives you the full Ruby language while Puppet has a ruby-based DSL. Does that matter for you? Only you can tell.

From my experience with Puppet/Ansible (and I don't know Chef yet) it's all mostly down to taste. For example, I really like the fact that Ansible needs only a working ssh key and nothing else. And I like that there is no central repository. Others may like the fact that Puppet has a repository. Also, I like the dead simple configuration file format. But that is all subjective.

So, to cut to the chase, my advice would be:

Use AWS Stacks if you are fully committed to AWS, will never change to another provider, and like the higher integration with AWS features. It uses Chef internally anyways, and exposes the Chef files to you.
Use Chef directly if you like to learn something new and want to stay a bit vendor independent. If you later decide to commit to AWS fully, then you should be able to do so.
Use Puppet (which AWS starts to support as well, seemingly) if you, well, like Puppet more, or if you can't stand Chef.


How does continuous integration relate to continuous delivery / deployment? Here is a quote from the current content of continuous-integration:

  ... process of merging developer's working code copies to a shared codebase frequently to prevent or minimize integration problems.

OK, I get that. But then there is also continuous-delivery and  continuous-deployment, and that's where I continuously get a bit lost:

How does continuous integration relate to continuous delivery and/or continuous deployment, assuming that somewhere along the line(s) via integration you end up delivering in a target environment where everything will be deployed.
What's the difference between continuous delivery and continuous deployment?

Back in the days, before DevOps was called DevOps, we used terminology which might possibly help to understand these new DevOps terms, such as:

promote to (or demote from) some pre-prod target, optionally combined with some type of regeneration process (compiles, binds, etc) to package all related components together in executable-like things. That's what should be similar/close to continuous integration, or not?
distribute to some target environment, using something like FTP (if standard copies cannot bridge the gap), but do not yet activate it in the target. That's what should be similar/close to continuous delivery, or not?
install (or activate) in some target environment, combined with things like binds, stop/start operations, etc. That's what should be similar/close to continuous deployment, or not?

Continuous delivery and continuous deployment both take continuous integration one step further, by adding a 'deployment to production' step to the process.
The difference between continuous delivery and deployment is that for delivery this step is done manually and for deployment is it automatic.

Difference between Continuous Integration, Continuous Delivery and Continuous Deployment. Picture copied from codeproject.com

Whether you do continuous delivery or continuous deployment is very much an implementation choice. If you do continuous deployment, changes in code will be deployed automatically after the acceptance tests are passed. This may or may not be desirable for your product. With continuous delivery, people can make a choice whether a particular code change  is deployed or not (and possibly where exactly it is deployed).

Since the difference between continuous delivery and deployment is small and many people are unaware of the exact difference, the two terms are sometimes used interchangeably.


Kubernetes - Deploying older version of app in new API I'm working with yaml files built for an older version of the API

apiVersion: extensions/v1beta1

I tried changing to

apiVersion: apps/v1

But couldn't figure out the correct "selector" formatting. So I changed back to v1beta1

Then I ran across this command:

kubectl convert -f . | kubectl -n robot-shop create -f -

All seemed to deploy correctly. But now when I try to reach my app, I get connection refused. After some investigation, I took a look at iptables and see this:

iptables -L | grep 30080
REJECT     tcp  --  0.0.0.0/0            0.0.0.0/0
/* robot-shop/web:8080 has no endpoints */ ADDRTYPE match
dst-type LOCAL tcp dpt:30080 reject-with icmp-port-
unreachable

After some more research, this seems to have something to do with kube-proxy and "selector".

Question is, what's the right course of action?
1. Change the fw rule to ACCEPT
2. make the correct changes to yaml file (need help with that)
 a. If so, do I delete the deployment and then make correct changes, then redeploy?

Here's my deployment yaml file:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  annotations:
    kompose.cmd: kompose -f ../docker-compose.yaml convert
    kompose.version: 1.10.0 (8bb0907)
  creationTimestamp: null
  labels:
    io.kompose.service: web
  name: web
spec:
  replicas: 1
  strategy: {}
  template:
    metadata:
      creationTimestamp: null
      labels:
        io.kompose.service: web
    spec:
      containers:
      - image: linuxacademycontent/rs-web:latest
        name: web
        ports:
        - containerPort: 8080
        resources:
          limits:
            cpu: 200m
            memory: 100Mi
          requests:
            cpu: 100m
            memory: 50Mi
      restartPolicy: Always
status: {}

And my service yaml file:

apiVersion: v1
kind: Service
metadata:
  annotations:
    kompose.cmd: kompose -f ../docker-compose.yaml convert
    kompose.version: 1.10.0 (8bb0907)
  creationTimestamp: null
  labels:
    io.kompose.service: web
  name: web
spec:
  type: NodePort
  ports:
  - name: "8080"
    port: 8080
    targetPort: 8080
    nodePort: 30080
  selector:
    io.kompose.service: web
status:
  loadBalancer: {}

Prior to apps/v1 the selector field was implicitly picked up from the pod template's labels. So in your case it would be:

spec:
  selector:
    matchLabels:
      io.kompose.service: web

Starting with apps/v1 an explicit selector field is required by the Kubernetes API.

I would suggest not to change the iptables on your own as it it the responsibility of your Kubernetes network plugin and is most likely done well.

As far as I am concerned you may simple update the Deployment manifest and no uninstall is required.


How to arrange builds for staging and production with Azure DevOps? On Azure DevOps I need to build and deploy a ClickOnce application. Because this application needs to be signed (including config) I must create a build for staging and a build for production.

I decided to create an artifact for my App with 2 tasks:

At the moment I'm using the XML transformation to apply my specific staging or production settings. It is working fine but I would like to use the Variables substitution system build in Azure DevOps. I already use it in Deployment part for all my IIS project deployment. But in Deployment I have "stage" and I can easily tell the system to use my staging variable in staging stage and production one for my production stage. I cannot do that in Build Pipeline section.

What can I do? Should I create 2 separate builds with 2 separate artifact?

I think I see what you are after. Build variables created in the Variables tab of the GUI designer are scoped to the entire definition. But you can set a Job Scoped variable from a script within a GUI based pipeline.

So within the context of each of the Agent Jobs you have defined above, you would add a powershell task that would allow you to set the variable(s) that you then use for the automagic variable substitution in your xml.

YAML is better

If you want to go the YAML definition route you can actually create Job scoped variables directly in yaml. This example below substitutes against a JSON file but the concepts are similar for xml.

This is the Json file it substitutes against test.json.

{
    "menu": {
        "id": "file",
        "value": "File"
    }
}

trigger:
- master

pool:
  vmImage: 'ubuntu-latest'

jobs:
- job: Production
  variables:
    - name: menu.id # Substituting this value in test.json
      value: 'productionValue'
  steps:
    - task: FileTransform@1
      displayName: 'File Transform:'
      inputs:
        folderPath: '$(System.DefaultWorkingDirectory)'
        fileType: json
        targetFiles: test.json
    - task: PowerShell@2
      inputs:
          targetType: 'inline'
          script: |
            Get-Content -Path test.json

- job: Staging
  variables:
    - name: menu.id  # Substituting this value in test.json
      value: 'stagingValue'
  steps:
    - task: FileTransform@1
      displayName: 'File Transform:'
      inputs:
        folderPath: '$(System.DefaultWorkingDirectory)'
        fileType: json
        targetFiles: test.json
    - task: PowerShell@2
      inputs:
          targetType: 'inline'
          script: |
            Get-Content -Path test.json

Custom Conditions

Another thing you might want to dig into is that Azure DevOps exposes task level custom conditions. You could add both transform tasks for Stage and Production to the pipeline definition and then only conditionally run the step you want. You could potentially collapse this into just one job and only build what you need to.

You need to decide on what that condition is. For example, you could do it based on the source branch (probably better) or just a variable that you can set at queue time.


How to add LDAP group for ssh access by ansible playbook? I have LDAP Group.

superproduct1

In LDAP Group has some users:

developer1, tester1, analitic1, project1

I want to create ansible playbook for add LDAP group to SSH access on host.

Found only

group – Add or remove groups

- name: Ensure group "somegroup" exists
  group:
    name: somegroup
    state: present

How to add LDAP group superproduct1 for ssh access?

You don't need to edit /etc/groups since the LDAP group already exists in your LDAP database. You want to let SSH know to use it, try using the AllowGroups setting in /etc/ssh/sshd_config.

From the manpage: "If specified, login is allowed only for users whose primary
group or supplementary group list matches one of the patterns.  Only group     names are valid; a numerical group ID is not recognized.  By default, login is allowed for all groups."

See this question for more information


I was trying ti set up presto with azure gen2 , but facing the below issue when I am querying I am using prestodb:0.234

Dependency jars:

   azure-storage-2.0.0.jar
   hadoop-azure-3.2.1.jar
   hadoop-azure-datalake-3.2.1.jar
   azure-data-lake-store-sdk-2.3.8.jar

I am getting below error while doing the querying.

com.facebook.presto.spi.PrestoException: cannot create caching file system
    at com.facebook.presto.hive.HiveCachingHdfsConfiguration.lambda$getConfiguration$0(HiveCachingHdfsConfiguration.java:70)
    at com.facebook.presto.hive.HiveCachingHdfsConfiguration$CachingJobConf.createFileSystem(HiveCachingHdfsConfiguration.java:94)
    at org.apache.hadoop.fs.PrestoFileSystemCache.get(PrestoFileSystemCache.java:59)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:373)
    at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295)
    at com.facebook.presto.hive.HdfsEnvironment.lambda$getFileSystem$0(HdfsEnvironment.java:73)
    at com.facebook.presto.hive.authentication.NoHdfsAuthentication.doAs(NoHdfsAuthentication.java:23)
    at com.facebook.presto.hive.HdfsEnvironment.getFileSystem(HdfsEnvironment.java:72)
    at com.facebook.presto.hive.HdfsEnvironment.getFileSystem(HdfsEnvironment.java:66)
    at com.facebook.presto.hive.BackgroundHiveSplitLoader.loadPartition(BackgroundHiveSplitLoader.java:298)
    at com.facebook.presto.hive.BackgroundHiveSplitLoader.loadSplits(BackgroundHiveSplitLoader.java:270)
    at com.facebook.presto.hive.BackgroundHiveSplitLoader.access$300(BackgroundHiveSplitLoader.java:100)
    at com.facebook.presto.hive.BackgroundHiveSplitLoader$HiveSplitLoaderTask.process(BackgroundHiveSplitLoader.java:199)
    at com.facebook.presto.hive.util.ResumableTasks.safeProcessTask(ResumableTasks.java:47)
    at com.facebook.presto.hive.util.ResumableTasks.access$000(ResumableTasks.java:20)
    at com.facebook.presto.hive.util.ResumableTasks$1.run(ResumableTasks.java:35)
    at com.facebook.airlift.concurrent.BoundedExecutor.drainQueue(BoundedExecutor.java:78)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: No FileSystem for scheme: abfs
    at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2660)
    at org.apache.hadoop.fs.PrestoFileSystemCache.createFileSystem(PrestoFileSystemCache.java:114)
    at org.apache.hadoop.fs.PrestoFileSystemCache.getInternal(PrestoFileSystemCache.java:89)
    at org.apache.hadoop.fs.PrestoFileSystemCache.get(PrestoFileSystemCache.java:62)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:373)
    at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295)
    at com.facebook.presto.hive.HiveCachingHdfsConfiguration.lambda$getConfiguration$0(HiveCachingHdfsConfiguration.java:66)

Any suggestions?

Since ABFS mentioned in the stacktrace as a caching filesystem, then I suppose two possible reasons and suggest solutions for them

Dependency mesh in maven

As noted in two answers (one, two) to a similar question

  When we use maven-assembly-plugin, it merges all our JARs into one, and all META-INFO/services/org.apache.hadoop.fs.FileSystem overwrite each-other. Only one of these files remains (the last one that was added). In this case, the FileSystem list from hadoop-commons overwrites the list from hadoop-hdfs, so DistributedFileSystem was no longer declared.

So, check your dependencies and if there are JAR with correct org.apache.hadoop.fs.FileSystem in your distibution.

PrestoDB uses outdated azure-site.xml

There is known issue #11648.
And as said, ABFS, ADLS and ALDS gen2 was only introduced to https://github.com/prestosql/presto, not to prestodb

So, suggestion is to ask your question on Presto Community Slack

P.S. try storediag

There is a hadoop manual on Azure ABFS

And among other tips they recommend:

  One useful tool for debugging connectivity is the cloudstore storediag utility.

  This validates the classpath, the settings, then tries to work with the filesystem.


Is jenkins-x the successor of jenkins? I strongly doubt that jenkins-x is the successor of Jenkins. To me it looks like a tool that is able to create a ci/cd pipeline for new apps.

Jenkins X is a CI / CD platform for Kubernetes.
This project is a sub-project of Jenkins.
see below
here


Consulting DevOps from the Ops side Background

One of the org units in my company is doing application support for a relatively large client; that is, they're keeping some of their business applications running 24x7, doing anything from the root account upwards (no hardware/network). Things like keeping volumes, RAM, CPU sufficient, routing bug tickets where they need to go, backup&recovery, all that stuff. All the applications (many) are in a quite stable state, little development is going on.

It's 100% classical, i.e., 100% Ops. The occasional contacts with development are laden with the usual problems (mistrust, prejudice, etc.).

The customer decided that it wants to be modern, and is in the process of introducing "DevOps" - without having a clear understanding of what that actually means. They just know that they don't want to have one large entity running a large cluster of servers, not really knowing the software indepth, but that individual teams should build and support their own applications. Note that there is no container technology or anything like that - they are getting into OpenShift, but softly-slowly. Everything is quite classical... trouble ticket systems, multi-tier support levels, a great many tickets solved regularly, but all of them very easy in itself (like volumes running full etc.), very occasional difficult problems that need developer input.

So I'm on a taskforce of ours that's supposed to come up with ideas. I have quite some experience consulting Dev teams to be more "DevOps"y - i.e., more automation, more tooling, keeping dev/test/prod environments identical, that kind of thing. From their point of view, it's pretty clear how to proceed. But in my current example, it's not quite clear to me what to suggest.  They do have problems, but more in the "Ops" side (like finding enough good staff to man the 24x7 shifts and so on). There does not really seem to be a "Dev-Ops" problem to be solved.

Question

In an Ops-heavy department with only a small amount of development going on, what are some DevOps practises, culture changes, process changes etc. that are very likely

beneficial, no matter what they are doing so far
relatively easy to introduce
not that likely to induce fear and instinctive resistance (which there seems to be quite a case going on here, after the first talks)

For example, in a Dev team, I'd show them what to do with containers, or 12-factor, or ... and what concrete problem it solves, and this usually shows them that there's not much to fear, and that there are real benefits to be reaped. From there on, it goes as far as they want to take it.

The only objective problem I saw so far was that they really have little to no insight in the software, i.e., they mostly cannot even read the source (due to lack of knowledge) which would, in some cases, help in solving/routing bug reports etc.

The main subjective problem seems to be that they think real bad about developers. (According to them, those are lazy, picky, don't want to do 24x7, solve every ops problem simply by "re-rolling containers" etc.).

The key to presenting solutions is always to identify problems first; if the team doesn't feel like you're solving any problems, then they're going to dismiss what you're suggesting.  The specific problems your Ops team is facing we don't know, but there are a number of common ones:

Getting non-actionable alerts
Spending lots of time in toil
Version upgrades are a pain
Deployments are a pain
No visibility into what the apps are doing

and so on.  Once you know what problems they're having, then you can target solutions that fix those.

  The main subjective problem seems to be that they think real bad about developers.

Cultural changes are incredibly difficult to make (usually the easiest way to is hire people with the philosophies you want and gradually prune off the old people).  But embedding operations engineers with developers can help this one immensely.  I've worked in several small companies where there was just one engineering team (and I happened to work primarily on ops work), and so this idea comes very naturally to me, but for people who have been working solely on an operations team with little contact with developers for a long time, it can be a difficult adjustment.  But move your desk to be with them, participate in their chat channels, eat lunch with them - this all helps both ops and devs realize that the other side are actually humans.  This also provides better opportunities for ops to use their expertise to solve problems for devs that otherwise wouldn't've surfaced (say, they're trying to do something and you provide a quick shell script that solves it much faster than the manual way they were going with), and that builds trust and emotional budget that you can spend when you need something from them.


GKE: Sharing the same filesystem across pods I'm quite new and try to get my head around kubernetes and GKE so here it goes:

I would like an application to write data to a persistent storage.

If I'd scale up, or down, or a pod dies,
I'd like this persistent storage to survive.

Also, the same data should be available to different pods.

How do I achieve this?

The data are images, so storing them in a database could be a valid solution for this case,
but I wondered whether - if data is big - there are also solutions with filesystems
to achieve this...

Thanks for answering,

S.

I think it's a pretty n00b question :(

I suppose you'd need a pod which manages the filesystem,
and make it available to the other pods which need the functionality to store data on it...

Br,

S.


Using GSuite or Github to Manage who can log into Linux or Windows VMs on Azure I want to manage the members of our team who are able to log into our cloud-hosted VMs, both Linux and Windows.  We currently use G Suite and GitHub, to manage access to our code and documentation etc. I would really prefer to not have to also manage a separate Active Directory Domain within our VNets or deal with revoking ssh keys or updating multiple usernames/passwords on every single VM we deploy.

Are there ways to use either GSuite or Github to control login access to our VMs?

If there isn't, what are the best practices to deal with 20+ developers who are all empowered with DevOps responsibilities?

I would really prefer to not have to also manage a separate Active Directory Domain within our VNets

You've already found the answer: centralized authentication.

If you use a tool like Okta, you can indirectly tie GSuite to AD (or an LDAP server).  And then of course you can tie OpenSSH there as well.

  or deal with revoking ssh keys

Centralized authentication solves this.

  or updating multiple usernames/passwords on every single VM we deploy.

And this as well.

You've already got the answer, even if you don't like it.


Is it a common practice to include the version of an API into the rest interface itself? I was looking for the current version of the github API, but it seems to be omitted.

https://api.github.com/

I am personally considering to add a new field to my own API, but I wonder whether that is a common practice or should be done differently.

Many APIs include a version number in the url of the service. Some examples:

https://api.twitter.com/1.1/statuses/home_timeline.json
https://graph.facebook.com/v2.2/me/adaccounts
https://vision.googleapis.com/v1/images:annotate?key=YOUR_API_KEY
https://maps.googleapis.com/maps/api/js?v=3.29
https://westeurope.api.cognitive.microsoft.com/vision/v1.0/analyze
https://www.europeana.eu/api/v2/opensearch.rss?searchTerms=Rembrandt

However many people on the Internet argue that this is incorrect because it violates REST/HTTP specifications; "embedding of API version into the URI would disrupt the concept of hypermedia as the engine of application state". It’s more appropriate to use the Accept header to request a particular version. For the latter people suggest to use a Vendor MIME type, for example something like Accept: application/vnd.mycompany-api-v3+json

Nevertheless, the reason why many API designers do include an API version in the url is because it's easy to implement and debug (clearly visible which version you are using).

Side note: in our organization we recently decided to support both a versionless endpoint that always point to the latest stable version of our APIs, as well as an endpoint with a version specification. This allows users to choose if they want to commit to our latest API (which may require making changes when a newer version is released), or to connect to a fixed API version (which may become deprecated over time).


How to pin OS package versions in Docker image Scenario

For obvious reasons I believe that every package installed in a Docker image should have their version nailed down.
That's why in our Dockerfile we always enforce the OS package manager to install a specific version of a package.
For example:

FROM node:6.10.0

RUN apt-get update \
      && apt-get -y install nginx=1.6.2-5+deb8u6 \
      && apt-get -y install supervisor=3.0r1-1+deb8u1

Problem

The upstream repository has changed and the nginx=1.6.2-5+deb8u6 package is now invalid.

I'm not sure what brings about this change but I think it has something to do with distros going out of their support-cycle or that old packages are removed/moved somewhere else.

Question

How do I nail the exact version of OS-leveled packages without running into this problem?

Note: I know these 2 options will work but I want to know if there's an easier way out:

Make a custom base image
Download the binaries and COPY them into the Docker image

I'm not sure that this qualifies as an "easier" way, but you could host the specific packages on a server (e.g., Artifactory, S3).  You would then have the Dockerfile connect to this location and use the packages from there.

This would allow you to avoid creating a custom base image and you would not have to manually download the packages.


What are fetching DevOps analogies? Some presenters use analogies to clarify a certain technology, e.g. Pizza as a Service 2.0 that explains the differences between different as-a-Service (aaS) stacks.

Advantages of this Pizza analogy is that it consists of multiple analogies, i.e. runtime aka pizza and homemade aka legacy.

When one Googles "DevOps analogy", various images are shown, but not of them is very catchy.

Definition of "fetching"

Show the image in a presentation
Talk 30 seconds about it
During the elevator pitch more and more people understand DevOps and it is completely clear by them.

Mostly for devs but knowledgeable to others with the "disaster girl" meme:
"Works on my machine.. Ops problem now!" This illustrates that lack of responsibility can endanger the whole company, and the value of software working only in a specific environment is not absolute.

Also, the matrix of hell. Adding Docker might seem as one column more, but containers will become to-go technology and the standard architecture on long term. So, you can run Docker containers also with Kubernetes or Apache Mesos.


use of nginx server_name I am new to nginx, I have configured my consul with nginx.

It's working but I have few doubts in my configuration.

here is my configuration

upstream consul {
  server 127.0.0.1:8500;
}

server {
  listen 80 default_server;
  server_name localhost;

  location / {
    proxy_pass http://consul;
  }
}

this is working fine if hit http://localhost, in my machine I am able to see consul UI.

But If I change my location to /consul, it is giving 404.

and what is the use of server name, if I change it to server_name mylocalserver;

I think I should be able to run my gttp request on this address http://mylocalserver. but that is also not working.

You need to add a host entry of mylocalserver in the host file against 127.0.0.1, like 127.0.0.1 mylocalserver.
For /consul, there should be a /consul endpoint in your code to serve your request.

As per your configuration, your http://localhost will be served because you have set proxypass for "/"(all). It will send all requests to the application/server and will be served.


jenkins pipeline code to build docker images I am trying to integrate docker build process in Jenkins. I know the things with docker plugin and configure docker agent and template to build it. I want these things to be done with Jenkins pipeline code.
Basically, I want below process
1. Pull the sources code from git and create maven build

Push artifactory .war to nexus
Pull another git repo having dockerfile and other supporting data
take artifactory (war) from nexus and add it to docker images build.
Push the newly updated image to docker repo
execute deployment on kubernetes.

I have steps to build maven code push it to nexus and deployemnt on k8s. But I am facing issue to build docker image using.  I have jenkin setup using docker image. So need to use docker agent and related plugin in jenkin code.

On building the docker image, you need to have the docker daemon enabled.
You can also use the jib, a Google project, which creates images for your Java applications without the Docker daemon


Redirect to a custom page on 503 error in Ingress When our site is getting deployed, the Ingress gets created and before I could get any response I get 503 for 5-6 minutes (time taken to startup).
We need to configure our ingress to redirect to a static page that is custom-made instead of the 503 error. Can I add an annotation to achieve this? If yes, please give me an example for redirect config using the status code.

There is no built-in configuration/annotation to handle this.
Although, you can use the server-snippet annotation to create a custom configuration that intercepts the error 503, proxying the request to a service that is serving your custom error page. Example:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress
  annotations:
    kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/server-snippet: |
      proxy_intercept_errors on;
      error_page 503 = @errorpages;

      location @errorpages {
        proxy_set_header X-Code $status;
        proxy_pass http://errors-svc.default.svc.cluster.local;
      }
spec:
  rules:
  - host: ameba.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: nginx
            port:
              number: 80


initContainer with huge size software like magento2 As a practice I have pods with these 2 containers:

php-fpm
nginx

The image of my project is called testimage. The solution uses the 'testimage' as initContainer and then copies the all folders and files of the testimage into a shared volume called volume1. Then this volume is mounted into nginx and php-fpm containers with a Kubernetes volume mount.

When the size of testimage goes beyond 1gb (for example a magento2 project with full pre-built static data and compiled code), it takes a long time for copying these files and it seems like a bad practice.

Should I put the project files into the php-fpm container (first container)? For nginx to use it again, I have to copy paste data into a Kubernetes volume and then mount it to nginx.

The end solution is putting all app code + fpm + nginx  into a single container than this makes a huge image with all of these. There is no per-container resource management and etc available with this solution.

Static assets should not be in the image, but rather on a PersistentVolume or better yet, on an object store such as AWS S3 then let a CDN to use it.

At the end of the day, Magento is a CMS, so the admin users will change the assets and other part of the content. It is not practical to bake assets these in to an container image doe the nature Magento.


Internal Server Error while running docker-compose to install Gitlab I'm installing Gitlab on my server using docker-compose based on Docker Docs.
My SSH port is not the default one (22) and I have changed it to something else like 2228. So the YML file became like this:
web:
  image: 'gitlab/gitlab-ce:latest'
  restart: always
  hostname: 'gitlab.mydomain.com'
  environment:
    GITLAB_OMNIBUS_CONFIG: |
      external_url 'https://gitlab.mydomain.com'
      gitlab_rails['gitlab_shell_ssh_port'] = 2228
  ports:
    - '80:80'
    - '443:443'
    - '2228:22'
  volumes:
    - '/srv/gitlab/config:/etc/gitlab'
    - '/srv/gitlab/logs:/var/log/gitlab'
    - '/srv/gitlab/data:/var/opt/gitlab'

When I run the docker-compose up -d, I receive this result and errors:
Starting docker_web_1 ...

ERROR: for docker_web_1  a bytes-like object is required, not 'str'

ERROR: for web  a bytes-like object is required, not 'str'
Traceback (most recent call last):
  File "site-packages/docker/api/client.py", line 261, in _raise_for_status
  File "site-packages/requests/models.py", line 940, in raise_for_status
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http+docker://localhost/v1.21/containers/9d4483106cf1683f46b5158a9550f9b4375d7c147ecb8b07df533840ad0806b6/start

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "compose/service.py", line 625, in start_container
  File "compose/container.py", line 241, in start
  File "site-packages/docker/utils/decorators.py", line 19, in wrapped
  File "site-packages/docker/api/container.py", line 1095, in start
  File "site-packages/docker/api/client.py", line 263, in _raise_for_status
  File "site-packages/docker/errors.py", line 31, in create_api_error_from_http_exception
docker.errors.APIError: 500 Server Error: Internal Server Error ("b'driver failed programming external connectivity on endpoint docker_web_1 (9b32d28cc2f635e085e82a2ef9f4d1fd3afc76f7693e279405ada5f61b96d215): Error starting userland proxy: listen tcp 0.0.0.0:443: bind: address already in use'")

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "bin/docker-compose", line 6, in <module>
  File "compose/cli/main.py", line 72, in main
  File "compose/cli/main.py", line 128, in perform_command
  File "compose/cli/main.py", line 1107, in up
  File "compose/cli/main.py", line 1103, in up
  File "compose/project.py", line 570, in up
  File "compose/parallel.py", line 112, in parallel_execute
  File "compose/parallel.py", line 210, in producer
  File "compose/project.py", line 556, in do
  File "compose/service.py", line 568, in execute_convergence_plan
  File "compose/service.py", line 510, in _execute_convergence_start
  File "compose/parallel.py", line 112, in parallel_execute
  File "compose/parallel.py", line 210, in producer
  File "compose/service.py", line 508, in <lambda>
  File "compose/service.py", line 620, in start_container_if_stopped
  File "compose/service.py", line 627, in start_container
TypeError: a bytes-like object is required, not 'str'
[5738] Failed to execute script docker-compose

I'm new in Docker and docker-compose, but I did everything as it is written in the official documentation. May you help me plesae to solve this issue?

In the error message you presented your system is reporting port 443 is already in use. You need to confirm which ports you have in use for other services.

  listen tcp 0.0.0.0:443: bind: address already in use

For what it is worth I recommend you use a nginx reverse proxy. Please reference https://stackoverflow.com/questions/46195852/how-to-run-gitlab-in-docker-container-with-nginx-proxy-over-ssl-with-letsencrypt for further details.


AWS: Onboarding on premise instance Is there a way to onboard the Raspberrypi device in AWS. So that the same device can be managed using ec2 console and it can seamlessly work with the rest of AWS services?

I am just an experimental user, I don't need much heavy computer but use AWS to host my website and blogs, etc.

AWS console is for managing AWS resources, which, arguably, your Raspberry Pi is not.

Some services can work with non-AWS resources. For example you can install System manager agent (though not sure if there is a RPi version) on on-prem servers.

Alternatively you can install AWS Greengrass for Raspberry Pi that gives it some AWS capabilities.

Hope that helps :)


CloudWatch trigger for a range of GuardDuty severities I want to edit my CloudWatch rule so that it only triggers an SNS topic for "GuardDuty findings" that fall in the High severy range, defined by AWS as the value of the severity parameter in the GetFindings response falls within the 7.0 to 8.9 range.

The sample Event Pattern in the documentation  only shows how to create a trigger for severities 5 and 8 as follows:

{
  "source": [ "aws.guardduty" ],
  "detail-type": [ "GuardDuty Finding" ],
  "detail": {  "severity": [ 5, 8 ]  }
}

How do I change this  {  "severity": [ 5, 8 ]  } to be a range between 7.0 to 8.9?

I used this post to get our Guard Duty alerts working, thanks!  But a recent change at AWS caused us to stop getting any alerts.  Turns out we needed to add the integer values along with floating point numbers. We used a CLI command like the following to do that:

aws events put-rule --name Test --event-pattern "{\"source\":[\"aws.guardduty\"],\"detail-type\":[\"GuardDuty Finding\"],\"detail\":{\"severity\":[7.0,7.1,7.2,7.3,7.4,7.5,7.6,7.7,7.8,7.9,8.0,8.1,8.2,8.3,8.4,8.5,8.6,8.7,8.8,8.9,7,8]}}"

Note the "7" and "8" at the end.  Alerts are working again.


Tool or way to support handling dependencies of modules in various platforms (for machine learning in Python) I am Data Scientist and I have a problem with setting of versions modules like Keras, Tensorflow PyTorch, etc.
Although I am using tools like virtuaenv or conda, my code isn't cross-platform, version of modules from mac don't fit at windows and I spending a lot of time to the handling of infinity number of problems. Docker isn't helping every time, because sometimes I have to send code to my client, which is non-technical(very weak at CS at all).

The main question is: is the way/tool/software/anything which supports handling dependencies of modules on various platforms?

Python is notoriously painful when it comes to dependency and version management.

I think its your job to make a portable application, and there are usually methods to do so (even if they  boil down to ugly branching logic based on the host OS type/etc).  Most of the time, you can avoid OS specific things with some effort though.

In terms of deploying code to clients, I would recommend looking at tools that bundle your code into a stand-alone application (pyinstaller, py2app, etc).  If you can build a standalone app for your clients for each OS, you won't have to worry about what is on their system (python versions, docker, etc).


Permission errors with non-root USER when running on Linux I have been getting some unexpected failures with the execution of my Docker images when running on my Ubuntu 16.04.3 LTS machine. I'm using Docker version 18.02.0-ce (edge), installed from apt. I've narrowed down the problem enough to demonstrate it with this Dockerfile:

FROM tomcat:8.5.24-jre8-alpine

RUN adduser -D test

RUN chown -R test "$CATALINA_HOME"

USER test

RUN ls "$CATALINA_HOME"/webapps

When I attempt to build this, here's the result:

$ docker build -t test test/
Sending build context to Docker daemon  163.4MB
Step 1/5 : FROM tomcat:8.5.24-jre8-alpine
 ---> 134d23a9a14e
Step 2/5 : RUN adduser -D test
 ---> Using cache
 ---> a6f70ba83b8d
Step 3/5 : RUN chown -R test "$CATALINA_HOME"
 ---> Using cache
 ---> 8a0fae3db0a8
Step 4/5 : USER test
 ---> Using cache
 ---> 45fdf333232b
Step 5/5 : RUN ls "$CATALINA_HOME"/webapps
 ---> Running in 48711dfb9569
ls: /usr/local/tomcat/webapps/ROOT: Permission denied
ls: /usr/local/tomcat/webapps/examples: Permission denied
ls: /usr/local/tomcat/webapps/docs: Permission denied
ls: /usr/local/tomcat/webapps/host-manager: Permission denied
ls: /usr/local/tomcat/webapps/manager: Permission denied
The command '/bin/sh -c ls "$CATALINA_HOME"/webapps' returned a non-zero code: 1

These "Permission denied" errors are very strange to me, since the chown command should have given ownership of the whole /usr/local/tomcat folder, including the subfolders.

I have tried this same scenario with many different versions of Docker on my Ubuntu host, all with the same result. However, when I tried building this image on my Mac OS machine, it worked fine.

One slight variation that I have found which works around the problem is to give the "test" user access to the "root" group, like so:

RUN adduser -D test -G root

With this simple change, the Dockerfile builds. So it seems obvious to me that somehow group membership permissions are involved, but I can't imagine why that would matter since the user is the owner. Also I can't imagine why building on Linux is different than building on Mac OS.

edit -
This seems to be the same issue described here: https://github.com/docker-library/tomcat/issues/35

As described in https://github.com/docker-library/tomcat/issues/35, this is related to the storage driver used (aufs).  Changing to use overlay2 solved the issue:

https://docs.docker.com/storage/storagedriver/overlayfs-driver/


Persist Changes to New Docker Image - neo4j I am trying to create a new image based on the neo4j:latest image. What is the proper way to persist to a new image and run a container based on the image?

I have these goals:

Save the new password I assigned to the database.
Keep data stored in the database.

I am using
    Windows 10
    Docker Community Edition Version 18.06.0-ce-win72 (19098)

Here is what I have tried:

Start a container from neo4j:latest.

I used this docker-compose.yml:

version: "3"
services:
  neo4j:
    image: neo4j:latest
    ports:
      - "7474:7474"
      - "7687:7687"
networks:
  webnet:

Connect to neo4j using a browser. I set a new password, as is required.

http://localhost:7474/browser/

Load data into the database using a script run on my host. I verified that the database now contains nodes and relationships.
Commit the container to an image.

docker commit --author="Jacob Quisenberry" c5c3f2998895 jquisenberry/graphdb:latest

Push the image to my repository.

docker push jquisenberry/graphdb:latest

Run the image

docker run --expose=7474 --expose=7687 -p 7474:7474 -p 7687:7687 jquisenberry/graphdb:latest

Connect using a browser.

I expected not to have to enter a new password and for my data to be intact.Instead, I am required to enter a new password. My database now lacks nodes and relationships.

Regarding to Neo4j with Docker and Insallation: Docker

  /data to allow the database to be persisted outside its container.

  /logs to allow access to Neo4j log files.

I understand that we should mount the /data to the host machine so that the new password is persisted as the following example.

docker run \
    --publish=7474:7474 --publish=7687:7687 \
    --volume=/path/to/neo4j/data:/data \
    --volume=/path/to/neo4j/logs:/logs \
    neo4j:latest

Note, the /path/to/neo4j is a physical directory on the host machine.

Edit1

To set password, the Insallation: Docker mentions as

  By default Neo4j requires authentication and requires you to login with neo4j/neo4j at the first connection and set a new password.

  You can set the password for the Docker container directly by specifying --env NEO4J_AUTH=neo4j/<password> in your run directive. Alternatively, you can disable authentication by specifying --env NEO4J_AUTH=none instead.

Then the docker run should be

docker run \
    --publish=7474:7474 --publish=7687:7687 \
    --env NEO4J_AUTH=neo4j/<password> \
    --volume=/path/to/neo4j/data:/data \
    --volume=/path/to/neo4j/logs:/logs \
    neo4j:latest

Anyhow for the /data, I only achieve with mounting it to the host machine.


Internal network between container Docker Compose with outgoing connection I am a newbie to Docker.
I am trying to run my application containing from multiple containers

Here is my docker-compose.yml file

version: '3.3'
services:
  php-service:
    container_name: 'php-service'
    build: './php-service'
    networks:
      - backend
    ports:
      - '8666:8666'
  go-service:
    container_name: 'go-service'
    build: './go-service'
    ports:
      - '8825:8825'
      - '8835:8835'
    volumes:
      - './go-service:/go/src/go-service'
    networks:
      - frontend
      - backend
    depends_on:
      - 'mongo'
      - 'consul'
  consul:
    image: 'consul:latest'
    networks:
      - backend
    ports:
      - "8300:8300"
      - "8400:8400"
      - "8500:8500"
      - "8600:53/udp"
  mongo:
    image: 'mongo:latest'
    container_name: 'mongo'
    networks:
      - backend
    ports:
      - '27100:27017'

networks:
  frontend:
    internal: false
  backend:
    internal: true

What I want to achieve.

The Outside World - is a network space of a hosting machine where the Docker daemon is running.

Go-Service, PHP-Service, Consul and Mongo Db communicate over internal network not exposed to the outer world. Let's call this network backend.
I want services inside backend network to be able to send outgoing requests to outer world
Another network is for the outside world, I need to expose only specific ports. In my case I want to expose 8825 and 8835 only. So that ports will be exposed as localhost:8825 and localhost:8835 on the hosting machine

Is it possible to get the desired configuration?

Or maybe I am doing something wrong, please suggest any other good way to achieve this.

Thank you so much

Internal Docker networks mean there is no gateway configured to reach the outside internet, you can only reach other containers from that network. In your scenario, none of your networks should have this definition.

Without setting the internal network flag, containers can reach out of the docker host, potentially to the internet if the host has that access. This is one-way access, containers can access the external network, but that external network cannot access the containers.

To access a container from outside of the Docker host, you need to publish the port on the host mapping into the desired container (or service). Publishing ports with the ports section in the docker-compose.yml file is only needed to provide external access, not access between containers. Applications inside containers need to listen on all interfaces (0.0.0.0) for both publishing ports on the host and for container to container communication. However, you can publish ports on a specific interface on the host, e.g.:

ports:
  - '127.0.0.1:8825:8825'
  - '127.0.0.1:8835:8835'

For container to container communication, it is not necessary to expose or publish the port. All that is needed is a user defined network (which you have) in common between the containers. Then the containers can communicate using Docker's built-in DNS to resolve the service name (e.g. consul and mongo) and the application port inside that container (e.g. consul:8300). To avoid the external network from accessing these containers, do not publish any ports from them.

Side note: from your docker-compose.yml, you should also consider removing container_name, depends_on, and build since they won't work if you try to transition to swarm mode.


Blue Green Deployment DB transactions management Imagining I have 2 sets of environment(Blue and Green)

           +
           |
           |
           |
           v
   +-------+---------+
   |     Proxy/LB    |
   |                 |
   +--+-------------++
      |             |
+-----++          +-+----+
|      |          |      |
|APP v1|          |APP v2|
|Blue  |          |Green |
+--+---+          +--+---+
   |                 |
+--+---+          +--+---+
| DB   |          | DB   |
| Blue | DB Sync  | Green|
|      <---------->      |
+------+          +------+

The version Blue is currently running and green is deployed and all DB replication completed. Now I want switch traffic to green environment. In this case, what is the best way to manage the DB transactions?(I have one thought below)

Forward the new traffic to green and wait for existing blue's environment's transactions complete. Once all are done, switch traffic completely to green(UPDATE: Can I achieve this with Nginx?)

If above is true, I'd have to do little extra work to forward only new traffic to green.

Or what is the best way of doing Blue Green deployment in this case?

PS:

We are using Nginx proxy
All servers are bare metal, no cloud!
2 standalone DB setups(Postgresql with master-slave)

As per Martin Flowers' definition of Blue-Green deployment

  The blue-green deployment approach does this by ensuring you have two production environments, as identical as possible. At any time one of them, let's say blue for the example, is live. As you prepare a new release of your software you do your final stage of testing in the green environment. Once the software is working in the green environment, you switch the router so that all incoming requests go to the green environment - the blue one is now idle.

Steps

You've said that DBs are synced using master-slave topology.
So, to switch traffic from one instance to another, we have to make  steps:

Switch web-traffic from APP v1 to APP v2
Shutdown deprecated DB Blue
Promote DB Green to master mode
Start new DB Blue v2 in slave mode
(you may ask - why shutdown DB Blue instead of using master-master replication or other ways to sync DB Blue v1 with DB Green in master mode. See explanation in DB replication)

Graceful switch web-traffic

To switch web-traffic from APP v1 APP v2 - just use nginx reload - it will graceful end all working connection and move traffic from App v1 to App v2
There are many scripting solutions for switch blue/green using shell/python scripts

E.g.:

meappy/docker-nginx-blue-green: Docker blue green upstreams for nginx with ngx_http_perl_module
Simple 0-Downtime Blue Green Deployments: Python simple 0-Downtime Blue Green Deployments example

The logic is simple: add green upstream in nginx config, use it as default and send reload to nginx:

Sample nginx.conf for reverse proxy/load balancer

http {

upstream appv1 {
    zone appv1 64K;
    server 10.10.0.1:80;
}

upstream appv2 {
    zone appv2 64K;
    server 10.10.0.3:80;
}

server {
...
location / {
    proxy_pass http://$appv1;
}
}

}

Replace proxy_pass http://$appv1; with proxy_pass http://$appvv;

And then

nginx -s reload

Stop DB Blue

systemctl stop postgresql

Promote DB Green to master mode

Trigger the promotion using your defined trigger in $PGDATA/recovery.conf

touch $PGDATA/failover

Remove failover trigger from Green DB

cd $PRIMARY_DATA
rm -f recovery.* failover

Make sure that hot stanby mode is on on Green DB in master mode

cat postgresql.conf | grep '#hot_standby = on'

DB replication

Make a backup of master Green DB

psql -c "SELECT pg_start_backup('Streaming Replication', true)" postgresql://postgres@GreenDB/postgres

Send a backup to Blue DB

rsync -a $PG_DATA_Green_DB/ $BlueDB_IP:$PG_DATA_Blue_DB/ --exclude postmaster.pid --exclude postmaster.opts

See also

Evolution of Fault Tolerance in PostgreSQL

DB Migration (More complex but more robust database migration scheme)

You may use more complex scheme for migration using Ansible and `Pglupgrade tool:

See for details:

Near-Zero Downtime Automated Upgrades of PostgreSQL Clusters in Cloud (Part I)
Near-Zero Downtime Automated Upgrades of PostgreSQL Clusters in Cloud (Part II)

P.s some definitions.

Just to...

  ... sort the buyers from the spyers, the needy from the greedy...

And blue/green deployment vs canary deployment vs A/B test

Definitions, kinds of deployments and tests

(see details in these answers )

Blue-Green Deployment - When deploying a new version of an application, a second environment is created. Once the new environment is tested, it takes over from the old version. The old environment can then be turned off.
A/B Testing - Two versions of an application are running at the same time. A portion of requests go to each. Developers can then compare the versions.
Canary Release - A new version of a microservice is started along with the old versions. That new version can then take a portion of the requests and the team can test how this new version interacts with the overall system.
Feature flagging - The action of "configuring" (cold, or even hot) which functionality is (not)available for which (group) of users. If you also do something like "feature flagging" you can deploy first, measure soundness of your release in backwards compatibility/bug perspective, and release new functionality gradually to different users, or vice versa (scale down or even rollback functionality and/or binaries). Feature flagging allows for splitting availability of functionality from deployment of binaries, and gives much more fine-grained decision making then only "deploy/rollback"

P.P.S Blue/Green vs Canary

source

Both blue-green and canary releases solve the same purpose
Although both of these terms look quite close to each other, they have subtle differences. One put confidence in your functionality release and the other put confidence the way you release.

Blue-Green Deployment

  When deploying a new version of an application, a second environment is created. Once the new environment is tested, it takes over from the old version. The old environment can then be turned off.

1. It is more about the predictable release with zero downtime deployment.

2. Easy rollbacks in case of failure.

3. Completely automated deployment process

4. In cloud environment where it is easier to script & recreate infrastructure, blue/green deployment is preferred as it allows the infrastructure to be in sync with the automation

Canary Release

  A new version of a microservice is started along with the old versions. That new version can then take a portion of the requests and the team can test how this new version interacts with the overall system.

1. The canary release is a technique to reduce the risk of introducing a new software version in production by slowly rolling out the change to a small subset of users before rolling it out to the entire infrastructure.

2. It is about to get an idea of how new version will perform (integrate with other apps, CPU, memory, disk usage, etc).

See also illustration (source):

Blue/Green:

Canary:


What are the cost effective ways to deploy services using Kubernetes on multiple deployment environments? We are converting a traditional docker-compose based service stack to a Kubernetes based services
Before the conversion we had three deployment environments

Local : Basically this is the developer's working machine.
Dev : Where the everyday releases happens and developers who belongs to other projects like web apps and mobile app uses endpoints from this environment to get their daily work done.
Staging : Project owners/clients see this space
Production : This is where the world facing actual services are living.

So when its come to Kubernetes, we need to specially invest on Kubernetes supported clusters and those are heavily expensive.
Since we need three deployment environments , do we need to purchase three Kubernetes cluster groupes?
Or how other traditional teams managing this requirement?

Based on my experience I'd suggest following scenario:
First you should make your apps deployment process similar for all environments, so same tool and set of files will be responsible for this part. And only difference should goes to parameters.
And second it's always a good practice to have kubernetes cluster per stage. Very common situation when your changes goes not only to kubernetes apps manifests, but to surrounding infrastructure also (LoadBalancer, Object Buckets, Disks Storage, etc).
You cat try to cut costs on kubernetes part by using same cluster for dev/stage but still situations with differences in infra resources could happen. And that could lead to more efforts on support and release deployment level.
In case you will use AWS, you can put kuberentes load to EC2 Spot Instances and that could heavily reduce your monthly bill.
As for toolset for k8s apps manifests, I've tried multiple tools and ways, and from my experience terraform+terragrunt works better here. Single tool for all infra resources, parameterization works easily, common code could be separated to reusable modules.
Other option is kustomize or helm, but those tools could help with kubernetes manifests only, and can't manage external resources.


Issue with custom domain SSL certification We are working on saas application which is hosted on app.example.com. Now we are trying to give the option to the users where their own domain or sub domain like dev.stack.com point to my application by adding A record to 1.2.3.4 IP address (ip address where app.example.com is hosted).

With the above setup, we are facing the issues with ssl certificate of the vendor domains. Is there any way we can create global certificate and add it to our servers which can work for any domain pointed to our servers.

Can you obtain a secure certificate for someone else's domain or subdomain without their help - hopefully not.

One of the core security pillars of a secure certificate is that it ensures that you are actually connected (through an encrypted connection) to the the site you wanted to connect to. For this purpose the minimum check every certificate issuer makes is to check for ownership of the domain by the entity requesting it.

For "manual" issuers, this often involves uploading a file to a web server on this domain, creating a DNS record or clicking on a link in an email to the registrant, administrator or technical contact in the whois record of the domain.

You can replicate this process for your needs for your SaaS application by using LetsEncrypt: It allows you to either

upload a file to a folder on a web server for the domain: In this case you would advise the client to point an A record (or probably better CNAME) to your server and once resolved, you can issue the certificate with letsencrypt by copying the file yourself
create a specific DNS record, which again allows you to create the certificate.

Lets encrypt certificates are valid for 90 days, but as you will need the A record in place in anycase to run your SaaS, you will be able to renew the certificate yourself regularly.

P.S.: Not too long ago, subsidiaries of Symantec were found (twice) to not always have checked domain ownership and then did not appear to take remedial action consistently (twice). This led to the "distrusting" of all Symantec certificates in Chrome and Firefox after giving owners of these certs some time to create new certificates.


Terraform: Provide credentials for docker hub registry How would I provide docker hub credentials to a private image in the following example?

data "docker_registry_image" "ubuntu" {
  name = "ubuntu:latest"
}

resource "docker_image" "ubuntu" {
  name          = "${data.docker_registry_image.ubuntu.name}"
  pull_triggers = ["${data.docker_registry_image.ubuntu.sha256_digest}"]
}

Docker registry credentials is seting up in provider configuration.

provider "docker" {
  registry_auth {
    address = "quay.io:8181"
    username = "someuser"
    password = "somepass"
  }
}

Also possible to set env variables DOCKER_REGISTRY_USER and DOCKER_REGISTRY_PASS