One azure-pipeline.yaml for staging and production Currently I have two separate Azure DevOps Pipeline config files:

azure-pipelines-staging.yaml
azure-pipelines-production.yaml

Really the only differences in them are the:
trigger:
  branches:
    include:
    - master # or staging

And the directory where my k8s manifests are:
manifests: |
  $(Pipeline.Workspace)/manifests/azure/prod/api.yaml

# or

manifests: |
  $(Pipeline.Workspace)/manifests/azure/staging/api.yaml

So ideally it would be something like:
trigger:
  branches:
    include:
    - master
    - staging
...
manifests: |
  $(Pipeline.Workspace)/manifests/azure/$(branch)/api.yaml

Not quite seeing how to get $(branch) name in there. Suggestions?

Ok, found it here:
Build.SourceBranch

The branch of the triggering repo the build was queued for. Some examples:

 - Git repo branch: refs/heads/master
 - Git repo pull request: refs/pull/1/merge
 - TFVC repo branch: $/teamproject/main
 - TFVC repo gated check-in: Gated_2016-06-06_05.20.51.4369;username@live.com
 - TFVC repo shelveset build: myshelveset;username@live.com

When your pipeline is triggered by a tag: refs/tags/your-tag-name
When you use this variable in your build number format, the forward slash characters (/) are replaced with underscore characters _).

Note: In TFVC, if you are running a gated check-in build or manually building a shelveset, you cannot use this variable in your build number format.


Is there a way to know which files have changed in a docker container compared to its base image? Is there a way to know which files have changed in a docker container compared to its base image?

I am looking both for file creation/modification/deletion, but also for metadata change like file permissions.

The docker container diff command shows the differences between a container and its base system.

In the output, files are reported as (A) if the file or directory was added, (D) if a file or directory was deleted or (C) if a file or directory was changed. File permission changes are reported as (C).

she$ CID=$(sudo docker run --rm -d ubuntu sleep 500)
sh$ sudo docker exec $CID touch newfile
sh$ sudo docker exec $CID useradd bob
sh$ sudo docker exec $CID chmod o-x /usr/bin/cut
sh$ sudo docker diff $CID
C /etc
C /etc/group
A /etc/group-
C /etc/gshadow
A /etc/gshadow-
C /etc/passwd
A /etc/passwd-
C /etc/shadow
A /etc/shadow-
C /etc/subgid
A /etc/subgid-
C /etc/subuid
A /etc/subuid-
A /newfile
C /usr
C /usr/bin
C /usr/bin/cut
C /var
C /var/log
C /var/log/faillog
C /var/log/lastlog
C /var/log/tallylog


Is there any way of getting code coverage with molecule? I'm currently working on testing Ansible roles with Molecule. I have one role with 2 scenarios that are executed in Jenkins but just before that, those are submitted to a SonarQube analysis. So I installed the YAML plugin for it to recognize the files, but now, I am asked to add code coverage to these reports.

I'm wondering if there's a way of having code coverage using Molecule with its default stack (Docker/Testinfra)?

You can use another verifier. Inspec should have a reporter that will report in the correct format. https://molecule.readthedocs.io/en/stable/configuration.html#inspec


When is it ethically acceptable to fork someone else github code and continue with that? https://github.com/kamaln7/ansible-swapfile

Seems not to be maintained. For example, an issue was created in May and no reaction. A new issue has just been created to ask whether this repo is still maintained.

When is it allowed to fork someone else repo and to publish it to Ansible galaxy for example?

It is always ethically acceptable to fork any code on Github (or Sourceforge or wherever you got it from) and do whatever you want with it, within the terms allowed by its license! This is what open source and permissive licenses are all about. The original author wanted the world to contribute to this piece of software, and you will be doing exactly that.

Fork it, respect the license requirements, and go on with it. It does not matter whether it is abandoned (like your example, since the last change was 3 years ago) or current.

If you do feel that you did some good improvements, send pull requests back to your upstream, as  usual. If the original author never comes back, then so be it. If he does, he can integrate them (or not if he so chooses). But this is strictly optional, you are neither legally nor ethically forced to do so.

About uploading it to some other locations (Ansible Galaxy or whatever - say Docker Hub for docker images and so on): the issue is probably mostly to avoid irritations due to the name being "replaced" (but I'd assume if the original author already has uploaded it to A.G., you can't just replace it, anyway).  So, in the latter case, just upload it with another name, which is still easily identifyable as related to the original - say, append some suffix which is not too obnoxious.

If the author, in your case, should later re-appear, and you had uploaded the software to the Ansible Galaxy, and he wants to take over, then you two will simply communicate and resolve the issue somehow (i.e., merge the two forks again and decide who keeps updating Ansible Galaxy). Chances are that he will be more than glad to leave the work to you.

Oh, and to be sure; this piece of software is under the MIT license:

  Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

There you go.


One NAT attachment for two private subnets I have two NAT gateway attachments for two private subnets in AWS. Is it possible to have one NAT gateway attachment for two private subnets so that I can save one EIP?

Remember that you should be engineering to your requirements around cost, resilience and performance.

If your constraint is around cost, then yes, a single NAT gateway, with both subnets having routing tables that route public IP traffic to the NAT gateway is a viable approach.  If you are this cost sensitive, are you sure that AWS is the right hosting environment?

If your constraint is around having all traffic arrive on the internet from a single elastic IP address, then may i suggest that having two NAT gateways in different subnets in different AZ's is a better choice. You will need one elastic IP address on the active NAT gateway, with routing tables for the private subnets pointing internet traffic to the active NAT gateway.  In the event of a network failure, have a script that moves the elastic IP address to the standby NAT, and then update the route tables to send internet traffic to the second NAT, making it now the active NAT.

Do not fall into the false economy of thinking that you can spin up the replacement NAT at failover time, as it only takes a few seconds. When large catastrophic failures happen within AWS, the processes for the creation of new resources into the surviving AZ's become far busier than normal, and new requests seem to be queued. you would have an unpredictable RTO with this approach. A single NAT should be treated as accepting that on rare occasions, a 12 hour outage is acceptable.


Pause/Suspend guest in KVM without virsh and libvirtd I have installed KVM without libvirt/virsh and virtualmanager.

Packages which installed are kvm qemu-kvm bridge-utils libguestfs-tools

How can I pause/suspend VM in this case?

After long searching and experiments, the best option for me became qemu monitor. See https://nico.schottelius.org/blog/control-and-shutdown-qemu-kvm-vm-via-unix-socket/ for more details regarding flexible management guest by using qemu monitor.


Assign DNS name to Docker Swarm containers with public IPv6 I'm trying to understand how I would set up a Docker container (running in Swarm mode) with publicly available IPv6.

I know Docker Swarm uses mesh routing, but I don't quite understand how I can dynamically assign DNS names to addresses.

As an example, let's say I want to run a Git server (with HTTP 80 443 and SSH 22) on git.example.com and a regular web server at example.com (with HTTP 80 443). Both have public IPv6 addresses, but I don't want to hard-code the IPv6 address (for example at CloudFlare). After all, this is DevOps!

How would I go about doing this? I assume I need to run my own DNS server for this.

Note: After doing some more research, it seems this is possible with the Docker HTTP Routing Mesh (HRM) which is unfortunately only part of the Enterprise Edition. Does anyone know a similar, open-source approach to this?

Assuming that "publicly available" is on the public internet it will be difficult.

DNS isn't really that reactive to changes and is challenging to run in a secure fashion on the public internet.

It also depends on what protocol you are running, because of an oddity around SRV records where pre-existing protocols were excluded from supporting it in the RFC.

But in general the public and the private solution I would implement are similar with an additional layer for a public internet solution. (assuming no use of a 3rd party or geo specific answers)

On the internet edge use Unbound DNS server on the edge, and have it relay queries to consul which would handle service registration.

If this is for private usage this would allow for some data-center awareness and fail-over.

If it is just for internal use you can just use consul and either configure your internal DNS servers to delegate subdomains or forward to the default domain.


Ansible tmp files created with odd permissions I have an ansible playbook which can be simplified to this:

- name: Provision server
  hosts: all_hosts
  become: yes
  roles:
    - { role: foo }

With the following in ansible.cfg, since it may be relevant:

[defaults]
allow_world_readable_tmpfiles = True
host_key_checking = False
log_path=./ansible.log
remote_user = ubuntu

That playbook successsfully configures 10 servers, but one of the servers fails with:

[WARNING]: sftp transfer mechanism failed on [xx.xx.xxx.xxx]. Use
ANSIBLE_DEBUG=1 to see detailed information
 [WARNING]: scp transfer mechanism failed on [xx.xx.xxx.xxx]. Use
ANSIBLE_DEBUG=1 to see detailed information
fatal: [host_1]: FAILED! => {"failed": true, "msg": "failed to transfer file to /home/ubuntu/.ansible/tmp/ansible-tmp-1500318083.01-204857088854554/stat.py:\n\nscp: /home/ubuntu/.ansible/tmp/ansible-tmp-1500318083.01-204857088854554/stat.py: Permission denied\n"}

So I ssh'd to the problematic server and a "healthy" server, and ls -al the contents of the latest temp folder for each. The healthy server had files with permissions of rwx --- ---. The unhealthy server had this:

---x------ 1 ubuntu ubuntu 58378 Jul 17 14:09 file.py
---x------ 1 ubuntu ubuntu 56779 Jul 17 14:09 stat.py

Why would those files get those permissions? I assume that is the problem, because then scp tries to write to them, but it cannot. That doesn't totally make sense with the error message, though, which seems to be suggesting that the scp fails.

I'm trying to figure out what debug steps to take, and if anyone can think of other places I can look in order to provide more information, I'm all ears.

The reason for the failure is that the server in question, unlike all the others, had sftp disabled.
Why the error messages are what they are, I don't know (why the errant scp transfer mechanism failed?), but I don't have time right now to investigate with the debug option enabled.

But that was the issue, and adding scp_if_ssh=True to the [ssh_connection] section of my ansible.cfg solved the issue:

[ssh_connection]
scp_if_ssh=True


Parallel stages running on same agent in Jenkins declarative pipeline I've been using a Jenkins pipeline with a parallel step for testing like this:

pipeline {
    agent { label 'php' }
    stages {
        stage('build') {
            steps {
                ...
            }
        }
        stage('test') {
            parallel {
                stage('unit tests') {
                    steps {
                        ...
                    }
                }
                stage('integration tests') {
                    steps {
                        ...
                    }
                }
            }
        }
        stage('deploy') {
            steps {
                ...
            }
        }
    }
}

The pipeline runs fine on a single agent, but now I want to add a step with manual input, which should not keep the agent running. The agents are cloud servers that get started and stopped on demand.

To do so, I assigned "none" agent to the pipeline and the actual agent to all stages but the one requesting input:

pipeline {
    agent none
    stages {
        stage('build') {
            agent { label 'php' }
            steps {
                ...
            }
        }
        stage('test') {
            parallel {
                stage('unit tests') {
                    agent { label 'php' }
                    steps {
                        ...
                    }
                }
                stage('integration tests') {
                    agent { label 'php' }
                    steps {
                        ...
                    }
                }
            }
        }
        stage('deploy staging') {
            agent { label 'php' }
            steps {
                ...
            }
        }
        stage('confirm release') {
            steps {
                input { message 'release?' }
            }
        }
        stage('deploy production') {
            agent { label 'php' }
            steps {
                ...
            }
        }
    }
}

Artifacts that are needed in a following stage are carried over with stash and unstash. This works, but with two nuisances:

Stashing and unstashing between each single stage causes overhead. I would rather have all stages until the manual input executed in a single workspace. But agent is only allowed in stages that contain steps.
Each parallel stage blocks a single agent. So if there is only one free agent, there is no parallel execution, unlike before.

I can live with (1) if it's not possible otherwise, but parallel execution on one agent is important for me. Is it possible somehow in my current setup? If not, what are my options to achieve something similar?

I modified your Jenkins file as shown below.

pipeline {
agent { label 'agent01'}
stages {
    stage('build') {
        steps {
            echo "build"
            sh "ls -lrt && touch build.txt"
        }
    }
    stage('test') {
        parallel {
            stage('unit tests') {
                steps {
                    echo "unit tests"
                    sh "ls -lrt && touch ut.txt"
                    sleep(time:10,unit:"SECONDS")
                }
            }
            stage('integration tests') {
                steps {
                    echo "integration tests"
                    sh "ls -lrt && touch it.txt"
                    sleep(time:10,unit:"SECONDS")
                }
            }
        }
    }
    stage('deploy staging') {
        agent { label 'master' }
        steps {
            echo "deploy staging"
            sh "ls -lrt && touch dep.txt"
        }
    }
    stage('confirm release') {
        agent { label 'master' }
        steps {
            echo "RELEASE?"
            sh "ls -lrt && touch release.txt"
        }
    }
    stage('deploy production') {
        steps {
            echo "deploy production"
            sh "ls -lrt && touch produ.txt"
        }
    }
}
}

You need not stash and unstash files since it runs on same node and
    workspace.
Use a general node for all the stages except for the one that needs a user input.

The output is

    Running in Durability level: MAX_SURVIVABILITY
[Pipeline] node
Running on agent01 in /var/jenkins_home/workspace/sandbox-pipeline
[Pipeline] {
[Pipeline] stage
[Pipeline] { (build)
[Pipeline] echo
build
[Pipeline] sh
[sandbox-pipeline] Running shell script
+ ls -lrt
total 0
+ touch build.txt
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (test)
[Pipeline] parallel
[Pipeline]     [unit tests] { (Branch: unit tests)
[Pipeline]     [integration tests] { (Branch: integration tests)
[Pipeline]     [unit tests] stage
[Pipeline]     [unit tests] { (unit tests)
[Pipeline]     [integration tests] stage
[Pipeline]     [integration tests] { (integration tests)
[Pipeline]     [unit tests] echo
[unit tests] unit tests
[Pipeline]     [unit tests] sh
[unit tests]     [sandbox-pipeline] Running shell script
[Pipeline]     [integration tests] echo
[integration tests] integration tests
[Pipeline]     [integration tests] sh
[unit tests] + ls -lrt
[unit tests] total 0
[unit tests] -rw-r--r-- 1 jenkins docker 0 Oct 19 13:53 build.txt
[unit tests] + touch ut.txt
[integration tests]     [sandbox-pipeline] Running shell script
[Pipeline]     [unit tests] sleep
[integration tests] + ls -lrt
[integration tests] total 0
[integration tests] -rw-r--r-- 1 jenkins docker 0 Oct 19 13:53 build.txt
[integration tests] -rw-r--r-- 1 jenkins docker 0 Oct 19 13:53 ut.txt
[integration tests] + touch it.txt
[unit tests] Sleeping for 10 sec
[Pipeline]     [integration tests] sleep
[integration tests] Sleeping for 10 sec
[Pipeline]     [unit tests] }
[Pipeline]     [unit tests] // stage
[Pipeline]     [unit tests] }
[Pipeline]     [integration tests] }
[Pipeline]     [integration tests] // stage
[Pipeline]     [integration tests] }
[Pipeline] // parallel
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (deploy staging)
[Pipeline] node
Running on Jenkins in /var/jenkins_home/workspace/sandbox-pipeline
[Pipeline] {
[Pipeline] echo
deploy staging
[Pipeline] sh
[sandbox-pipeline] Running shell script
+ ls -lrt
total 0
+ touch dep.txt
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (confirm release)
[Pipeline] node
Running on Jenkins in /var/jenkins_home/workspace/sandbox-pipeline
[Pipeline] {
[Pipeline] echo
RELEASE?
[Pipeline] sh
[sandbox-pipeline] Running shell script
+ ls -lrt
total 0
-rw-r--r-- 1 jenkins 115 0 Oct 19 13:58 dep.txt
+ touch release.txt
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (deploy production)
[Pipeline] echo
deploy production
[Pipeline] sh
[sandbox-pipeline] Running shell script
+ ls -lrt
total 0
-rw-r--r-- 1 jenkins docker 0 Oct 19 13:53 build.txt
-rw-r--r-- 1 jenkins docker 0 Oct 19 13:53 ut.txt
-rw-r--r-- 1 jenkins docker 0 Oct 19 13:53 it.txt
+ touch produ.txt
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
Finished: SUCCESS


Why does Helm time out on almost every action? I'm trying to set up helm for the first time and I'm having troubles.

First, I've created account with cluster-admin role. (According to https://github.com/kubernetes/helm/blob/master/docs/rbac.md#example-service-account-with-cluster-admin-role ).

After that I've initialized brand new helm tiller by using helm init --service-account=tiller and it was successful.

Now when I'm trying to install something:

First try:

$ helm repo add gitlab https://charts.gitlab.io
$ helm install --name gitlab-runner -f gitlab-runner-values.yaml gitlab/gitlab-runner

where gitlab-runner-values.yaml looks like this:

gitlabUrl: https://my-gitlab.domain.com
runnerRegistrationToken: "MY_GITLAB_RUNNER_TOKEN"
concurrent: 10

Second try (as I was not sure if there is an issue with custom repo, so I tried from official):

$ helm install stable/kibana

I'm getting this error:

Error: forwarding ports: error upgrading connection: error dialing backend: dial tcp 192.168.0.18:10250: getsockopt: connection timed out

I noticed that 192.168.0.18 is visble on pod list:

kube-system   kube-proxy-kzflh                        1/1       Running   0          7d        192.168.0.18   kube-worker-7
kube-system   weave-net-jq4n4                         2/2       Running   2          7d        192.168.0.18   kube-worker-7

and that tiller is running on the same node:

kube-system   tiller-deploy-5b48764ff7-qtv9v          1/1       Running   0          3m        10.38.0.1      kube-worker-7

I was told that I probably don't have permission to pods/port-forward and list pods, but kubectl auth can-i create pods/portforward tells me that I can do this (the same with list pods)

Also helm list is throwing the same error as install.

You have found a Kubernetes issue #22770, where there is a workaround mentioned here and it goes like follows:

What you're experiencing is a known issue with k8s where for some operations it expects to be able to resolve your node names in the global DNS.
And suggested a work around would be to:

Add entries to /etc/hosts on the master mapping your hostnames to their public IPs
Install dnsmasq on the master (e.g. apt install -y dnsmasq)
Kill the k8s api server container on master (kubelet will recreate it)
Then systemctl restart docker (or reboot the master) for it to pick up the /etc/resolv.conf changes

As last comment on the issue #22770 there is

I can confirm that adding the names of the our nodes to /etc/hosts on the kubernetes master resolves this problem for us too :)

, so there is not a fix in k8s coming in any upcoming version. At least from this particular ticket.


Unable to create databags in chef I am very new to chef and trying to secure passwords using databags. Unfortunately, I tried in different ways but unable to succeed. I followed different tutorials first and second. I understood only two steps but later on I was unable to move forward.

I created secret key using openssl
In link1 step 2 says knife data bag create passwords mysql --secret-file /tmp/my_data_bag_key
but I don't know where it is creating the databag.

Can some one guide me from step two? I am trying to secure my database password. I got stuck after creating secret file.

I'm using the following ruby script to encrypt/decrypt chef data bags locally:
https://gist.github.com/anonymous/4923ebeee53fe4d43a1cfb70a2abaadd

to encrypt:

ruby encrypt.rb PATH_TO_SECRET_FILE < UNECRYPTED_DATA_BAG.json > ENCRYPTED_DATA_BAG.json

and to decrypt:

ruby decrypt.rb PATH_TO_SECRET_FILE < ENCRYPTED_DATA_BAG.json > DECRYPTED_DATA_BAG.json


How to create AWS application load balancer using saltstack I was trying to create application load balancer with salt stack. I am using salt boto_elbv2 which is having option to create target groups but dont have option to create load balancer, how can i achieve it ?

Please follow official doc to create LB

Please create a IAM role or mentioned secret and access key as below

elb.keyid: HHTYHLSHDJFJFJFJFJFJFFSVJS
elb.key: basjbsdfbdsfdsjfbkgbsfbdsjfbfdgdfgdfs

Then Create config for ELB

Ensure myelb ELB exists:
    boto_elb.present:
        - name: myelb
        - region: us-east-1
        - availability_zones:
            - us-east-1a
            - us-east-1c
            - us-east-1d
        - keyid: GKTADJGHEIQSXMKKRBJ08H
        - key: askdjghsdfjkghWupUjasdflkdfklgjsdfjajkghs
        - listeners:
            - elb_port: 443
              instance_port: 80
              elb_protocol: HTTPS
              instance_protocol: HTTP
              certificate: 'arn:aws:iam::1111111:server-certificate/mycert'
              policies:
                  - my-ssl-policy
                  - cookie-policy
            - elb_port: 8210
              instance_port: 8210
              elb_protocol: TCP
        - backends:
            - instance_port: 80
              policies:
                  - enable-proxy-protocol
        - health_check:
            target: 'HTTP:80/'
        - attributes:
            cross_zone_load_balancing:
              enabled: true
            access_log:
              enabled: true
              s3_bucket_name: 'mybucket'
              s3_bucket_prefix: 'my-logs'
              emit_interval: 5
            connecting_settings:
              idle_timeout: 60
        - cnames:
            - name: mycname.example.com.
              zone: example.com.
              ttl: 60
            - name: myothercname.example.com.
              zone: example.com.
        - security_groups:
            - my-security-group
        - policies:
            - policy_name: my-ssl-policy
              policy_type: SSLNegotiationPolicyType
              policy:
                Protocol-TLSv1.2: true
                Protocol-SSLv3: false
                Server-Defined-Cipher-Order: true
                ECDHE-ECDSA-AES128-GCM-SHA256: true
            - policy_name: cookie-policy
              policy_type: LBCookieStickinessPolicyType
              policy: {}  # no policy means this is a session cookie
            - policy_name: enable-proxy-protocol
              policy_type: ProxyProtocolPolicyType
              policy:
                ProxyProtocol: true

# Using a profile from pillars
Ensure myelb ELB exists:
    boto_elb.present:
        - name: myelb
        - region: us-east-1
        - profile: myelbprofile

# Passing in a profile
Ensure myelb ELB exists:
    boto_elb.present:
        - name: myelb
        - region: us-east-1
        - profile:
            keyid: GKTADJGHEIQSXMKKRBJ08H
            key: askdjghsdfjkghWupUjasdflkdfklgjsdfjajkghs

Register instance with ELB.
salt.states.boto_elb.register_instances(name, instances, region=None, key=None, keyid=None, profile=None)

add-instances:
  boto_elb.register_instances:
    - name: myloadbalancer
    - instances:
      - instance-id1
      - instance-id2

You may find this blog post useful also.


How does jenkins publish over ssh store passphrases? When I look in jenkins.plugins.publish_over_ssh.BapSshPublisherPlugin.xml I can see several entries, that have secretPassphrase populated with a string 44 characters long. By using hudson.util.Secret.fromString in the /script tool, I can decrypt them and see that password. Fine so far.

However... I also have an entry there where the secretPassphrase is populated with a string only 24 characters long, which cannot be decrypted. Trying gets me an empty result, which is strange, because decrypting "asdfasdf" returns "asdfasdf". So there must be something special about this string which is preventing the decryption for working, or from returning the default. But deployments to this server work; so the passphrase is correct. But the secretPassphrase is NOT the actual password on the server.

Which means jenkins must be using some other method of decrypting that passphrase, besides the standared hudson.util one. What is it? How can I recover this password?

According to the source code for the Publish Over SSH plugin, the passphrase is encrypted and decrypted using hudson.util.Secret, and the encrypted value is stored in the plugin's XML configuration file.

Supposedly there is ongoing work to switch to using Jenkins' built-in credentials store; however, I don't see any active pull requests for such a change.


How should CI/CD deploy to production environment - copy artifacts or git pull from the server? We have an internal CI/CD server running private GitLab. It has the source code of our NodeJS project. We would like to run CI/CD and have it deploys to our on-premises production server. (Let's assume we are using the master branch).

Here are two most common methods I found on the internet:

Run git pull on the production server, run whatever build tasks needed (npm install, minify, etc...), and restart the service
Build the source codes on CI/CD server, rsync the built artifacts to production server and restart

Surprisingly, most of the examples I found on the internet uses method 1. This method requires the production server to have public internet access which may not be possible for many cases.

My question: is method 1 really the best way to deploy a NodeJS application to a production server? If yes, what is the reason?

There are lots of bad examples on the Internet.  Doing a git pull to distribute code is fabulous for development, but prone to all sorts of issues in practice.  It should not be used for production deploys without careful consideration of the negative consequences.

What happens when somebody checks out a different branch?  You'll be sad when you keep git pulling and nothing new shows up.
How do you verify that the machine is running the expected release?  You can get this from git, but it is easier with most artifact systems.
How do you release to machines that are on an air-gapped network?  Simulating your git server on a private network will be a fun time for you.

I'm not sure what format npm uses for artifacts, but it is also probably a good idea to back them up off of the CI/CD server and use a dedicated set of machines for serving up the artifacts.  If you spin up 100 machines at once would your CI/CD server handle it or fall over?  Isolating the artifact serving will avoid your CI/CD server being swamped and useless until the deploy is over.


Is it common for DevOps engineers to write code for user requirements? After reading some articles about DevOps, I understand that DevOps engineers do write code. However, it's a bit unclear to me about what kind of code a DevOps engineer write.
Assume that there are (very roughly) 2 categories of code:

For system operation: like scripting for deployment, configuration of Puppet or Ansible, etc.
For user requirement: e.g. allow user to upload avatar images, add an "export" button to download as .csv files.

I found that some DevOps engineers only do the first category, while some others do both.
My question: is it common for DevOps engineers to write code for user requirements?

Short Answer
It depends on the organisation if you are a dedicated infrastructure specialist without any software development experience, you are unlikely to be writing feature code.  However, if you are a Software Engineer who also knows Terraform, you might be called on to do both on the same day.
Long Answer
DevOps is not strictly a job title; in the most general sense, the "DevOps Mindset" forms part of an agile or lean operating model.  In this kind of model, DevOps is a way of working for everyone that focuses on the Three Ways:

Systems Thinking - understanding the creating value is the product of the system as a whole, not any one part of the system.  When I say systems, I am talking about systems that include humans and technical components, i.e. people writing automation or features in code and the technology that executes that code.
Amplification of Feedback Loops - creation and tightening of information flowing between the consumer and the parts of the system.
Culture of Continuous Experimentation and Learning - enabling a data-driven culture that encourages agility and learning through experimentation.

Adoption Patterns
The three ways don't specify how the organisation should be structured; they just provide the foundational principles of DevOps.  Coupled with the fact that most companies don't change quickly, this means there are several DevOps "Adoption Patterns":

DevOps Bridge - where a team of DevOps engineers write the code to automate deployment and operations and often operate that.
DevOps Operations Teams - where a team writes code to operate the software, but the development team writes the code, infrastructure and deployment automation.
Tools Teams - where a group of DevOps engineers write tools that developers use.
Embedded DevOps - where a DevOps engineer is embedded in the team to write infrastructure and deployment automation and built out the operational capability.
Ubiquitous DevOps - where the group as a whole is accountable for feature development and infrastructure/deployment automation.  They might have a DevOps Engineer or developers might "do DevOps".

Google and Amazon
There are also two general approaches described by Amazon and Google:

"You build it, you run it" - this is a mantra that Werner Vogels, Amazon's CTO, used to describe how Amazon scaled Development and Operations teams. i.e. the group as a whole is responsible for building features, delivering it into Production and operating it while it is there.
Site Reliability Engineering - this is Google's (and many other organisations) technology operating model that does split Development and Operations, but in such a way that there is shared accountability for the quality and warranty of code.

Amazon's "You Build It; You Run It" Model
Teams within Amazon are accountable for delivering a service, not software or infrastructure.  Therefore they need to have people in the group who are capable of fulfilling every technical role: Software Architects, Back-end Developers, Front-end Engineers, Infrastructure Engineers, Automated Testers, Operations Engineers, and so on.  Some people in those teams can fulfil more than one role; others may be much more specialist.
Within Amazon's model, it's the team as a whole's responsibility both implement features and write automation (amongst other things) they may:

Choose to hire a Software Engineer to write the code for the features; plus a DevOps Engineer to write the code for the automation, or.
Hire two engineers who can both do both roles within the team.

Google's Site Reliability Engineering Model
Google did their own thing for about eight years and came up with Site Reliability Engineering, SRE.  Broadly Google employs Software Engineers (SWEs) and Site Reliability Engineers (SREs), Software Engineers are responsible for building code, automating infrastructure and automated deployment with the promise that if the project is important enough and they have proven they can operate it in Production themselves that Site Reliability Engineers will take over the operation of the project.
SRE gives Google the best of both worlds; teams are accountable for getting a product live and making sure it's reliable.  Site Reliability Engineers can then take over up to ten live products meaning Google reduce their on-call costs and free up Software Engineers to concentrate on developing new features or products.
If you refer to Google's Site Reliability Engineering triangle:

Product - is what you think of as high-level "feature" development, i.e. understanding what the user wants and implementing it.
Release Processes - is what you probably think of as code that improves the flow of features into Production which could be deployment code or infrastructure.
Development - is a bit of all three worlds: code that implements a feature, deployment code or infrastructure as code.

My Journey
My journey to where I am now (heading up Managed Services) was first as a Network Technician and IT Manager, then becoming a .NET Developer, and then a DevOps consultant, finally taking up my current role.  I'm at the point in my career where I can turn my hand to more or less any task.
When I hire, I look for T-Shaped individuals, i.e. people who are experts in one or two disciplines but are moderately capable in the full range of disciplines we need to be successful.
Summing Up
When you're just starting, start with the end in mind:

If the goal is a team that is cross-functional and capable of building and running a product - then work out what that team needs to do and make sure that there is more than one person that is proficient in each capability.

If the goal is to become a DevOps engineer, start with the basics of Infrastructure of Code, Continous Deployment and Operations.  As you learn, teach those in your team how to do those things themselves and ask them how they do their role (this includes QA, Front-end, Backend and Hardware engineers) eventually you become that T-Shaped individual.


How can we get the jenkins console output in a text file? How can i get the Jenkins console output in a text file?
I want to share it with someone, is there any way to do it?

if you want just access the log and download it as a txt file to your workspace from the job's URL:

${BUILD_URL}/consoleText

On Linux, you can use wget to download it to your workspace

wget ${BUILD_URL}/consoleText

The actual log file on the file system is in the Master machine. You can find it under:

$JENKINS_HOME/jobs/$JOB_NAME/builds/lastSuccessfulBuild/log


How do I release multiple projects to same domain using AWS when developer commit the latest code change I have multiple projects on Github and I need to release each project on same domain different paths when developer commit code changes.

Example:

I have Github repositories like below

CRM
CMS
Main-Website

currently, our website is like this

example.com/CRM
example.com/CMS
example.com --> is the Main-Website repository.

Please someone tell me how to do this in AWS using there services. Currently im doing this manually login to server and git pull

Thanks.

You may wish to try scripting the actions you want. Then use git hooks to trigger your scripts to release the latest code to your environment. Alternatively, you could use something like Jenkins to configure a "push on green" system if you are working with compiled code. Lastly, (though I haven't used them) Amazon claims that their CodePipeline product can do this (and perhaps CodeBuild can as well), though my personal preference is to keep things as generic as possible and avoid drinking Amazon's Kool-Aid in order to keep things portable. Like a drug dealer, the first hit may be free with Amazon, and things are great until they have you hooked and decide to jack up the price. Utilizing my own open source system ensures that I can easily take my ball and go to another cloud or home altogether (and host it myself) in the event that I need to make a rapid exit from Amazon's cloud and avoids lock-in.


Automation and the .NET framework which tools to use? I am aware of the de facto and popular tool choices for various programming languages such as Go, NodeJS, Java, Python etc.
However, I do not know what toolchain is reasonable or even (hot) in the .NET world.
I have heard that people use Octapus deploy, is this still a valid choice?
Is NuGet still the defacto package manager?
What about code inspection, automatic QA etc.?

I would like to get a feeling of the complete toolchain for .NET and what is popular right now when considering development, automatic test, and delivery.

Somewhat stealing from Ian Margett's answer as the architecture is common amongst most Microsoft/.NET development organizations, the high-level target operating model for looks something like this:

The goal is to create a Continuous Deployment pipeline, using existing off-the-shelf software, namely TeamCity, ProGet, SonarQube and Octopus Deploy:

GitHub is the source code management tool, however, it could be BitBucket or Visual Studio Team Services.  The branching model and code review process are out of scope at this high level.
TeamCity is chosen as the Build System due to its tight integration with Octopus Deploy and good all-round support for .NET, msbuild and PowerShell.  TeamCity is also used as the orchestrator of deployments in Octopus Deploy.
ProGet is the package management solution which both stores Octopus Packages and proxies public package/image repositories.  The rationale for not using the built-in TeamCity NuGet store is purely for scalability reasons.
SonarQube provides continuous code quality management and reports are published as part of the TeamCity build outputs.
Octopus Deploy is used as the deployment tool for both infrastructure and code into the target platforms.

I've seen this broad approach implemented across two companies and successfully implemented it in two additional companies - in the most recent case we swapped TeamCity out for AppVeyor which worked, albeit a little painful when setting up firewall rules.


How to export EC2 snapshot into S3 bucket? I've created a snapshot of EC2 instance via:

aws ec2 create-snapshot --volume-id vol-xyz

In aws ec2 create-snapshot help we can read:

  Creates a snapshot of an EBS volume and stores it in Amazon S3.

Although I cannot see in which bucket it has been imported.

In addition, there is a command aws ec2 import-snapshot which imports a disk from the S3 bucket.

So basically I'm looking for some way of exporting EC2 snapshot into S3 bucket where I can see the image file in order to download it.

How I can do it?

EBS Snapshots aren't really disk image files.  They're logical containers of pointers to chunks of compressed and deduplicated backup data that EBS stores in S3 when you create snapshots -- but these are stored in buckets that EBS owns and controls.

They are user-accessible via only one mechanism -- creating a new EBS volume from the snapshot.  When you do this, the new volume is linked back to its source snapshot and EBS starts loading the data onto the new volume in the background, from the snapshot storage in S3.  If you access a part of the volume that EBS hasn't yet restored, it immediately goes and fetches the chunks and puts them on the volume.

To create a disk image file of a snapshot, you create a new volume from the snapshot, attach it to a running instance without mounting it, and then use the same tools you would use for a standard hard drive -- tools like dd to read raw blocks from the snapshot, and write them to a file, which you can then download from the instance, e.g. using scp.


How to exclude a node in kubernets using podAntiAffinity? I'm trying to exclude a set of nodes when I submit a job in kubernetes. The set of nodes do not (yet) have a label to differentiate them, so all I can exclude them on is hostname (for now).

All the examples on using podAntiAffinity use labelSelector which expects a label on the pod to differentiate it.

How can I limit my job from deploying to a specific set of nodes? The closest example I could scrape together from the docs is this non working example:

affinity:
  podAntiAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
    - labelSelector:               <--- this is wrong, there are no labels
      matchExpressions:
      - key: "kubernetes.io/hostname"
        operator: In
        values:
          - hostname1
          - hostname2

To exclude a set of nodes when submitting a job in kubernetes.

you should use Node affinity which is conceptually similar to nodeSelector –and will allow you to constrain which nodes your pod is eligible to be scheduled on, based on labels on the node and you should be able to use hostname.

On below example i run a job to 5 completions and all are started on node-01 of my cluster with node affinity defined.

Example : job.yaml

apiVersion: batch/v1
kind: Job
metadata:
  creationTimestamp: null
  labels:
    run: busybox
  name: busybox
spec:
  completions: 5
  template:
    metadata:
      creationTimestamp: null
      labels:
        run: busybox
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: kubernetes.io/hostname
                operator: In
                values:
                - k8s-node01
      containers:
      - image: busybox
        name: busybox
        resources: {}
      restartPolicy: OnFailure

Result :

ubuntu@k8s-master:~$ kubectl get all -o wide
NAME                READY   STATUS      RESTARTS   AGE     IP               NODE         NOMINATED NODE   READINESS GATES
pod/busybox-2gdp6   0/1     Completed   0          3m20s   192.168.85.204   k8s-node01   <none>           <none>
pod/busybox-l654d   0/1     Completed   0          3m55s   192.168.85.201   k8s-node01   <none>           <none>
pod/busybox-n7dbr   0/1     Completed   0          3m29s   192.168.85.203   k8s-node01   <none>           <none>
pod/busybox-pfc79   0/1     Completed   0          4m11s   192.168.85.200   k8s-node01   <none>           <none>
pod/busybox-r4xln   0/1     Completed   0          3m16s   192.168.85.205   k8s-node01   <none>           <none>

NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE   SELECTOR
service/kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP   31h   <none>

NAME                COMPLETIONS   DURATION   AGE     CONTAINERS   IMAGES    SELECTOR
job.batch/busybox   5/5           60s        4m11s   busybox      busybox   controller-uid=e12f136b-d828-4f7b-a49b-303499496a8a

I had two worker nodes on my cluster on job yaml i defined affinity to node01 only.

$ kubectl get nodes
NAME         STATUS   ROLES    AGE   VERSION
k8s-master   Ready    master   31h   v1.17.2
k8s-node01   Ready    <none>   31h   v1.17.2
k8s-node02   Ready    <none>   31h   v1.17.2


Does building a Docker image require the Docker daemon to be running? I am trying to get my feet wet with Docker/containers. I started with this tutorial:

Getting Started · Spring Boot with Docker

It works fine, but I was surprised it is necessary to have the "dockerd" daemon running to build an image. It connects to it using localhost and a port number as part of executing the Maven build. Why is this required (and can it be avoided)? I mean, in many cases the image will have to be run on a separate platform anyway. It is particularly annoying, because I am building on Windows where Docker requires you to have Hyper-V enabled, which causes other issues (for instance that you can't use VMware on the same machine).

Due to the problems, I resorted to building it all inside a Linux VM (which then ran the Docker daemon).

Another thing which might be related to the above: It seems the build process doesn't leave the image on disk; it is installed directly into the repository managed by the Docker daemon. I can then export the image from the command line, and I suppose (haven't tried) this image could be moved to another machine and run from there. But how do I build "just" the image without having it deployed anywhere, etc.? It just needs to produce the artifacts in the local file system and leave it at that. Maybe that would also remove the requirement for having Docker running?

The command the tutorial uses for running Maven is:

./mvnw install dockerfile:build

However, trying alternatives to "install" (such as "package", etc.) seemingly doesn't change the behavior.

Each command in the Dockerfile is processed as a separate step. Every step from the build process gets executed in a dedicated container. It starts from Step 1 (the first command in the Dockerfile). That step is run in a container and if it is successful, a commit to a temporary image (let's call it image A) takes place. Then, Step 2 is executed in a container which is run from image A ... and so on until the final step is executed successfully. And as Tensibai commented - how do you think to run a container without the daemon?

So, yes - building a docker image requires the docker daemon to be running.


kubelet connecting to docker.sock instead of containerd.sock Kubernetes version: latest
Installation method: kubeadm
Host OS: debian9
container runtime: containerd
Kubelet fails when starting and after inspecting the logs using journalctl -xeu kubelet i see that it tries to connect to the docker.sock socket instead of containerd.sock. I tried running kubeadm --config=config.yaml using this config file
apiVersion: kubeadm.k8s.io/v1beta1
kind: InitConfiguration
nodeRegistration:
        criSocket: "unix:///var/run/containerd/containerd.sock"

I have also tried running the kubeadm with the container runtime  85  argument, also doesn’t work.
any other ideas are appreciated and thanks

In kubeadm docs

--cri-socket stringPath to the CRI socket to connect. If empty kubeadm will try to auto-detect this value; use this option only if you have more than one CRI installed or if you have non-standard CRI socket.


Unable to run `conda activate` from Jenkins pipeline I'm running Conda 4.6.14 on Ubuntu 16.04.

I thought about using virtual environments with Conda in my Jenkins pipelines that needed Ansible, by having all nodes use the same environment, ansible-env to run playbooks. While the process seems pretty straight forward, I'm unable to call conda activate ansible-env, because it results in the following error:

+ conda activate ansible-env

CommandNotFoundError: Your shell has not been properly configured to use 'conda activate'. To initialize your shell, run

    $ conda init <SHELL_NAME>

Currently supported shells are:
  - bash
  - fish
  - tcsh
  - xonsh
  - zsh
  - powershell

See 'conda init --help' for more information and options.

IMPORTANT: You may need to close and restart your shell after running 'conda init'.

Here is the code I'm using to activate...pretty basic

sh("conda activate ansible-env")
sh("ansible --version")
sh("conda deactivate")

I ran conda init --system on the build node as the Jenkins user and then rebooted. No change.

I've spent a great deal of time trying to google a solution, but everything I've tried hasn't worked.

What I've attempted:

Tried running sh("conda init bash") in the pipeline.
Executing . /home/jenkins/miniconda3/profile.d/conda.sh from within the pipeline.
Made sure the bin directory for miniconda3 is set in $PATH.
Ran dpkg-configure dash to make sure /bin/bash is the default shell on the node.
Tried sourcing the /home/jenkins/.bashrc file
Calling the conda executable via the absolute path.
Tried using source ansible-env or activate ansible-env without conda.

I'm pretty sure the issue has to do with the following lines added to ~/.bashrc of the jenkins user after running conda init --system, but still not being accessible from that same user when a job is running. Unfortunately, I've hit a wall and not sure where to go from here.

NOTE: Please don't suggest switching to VirtuanEnv unless there's a specific reason Conda won't work with Jenkins. I'm aware of that route, but went with conda for...reasons.

What we do is download and install miniconda in the first step in our (declarative) pipeline and create the environment(s) from a yaml file.

sh '''#!/usr/bin/env bash
wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh -nv -O miniconda.sh
bash miniconda.sh -b -p $WORKSPACE/miniconda
conda config --set always_yes yes --set changeps1 no
conda update -q conda

conda env create -f envs/ansible-env.yaml
'''

In each step we start the conda environment like so:

sh '''#!/usr/bin/env bash
source $WORKSPACE/miniconda/etc/profile.d/conda.sh
conda activate miniconda/envs/ansible-env/

# do stuff
'''


Best practice for Jenkins docker to deploy to other container I'm currently experimenting with docker with the potential of using it in future professional projects and am a bit confused about best practices.

In my experimental setup I have three docker containers, all currently running on the same server, though that might change in future projects:
- Git server (Gitea)
- Jenkins server
- Web app (Tomcat)

Pulling the source code from git and building the war is no problem. Now I'm running into difficulties actually deploying it.

My current solution:
I've modified the tomcat docker so I can deploy the resulting war to the tomcat docker and restart it inside the docker. I've read that this is bad practice and I understand the reasoning, so I need another solution.

Building the docker container from another docker container is also said to be bad practice and I also understand the reasoning for that, so that's a no-go, too.

An idea is to ssh to the host system and then either deploy the webapp to a folder shared with the tomcat container or rebuild the container. That would probably work and be future-proof if the tomcat docker moves to a different host, but it doesn't feel docker-like from my limited docker understanding.

The last idea is to make it so the tomcat docker container builds the webapp itself, i.e. does the git clone, mvn package and so on itself. That still needs to be triggered, possibly by an ssh connection to the host.

In both those cases I'd need to expose a docker-capable user to remote login, which I don't really want to do.

I can only find information on bad practice, not on how to actually solve this specific problem in a good way. What is the actual correct way to do this?

Where you say:

  Building the docker container from another docker container is also
  said to be bad practice and I also understand the reasoning for that,
  so that's a no-go, too.

Perhaps there are bad ways of doing that but it can be a solid pattern. By way of example if you use openshift Kubernetes it runs s2i as builder images on your cluster. s2i checks out your code from git, compiles it, and pushes it to a new layer in your docker repo. It has an incremental flag where it will tar up your library deps folders at the end of a build and store as a docker layer. Then the next time you build it does a clean checkout then untars the libraries from the last build. That speeds up the next build a lot as it has the local maven cache from the last build. It also means the s2i build container can be launched on a different host in the cluster at each execution and it just connects to git and the central container registry and does a fast repeatable build.

s2i is a stand alone tool that can use your local docker demon. So it acts as a single commandline ”compile my stuff into an updated docker image”. There are community supported s2i images for php7, node.js, Java etc. I recall that the java s2i will run a maven build with the “openshift” profile and expects your to copy the built WAR to “./deployment”. When you run the image that s2i writes out to your container registry it launches wildfly that is in the base layers and it deploys whatever finds in the deployment folder (but double check its been a while since I looked at it).

In your case If you are using Jenkins I suggest you take advantage that it supports building docker images. Append a docker pipeline step to create your image and push it to your chosen container registry. Enterprises that use Jenkins to build and Nexus for storing JARs and WARs are now using Nexus as their container registry. They can stick with Jenkins and Nexus and simply reconfiguration things to move into the world of containers.

I should also point out that you can be more “microservice-like” by skipping installing tomcat. You can use maven jetty plugin or maven tomcat plugin to just run the code with “mvn jetty:run” or “mvn tomcat7:run”. You can set a full blown config file for the servlet container via the plugins. Maven let's your set whatever jvm options you want. (note: after Tomcat 7 there doesn't seem to be support for this modern approach see the next paragraph where the mainstream now uses Jetty for this).

If you like spring you can use spring boot to build an excutable JAR that runs your webapp. The JAR will be large as it has everything to run the webapp including the servlet container packed into a single JAR. You can run it as “java -jar” and it runs your webapp in an embedded servlet container. Better yet configure it's maven plugin to make an executable jar you can simply run as “./my-app-1.0.0.jar” and it will lunch. That makes building a docker image trivial. You don’t have to install tomcat you only need a container with Java on it and copy in your spring boot executable JAR and run it as the CMD. Spring Boot is very mainstream for Java microservices and it runs Jetty under the bonnet.


Gitlab CI push to docker-registry fails I have GitLab server and a separate docker-registry. I have created a testuser with a testpassword.
My  .gitlab-ci.yml
image: docker:latest
services:
  - docker:18.09.7-dind

variables:
  DOCKER_HOST: tcp://docker:2375/
  DOCKER_DRIVER: overlay2
    # See https://github.com/docker-library/docker/pull/166
  DOCKER_TLS_CERTDIR: ""
  REGISTRY: myregistry:5000/golang-test

stages:
  - build
 # - test
before_script:
  - docker login -u testuser -p testpassword  myregistry:5000

build_project:
    stage: build
    tags:
      - docker-ci
    script:
      - docker ps
      - docker build -t $REGISTRY .
      - docker push $REGISTRY

When CI pipeline runs, I get following error.

WARNING! Using --password via the CLI is insecure. Use
--password-stdin. Error response from daemon: Get https://myregistry:5000/v2/: dial tcp: lookup worker1 on
192.168.123.456:53: no such host ERROR: Job failed: exit code 1

concurrent = 1
check_interval = 0

[session_server]
  session_timeout = 1800

[[runners]]
  name = "docker-ci"
  url = "https://gitlab.sample.com:8000/"
  token = "89MfxRTby83ny3eTkTf2"
  tls-ca-file = "/etc/gitlab-runner/certs/gitlab.sample.com.crt"
  executor = "docker"
  pre_build_script = "export DOCKER_HOST=tcp://docker:2375"
  [runners.custom_build_dir]
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]
  [runners.docker]
    tls_verify = false
    image = "ubuntu:18.04"
    privileged = true
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/cache"]
    shm_size = 0

This was after suggested modifications here.
I can successfully login to my docker-registry using docker login -u testuser -p testpassword  https://myregistry:5000 from gitlab running server.

Are you sure that your registry is publicly available?
Checklist:

Check if the domain is reachable and responding the right IP address -- nslookup myregistry
Check if your port is open or not blocked any kind of software (let's say via firewalls or AWS security group) -- telnet myregistry 5050
Check if your registry is not restricted for a particular CIDR of IP
Test your login locally and ensure that there is no problem with it


Best learning resources for continuous integration tools like Jenkins I am a .NET developer , I have decided to learn DevOps continuous integration tools.

One of my folk asked me to learn below tools and language:

Jenkins
Shell Scripting
Python

I have found below learning resources to be usefull.

Free Videos

https://www.youtube.com/watch?v=p7-U1_E_j3w
https://www.youtube.com/watch?v=89yWXXIOisk&t=89s

Free Course

https://www.udacity.com/course/intro-to-devops--ud611

Paid Course

https://www.udemy.com/the-complete-jenkins-course-for-developers-and-devops/

Training Materials

https://gist.github.com/ssmythe/ae1449b116ece85dc10d

Is their any other resources available by which one can master Jenkins and continuous integration ?

Any resource available to get started shell scripting and Python language ?

Here are the  resources which might be useful for . continuous integration
Below . you might be visited but as novice . it will be easy to understand.

jenkins:-

https://www.tutorialspoint.com/jenkins/jenkins_git_setup.htm

Shell Scripting :-
 I would . suggest . to start with basic unix commands  like . cp,mv, file related and . permissions etc.. (Basic stuff).

Its good to have . some practise  on the system so that you can  get an Idea . Because as a Devops engineer we should be strong enough on Basic . stuff.


How to move the "All" tab to the right in Jenkins? Is it possible to move the "All" Jenkins tab from the main view all the way right? It's just my browsing preference to see it at the end to not obstruct my always specific habit of using views. In other words, I would like my first view to be what I see at the beginning on the left.

One option is to add a new view, remove the "All" view, create a new view, select  "All" and let it start with "z", e.g. "z-all" as the view is sorted alphabetically.

There is an open issue that indicates that the ordering of tabs in Jenkins is alphabetically sorted and cannot be changed at the moment, e.g. change the order manually.


Git clean/smudge filters for ansible vault secrets I am trying to setup clean/smudge filter in git to have automatic encrypting and decrypting of files containing secrets thru ansible-vault command.

Peculiarity of the ansible-vault command is that it is not idempotent (it creates a different binary each time it is invoked on the same data).

I started with the implementation suggested in this blog page. Unfortunately it did not work correctly, as whenever smudge is called (be it a git checkout , or just git status), the secret files looks as modified for git, even if it is not.

So I wondered if git would be comparing the binary he has in the index with the clean filtered current file, and I tried to build on those script like follows:

#!/bin/sh -x
# clean filter, it is invoked with %f

if [ ! -r "$HOME/.vault_password" ]; then
  exit 1
fi

tmp=`mktemp`
cat > $tmp

# get the plain text from the binary in the index
tmphead=`mktemp`
git show HEAD:$1 > $tmphead
contenthead=`echo "embedded" | ansible-vault view $tmphead --vault-password-file=$HOME/.vault_password`
export PAGER=cat
echo -n "$contenthead" | tee $tmphead

# if current and index plain text version differ
if [ "`md5sum $tmp | cut -d' ' -f1`" != "`md5sum $tmphead | cut -d' ' -f1`" ]; then
  tmpcrypt=`mktemp`
  cp $tmp $tmpcrypt
  # generate a new crypted blob
  echo "embedded" | ansible-vault encrypt $tmpcrypt --vault-password-file=$HOME/.vault_password > /dev/null 2>&1
  cat "$tmpcrypt"
else
  # just return the HEAD version
  cat "$tmphead"
fi

rm $tmp $tmphead $tmpcrypt

The difference here is that it tries to compare the current and HEAD versions of the plain text (unencrypted) secret files, and only in case they differ output a new binary blob encrypted with ansible-vault.

Unfortunately, after this change git continues to think the secret file is always modified. Even after git adding the file again, so that the git blob is computed, git thinks the file is different and let the change go into the commit. Note that git diff return empty changes, as it should.

For reference, this is smudge:

#!/bin/sh

if [ ! -r "$HOME/.vault_password" ]; then
  exit 1
fi

tmp=`mktemp`
cat > $tmp

export PAGER='cat'
CONTENT="`echo "embedded" | ansible-vault view "$tmp" --vault-password-file=$HOME/.vault_password 2> /dev/null`"

if echo "$CONTENT" | grep 'ERROR: data is not encrypted' > /dev/null; then
  echo "Looks like one file was commited clear text"
  echo "Please fix this before continuing !"
  exit 1
else
  echo -n "$CONTENT"
fi

rm $tmp

and this is diff:

#!/bin/sh

if [ ! -r "$HOME/.vault_password" ]; then
  exit 1
fi

export PAGER='cat'
CONTENT=`echo "embedded" | ansible-vault view "$1" --vault-password-file=$HOME/.vault_password 2> /dev/null`

if echo "$CONTENT" | grep 'ERROR: data is not encrypted' > /dev/null; then
  cat "$1"
else
  echo "$CONTENT"
fi

The problem here is caused by the random salt in ansible-vault encryption. You can hack VaultEditor class to pass the salt to it from an argument in ansible-vault. The random salt is generated in lib/ansible/parsing/vault/__init__.py on this line. It is called from lib/ansible/cli/vault.py where you can easily add the argument for fixed salt. If you do change it, please submit an upstream patch to Ansible, I'd love to use it.

This problem is further discussed here on hacker news. And there are other implementations with tools that take fixed salt, namely gitcrypt, transcrypt. Here is also a link to one more implementation using ansible-vault called ansible-vault-tools, but this one has the same salt issue as far as I know.


What's the best way to restrict an environment to a specific CDN? We have had problems a couple times now where the links to the CDN in our components don't get properly updated between the team, QA, and staging environments.

Is there a way to force an error if the QA environment tries to access the team CDN or vice-versa? Or some other well-known solution to this problem?

If the different CDNs are hosted at different IP addresses one possibility might be to configure the local hosts table for the machines in a certain environment in such way as to overwrite the DNS resolution for the CDNs which shouldn't be accessible and pointing them to either:

a non-existent address, causing connection timeout failures if the invalid CDNs are referenced
a server where the respective files cannot be found, causing 404 errors (faster than the connection timeouts)


Retaining at least one successful build in Bamboo In Jenkins, you could configure the system to retain a certain number of either successful or failed builds.

With Bamboo, it seems like you can just retain a certain number of builds which rotates the last successful build after 10 failed builds out of the system.

Is there a way in Bamboo to retain the last successful build for an unlimited time?

UPD wait, but is that true with the 10 builds limitation, really?
I have just realized that I could click the UI through the whole build history, both successful and unsuccessful builds.
http://imgur.com/a/JnJ9v

IMHO if your successful build doesn't push to an artifact repository, there's a problem in your pipeline.

Jenkins or Bamboo do not aim at keeping built artifacts forever and they don't give you any versioning delivery, as such that's a point of failure in your delivery system.

I'd advise to review your method to use an artifact repository and that will solve this XY problem of keeping a successful build inside the build system used as an artefact source.

Note: I can't speak for Bamboo's team, but adding this capacity involves raising the code complexity for a 'side usage'. As such it can makes sense Bamboo's team avoid it to avoid adding code complexity for a feature not in their overall goal. I don't think they avoid it to enforce a best practice, but more to avoid raising unnecessary (from the product point of view) complexity to their code base.


How to handle item not in list in ansible variable? This question may look strange as something like this is not implemented in real life. I have tried to make it very general.

Here is my ansible-playbook:

tasks/main.yml

- name: "Setting place name of {{ country }}"
  set_fact:
     place: "get_place_name"

- name: "Setting places name of {{ country }}"
  set_fact:
     places: "get_places_name"

- name: Details of "{{ country }}"
  debug:
    msg: "{{ item }}"
  when:
    -  place in "{{ item.Other_Places }}" or places in "{{ item.Other_Places }}"
  with_items: "{{ Places }}"

vars/main.yml

---
Places:
- Country: "United States of America"
  Capital: "{{ capital }}"
  Other_Places: ['place1', 'place2']

- Country: "China"
  Capital: "{{ capital }}"
  Other_Places: ['places1', 'places2', 'places3', 'places4']

Here if "place" variable is defined as "place1" then it will run and output will be something as below:

"msg": {
        "Country": "United States of America",
        "Capital": "place1",
        "Other_Places": "['place1', 'place2']"
    }

The problem is I am confused on how to handle something that is not listed in "Other_Places"? Suppose I input "place10" as variable name "Capital" then it is not defined then it should simply output "Sorry, no matching place found."

If we do something like below :

- name: Exceptional case
  debug:
    msg: "{{ capital }} not in {{ item.Capital }} "
  when:
    -  place not in "{{ item.Other_Places }}" and places not in "{{ item.Other_Places }}"
  with_items: "{{ Places }}"

This will output all at once because "place" is not defined in "capital".

use the condition

- when: variable is defined
  run_module_when_defined:

or

- name:
  block:
    - task1
    - task2
  when: variable is defined


Store named volumes on another drive I'm really familiar with Docker on Linux but my company's IT department uses Windows. They're interested in converting VM-based services over to Docker so I'm trying to get them set up with a good Windows-based Docker infrastructure.

I've started with a couple containers managed by docker-compose. I would like to use named containers rather than hardcoding paths. For example:

volumes:
  portainer_data:

services:
  portainer:
    image: portainer/portainer:latest
    volumes:
      - portainer_data:/data

They created a 900GB E: drive to hold persistent data. How can I get all the named volumes for all containers to be stored in E:\Volumes?

I figure it would make it super easy for them to configure backup. Instead of having to script a bunch of steps using docker cp I could just tell them, "Back up the E: drive". Easy.
If I understand Docker Desktop and Hyper-V correctly, volumes are stored inside a MobyLinuxVM. The VM is on the C: drive and only has 48GB to play with.

(Please let me know if there's a better way to tackle this in general. I'm open to frame challenges. Down the road I'd like to add Kubernetes or Docker Swarm to the mix, but one step at a time.)

OS: Windows Server 2019 Datacenter

If I understand Docker Desktop and Hyper-V correctly, volumes are stored inside a MobyLinuxVM.

That's right and it seems to be possible to  move the whole VM to another drive as described here.

Maybe you could solve you problem by using a volume driver as described in the official docs.

Another approach is to use a Linux VM with Docker and tell your Docker on Windows installation to use another daemon by setting the DOCKER_HOST environment variable. This will give you more control and will allow you to use everything available on Linux (mounting, symbolic links, ...). But of course this will not work if you or your IT department want to use Windows containers some day.


How do you monitor Node Micro services running inside of Docker Containers? This article about "How healthy is your Dockerized application?" explains the trouble with monitoring, but it doesn't provide any good examples of how to actually monitor a microservice inside of the docker container.

We are currently using PM2 monit to monitor our microservices, but as we put them into docker containers we lose the ability to access this data within one screen for all the various microservices which each run in their own docker container.

Dockerswarm monitoring will tell us the state of the containers, but not the microservice running inside of them.

What's a solid proven way of solving this problem?

One commonly used solution(not container-specific) is to build a health check API within your service which tests all the functionality you want monitored (say availability of DBs and other dependencies) and the app itself, and returns some expected output (say < service >: < status >). You can then trigger alerts from a monitoring service like Nagios if this API does not return an ok for all the services. This will also fail if the microservice itself is unhealthy.

This approach also has the benefit of running a functional test of your service (by hitting an API end point).

This approach does not cover some edge cases though - eg. the microservice runs (but particular APIs fail).


How to define failure situation on jenkins I created a project in Jenkins and set it to run tests on every pushes. but even when one of the pushes does not pass the tests, Jenkins build returns success. How can define failure situation if tests do not pass in Jenkins?

By default Jenkins assumes whether the build is SUCCESS or not from build process exit code.

Which means that build exit with code 0 considered as SUCCESS, rest all are considered as FAILURE.

If your build step/process exit with code 0, even during the failure case then Jenkins will report the build is SUCCESS.

So I my suggestion is to run the all build steps manually and try to get the exit code on each step (for both PASS and FAIL case).


AWS-CLI : to download IAM user with policy aws : A single command(aws-cli) to get the IAM user name with the what all permission he/she have , would be great if can be downloaded as an excel or csv ?

You can use the below command to get all the policy attached to an iam user. (You must be having permissions to list-users and list-user-policies before using the below commands)

aws iam list-attached-user-policies --user-name username | jq -r '.AttachedPolicies[].PolicyArn' | awk -F'policy/' {'print $NF'}

(Make sure to replace username with some actual iam username in the above command)

If you want to get the list of all users and user-policies attached to those users, you can use the below one line command. The below command will get all the iam users and then will find the user-policies attached to that user and will put in a CSV file. The first column will contain username and second column will contain `policies separated by spaces.

for u_name in $(aws iam list-users | jq -r '.Users[].UserName') ; do permissions="$(aws iam list-attached-user-policies --user-name ${u_name} | jq -r '.AttachedPolicies[].PolicyArn' | awk -F'policy/' {'print $NF'} | tr '\n' ' ')" ; echo "${u_name}, ${permissions}" >> iam_users.csv ; done


What is Unit testing in the context of YML configuration Alerting rules? I read a very short blog posting about Unit testing alerts with Prometheus

Can you explain me why I need unit testing for *.yml? What is the benefit of unit testing for configuration files?

I not understand what are the gains when implementing that.

The important part of the blog post you link to isn't testing the syntax of the YAML file; it's testing the semantics of the Prometheus configuration.

The core configuration in that post includes a fragment:

- alert: MyAlert
  expr: avg without(instance)(up) < 0.75
  for: 2m
  labels:
    severity: page

How do you know this is the right configuration?  If it's misconfigured then either your on-call team gets paged when the system is healthy, or it doesn't get paged when the system is down; both are bad.

The test setup shown in the blog post would let you test your Prometheus alerting setup, with artificial sample data (the "bar" instance is up for 5 minutes, then down for 5 minutes, and it's expected to page after it's been down for longer than 2 minutes), before you deploy it for real and without actually paging anyone.  That gives you some confidence your monitoring setup will do what you expect, before you deploy it for real.


Jenkins Plugin Install Failed I am trying to set up new Jenkins. When I download Suggested Plugins OR try to install any plugin I am getting below error.
I am using Jenkins Latest Version 2.204.1. I have set up this on VM. I am getting this error for all the plugins even when I am trying to install suggested plugins during the initial setup, that time also I am getting the same error. I am installing Jenkins on Microsoft Windows Server VM. I have downloaded Jenkins from Browser, extracted zip file and installed it using jenkins.msi.

    at java.net.DualStackPlainSocketImpl.waitForConnect(Native Method)
    at java.net.DualStackPlainSocketImpl.socketConnect(Unknown Source)
    at java.net.AbstractPlainSocketImpl.doConnect(Unknown Source)
    at java.net.AbstractPlainSocketImpl.connectToAddress(Unknown Source)
    at java.net.AbstractPlainSocketImpl.connect(Unknown Source)
    at java.net.PlainSocketImpl.connect(Unknown Source)
    at java.net.SocksSocketImpl.connect(Unknown Source)
    at java.net.Socket.connect(Unknown Source)
    at sun.net.NetworkClient.doConnect(Unknown Source)
    at sun.net.www.http.HttpClient.openServer(Unknown Source)
    at sun.net.www.http.HttpClient.openServer(Unknown Source)
    at sun.net.www.http.HttpClient.<init>(Unknown Source)
    at sun.net.www.http.HttpClient.New(Unknown Source)
    at sun.net.www.http.HttpClient.New(Unknown Source)
    at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(Unknown Source)
    at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(Unknown Source)
    at sun.net.www.protocol.http.HttpURLConnection.plainConnect(Unknown Source)
    at sun.net.www.protocol.http.HttpURLConnection.connect(Unknown Source)
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(Unknown Source)
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream(Unknown Source)
    at sun.net.www.protocol.http.HttpURLConnection.getHeaderField(Unknown Source)
    at java.net.URLConnection.getHeaderFieldLong(Unknown Source)
    at java.net.URLConnection.getContentLengthLong(Unknown Source)
    at java.net.URLConnection.getContentLength(Unknown Source)
    at hudson.model.UpdateCenter$UpdateCenterConfiguration.download(UpdateCenter.java:1194)
Caused: java.net.SocketTimeoutException: connect timed out
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
    at java.lang.reflect.Constructor.newInstance(Unknown Source)
    at sun.net.www.protocol.http.HttpURLConnection$10.run(Unknown Source)
    at sun.net.www.protocol.http.HttpURLConnection$10.run(Unknown Source)```[![enter image description here][1]][1]

  [1]: https://i.stack.imgur.com/2TgUP.png

You can try 3 different things-

Try pinging https://updates.jenkins-ci.org/download/plugins/ and see if it gets any response. If it doesn't respond then make sure your internet is working fine.
You can try installing the plugins by connecting to a VPN.
Lastly, you can reinstall Jenkins to see if the error persists.


How to login to k8s proxy nowadays? Back in the day when I ran kubectl proxy, I could login to http://localhost:8001 immediately, but know a login prompt is shown. When skipped is clicked the following message is shown:

  configmaps is forbidden: User
  "system:serviceaccount:kube-system:kubernetes-dashboard" cannot list
  configmaps in the namespace "default": Unknown user
  "system:serviceaccount:kube-system:kubernetes-dashboard"

Attempt to solve the issue

When the kubeconfig is copied, then the login fails as well

What is the Kubernetes version? Is it >1.8 ? (which has RBAC enabled by default).

The error message says that the Kubernetes Dashboard service account is not allowed to list configmaps in default namespace. RBAC by default is deny-all. You need to explicitly grant permissions.

You can create the following ClusterRoleBinding to grant dashboard full admin privileges.

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: kubernetes-dashboard
  labels:
    k8s-app: kubernetes-dashboard
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: kubernetes-dashboard
  namespace: kube-system

Save the above block into dashboard-rolebinding.yaml and create it on the cluster:

kubectl create -f dashboard-rolebinding.yaml

IMPORTANT: Make sure your dashboard is secured from outside before doing this. Otherwise anyone can have complete access to your cluster. Here is a great article about securing  kubernetes dashboard, from Heptio: https://blog.heptio.com/on-securing-the-kubernetes-dashboard-16b09b1b7aca writted by Joe Beda himself.


Why HashiCorp vault returning null? I have a HashiCorp vault setup and it seems to return null all the time except when I ssh into the VM and then I set the values again followed by checking the values directly on the vault server.

Any ideas what might be happening?

I am a bit lost on this and not even sure what logs or config files are helpful. This is one of my first experiences with vault and so far it seems finicky at best.

If you are unsure what happened, try deploying it in AWS via AWS vault quickstarts. They deploy vault end-to-end, so now you can compare it to yours and find out what was the problem.


I have an custom AMI and want to run certain config commands at initial boot, depending on the autoscale instance that starts it. How do I do this? I have a custom AMI that contains the dependencies for several applications.

The only difference between the applications is several configuration options that can be configured by a bash script. (i.e. enabling/disabling certain supervisor groups.)

Each different app is triggered by an different autoscale instance.

How do I get the instance to run a different script on boot by an autoscaler?

If I am understanding you correctly, you have one AMI used in several autoscaling groups and want to run a specific “bootstrap” sequence on it depending on the autoscaling group it belongs to.

This is not difficult to accomplish because the launch configuration you created to map the autoscaling group to the AMI can accomodate user data which can be a shell script.

In the shell script provided as user data, it is customary to retrieve meta data so that the instances can correctly identify itself in logs and arrange other similar last minute details.  This meta-data could also contain a reference to the auto-scaling group, which you could take advantage of to reduce the little overhead due to autoscaling-group management.


Deploying and testing features on a server by more than one developer I have 4 environments (prod/preprod/SIT/DEV), also two main branches (Master & Develop), two developers work on features (One & Two).

Developer One creates a feature (Feature A), he then merges the feature into the Develop branch.

Developer Two also creates a feature (Feature B) and he does the same, he merges the created feature into the Develop.

Now, the Develop branch will have two features (A & B), when the business owner decides to go with one feature ONLY (feature A), what should be the right approach to follow?

what is the best GIT workflow that we should follow? especially when implementing DevOps solutions.

P.S. I only mentioned two developers with two features, while in real life, I might have many more at the same time, hence I need to know the best approach to follow.

Since you have a single released version product one thing to consider would be keeping the Develop branch or not:

keeping it would have the advantage that at any moment you can bring in a hotfix in master even if Develop isn't stable enough for a merge into master. But this means you'd need to have 2 focus points instead of a single one - both branches need to be kept stable. There will be additional efforts for constant merges back and forth between the 2 branches to keep them from diverging. The overhead of context switching between them can be significant.
getting rid of it (see How to get rid of develop branch for simplified Git flow) and doing the integration directly in master would reduce the overhead of carrying together 2 branches. The releases would simply be tagged versions of the master branch. But getting in a hotfix for the current release version would require the branch to be stable enough (virtually ready for a new release) at any moment. Which would be desired anyways and is achievable (with some effort).

In the context of continuing the practice of developing features in isolation (on separate, potentially long-lived feature branches) and going through the pains of integration hell one thing that could be improved would be a better communication between the business owner and the development team: deciding to drop feature B after it was integrated into the product is practically equivalent with adding a new feature C (late in the game).

Another (much better IMHO) option would be to switch to the Continuous Integration practice: features would be developed directly in master (using very short lived branches carrying tiny incremental changes much easier to handle, ideally merged back into master in less than 1 day), behind feature toggles/feature flags:

during development the features would be disabled by default, making master virtually equivalent to the current release version, ready to receive hot fixes if/when needed.
the flags can be individually enabled in various environments to determine the respective feature progress/readiness for release. This can be done simultaneously for virtually any number of features (or feature combinations), regardless of their size or development stage or duration, as different verification steps of the automated CI system.
whenever a feature is ready and the business owners decide to accept it the respective flag is turned on in a tiny commit and a new tag "brings" the feature into the next release. No massive branch merges, no risk of surprises, everything should be just business as usual - all kinks should have been already ironed out. A much more predictable (and less stressful) development process.
keeping the flag supporting code in place for a while also allows staging the dropping of the feature in a subsequent release by disabling the flag in a CI system verification stage. But eventually a cleanup of the feature flag and the supporting code can/should be done as a background code base maintenance task, making the feature permanent.

IMHO such approach is the only reasonable option in larger scale projects where many features are being developed simultaneously.


Istio: envoy proxy to https container in same pod I've got a container that runs a https service on port 9999.
The service is as follows:

apiVersion: v1
kind: Service
metadata:
  labels:
    role: pingfederate2
  name: pingfederate2
spec:
  ports:
    - name: https
      port: 443
      targetPort: 9999
  selector:
    role: pingfederate2

The envoy log generated when going through the gateway is

[2020-03-19T08:42:37.356Z] "- - -" 0 - "-" "-" 2046 7 11 - "-" "-" "-" "-" "127.0.0.1:9999" inbound|443|https|pingfederate2.default.svc.cluster.local 127.0.0.1:39954 10.25.132.9:9999 10.25.131.133:56890 outbound_.443_._.pingfederate2.default.svc.cluster.local -

To me, it looks like envoy is trying to communicate with the pod over http, not https.
Is there a way to force Envoy to use https to locally communicate with the container? I could be barking up the wrong tree though.

Changing the destination rule from ISTIO_MUTUAL to SIMPLE fixed this issue.
There's obviously a problem with my config with the mTLS.


Configure Kubernetes with docker container on Ubuntu VM I am using the YAML file to deploy the container on Kubernetes with some replication factor on a hosted machine.
YAML File
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mojo-deployment
  labels:
    app: mojo
spec:
  selector:
    matchLabels:
      app: mojo
  replicas: 3
  template:
    metadata:
      labels:
        app: mojo
    spec:
      containers:
        - name: mojo
          image: mojo:1.0.1
          ports:
            - containerPort: 9000

---
#Services Info
apiVersion: v1
kind: Service
metadata:
  name: mojo-services
spec:
  selector:
    app: mojo
  ports:
    - protocol: TCP
      port: 80
      targetPort: 9376

---

#Ingress Configuration
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: mojo-ingress
  annotations:
    kubernetes.io/ingress.class: mojo
spec:
  backend:
    serviceName: mojo-services
    servicePort: 80

Steps:

Build Docker image using `docker build -t mojo:1.0 .
docker image ls show me an image id.
Skipping docker build command to deploy image on container. Do I need to do it? or kubectl service will take care of it.
Run kubectl apply -f Prod.yaml. It shows

deployment.apps/mojo-deployment created
service/mojo-services created
ingress.networking.k8s.io/mojo-ingress created

kubectl get service returns

kubectl get pod returns

kubectl get deployment returns

Questions?

Do I need to build the container before deploying YAML file? I tried it but still kubernetes not running.

Why all pods are showing Pending status.

Deployment is also showing pending status.

Though I am trying to access the Ingress with :80 and cannot access it.

You have to push your image in a registry in order to get the deployment working. Kubernetes does not deal with images (i.e. building, pushing...).

Try kubectl describe deployment mojo-deployment, you'll get more informations about why it's pending (surely because it does not know how to get your Docker image).


What does it mean that "outside of memory-managed code execution runtimes (JVM, Go, etc.) the usefulness of unikernels starts to rapidly decline"? The benefit of unikernels is managing a large number of applications in a protected space. (Some might say all the benefits of Docker without all the overhead). (Please don't get me wrong - I'm a huge fan of docker and use it 20 times a day in my work - I'm using this question as a a way to explore an idea).

The following commentator writes:

  Another minor point is that outside of memory-managed code execution runtimes (JVM, Go, etc.) the usefulness of unikernels starts to rapidly decline.

Now to me, a server application written in C++ (not memory managed) vs Java (memory managed) has no impact on the utility of a unikernel. In both you get the isolated protected way to manage your application lifecycle. Perhaps I'm missing something.

My question is: What does it mean that "outside of memory-managed code execution runtimes (JVM, Go, etc.) the usefulness of unikernels starts to rapidly decline"?

Well, I'm not a big expert on the matter and, in fact, I don't understand the statement. I've read it several times and there is no further explanation, so I cannot say it's make sense or not for me because I just don't know where the idea comes from.

Anyway I'll try to expose what I think here.
The main difference about managed and unmanaged code is who is in charge of the memmory management. If it's the developer, we're talking about unmanaged code (C language). If there is a garbage collector or any other similar mechanism, it's managed (Go, Java, .Net platform, Python even bash if you want to consider it!!). So I don't undertand how important is who reserves/frees/whatever the memmory.

And, IMHO, pros of unikernels are supossed to be:

Performance.

Kernel modes are supressed (rings for x86, modes for arm, ...) and the application that will be running in the system, as well as any library needed, is compiled and statically linked into the kernel and the overhead introduced by these mechanism (syscalls basically) are supressed.
Kernel is reduced as much as possible. No USB, bluetooth, support for lots of FS and unnecessary code is just not compiled. Kernel gets tiny.

Security. This is a sensitive topic. Again, I think unikernels are safer than the traditinal ones. If it's a well-done job, there won't be nothing than the minimun functionalities implemented into the system, so this should reduce possible break-points. However, if any is found, this will be more dangerous. Every single process is running by the privileged privileged (and only, multiuser capabilities are disabled in unikernels) user in the system. Privileged Remote code executions can happend. However, it should be in a container-like system. You just can restart it and (in no persistent systems) all the work done by the attacker should be gone (but still an upgrade should be needed...).

As I said, I cannot figure out the reason of this statement, but I think it's related to performance rather than security. In that case, I'm wondering why should be a performance improvement reducing syscalls compared to containers if:

Containers share the running kernel within the host where are running, so syscalls are done just once as there is just one instance of the kernel running.
Unikernels, as said, do not use syscalls and every procces runs in privileged mode (can access the harware directly). But, in this case (I THINK) as far as kernels are different, you need a virtualization layer, so total running kernels are 2. Unikernel has no syscalls but the host kernel does, so syscalls are not supressed and will run once. Of course, I'm assuming that no-one will run a unikernel system on a bare-metal.

So, the number of syscalls should be pretty similar to the containers scenario. Isn't it?


Ansible: Other option available for telnet check of open ports? I'm new to Ansible. Here's my task ...

I have 400+ hosts, and I need to verify if 5 different ports are open from their end to our web server.

Individually, I could log in and run:

telnet mywebserver.com 443
telnet mywebserver.com 80
telnet mywebserver.com 8443

..and so on..

What module or plugin could be used in Ansible so I could automate this, and have it report the results (whether open or closed ports) back to my Ansible server?

You can use the Ansible wait_for module which checks a specific TCP port is open.
Since in this case, all ports should be open already, we can use a minimal no. of retries, just enough to cover network issues:
- name: Check all port numbers are accessible from the current host
  wait_for:
    host: mywebserver.com
    port: "{{ item }}"
    state: started         # Port should be open
    delay: 0               # No wait before first check (sec)
    timeout: 3             # Stop checking after timeout (sec)
  ignore_errors: yes
  with_items:
    - 443
    - 80
    - 80443

By default, Ansible will check once every second (configurable in Ansible 2.3 using the sleep attribute), so this will check 3 times per port.
Run this in a playbook against your inventory of 400+ hosts - Ansible will check in parallel that all hosts can reach mywebserver.com on those ports.

the parallelism is subject to the forks setting in your ansible.cfg.

We use ignore_errors: yes here so that any errors are marked in red but do not stop execution.
Open ports are reported as ok items in output and closed ports are reported as failed (you must use -vv flag on ansible-playbook to see this output).
Fine-tuning output
If you want more specific output for the success and failure cases, the code must be more complex, adding a second task:

wait_for task must register a variable
the second task produces output using debug based on success/failure condition (e.g. using Jinja2 conditional expression)
then you need to put both these tasks in an include file (without any with_items loop), and write a main playbook task that uses an include ... with_items to call the include file once per port.


How to rebuild multiple jobs at the same time in Jenkins? I'm using Jenkins Rebuild Plugin to execute a parametrized builds with a single Rebuild button. However when I need to rerun dozens of parametrized jobs (either selected, or from the same folder), it's still not efficient.

Is there any simpler way (eventually using different plugin) to re-run multiple jobs at the same time using the last used parameters?

You can use a pipeline to tie the jobs together, executing them either in parallel or sequentially through steps.
You can pass the relevant parameters to each job as you call it from the pipeline script.
Then you just need to rebuild a single pipeline job. You can tie the jobs together with a common build name through a variable as well.

build job: 'buildjob1', parameters: [
  [$class: 'StringParameterValue', name: 'BUILD_NAME', value: ${env.JOB_BASE_NAME}-${env.BUILD_NUMBER}"],
  [$class: 'StringParameterValue', name: 'FIXEDPARAM', value: 'some-string'],
  [$class: 'StringParameterValue', name: 'PARAM1', value: "${PARAM1}"]
]


Ingress ip returns HTTP error DNS_PROBE_FINISHED_NXDOMAIN I have the next ingress (output of "kubectl get ingress")
NAME   HOSTS   ADDRESS          PORTS   AGE
main   *       192.168.39.190   80      46m

But when I try to visit that IP address in chrome I get
Cannot access the website. Check if there's a write error in name.local
DNS_PROBE_FINISHED_NXDOMAIN

Any idea why? Everything is green on the dashboard, there are errors.

There are two main reasons for this issue to happen:

Your dns are not exposed.
Your code is redirecting to a https address (and since you are working in your local machine, most likely you don't have a secure certificate)

CASE 1 - Exposing DNS
On linux you can solve it like:

minikube ip

vim /etc/hosts

Add the ip from the first step and the hostname you want to expose, and save the file.
# For example
192.168.39.70   elasticsearch-es-http

No need to restart any service, it should work just like that. Add all the hostnames you need.
CASE 2 - HTTPS on minikube
If your problem is 2, then you have to find a way for you application to not use HTTPS while you are using minikube.


Worker Node in Kubernetes I am new to Kubernetes. I have followed one document and setup master node in Kubernetes. Now I want to create a Worker node, so How can I create it? I don't know anything about to set up a worker node.

If you are running on VMs or outside of a cloud provider then manually setting this up involves several steps.  The tool you will need is kubeadm (see documentation linked) and you will need to init the nodes and then kubeadm join the nodes to the master.

If you are using a cloud provider, then follow the providers documentation on how to add additional nodes.

A great guide called "Kubernetes The Hard Way" is another good resource for learning how to setup a cluster without any automated tooling.


Docker container is running but when using curl not accessible I tried to host docker container in my EC2, just for UAT (for production, we will fargate). My concern is I want to host my container in EC2 and run all test cases, after successfully will be deployed to fargate.

Here is appspec.yml

version: 0.0
os: linux
files:
  - source: /
    destination: /root/MY-BACKEND
    overwrite: true
hooks:
  BeforeInstall:
    - location: ./scripts/before_install.sh
      timeout: 300
      runas: root
  AfterInstall:
    - location: ./scripts/after_install.sh
      timeout: 300
      runas: root
  ApplicationStart:
    - location: ./scripts/app_start.sh
      timeout: 300
      runas: root
  ApplicationStop:
    - location: ./scripts/app_stop.sh
      timeout: 300
      runas: root
  ValidateService:
    - location: ./scripts/validate_server.sh
      timeout: 300
      runas: root

and after_install.sh

  cd /root/MY-BACKEND
  docker build -t my-backend .

and app_start.sh

  cd /root/MY-BACKEND
  docker run -d -p 80:2019 --name my-docker my-backend

and validate_server.sh

  curl -m 5 http://0.0.0.0

After that, I've encountered following error message from Validate Server [stderr]curl: (52) Empty reply from server

And when changed to curl -m 5 http://localhost, encountered following error message [stderr]curl: (56) Recv failure: Connection reset by peer

But when I open browser, my app is working either tried curl from my local and remote server, both are working. Please let me know which I missed to add some validation.

I've found solution. Problem is backend took 10 mins to make fully loaded. When curl -m 5 http://0.0.0.0 in validate_server.sh, backend is not ready yet to response back that's why [stderr]curl: (56) Recv failure: Connection reset by peer throw. So, I've added sleep 20s above curl -m 5 http://0.0.0.0 inside validate_server.sh, now it was working.


Create service per pod in statefulset I have a statefulset app which can have any number of replicas. The pods produced should be able to communicate to external comptures. Those pods must act statefully when clients communicate with them.
I tried to create headless service and nodeport services, but it still has only one service to communicate with and also couldn't make it to communicate externally.
How can I make it allocate external ip and port per each port?

Just to clarify if I got you right.
Do you want to have a separate service for every pod/replica in your statefulSet?
If so - this is not how K8s sees services being used. The service is used for grouping and load balancing purposes.
In your case, if you want to have 1-on-1 connection between an external client and a pod, I suggest deploying your statefulSet a number of times with replicas=1 and exposing a service (as NodePort/LB) every time.