docker swarm init made docker go corrupted I started learning docker recently and I set up a virtual machine (VirtualBox) with Lubuntu image:

Linux verion: Linux 4.18.0-10-generic i686
Docker version: 18.06.1-ce, API-version: 1.38, Go version: go1.10.4, Git commit: e68fc7a

So unless I have started acquaintance with swarm everything was going smoothly. However after I executed docker swarm init daemon seemed going down and whatever command I'm now trying to execute, it responds with: Cannot connect to Docker daemon at..

When I'm trying to run sudo dockerd it shows the output where there're some warnings like failed to load plugin io.containerd.snapshotter.v1.btrfa and one error saying error creating cluster object... error="name conflicts with an existing object" module=node node.id=.... Eventually the output ends with floating point exception with some address in memory.

Restarting does not help. The only things that makes docker run is reverting to the snapshot that I luckily made before calling swarm init.

So I would appreciate your help or advice how I do fix everything out.

This response is for
Kernel: Linux 4.4.132-1075-rockchip-ayufan-ga83beded8524
Architecture: aarch64 aarch64 aarch64 GNU/Linux

I had the same issue when issuing commands to a cluster in swarm mode.

Running docker info would result in Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

Running sudo systemctl start docker would result in a failed service state and output reading

Job for docker.service failed because the control process exited with error code.
See "systemctl status docker.service" and "journalctl -xe" for details.

Running systemctl status docker.service would result in

docker.service - Docker Application Container Engine
   Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Thu 2019-05-16 06:03:46 UTC; 1min 12s ago
     Docs: https://docs.docker.com
  Process: 3874 ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock (cod
 Main PID: 3874 (code=exited, status=2)

Running journalctl -xe wouldn't result in any information related to docker

Running sudo dockerd -D would result in lots of output with a certain line reading
error="name conflicts with an existing object" module=node node.id=105rworelt1fvw9ttjhnf61aw

Looking in /var/lib/docker there is a folder called swarm and doing a grep on all files and folders in the swarm directory for the
string 105rworelt1fvw9ttjhnf61aw came up with two files with the string 105rworelt1fvw9ttjhnf61aw in them.
sudo grep -r "105rworelt1fvw9ttjhnf61aw" /var/lib/docker/swarm/

Those two files were /var/lib/docker/swarm/.tmp-state.json504201306 and /var/lib/docker/swarm/state.json

At this point I wasn't feeling very surgical so I removed all files and folders under the /var/lib/docker/swarm and then tried sudo systemctl start docker again

A bit surgical: sudo rm -ri /var/lib/docker/swarm/

Note: Don't delete the swarm directory. If you do just create it again.

After this a docker info would work and I was able to docker swarm init again and deploy a stack without issues


What is the message of this picture? I have found this picture in a comment to this Reddit thread and would like to better understand its message. I do not know the image's original source, maybe it will come up later.

The Reddit thread asks:

  Why is the Docker Mind Shift so hard?

The OP then links to The Docker Mind Shift and states that

  The Docker Mind Shift helps explain the challenges many people struggle with, in DevOps and containers. Thoughts?

(Currently,) the highest upvoted comment states

  to be honest my biggest challenges have been with the speed of tooling changes rather than the concepts. In theory, i get the workflow and ideas behind docker fairly well, but in practice sometimes it does feel like you're throwing a whole load of plates up in the air and then walking off before they hit the ground again. I get that as a new tech idea it moves fast, but just trying to keep up with the ever moving goalposts can be exhausting. It does seem to have gotten better in the last 2 years though to be fair.

In response to this comment, sirex007 responds

  after finishing any reasonably sized project in docker, i'm always reminded of this image

and links to the above image. This image is attempting to comically illustrate the confusion many feel with the world of DevOps. There is a smorgasboard of toolkits and frameworks out there, and the bullets in the image are a commentary of the nonsense of the diagram to the right.

This is seen in the first bullet:

  Did you just pick things at random?

The subsequent bullets then expand on this question. For example, Redis is a caching system and stores key-value pairs. Similarly, MongoDB is a type of NoSQL database which also supports key/value pairs and caching, so it is a little silly to have both Redis and MongoDB.

It seems that sirex007 feels that this graphic illustrates the point that I get that as a new tech idea it moves fast, but just trying to keep up with the ever moving goalposts can be exhausting. and my biggest challenges have been with the speed of tooling changes as there are a lot of trendy tools appearing in the graphic, which is making the point that the implementer doesn't seem to understand the toolkits in use in his or her stack well.

This is a fair point: with the DevOps goal of rapid iteration and deployments, the appearance of new tools on a frequent basis and the changes within those tools could certainly seem like they are moving at a breakneck pace. I would argue that this is evidence of the success of the DevOps model however. If these projects are able to successfully iterate rapidly using DevOps, it is proof of the model's successes - though perhaps the cost is trying to keep up with that rapid pace.


Javascript based alternative to Fabric Is there a similar DevOps tool to fabric with JavaScript as its scripting language? I'm specifically interested in the remote execution side.

Most of the tools I've found rely on python (e.g fabric) or Ruby (e.g. Capistrano, Chef). However, in my team we don't use these languages for anything else. While these languages may be great, they are not as universal as JavaScript in the web development industry.

I mention fabric because it's ideal for what I want except that I have minimal python skills (sadly).

Fabric (and Capistrano, presumably the unnamed Ruby tool you came across) are a bit unusual in that they're task runners with extra features for easily running tasks on remote hosts.  I'm not aware of any other popular tools that do exactly the same thing, but depending on your needs there are a few other options that may work for you.

Grunt and Gulp are the foremost task runners in the Javascript ecosystem.  While not designed specifically to deploy software, they both have plugin-based systems that allow you do this, and you'll find plenty of articles of people using either one to do so.  For instance, this tutorial uses the grunt-ssh plugin to integrate ssh with grunt, while this one relies on gulp-rsync.

Using task runners to deploy code tends to be an approach favored by developers who are wearing an ops hat.  When you ask an operations engineer, they're more likely to suggest hacking this behavior into a configuration tool.  This can take many forms (for instance, packaging your app into an rpm or deb, then installing it like any other software), but I'm going to focus on what I think would be the simplest and most direct method for you.

Ansible is ostensibly a configuration management tool, but it really shines at task orchestration, which I would define as "running a set of actions across a set of servers".  The official documentation is a good place to get started, but to give you an idea of what this might look like, you might have a playbook (a YAML file that defines a set of tasks) like this:

- hosts: webservers
  tasks:
    - name: update code checkout
      git:
        repo: ssh://git@github.com/mylogin/hello.git
        dest: /var/www
        version: master

    - name: restart webserver
      command: restart-command-you-run

When you run this file with ansible-playbook, it will look up your host list (as simple as an ini file, but can also be a dynamic list written in any programming language) to determine what servers exist and what groups they're in, and then run the defined tasks on the servers that you tell it to.

Now, you specified that you wanted to stay in the Javascript ecosystem.  However, while Ansible is a new thing for you, it doesn't have all the complexities of another programming language, since configurations are written in YAML.  It will also allow you to do much much more later, and unlike several of its competitors, uses ssh to communicate, so there's very little setup involved (nothing to install on the remote machines).


How to append namespace before metric name in prometheus? I have static config in prometheus.yml
  - job_name: 'myjob'
    static_configs:
      - targets: ['monitoring.api.endpoint.net']

And I recieve various metrics for example cpu_usage, disk_write_ops, etc. Is it possible to put them all into namespace like compute_cpu_usage with relabel tool?

Found this solution:
static_configs:
  - targets: ['monitoring.api.endpoint.net']
metric_relabel_configs:
- source_labels: [__name__]
  regex: '(.*)'
  replacement: 'compute_${1}'
  target_label: __name__


Whats the point of circleci orbs? I'm looking at: CircleCI Orb for kubernetes

Whats the point of this? Whats the benefit of an orb which is a CircleCI  specific feature rather than say a bash function/script (which is more interoperable/common)?

Seems like unnecessary lockin, but maybe I'm just being naive here?

Let me answer each your question in parts.

What the point is?

As per Orbs Introduction:

  orbs are shareable packages of configuration elements, including jobs, commands, and executors.
  CircleCI provides certified orbs, along with 3rd-party orbs authored by CircleCI partners

The advantages of this approach are that an authorized package of jobs, commands, and executors is offered.

Assuming hundreds of orbs, it looks like a community-approved plugin/addon ecosystem. E.g. Ansible roles or Jenkins plugins or Wordpress plugins

orbs vs shell functions/script

Since orb packages:

jobs,
commands,
and executors,

then it can hardly be replaced by a simple combinations of shell commands/functions.

Seems like unnecessary lockin?

As for lock-in concerns: you are right.

It looks like lock-in. And it is lock-in.

But as I know, there isn't any portable solution than can be easily migrated from one CI/CD system to another. E.g. try to convert .gitlab-ci.yml to travis.yml.

But lock-in's, as usual, tend to give you an advantage in speed or ease of deployment.

Portability

Since there are four executor types (docker, machine, windows and macos), it gives you more portable and cross-platform approach.


How to get traefik ingress to load balance across several ClusterIP services I want to be able to load balance several nodes of a backend service represented by ClusterIP services using a traefik.containo.us/v1alpha1 ingress. I know that ClusterIP service can load balance easily with deployments and ReplicaSet  but the way this application is designed each node needs to be it's own Deployment as it is a stateful node.
So I either need a way to have the flow be:
Ingress --> [ClusterIP-Node1, ClusterIP-Node2, ClusterIP-Node3, ClusterIP-Node4]

or
Ingress --> ClusterIP-LoadBalancer --> [ClusterIP-Node1, ClusterIP-Node2, ClusterIP-Node3, ClusterIP-Node4]

How can I use traefik.containo.us/v1alpha1 to automatically loadbalance between these ClusterIP services?
For example, if I currently have one Ingress for each `ClusterIP, like this:
...
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: my-service-node1-ingress
spec:
  entryPoints:
    - web
    - websecure
  routes:
    - kind: Rule
      match: Host(`myservice-node1.mycompany.example`)
      services:
        - kind: Service
          name: myservice-node1-clusterip
          passHostHeader: true
          port: 80

---
apiVersion: v1
kind: Service
metadata:
  name: myservice-node1-clusterip
  labels:
    app: myservice
    tier: backend
    name: myservice-node1
spec:
  selector:
    app: myservice
    tier: backend
    name: myservice-node1-deployment
  type: ClusterIP
  ports:
    - name: rpc-listener
      protocol: TCP
      port: 80
      targetPort: 8545
... N+1

And I want to change that to a single Ingress Point similar to this:
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: my-service-ingress
spec:
  entryPoints:
    - web
    - websecure
  routes:
    - kind: Rule
      match: Host(`myservice.mycompany.example`)
      services:
        - kind: Service
          name: myservice-node1-clusterip
          passHostHeader: true
          port: 80

Is it possible to have the traefik.containo.us/v1alpha1 load balance across the services similar to this? I saw something that looked promising on this page in the docs that you can have n number of services[n].name for a matching route routes[n].services. Does that mean you can do something similar to the following?
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: my-service-ingress
spec:
  entryPoints:
    - web
    - websecure
  routes:
    - kind: Rule
      match: Host(`myservice.mycompany.example`)
      services:
        - kind: Service
          name: myservice-node1-clusterip
          passHostHeader: true
          port: 80
        - kind: Service
          name: myservice-node2-clusterip
          passHostHeader: true
          port: 80
        - kind: Service
          name: myservice-node3-clusterip
          passHostHeader: true
          port: 80
        - kind: Service
          name: myservice-node4-clusterip
          passHostHeader: true
          port: 80

Also, is there a way to pass the sessionAffinity with it like there are in services sessionAffinity: ClientIP?

The sample I posted initially works as is:
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: my-service-ingress
spec:
  entryPoints:
    - web
    - websecure
  routes:
    - kind: Rule
      match: Host(`myservice.mycompany.example`)
      services:
        - kind: Service
          name: myservice-node1-clusterip
          passHostHeader: true
          port: 80
        - kind: Service
          name: myservice-node2-clusterip
          passHostHeader: true
          port: 80
        - kind: Service
          name: myservice-node3-clusterip
          passHostHeader: true
          port: 80
        - kind: Service
          name: myservice-node4-clusterip
          passHostHeader: true
          port: 80


Is it possible to upgrade Jenkins from Jenkins it-self? On Manage Jenkins I've a message that:

  New version of Jenkins (2.47) is available for download (changelog).

Is there any way of upgrading Jenkins from the GUI similar as it's done for the plugins?

I'm using Jenkins ver. 2.43 on Linux.

Yes, it is possible. In Windows, this is what I see when there is a new version available:

When you click the Upgrade Automatically button, Jenkins downloads and install the new war file itself.

Please, check Jenkins wiki for further details. There are some problems if you installed Jenkins from a Debian package.

  If you run Jenkins with java -jar jenkins.war, Jenkins will be able to update itself from the Manage Jenkins page. This is the simplest way to do automatic upgrades.

*nix/Mac

  If you've installed via the OS X installer, the jenkins running the process does not own the .war file. To get auto-upgrade working, just fix the permissions so the jenkins user can write the WAR (in /Applications/Jenkins).

Debian Package

  Using the Upgrade-Button from within Jenkins (available since 1.318) will not work if Jenkins was installed from a Debian package (results in permission denied errors when trying to download the new WAR file)!

Windows Auto-Upgrade

  If you install Jenkins as a Windows service, Jenkins will be able to update itself from the Manage Jenkins page.


ansible AWX is connecting through ssh as root instead of user defined in credential I'm running ansible 2.6.2 and awx 1.0.7.2 on rhel 7.5. AWX is connecting to my ssh proxy as root instead of the defined Credential user.

My playbook contains the following

become: yes
become_method: sudo
remote_user: remoteuser
ansible_user: remoteuser

ssh.config in my playbook contain the following

match host !proxyA,!proxyB,* exec "ssh proxyA nc -w1 %h %p &>/dev/null
  ProxyJump proxyA
match host !proxyA,!proxyB,* exec "ssh proxyB nc -w1 %h %p &>/dev/null
  ProxyJump proxyB

My AWX credential user which uses an ssh key and associated with the AWX JOB Template is this (which should be used for ssh connections):  remoteuser

I have no defined entries in /etc/ansible/ansible.cfg or any ansible.cfg defined in the current working directory (of the playbook) or in $HOME.

Here is the proxyA/B syslog entries showing the user that AWX is connecting as:

sshd: Failed publickey fro root from 1.2.3.4

And here is the AWX ansible output showing the connection as 'root' instead of the 'remoteuser'

root@proxyA's password: root@proxyA's password:  root@proxyA's password:

I can run the playbook through ansible (outside of AWX) without errors.

How do I tell AWX to use the right user name?

According to https://docs.ansible.com/ansible/latest/user_guide/playbooks_intro.html

it should be remote_user: if it still does not work, run the playbook with -vvvv and add the log to the Q&A.


Ansible fails in Jenkins pipeline when run from docker I'm trying to deploy a build via Jenkins pipeline using agent docker and Ansible playbook but it fails on Gathering Facts stage as shown below:
TASK [Gathering Facts] *********************************************************
fatal: [destination.box.local]: UNREACHABLE! => {"changed": false, "msg": "argument must be an int, or have a fileno() method.", "unreachable": true}

Similar Jenkins pipeline using agent any and Ansible not from docker (local installation) will do the job w/o any hiccups.
Agent section from Jenkins pipeline looks like:
pipeline {

    agent {
       docker {
          image 'artifactory.devbox.local/docker-local/myrepo/jdk8:latest'
          args '-v $HOME/.m2:/root/.m2 -v /etc/ansible:/etc/ansible -v $HOME/.ansible/tmp:/.ansible/tmp -v $HOME/.ssh:/root/.ssh'
       }
    }

   parameters {
      string(
              name: 'PROFILE',
              defaultValue: 'dev',
              description: 'Select profile to build?'
      )
      string(
              name: 'HOST',
              defaultValue: 'dev',
              description: 'Which Outbound Host(s) group to deploy to?'
      )
   }

   stages {

      stage('Artifactory configuration') {
         steps {
            rtMavenDeployer(
                    id: "MAVEN_DEPLOYER",
                    serverId: 'Artifactory',
                    releaseRepo: "libs-release-local",
                    snapshotRepo: "libs-snapshot-local"
            )

            rtMavenResolver(
                    id: "MAVEN_RESOLVER",
                    serverId: 'Artifactory',
                    releaseRepo: "libs-release",
                    snapshotRepo: "libs-snapshot"
            )
         }
      }

      stage('Maven build') {
         steps {
            withEnv(['JAVA_HOME=/usr/lib/jvm/java-1.8-openjdk', 'MAVEN_HOME=/usr/share/maven']) {
               rtMavenRun(
                       pom: 'pom.xml',
                       goals: 'clean',
                       deployerId: "MAVEN_DEPLOYER",
                       resolverId: "MAVEN_RESOLVER"
               )
            }
         }
      }

      stage ('Publish build') {
         steps {
            rtPublishBuildInfo (
                    serverId: 'Artifactory',
            )
         }
      }

      stage('deploy') {
         when{
            anyOf {
               branch 'master'
               branch 'develop'
            }
         }
         steps {
            ansiblePlaybook(
                    playbook: 'playbook.yml',
                    extraVars: [
                            profile: "${params.PROFILE}",
                            host: "${params.HOST}"
                    ]
            )
         }
      }

   }
}

After adding ansible_ssh_common_args='-o StrictHostKeyChecking=no' to the Ansible inventory (or setting host_key_checking = False in the config) I have got that error:
TASK [Gathering Facts] *********************************************************
fatal: [destination.box.local]: UNREACHABLE! => {"changed": false, "msg": "'getpwuid(): uid not found: 700'", "unreachable": true}
fatal: [ansible_ssh_common_args=-o StrictHostKeyChecking=no]: UNREACHABLE! => {"changed": false, "msg": "[Errno -3] Try again", "unreachable": true}

Any thought what I need to add to it to let Ansible run a playbook?

In my case it ended up that Jenkins was running docker agent with specific UID and GID. To get that fixed it required to rebuild that docker image with creating internal Jenkins user with the same UID and GID
For that purpose on top of the Jenkinsfile to crate that docker image I have added:
def user_id
def group_id
node {
   user_id = sh(returnStdout: true, script: 'id -u').trim()
   group_id = sh(returnStdout: true, script: 'id -g').trim()
}

and then during the build stage I have passed additional arguments to the docker as
--build-arg JenkinsUserId=${user_id} --build-arg JenkinsGroupId=${group_id}

then in the Dockerfile for that build:
FROM alpine:latest

#pick up provided ARGs for the bild

ARG JenkinsUserId
ARG JenkinsGroupId

//do your stuff here

#create Ansible config directory
RUN set -xe \
    && mkdir -p /etc/ansible

#create Ansible tmp directory
RUN set -xe \
    && mkdir -p /.ansible/tmp

#set ANSIBLE_LOCAL_TEMP
ENV ANSIBLE_LOCAL_TEMP /.ansible/tmp

#create Ansible cp directory
RUN set -xe \
    && mkdir -p /.ansible/cp

#set ANSIBLE_SSH_CONTROL_PATH_DIR
ENV ANSIBLE_SSH_CONTROL_PATH_DIR /.ansible/cp

# Create Jenkins group and user
RUN if ! id $JenkinsUserId; then \
    groupadd -g ${JenkinsGroupId} jenkins; \
    useradd jenkins -u ${JenkinsUserId} -g jenkins --shell /bin/bash --create-home; \
  else \
    addgroup --gid 1000 -S jenkins && adduser --uid 1000 -S jenkins -G jenkins; \
  fi

RUN addgroup jenkins root

# Tell docker that all future commands should run as the appuser user
USER jenkins

and finally update docker agent for the main pipeline which had issue:
   agent {
      docker {
         image 'artifactory.devbox.local/docker-local/myrepo/jdk8:latest'
         args '-v $HOME/.m2:/root/.m2 -v $HOME/.ssh:/home/jenkins/.ssh  -v /etc/ansible:/etc/ansible -v $HOME/.ansible/tmp:/.ansible/tmp -v $HOME/.ansible/cp:/.ansible/cp'
      }
   }


How to access ephemeral storage of EC2 instance? I've got ephemeral storage setup in Scalr as part of the role. The instance is spinned up by Chef and it seems it is running fine (it's using ephemeral0 device mounted as D: drive).

I've tried to use aws ec2 describe-instances command, but here is no any indication that the ephemeral device is attached to the instance (unless I don't know what I'm looking for).

How I can access ephemeral storages of the running instances using via AWS CLI/API?

You are correct that the describe-instances API doesn't include information about ephemeral volumes. Once an instance is created, the only way to know that it has such volumes is available from the metadata service that is accessible from the instance itself.

According to User Guide for Linux Instances, the mapped ephemeral volumes are shown in instance metadata.

Since you created the instances using Chef, another option would be to check the Chef state database - most probably the information about a created instance includes the ephemeral device information. But this method highly depends on your Chef version and the cookbook you use to create the instances, so YMMV.


User data in original windows ec2 instance does not get executed in custom ami I created a windows Ec2 instance with user data in it. In my ec2 instance I downlaoded git and installed it. Later in EC2 instance, I provided it with the user data which was supposed to download a git repository and place it in C:

This is my user data in original EC2 instance.

 <powershell>
Remove-Item C:\c -Recurse -ErrorAction Ignore
C:\"Program Files"\Git\bin\git.exe config --global http.sslVerify false
C:\"Program Files"\Git\bin\git.exe clone https://github.com/(somename)/(someapp).git /c
</powershell>
<persist>true</persist>

The EC2 instance worked without any problem. Then I decided to create a custom AMI from it. I created one and launched another instance from it. I then tried to run the instance to which it ran succesfully. But then in the new EC2 instance that I created from custom AMI, the user data from the original EC2 instance was not executed. I tried changing the setting of EC2 instance which was found in C://programdata . I configured it to run user data on every boot. But then still user data was not executed. I am not sure what's going on here. Any help will be appreciated.

User Data Persistance

If I create an image with Packer or other tool, I'll include normally a userdata initialization script that would allow connecting via WinRM. This is required as the default AMI settings wouldn't allow the tooling to connect. In this scenario Packer would create the instance from an AMI with my provided userdata.

However, when I then take an AMI of this configured instance and try to use it as a new instance, this new image has it's own userdata. If I wanted to run the exact same command, I would need to create the userdata in this new instance.

Userdata isn't persisted in your AMI, it's more an instance level configuration value.

Follow-Up Comments on This

The goal should be to create your AMI with the tooling, for example git, and then your new instance should not even need to run that command in the first place as you've "baked" it into your "golden ami". Consider this in your approach. Also consider decoupling any installation directly from userdata and use something like AWS Simple Systems Manager Command documents to run the installation. This would decouple your installation code from your infrastructure allowing you to apply automatically as well as on demand without having to repeat your work.

User Data Content

The code example above has a typo in the closing tag for PowerShell. I'd try replacing this and see if that's the root issue. If there's further issues after fixing that, post a comment on this answer and I'd be glad to look further.

Also as far as powershell syntax consider using join path for proper path parsing.

For example $exe = Join-Path $ENV:ProgramData 'git/git.exe

The. You can run

&$exe --config --global ....

I'd also recommend that you consider looking at Amazon systems manager documents. You can create a  document that initializes a setting such as get config.

Then you create a AWS association to this document on a tag, at that point it will run automatically anytime a new instance is created without you having to continue to maintain it. It also allows you to update the document independently of the user data, and then ensure you can run it manually as needed as well. I'm a big fan of decoupling any type of configuration scripts from the userdata and using that as a last resort.

One last item...

Remove-Item C:\c -Recurse -ErrorAction Ignore

Highly recommend you revisit that. That's seems super easy to mess up and start deleting stuff you didn't mean to. I'd make sure you use a fully qualified path with Join-Path and also use the -LiteralPath instead of positional argument by default. If anyone created a folder c on c drive I'd reject that in a heartbeat in a code review.


How set and get var with Path? I write script for deploy files by ssh via Jenkins

#!groovy

node('superhost01'){
    String HOSTNAME="host01"
    String USERNAME="tech_user"
    withEnv(['PATH=/data/jdbc_connector'])
    stage('Prepare') {
        checkout scm
    }
    stage('Deploy') {
        sh """
        scp -r config.yaml ${USERNAME}@${HOSTNAME}:$PATH
        """
}

}

BUT Jenkins returned error in line

withEnv(['PATH=/data/jdbc_connector'])

error:

[Pipeline] node
Running on superhost01 in /data/jenkins/workspace/jdbc_connector
[Pipeline] {
[Pipeline] withEnv
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
[BFA] Scanning build for known causes...
[BFA] No failure causes found
[BFA] Done. 0s
java.lang.IllegalStateException: There is no body to invoke
    at org.jenkinsci.plugins.workflow.cps.CpsStepContext.newBodyInvoker(CpsStepContext.java:283)

how to correctly assign the value of a variable?

withEnv doesn't persist environment variables.  Instead, you need to tell withEnv what steps to run with that environment by passing it a block.   Here is what I think you want your code to look like:

#!groovy

node('superhost01'){
  String HOSTNAME="host01"
  String USERNAME="tech_user"
  withEnv(['PATH=/data/jdbc_connector']) {
    stage('Prepare') {
      checkout scm
    }

    stage('Deploy') {
      sh """
      scp -r config.yaml ${USERNAME}@${HOSTNAME}:$PATH
      """
    }
  }
}

As a side note, I suspect you actually want to set PATH to /data/jdbc_connector:$PATH, otherwise your checkout step will not be able to find git and your deploy step will not be able to find scp.


How to set minikube to auto start on windows log on? How I can set minikube to auto start on windows 10 login? I checked but I can't find anything on google.

I saw that there is a default parameter autostart-enabled=off but I dont know where I can set it.

At the moment after I restart my pc I can see in Hyper-V Manager that the minikube machine is running but when I try minikube dashboard in terminal I get:

  minikube is not currently running so the service cannot be accessed

Should the normal startup functionality of w10 work?

To change which apps run at startup, press and hold (or right-click) the Start  button, select Task Manager, and then select the Startup tab. Choose an app, then select Enable or Disable.
To add or remove an app from the Startup tab, press the Windows Logo Key  + R, type shell:startup, and then select OK. This opens the Startup folder. Select Start . Find the app you're looking to add or remove, and press and hold (or right-click) it. Select More > Open File Location. In the file location folder, press and hold (or right-click) the app and select Copy. Paste the shortcut to your app into the Startup folder to have it run at startup.

Info from:
https://support.microsoft.com/en-us/help/4026268/windows-change-startup-apps-in-windows-10


How to define variable in Jenkins declarative pipeline? I defined variable in declarative Jenkins pipeline script but having issues with simple variable declaration.

Here is my script:

pipeline {
    agent none
    stages {
        stage("first") {
            def foo = "foo" // fails with "WorkflowScript: 5: Expected a step @ line 5, column 13."
            sh "echo ${foo}"
        }
    }
}

but it's shows error:

org.codehaus.groovy.control.MultipleCompilationErrorsException:
startup failed:
WorkflowScript: 5: Expected a step @ line 5, column 13.
    def foo = "foo"
    ^

The variable must be defined in a script section.

pipeline {
    agent none
    stages {
        stage("first") {
            script {
                 foo = "bar"
            }
            sh "echo ${foo}"
        }
    }
}


Is it possible to recover accidentally deleted HashiCorp's Vault secrets? According to this tutorial, it is possible to delete a secret, by running:

vault delete secret/hello

What will if someone in the organization accidentally removed a secret. Is it still possible to recover this?

If your secret backend is a separate service, which it should be, then with regular backups you should be able to stop vault, replace the backend with a backup and be good to go.

I've done this for a file backend but don't know what would happen for other backends so ymmv.


Should I use multiple beats input in Logstash? After some researches around the beats input plugin and specially this rewrite I wonder if I should use only one beat input or multiples to handle multiples entry types.

I'll have events coming from roughly 500 machines, with a 20/80 windows/linux distribution. I plan to use multiples beats shipper, filebeat, metricbeat and maybe packetbeat.

Is there an interest at using one input per type/os couple or would just one input and "if type=..." in the filter pipeline be enough ?

So still not benchmarked much, but after a quick code inspection and a bunch of reading about netty and logstash in version 5 the input is not the bottleneck to worry about.

Logstash team did put a bunch of work in the way the filters and outputs plugins are run in parallel, the beats input plugin wait for a batch of events, and the performances problem have indeed been solved in version 3.1.0 by the rewrite I quoted in the question.


why can't map local path to docker's inside? I want to build private docker hub. and I did it with:

 docker run -it --restart=always -p 7901:5000   -v /data/registry:/tmp/registry registry /bash/bin

And my disk informations as below:

[root@10-19-47-24 docker]# df -lh
Filesystem      Size  Used Avail Use% Mounted on
/dev/vda1        20G  4.4G   16G  22% /
devtmpfs        3.9G     0  3.9G   0% /dev
tmpfs           3.9G     0  3.9G   0% /dev/shm
tmpfs           3.9G  401M  3.5G  11% /run
tmpfs           3.9G     0  3.9G   0% /sys/fs/cgroup
/dev/vdb         60G  572M   60G   1% /data
tmpfs           783M     0  783M   0% /run/user/0
tmpfs           783M     0  783M   0% /run/user/1001
tmpfs           783M     0  783M   0% /run/user/1000
overlay          20G  4.4G   16G  22% /var/lib/docker/overlay2/9923f757c58c0792526f4cb32750401847f127b9e33c0dc78a29d7933bcde5bf/merged
shm              64M     0   64M   0% /var/lib/docker/containers/51959fbdc1663f968eff770d54c56bc04ddaba2c2d3623058c90a00f582fa40c/mounts/shm

And my version of docker is:

[root@10-19-47-24 docker]# docker --version
Docker version 18.06.0-ce, build 0ffa825

And I can push hello-world to the hub, but I can't see anything inside the /data/registry. My question is where is my registry? Is there any mistake of my command?

Oh! I have search for a while, it's should be like:

docker run -d --name=ak_registry  --restart=always -p 7900:5000   -v /data/.docker:/var/lib/registry    registry

Used /var/lib/registry instead of /tmp/registry


How to pass Gitlab env var with sensitive characters to bash script? I have the following situation:
I'm storing sensitive data in Gitlab's environment and then passing these variables to a deployment script in bash.

Gitlab's env var:

Key: sensitive_var
Value: aaa$bbb*%

Then in the .gitlab-ci.yml file the env vars are passed to the deployment script as follows:

.deploy:
  script:
    - deploy.sh PASS=${sensitive_var}

The deployment script parses the arguments as follows:

for arg; do
  TEMP_VAR=$(echo ${arg} | sed -e 's?=? ?')
done

The problem is that when we check the output of the deploy.sh script the variable's value gets expanded resulting in aaa*%.

Solutions tried so far:

Storing the value in Gitlab's environment with single quotes ' but then the value was 'aaa*%'.
Passing the environment variable with double quotes:

.deploy:
  script:
    - deploy.sh PASS="${sensitive_var}"

.deploy:
  script:
    - deploy.sh PASS="$sensitive_var"

But the result was the same: aaa*%.

Combination of the above solutions didn't work either.

Why is this happening, and how I can fix it?

Surprisingly indeed GitLab works weirdly with the env vars.
It evaluates them internally, though it shouldn't in my view.

Value set as (in project Settings > CI/CD > Variables): 'aaa$bbb*%'
becomes 'aaa*%'
(with single quotes left!)

Couple of of proof links:

https://gitlab.com/gitlab-org/gitlab-ce/issues/45173#note_101659865
https://gitlab.com/gitlab-org/gitlab-ce/issues/27436

Workarounds:

1)
You can duplicate dollar ($) sign, i.e.: aaa$$bbb*%
(as mentioned on the first link above).

2)
Use base64 to encode and decode the value.

Do this first:

$ echo 'aaa$bbb*%' | base64
YWFhJGJiYiolCg==

Then store this value in project Variables.

And then in your script decode it:

PASS=$(echo ${sensitive_var} | base64 -D)


What are the levels of monitoring and their importance? We are going through a process of change in our IT Operations Management. One of the items we are trying to improve is our monitoring of our infrastructure.

So far I've been thinking of only 3 levels of monitoring and I believe that one tool can not do all very well.

To hire such tools in the market, we are in the process of listing these levels of monitoring, what is the relevance of this monitoring for the business and applications that achieve our objectives.

I believe to date that we have 3 macro levels of infrastructure monitoring:

1) Application Level

APM softwares

2) Asset Level (Hardware, VMs, OS)

3) Network Level

NMS softwares

Is my thinking correct?
Are there other levels of monitoring? We do not have microservices or containers, but I would like this to be a generic post.
What tools are currently on the market that will monitor each level?

In our environment we use Appdynamics and logz.io for all levels of monitoring and it's a best tool,

Basically it monitor it for all levels,

Databse
Application especially works well on Java based services
Servers, containers
remote services
Business transactions
service endpoints, etc.,

Also it monitors all the components like, CPU, Memory, Load average, Network errors per minute, Network utilization, top 10 process usage of CPU and Memory.

All monitoring you can get it in the graph format.


Build user for Visual Studio Online Using visual studio online, do I need a separate user for the purpose of ferrying code/builds across machines.  I understand that the built in service account builds code and can publish, but with things like Jenkins that require access tokens to retrieve source code (for projects like unity (no we are not able to use unity build)) I find it would be best to have a 'service' account where I can create personal access tokens that isn't tied to any one developer.

Do I need a 'build' user or is there another way to ferry code to Jenkins servers or using personal access tokens for deployment groups?

In my opinion, there should always one key per user:

Security: If an employee will leave the company then the key could be revoked and the person who left the company cannot trigger builds anymore
Access-control and responsibilties: developers from team b are allowed to deploy apps to the production environment of team b, while members of team a are responsible for their own app and are restricted to environment a.


Python Script in Jenkins I am trying to run a python script in Jenkins. My Jenkins job is integrated with Jira & on build its gives JIRA_ISSUE_KEY. I want to save the value of JIRA_ISSUE_KEY in a variable.

import subprocess
e = 'echo "$JIRA_ISSUE_KEY"'
x = subprocess.call(e, shell=True)

Whenever I am doing print e it gives o/p as "echo "$JIRA_ISSUE_KEY"" & when I am doing print e its gives o/p as 0.

My overall output in Jenkins console is

JIRA comment is added
Running as SYSTEM
Building in workspace /var/lib/jenkins/workspace/Jenkins Jira
Integration Test
[Jenkins Jira Integration Test] $ python /tmp/jenkins2864498002834652992.py
MCO-15498
echo "$JIRA_ISSUE_KEY"
0
Finished: SUCCESS

I want to save the value MCO-15498 to some variable. How can I do that?

using bash script (or pipeline step sh), I would do:

JIRA_ISSUE_KEY=$(python my_python_script.py)
echo "$JIRA_ISSUE_KEY"


Is it possible to containerize using docker without a local setup? I have a Windows machine and I am just starting to learn docker. I have gone through the internet that the best way to start is by installing docker-desktop.  Is there a way i can use docker remotely on any cloud providers without having to install it on my machine ?
From what I understand I may need to upload my jar files to the remote servers and everything else should be able to be done on the cloud server.
Is there any such subscription I could purchase to use docker remotely ? Please let me know all the possibilities.

Checkout Play with docker here. If you just started then it is best to install docker on your local machine to get more of a native experience and understanding of docker networks and overlays etc... and after that you can move on to any cloud like aws ECS service, azure etc....


Unable to run the playbook in remote machine using Ansible job template. Connection Failed Playbook.yml

---
- name: connectionTest
  hosts: Windows
  tasks:
  - name: "Windows Connect"
    win_command: hostname
    register: wincmd_out

Variable declared in Inventories Project/inventories/XXX_Local/

[Windows]
XXVMxxxxxxx

Error Message:

fatal: [XXVMxxxxxxx]: UNREACHABLE! => {
    "changed": false,
    "msg": "Failed to connect to the host via ssh: OpenSSH_7.4p1, OpenSSL 1.0.2k-fips  26 Jan 2017\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 58: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug1: Control socket \"/tmp/awx_1636443_S7HHdB/cp/6b18046463\" does not exist\r\ndebug2: resolving \"XXVMxxxxxxx\" port 22\r\nssh: Could not resolve hostname XXVMxxxxxxx: Name or service not known",
    "unreachable": true
}

Looking at your code it seems you try to connect to a Windows host. Usually you connect windows via WinRM and not via ssh

The user guide for this can be found here: https://docs.ansible.com/ansible/latest/user_guide/windows_winrm.html

  For Ansible to communicate to a Windows host and use Windows modules, the Windows host must meet the following requirements:

  Ansible’s supported Windows versions generally match those under current and extended support from Microsoft. Supported desktop OSs include Windows 7, 8.1, and 10, and supported server OSs are Windows Server 2008, 2008 R2, 2012, 2012 R2, 2016, and 2019.
  Ansible requires PowerShell 3.0 or newer and at least .NET 4.0 to be installed on the Windows host.
  A WinRM listener should be created and activated. More details for this can be found below.

Then you need to tell ansible to connect wia winrm

...
ansible_connection=winrm
...

Checkout the details here: https://www.ansible.com/blog/connecting-to-a-windows-host


How to see previous version of a Docker image? I am looking for a netframework Docker image with .net 4.7.2 and netcoreapp2.2. I searched on the main docker website: DockerHub, .NET Framework
By Microsoft without success. I also check the Microsoft website just in case.

Is there a way to see previous version of a Docker image? (Especially for the .NET Framework)

Something to allow me to fill this kind of table:

| docker image                         | framework | core  | aspcore |
|--------------------------------------|-----------|-------|---------|
| microsoft/dotnet-framework:4.8-sdk   | 4.8       | 3.0   | ?       |
| microsoft/dotnet-framework:4.7.2-sdk | 4.7.2     |  2.1? | ?       |
|                                      | 4.7.1     | ?     | ?       |
|                                      | 4.7       | ?     | ?       |
|                                      | 4.6.2     | ?     | ?       |
|                                      | 4.6.1     | ?     | ?       |
|                                      | 4.6       | ?     | ?       |
|                                      | 4.5.2     | ?     | ?       |
|                                      | 4.5.1     | ?     | ?       |
|                                      | 4.5.0     | ?     | ?       |

docker image inspect - will show available information on the container (docs)

docker image history [your OPTIONS] imagename - Shows history of the container (docs)


Legacy build environment in Docker container, incompatible with Jenkins agent I have a working Dockerfile that creates a build environment for a legacy app, and now I'd like to use that environment from Jenkins, ideally allowing for incremental builds.

The environment is too old to run a recent version of Java, which means it cannot be used as an agent directly.

Is there a way to run a series of build steps inside a container without the container itself running an agent?

According the the Jenkins docs, you could use your custom docker image as the execution environment, assuming the server Jenkins is running on has Docker installed/running.

pipeline {
    agent {
        docker { image 'your-custom-image' }
    }
    stages {
        stage('Build') {
            steps {
                sh 'commands to run in your docker image here'
            }
        }
    }
}


Dump Kubernetes log files for all services/pods/jobs etc We are remotely deploying a kubernetes cluster and when the installation fails, we would like to see the logs of all the services and migrations and pods etc. that we are trying to install.
Instead of manually specifying each specific service etc. Is there an easy way of getting all the logs of the cluster and saving them in a tar file.

You are possibly looking for a third party stack on cluster to help you, if so there is long list you can explore and play to your need
Various Kubernetes logging tools you could consider from below list:

Fluentd
ELK Stack
Stackdriver

I have used this link to setup EFK on my local cluster.


Why does `docker-login` fail in BitBucket pipelines? Aim

to login to a private docker registry using BitBucket pipelines.

Attempt

Based on this documentation the following bitbucket-pipelines.yml:

image: docker:latest

pipelines:
  default:
    - step:
        script:
          - docker login -u $D_REGISTRY_USER -p $D_REGISTRY_PASS $D_REGISTRY_URL

options:
  docker: true

was created. When BitBucket pipelines run this it results in:

+ docker login -u $DOCKER_REGISTRY_USERNAME -p $DOCKER_REGISTRY_PASSWORD
$DOCKER_REGISTRY_URL
Warning: failed to get default registry endpoint from daemon
(Error response from daemon: authorization denied by plugin pipelines: Command
not supported.).
Using system default: https://index.docker.io/v1/
Error response from daemon: authorization denied by plugin pipelines: Command
not supported.

At the moment the docker login and docker push commands are only available to Pipelines Alpha members.

You can read about the features available to Alpha at "Pipelines Alpha".

In order to participate, the above link includes instructions on how you can opt-in to the added functionality.

Personally I received an e-mail from Atlassian a day after signing up saying my projects now include Alpha features. So this feature works in my pipelines without any issues.


Docker-compose containers can't connect to each other According to the Docker Compose Docs: Networking:

By default Compose sets up a single network for your app. Each container for a service joins the default network and is both reachable by other containers on that network, and discoverable by them at a hostname identical to the container name.
For example, suppose your app is in a directory called myapp, and your docker-compose.yml looks like this:
version: "3"
services:
  web:
    build: .
    ports:
      - "8000:8000"
    depends_on:
      - db
  db:
    image: postgres
    environment:
      POSTGRES_PASSWORD: testpass
    ports:
      - "8001:5432"

When you run docker-compose up, the following happens:

A network called myapp_default is created.
A container is created using web’s configuration. It joins the network myapp_default under the name web.
A container is created using db’s configuration. It joins the network myapp_default under the name db.

I put that exact docker-compose.yml in a directory with this Dockerfile:
FROM alpine
RUN ping -c 1 db

When I run docker-compose up, I get a connection error to the db container.
$ docker-compose up
Building web
Step 1/2 : FROM alpine
latest: Pulling from library/alpine
df20fa9351a1: Pull complete
Digest: sha256:185518070891758909c9f839cf4ca393ee977ac378609f700f60a771a2dfe321
Status: Downloaded newer image for alpine:latest
 ---> a24bb4013296
Step 2/2 : RUN ping -c 1 db
 ---> Running in cf64bfcb72eb
ping: bad address 'db'
ERROR: Service 'web' failed to build : The command '/bin/sh -c ping -c 1 db' returned a non-zero code: 1

If I put a node app in the Dockerfile, I can't make database connections to the db URI at postgres://postgres:postgres@db/postgres either.
I've tried using links and depends_on as well as the healthcheck for postgres advised in other SE answers.  (EDIT: Added depends_on to compose file).  Nothing has worked.  Why can't I connect to the db container?

The Dockerfile specifies how a container image is built.  The Dockerfile’s RUN command is trying to ping the db container, which is not yet running during the build phase.
Once the container images have been created, compose will start the containers and they will be able to address each other via their service names (db and web).
To accomplish what you want, an adjustment to your Dockerfile will need to be made.  Replace RUN with CMD.  This will allow the container to execute your ping command when it launches.
FROM alpine
CMD ping -c 1 db

Now your docker-compose up will build the web image, launch the db container and then launch your web container which will successfully execute a ping:
> docker-compose up
Creating network "simple_default" with the default driver
Building web
Step 1/2 : FROM alpine
 ---> a24bb4013296
Step 2/2 : CMD ping -c 1 db
 ---> Running in 0c210f0936fd
Removing intermediate container 0c210f0936fd
 ---> 7c0ed50ad4e8
Successfully built 7c0ed50ad4e8
Successfully tagged simple_web:latest
WARNING: Image for service web was built because it did not already exist. To rebuild this image you must use `docker-compose build` or `docker-compose up --build`.
Creating simple_db_1 ... done
Creating simple_web_1 ... done
Attaching to simple_db_1, simple_web_1

... output omitted

web_1  | PING db (172.19.0.2): 56 data bytes
web_1  | 64 bytes from 172.19.0.2: seq=0 ttl=64 time=0.218 ms
web_1  |
web_1  | --- db ping statistics ---
web_1  | 1 packets transmitted, 1 packets received, 0% packet loss
web_1  | round-trip min/avg/max = 0.218/0.218/0.218 ms
db_1   |
db_1   | Data page checksums are disabled.
db_1   |
db_1   | fixing permissions on existing directory /var/lib/postgresql/data ... ok
db_1   | creating subdirectories ... ok
db_1   | selecting dynamic shared memory implementation ... posix
db_1   | selecting default max_connections ... 100
db_1   | selecting default shared_buffers ... 128MB
db_1   | selecting default time zone ... Etc/UTC
db_1   | creating configuration files ... ok
db_1   | running bootstrap script ... ok
db_1   | performing post-bootstrap initialization ... ok
simple_web_1 exited with code 0


Is it possible to use multiple if statements in a salt state? I have a Salt state which checks multiple files exist on our webservers, we have 4 webservers and 2 environments so a combined total of 8. This state is applied to all by using a grain 'webserver'.

As the webservers are different (same underlying back end, different directory/file names) I have implemented an if statement which makes use of the inbuilt grains functionality 'host'.

{% if grains['host'] == 'dev-server1', 'test-server1' %}

dart-index.html:
  file.exists:
    - name: /opt/tomcat/webapps/example1/index.html

{% endif %}

When applying my state to all 'webserver' minions, this worked as expected. It checked for the above file on the target minions (dev-server1/test-server1) and ignored the others. However I have since added another if statement below this for dev-server2/test-server2:

{% if grains['host'] == 'dev-server2', 'test-server2' %}

tees-index.html:
  file.exists:
    - name: /opt/tomcat/webapps/example2/index.html

{% endif %}

Now whenever I run the state against the minions it fails as it attempts to check whether both of the above files exist across all minions (which of course won't exist).

So, is it possible to use multiple if statements in a state? Alternatively is there another method in which I could achieve the same goal? I would like to identify multiple files with a specific directory/file name in each type of webserver.

You can have more than if statement per state. The issue is that your conditional doesn't seem actually be checking the hostname, it's just passing everything. Try this:

{% if grains['host'] in ['dev-server2', 'test-server2'] %}


AWS t4g instance type not eligible According to the blog post since september/2020 and up to december/2020 instance type t4g will be available. I'm trying to create a machine, on one of the right zones (Oregon) according the FAQ, and using the right AMI, also according to the FAQ
So why I cannot choose those machines?
As you can see on the screenshot, t3a are eligible, but all t4g not:

Solution was pretty simple: you have to choose ARM architecture on the first step, AMI selection. Accustomed to choose x86 architecture, I didn't pay attention to that radio button. Also I didn't bother to hover over the forbidden icon on the first column in the image posted on question, which clearly stated what was the problem:

So solution is to select ARM on the previous step:


Amazon EKS from local kubectl does not cache MFA token/session I have set up an EKS cluster in AWS. My user-account there is secured with MFA. And I have an assumerole, that is allowed to manage my infrastructure.

So I installed and configured my system as stated in the AWS userguide: https://docs.aws.amazon.com/eks/latest/userguide/getting-started.html

I have kubectl in version 1.11.x installed, so I can use that binary to communicate with aws (and I do not need the patched amazon version). I then installed aws-cli and aws-iam-authenticator.

user@notebook [04:44:51 PM] [~]
-> % kubectl version --short --client
Client Version: v1.11.0
user@notebook [04:45:13 PM] [~]
-> % aws-iam-authenticator help
A tool to authenticate to Kubernetes using AWS IAM credentials
...
user@notebook [04:45:25 PM] [~]
-> % aws --version
aws-cli/1.15.51 Python/3.6.6 Linux/4.17.8-1-ARCH botocore/1.10.50
user@notebook [04:45:30 PM] [~]
-> %

I then created the configuration files from the how-to ~/.aws/credentials, ~/.aws/config, ~/.kube_aws/config and created an alias for myself to use kubectl with the new config (since I have more clusters to manage): alias kubectlaws='KUBECONFIG=/home/user/.kube_aws/config kubectl $@'.

So here comes my problem: When I connect to the cluster using the aws-cli binary, I have to enter my MFA token and then the session is cached (I read for about 15 minutes).

user@notebook [04:58:52 PM] [~]
-> % aws eks describe-cluster --profile my_profile --name ClusterName  --query cluster.status
Enter MFA code for arn:aws:iam::<MY_MFA_ARN>:
"ACTIVE"
user@notebook [04:59:09 PM] [~]
-> % aws eks describe-cluster --profile my_profile --name ClusterName  --query cluster.endpoint
"https://..."
user@notebook [04:59:30 PM] [~]
-> %

But when I do this with my kubectl alias, I have to enter the token every time I run a command! Furthermore, I cannot re-use a code, so I have to wait about 30 seconds, until the next one is provided.

user@notebook [05:03:56 PM] [~]
-> % kubectlaws get nodes
Assume Role MFA token code: 123456
NAME                             STATUS    ROLES     AGE       VERSION
...
user@notebook [05:04:05 PM] [~]
-> % kubectlaws get nodes
Assume Role MFA token code: 123456
could not get token: AccessDenied: MultiFactorAuthentication failed with invalid MFA one time pass code.
        status code: 403, request id: <SOME_ID>
No resources found.
Unable to connect to the server: getting credentials: exec: exit status 1
user@notebook [05:04:14 PM] [~]
-> %

This is really annoying, as you might imagine. Has anyone any idea what I did wrong there? Is this a bug? Or is this some kind of mis-configuration?

Thanks in advance!

Indeed, this is a pain. I've been fighting with it for the whole day yesterday, but managed to get it working, let's see if our cases are similar enough.

My guess is that the usual aws configure configuration with a MFA ARN + the AWS SDK used to develop kubectl are not entirely compatible: I have two profiles set for this, one that has its mfa_serial_arn and role_arn that works as follows:

First aws command I type, I get prompted for MFA token
From there onwards, while session is active, I can work without typing in more tokens.
kubectl asks for a token EVERY SINGLE TIME.

This is bad.

Now, I set a different profile, without any of the above, and I do the following:

```

keys=($(aws sts assume-role --role-arn <ARN_OF_DELEGATED_ROLE> \
   --role-session-name  <RANDOM_UNIQUE_NAME> \
   --serial-number <ARN_OF_MFA_DEVICE>\
   --token-code  \
    --query 'Credentials.[AccessKeyId,SecretAccessKey,SessionToken]' --output text))

# profile used here will be configured in ~/.aws/credentials
export AWS_PROFILE=<PROFILE YOU WANT>

aws configure set aws_access_key_id ${temp_keys[0]}
aws configure set aws_secret_access_key ${temp_keys[1]}
aws configure set aws_session_token ${temp_keys[2]}

```

This creates a temporary valid aws profile, that can be used BOTH with the aws cli, and kubectl.


How do you limit access to the production environment in the company you work for? In the company I work for, devops engineers (currently only 2 members, who are me and another coworker) are the only people who have access to the production database.
So when any other developers need to execute a MySQL query on the production database. They would send the query to the 2 engineers to let them execute it.
Here are the situations when we need to execute commands to the production database:

The database contains corrupted data, which produce bugs. They execute commands to fix the bugs.

A bug is reported. And they want to see the current values inside the database.

One of our customers wants to modify his/her data. But our web application doesn't have the ability to do the modification. So we have to send MySQL commands directly to the database to complete the customer's requirement.

The QA team created test accounts on the production environment. And they want to change the account's status so that they can do other tests.

This creates a lot of interruptions for me and the other coworker. When we develop programs during the daytime, we often have to switch context just to execute some queries.
I don't think this is a good architecture for the company. How do you control the permissions to the production environment in your company?
Our production database consists of sensitive customer information. If the data were leaked out, our company might be fined for millions of dollars.

If your question is how to manage database changes consider something like Flyway. This lets you control your changes via tracked config files in your repository and apply them via an automated & controlled process - use your normal code review & promotion steps.

If the question is "how do I get devs to stop bugging me to run arbitrary SQL commands" then you may want to consider scripting out a way to automate it or giving them a 3rd party UI to use with an account that's locked down to prevent changes & restricts them from seeing any sensitive tables. YMMV depending on the layout of your DB.


How to track (non ECS) container costs on EC2? I've been evaluating Netflix Ice - as a billing tool - but it appears to only work at the Machine instance level - not at the container level.

I'm looking for a tool to help generate billing reports for using docker containers across a cluster of docker servers on EC2. (Non ECS)

My question is: How to track (non ECS) container costs on EC2?

EDIT:
I have different groups each running their container on the same host. So I want a way to split instance costs by container usage.

Since you are not asking about ECS containers, this means that you must have a bunch of EC2 servers that you manage in some way (ECS/Kubernetes/Nomad/etc...). These servers are most probably included in an Auto Scaling Group.

Once you run containers on these servers, AWS does not charge you for the container. AWS only charges you per second of EC2 instance time while the instance was in a running mode.

So in fact you could say that containers are "free", you don't pay for containers, you pay for the infrastructure underneath (EC2 servers).

Netflix ICE is a great tool to track EC2 instance costs, as well as the AWS Billing dashboard.


Continuous Integration and Isolated Feature Testing I've been given the task at work to move everyone from using TFS Source Control to Git as the Development Team has been lead to believe that Git may well be the silver bullet that will fix alot of the teams issues.
I carried out the migration a month back and fortunately Git has fixed one thing for the team - code reviews are now alot more structured as pull requests in Git give the developer and reviewers a auditible platform to merge code into the master branch. There is however an issue of merge conflicts still, some of which taking up to 2 or 3 hours to resolve. The team are questioning if the tool (Git) is to blame whereas under analysis, I believe there are many other fundamental processes  and pratices at fault within the team. To name but a few that don't sit well with me:

All products (Visual Studio Projects) maintained by the company sit under one Visual Studio Solution. This is around 40+ VS Projects ranging from Class Librarys, Services and Database tiers. If an update needs to be made to Production for one product, the whole Solution (along with the other products) needs to be released.
There a 4 Scrum Teams working on the same VS Solution looking after their own Products. The Sprints all start on different days in a two-week cycle and therefore at any point in time, each team's Sprint is at a different state. Due to the issue raised in the first bullet point however, complexity ensues when it comes to the monthly release to Production as code for each product can be at separate stages.

Anyway, that paints part of the scene to help pose my question. We're now using Git using the following structure:

Feature branch - Used for each User Story being worked upon in a Sprint
Develop branch - All Feature branches are merged into here once the Pull Request has been approved and the change tested in isolation on a test server
Master branch - Successfully tested features are cherry-picked into the Master branch

My question is, how does DevOps / CI handle testing newly coded features in isolation? I've never worked in a place where we've tested new changes in total isolation therefore I can't get my head around why it's so important to my new Dev Team. In my opinion, the sooner we can test new changes with the rest of the codebase, the better.

First of all, git is certainly not to blame for merges taking a long time. Clean merging is the hallmark of git, unless it is used in some broken way. So I encourage you to look at what exactly seems to be the underlying problem. If your devs regularly make sweeping changes to the same files, then it's likely more a problem of cohesion/coupling than of the underlying merge tool.

Second; I like to do isolated feature-testing, if by that you mean that your test suite is ran against the feature branch (with only the branch changes) in a clean environment. In fact, I like to test all branches in this way. How could the develop or master branch become green if one of the features going into it are not? It goes without saying that develop / master need to be tested as well.

You could even go so far as to do a pre-merge test by doing a merge of feature into temp_develop_with_feature (or whatever you may call it), that is, simulate the merging of a branch into develop (under a different name) and test that.

Obviously, all of this assumes that sufficient testing ressources (VMs etc.) are available and fast enough, and that everything is 100% automated, both of which should be rather easy to achieve with something like Jenkins, Gitlab or whatever tool you use (if not, then it's maybe time to look at another one...).


Geographically, where are the travis-ci jobs run? Where are the travis-ci servers located?  Does travis-ci leverage AWS EC2?

Where are the travis-ci servers located? Does travis-ci leverage AWS EC2?

According to this documentation, the servers are located in the USA and AWS EC2 is used as well.

  Your code, depending on which platform or language runtime you're
  using, is run on virtualized servers running in:

  Amazon EC2 datacenters in Ashburn, VA, USA,
  MacStadium datacenters in Atlanta, GA, USA,
  Google Compute Engine datacenters in Berkeley County, SC and Council Bluffs, Iowa, USA.


How to tell kubectl to use different api groups I am trying to list down resources which is in different api group. I am running the below command

[jk@ome-esrs-adapter-master csadmin]# kubectl get issuers -n default
Error from server (NotFound): Unable to list "certmanager.k8s.io/v1alpha1, Resource=issuers": the server could not find the requested resource (get issuers.certmanager.k8s.io)

As you can see it is not listing any issuer which is in default space because it is searching in this api group "certmanager.k8s.io/v1alpha1" and acutal resource is in "cert-manager.io/v1alpha2" this API group so, I have specify api group in rest call and it is listing the issuer

[jk@ome-esrs-adapter-master csadmin]# curl http://localhost:8080/apis/cert-manager.io/v1alpha2/issuers | jq .
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   975  100   975    0     0  47140      0 --:--:-- --:--:-- --:--:-- 48750
{
  "apiVersion": "cert-manager.io/v1alpha2",
  "items": [
    {
      "apiVersion": "cert-manager.io/v1alpha2",
      "kind": "Issuer",
      "metadata": {
        "annotations": {
          "kubectl.kubernetes.io/last-applied-configuration": "{\"apiVersion\":\"cert-manager.io/v1alpha2\",\"kind\":\"Issuer\",\"metadata\":{\"annotations\":{},\"name\":\"ca-issuer\",\"namespace\":\"default\"},\"spec\":{\"ca\":{\"secretName\":\"nginxsecret\"}}}\n"
        },
        "creationTimestamp": "2020-04-02T13:05:56Z",
        "generation": 1,
        "name": "ca-issuer",
        "namespace": "default",
        "resourceVersion": "67806",
        "selfLink": "/apis/cert-manager.io/v1alpha2/namespaces/default/issuers/ca-issuer",
        "uid": "2970a3e5-6d0c-4f2b-964d-4598be9beaa3"
      },
      "spec": {
        "ca": {
          "secretName": "nginxsecret"
        }
      },
      "status": {
        "conditions": [
          {
            "lastTransitionTime": "2020-04-02T13:05:53Z",
            "message": "Signing CA verified",
            "reason": "KeyPairVerified",
            "status": "True",
            "type": "Ready"
          }
        ]
      }
    }
  ],
  "kind": "IssuerList",
  "metadata": {
    "continue": "",
    "resourceVersion": "284959",
    "selfLink": "/apis/cert-manager.io/v1alpha2/issuers"
  }
}

can anyone please help how to tell kubectl to look into diffrent API groups? Because I am sure that resource is there as rest api is working and when I reapply yaml file it says Issuer unchanged.

It works in dotted notation (Object.apiVersion.APIGroup), for e.g. deployments this way: kubectl get deployments.v1.apps In your case just try:

kubectl get issuer.v1alpha2.cert-manager.io -n=default


How do I write a buildspec.yml file? I have Java Spring web service project on EC2

I tried writing a buildspec.yml for it. However the AWS CodeBuild service is not giving successful results. In the install phase itself, it tells apt-get and yum command not found. The problem is that both yum and apt-get are not being detected. However the exemplary file in the docs is using apt-get to in the install phase. How can I make it work?

I am not able to figure it out... why does it fail for both yum and apt-get?

Few ideas to help you

When you define a "CodeBuild" project, you can specify which base image it will use to execute your commands. The one you are using is not Ubuntu yet you are trying apt-get. Try a different base image.
Provide us with a better log stream.

Exporting Logs from CloudWatch

I found that the easiest way is to export logs through command-line:

click on 'View Logs' link, to find out your Log Group and Log Stream
install "awscli" and "jq"

Run substituting your log-group-name and -log-stream-name:

aws logs get-log-events --log-group-name /aws/codebuild/Dumper  \
    --log-stream-name 0bd74b8c-74b9-4f9b-b275-167b45901aa3 | \
    jq '.events[].message' -r | grep -v '^$'

This should give you the output, which you can include in the question.


WIndows or Ubuntu server when developing on windows? I am developing a MEAN app a windows laptop. Should I use Ubuntu or windows server on my cloud provider? Are there downsides of choosing one over the other?

I dont see downsides of using your Windows laptop. Its a matter of preference. If you know how to deal with Windows Environment variables, paths, EOL characters, you should be good.
Many developers use Mac or Linux because they deal better with *unix like systems or because similar to the server they will run their apps.
At the end of the day after you build your MEAN solution the output should be the same regardless the OS.
Windows Server or Ubuntu server are just a OS to 'professionally' run your app. Absolutely no need to spend extra money to develop a MEAN app on those OS, I recommend to stick your your computer for development.


How to scale the control plane of a service mesh? Scalability in the data plane is easy to achieve by simply having more replicas. The configuration the proxies receive from the control plane makes that possible . However, if the number of services (and their sidecar proxies) get too big, the control plane can become a bottleneck by not being able to send metrics or other configuration data fast enough to the sidecars.

Is there a possibility to horizontally scale the control plane of a service mesh so that the total amount of possible services can be increased?
I didn't find anything regarding that.

It's true that when you scale your services is possible to have a bottleneck on the monitoring, but for avoiding that you have to check if your monitoring system is pull or push, in case it is push maybe is not the best option but pull is not the right answer for all situations.

One of the most used monitoring systems today is prometheus, you can se how it works here:

https://prometheus.io/docs/introduction/overview/

But is not the only one and you need to see how it should fit in your stack.

Check the possible integrations with istio

https://istio.io/docs/tasks/telemetry/metrics/querying-metrics/

In case you have already some of that probably need to think on scaling this part instead of the sidecars or istio itself.


Sonar:6.7 docker container restart loop I am running an init.sh script in order to create mapped directories for Sonar and Postgres.

Both of the containers build fine and are able to be started, but for some reason, our Sonar container is in a constant restart loop after about 8-16 seconds.

In the logs we noticed that there was a missing plugin error for "findbugs", although the compability matrix states that findbugs works with Sonar 6.7.

        2017.11.23 17:16:21 WARN  web[o.a.c.l.WebappClassLoaderBase] The web application [ROOT] appears to have started a thread named [Timer-0] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
     java.lang.Object.wait(Native Method)
     java.util.TimerThread.mainLoop(Timer.java:552)
     java.util.TimerThread.run(Timer.java:505)

This seems to be a popular error when I do a sudo docker logs sonar_container

Restarting:

VirtualBox:~/Sonar/docker-sonar-scpvm014$ sudo docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                                            NAMES
fa4b1c86433d        sonarqube:lts       "./bin/run.sh"           16 seconds ago      Up 5 seconds        0.0.0.0:9092->9092/tcp, 0.0.0.0:8060->9000/tcp   sonar_container
87db675ef2dd        postgres:9.5        "docker-entrypoint..."   18 hours ago        Up 11 minutes       0.0.0.0:5432->5432/tcp                           pgsql_container
:~/Sonar/docker-sonar-scpvm014$ sudo docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                                            NAMES
fa4b1c86433d        sonarqube:lts       "./bin/run.sh"           19 seconds ago      Up 9 seconds        0.0.0.0:9092->9092/tcp, 0.0.0.0:8060->9000/tcp   sonar_container
87db675ef2dd        postgres:9.5        "docker-entrypoint..."   18 hours ago        Up 11 minutes       0.0.0.0:5432->5432/tcp                           pgsql_container
~/Sonar/docker-sonar-scpvm014$ sudo docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                                            NAMES
fa4b1c86433d        sonarqube:lts       "./bin/run.sh"           23 seconds ago      Up 12 seconds       0.0.0.0:9092->9092/tcp, 0.0.0.0:8060->9000/tcp   sonar_container
87db675ef2dd        postgres:9.5        "docker-entrypoint..."   18 hours ago        Up 11 minutes       0.0.0.0:5432->5432/tcp                           pgsql_container
~/Sonar/docker-sonar-scpvm014$ sudo docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS                        PORTS                    NAMES
fa4b1c86433d        sonarqube:lts       "./bin/run.sh"           25 seconds ago      Restarting (0) 1 second ago

Does anyone have any experience with fixing this issue?

I ran into the same issue and found this docker-compose file that is able to start postgres and sonarqube.

https://github.com/SonarSource/docker-sonarqube/blob/master/recipes.md

version: "2"

services:
  sonarqube:
    image: sonarqube
    ports:
      - "9000:9000"
    networks:
      - sonarnet
    environment:
      - SONARQUBE_JDBC_URL=jdbc:postgresql://db:5432/sonar
    volumes:
      - sonarqube_conf:/opt/sonarqube/conf
      - sonarqube_data:/opt/sonarqube/data
      - sonarqube_extensions:/opt/sonarqube/extensions
      - sonarqube_bundled-plugins:/opt/sonarqube/lib/bundled-plugins

  db:
    image: postgres
    networks:
      - sonarnet
    environment:
      - POSTGRES_USER=sonar
      - POSTGRES_PASSWORD=sonar
    volumes:
      - postgresql:/var/lib/postgresql
      # This needs explicit mapping due to https://github.com/docker-library/postgres/blob/4e48e3228a30763913ece952c611e5e9b95c8759/Dockerfile.template#L52
      - postgresql_data:/var/lib/postgresql/data

networks:
  sonarnet:
    driver: bridge

volumes:
  sonarqube_conf:
  sonarqube_data:
  sonarqube_extensions:
  sonarqube_bundled-plugins:
  postgresql:
  postgresql_data:


Measure service unavailability during upgrade I am deploying a microservices - based application using an orchestrator (Rancher specifically).

During the service upgrade (when new images are being pulled and services re-discover one another), there is a small service outage.

What is the best / recommended way to measure the downtime?

I am performing e.g. a

watch -n1 'wget --spider http://some.endpoint.whoa'

but I want to time the duration of e.g. 502 responses.

What you suggest might be the simplest way to collect the data, but you will have to do quite a bit of work to extract the availability over certain periods.

I think it's fair to say that if you want availability, you need a monitoring system. This means having an extra service in your catalogue to continuously probe the availability of your microservices over time. Storing them in a time-series database would allow you to make queries to establish availability over various periods.

There are many tools which could do this for you. A good starting place would be the CNCF monitoring landscape


Create pull request for docker dependency package if outdated Platform: GitHub

CI/CD: GitHub Actions

Additional technologies: Docker with Ubuntu base image

I'm creating a Dockerfile and I need to install a package (such as Terraform). Every time the 'current_version' within "https://checkpoint-api.hashicorp.com/v1/check/terraform" changes, then I want a pull request made to the repo (to change a JSON file with version info).

I've also looked into options such as Dependabot and Renovate. Maybe create a Cron job?

I would appreciate some advice as to how to do this.

Here's how the code would look on a high-level:

FROM ubuntu:18.04

RUN apt-get install # install packages such as curl and wget

wget -O terraform.zip https://releases.hashicorp.com/terraform/${JSON_FILE.TF_VERSION}/terraform_${JSON_FILE.TF_VERSION}_linux_amd64.zip

Assuming a docker file like this...

FROM ubuntu:18.04

ARG TF_VERSION=0.12.25

RUN apt-get install wget -y

RUN wget -O terraform.zip https://releases.hashicorp.com/terraform/${TF_VERSION}/terraform_${TF_VERSION}_linux_amd64.zip

In your GitHub action you would need to get the current version from your Dockerfile.

export CURRENT_TF_VERSION=$(grep -Eo '[[:digit:]]*\.[[:digit:]]*\.[[:digit:]]*' Dockerfile)

Then the latest version of terraform.

export LATEST_TF_VERSION=$(curl -s 'https://checkpoint-api.hashicorp.com/v1/check/terraform' | jq -r .current_version)

Now in bash or using if and outputs in GitHub actions compare the two values. If they are not the same then find and replace the current version with the latest version in your Dockerfile.

sed -i "s/${CURRENT_TF_VERSION}/${LATEST_TF_VERSION}/" Dockerfile

Then you can use the create-pull-request GitHub action to create a pull request.

The create-pull-request documentation doesn't mention what happens if you don't have any changes but I asked here. If it doesn't throw an error when you don't have changes then you could simply get the latest value and run sed.

sed "s/[[:digit:]]*\.[[:digit:]]*\.[[:digit:]]*/${LATEST_TF_VERSION}/" Dockerfile


How to enable shallow clone using JCAC? Another question that is related to this one. We try to configure the shallow clone using Jenkins Configuration As Code (JCAC). Although a configuration section is available, shallow clone seems to be omitted:

Gitter

According to the Jenkins Job DSL documentation shallow has to be enabled:

shallow(true)


How to define Service Level Objectives without historical data? Assuming I can define Service Level Indicators for a user journey are there any structured methods for setting a Service Level Objective within that SLA without any historical data to indicate what is realistic or feasible?

For Availability we can often turn to various mathematical approaches that enable us to understand the maximum theoretical availability of the system, although even this is tied back to historical data in the form of Mean Time between Failure (MTBF) and Mean Time to Recover (MTTR).

Without much data you are really guessing, the general thought I have heard before is to start at 80% SLO then work your way up as you have data to back up your goal. There really is not set way of figuring out a SLO without some data. Even taking and using a week's data would be good enough to give you a reasonable idea.


Provisioning raspbery pi sdcard I was searching for a tool suitable for provisioning sdcard for raspberry pi.

I've successfully managed to implement most of it using Ansible playbook. Need only some time for last polishing.

Process looks like this:

Format sdcard
Mount sdcard in temporary directory
Debootstrap sid
Chroot
Finish configuration

It seems to work well for now, but I'm a bit worried that it is not in the scope of Ansible. Is there any more suitable tool for this task?

If you're previsioning your raspberry pi SD cards then you would probably be best using something like packer to build the actual image (you can use ansible as the provisioner for packer if you are really keen) then once you have your pi image, burn it onto the SD card.

If you need to make subsequent configuration changes them that would be where ansible fits most neatly in managing the ongoing configuration of the pi's


Why are IaaS hosting services are usually fixed-priced and PaaS are usually dynamically-priced? Basic plans in IaaS host-providers like DigitalOcean and Linode are usually fixed-priced and basic plans in PaaS host-providers like Microsoft Azure or Amazon Web Services are usually dynamically-priced.

Why are IaaS hosting services are usually fixed-priced and PaaS are usually dynamically-priced?

Fixed price tiered billing is an older billing model that suits a monthly billing cycle where usually the level of resources provided does not change over a month. Even if it is changed, this is usually met with friction from the billing department in form of the higher pricing tier being applied to the whole month.

By comparison AWS can be credited as the first provider to offer simple building blocks on an hourly basis (now per second). This allows for much more sophisticated usage of resources in form of:

auto scaling to the best amount of resources required.
turn services off out of hours or until turned on manually.
ultimately to offering functions as a service, the most finegrainerld resource billing usage available.

So I don't see these two items as being directly comparable, but one is a progression from the other.


Puppet master certificate generation not working Puppet Master:

Docker container with cent-OS-6
configured in /etc/hosts
configured /etc/puppet/puppet.conf with server,dns_alt_names and
certname

Puppet Agent:

Docker container  with ubuntu OS
configured serverip and agent ip in /etc/hosts
configure /etc/puppet/puppet.conf with server value

NOW trying to Generate Certificate in SERVER Machine with

sudo -u puppet puppet master --no-daemonize --verbose

Displays:Notice: Starting Puppet master version 3.8.7

Nothing happens  related to Certificate Generation.

I believe there aint any mistake in hosts file and other configuration missing. here might have things gone wrong.

In the client machine
If I run this command puppet agent -t . I get the following error. As the certificate is not generated in server. I think I have to fix that first than this .

Warning: Unable to fetch my node definition, but the agent run will continue:
Warning: Server hostname 'www.puppetmaster.com' did not match server certificate; expected one of puppetcertificate, DNS:puppet www.puppetmaster.com, DNS:puppetcertificate
Info: Retrieving pluginfacts
Error: /File[/var/lib/puppet/facts.d]: Failed to generate additional resources using 'eval_generate': Server hostname 'www.puppetmaster.com' did not match server certificate; expected one of puppetcertificate, DNS:puppet www.puppetmaster.com, DNS:puppetcertificate
Error: /File[/var/lib/puppet/facts.d]: Could not evaluate: Could not retrieve file metadata for puppet://www.puppetmaster.com/pluginfacts: Server hostname 'www.puppetmaster.com' did not match server certificate; expected one of puppetcertificate, DNS:puppet www.puppetmaster.com, DNS:puppetcertificate
Info: Retrieving plugin
Error: /File[/var/lib/puppet/lib]: Failed to generate additional resources using 'eval_generate': Server hostname 'www.puppetmaster.com' did not match server certificate; expected one of puppetcertificate, DNS:puppet www.puppetmaster.com, DNS:puppetcertificate
Error: /File[/var/lib/puppet/lib]: Could not evaluate: Could not retrieve file metadata for puppet://www.puppetmaster.com/plugins: Server hostname 'www.puppetmaster.com' did not match server certificate; expected one of puppetcertificate, DNS:puppet www.puppetmaster.com, DNS:puppetcertificate
Error: Could not retrieve catalog from remote server: Server hostname 'www.puppetmaster.com' did not match server certificate; expected one of puppetcertificate, DNS:puppet www.puppetmaster.com, DNS:puppetcertificate
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run
Error: Could not send report: Server hostname 'www.puppetmaster.com' did not match server certificate; expected one of puppetcertificate, DNS:puppet www.puppetmaster.com, DNS:puppetcertificate

First, try setting dns_alt_names in /etc/puppet/puppet.conf:

[main]
    dns_alt_names = www.puppetmaster.com
[master]
    autosign = true

Then see if your puppet-agent -t run works properly. be sure that you have properly set your server on the puppet client in /etc/puppet/puppet.conf too:

[agent]
    server = www.puppetmaster.com


Conditionally stopping SQL replication during deployment The project that i recently started working on uses a dacpac to deploy / update a database in MS SQL Server. The database is replicated, whenever the dacpac contains modifications to any table, we manually terminate the replication before continuing with the deployment. After the deployment, we run a script to start with a fresh replication. On the other hand, if only stored procedures have been modified, the replication can continue during the deployment.

The manual intervention is annoying me. Has anyone already solved this issue in an Azure Devops Release Pipeline?

The fallback is to use SqlPackage.exe /action;deployreport, but that means I have to write a file to disk to get the diff and subsequently inspect that outcome.

I eventually found the solution :)

In an Azure Devops Release pipelines I've created a Command Line task that executes SqlPackage.exe /Action:DeployReport /p:DoNotAlterReplicatedObjects=false ... which creates an XML file that contains info on which tables etc would be changed by a deployment. It's important to note that DoNotAlterReplicatedObjects should be set to false, otherwise SqlPackage will fail.

In a second task in the Azure Release pipeline, I analyze the generated DeployReport (e.g. the XML file) using Powershell and XPath**. The Powershell task will create an boolean output variable that will indicate whether replication needs to be stopped or not. In my case, I analyze the XML and if a table is created/altered/dropped, the replication will be stopped.
The syntax for setting the output variable is

Write-Host "##vso[task.setvariable variable=haltReplication;isOutput=true]$haltReplication"`

Third step is yet another Powershell script that will stop the replication. This third script will only run if $haltReplication evaulates to true. In the Control Options, the custom condition is and(succeeded(), eq(variables['haltReplication.haltReplication'],'1'))

**The XML file generated by SqlPackage.exe contains namespaces, which makes working with XPath a bit cumbersome. The following SO answer helped me a lot https://stackoverflow.com/questions/36417189/ignore-namespace-in-xpath-as-this-can-be-dynamic?rq=1


Best practice for installing new jenkins server When creating a fresh new Jenkins server is it better to install it on a VM or Docker?

I know that for most situations when you want just one bit of software on a server the answer would be Docker. Because of all it's plugins and their updates it is a rapidly changing piece of software so it feels like this is the exception.

If you were to use a Docker container, I know you can save all the jobs and configurations on external volumes. How would you go about keeping the updated plugins so you won't have to reinstall them if it falls? Is there a better way than Docker commits/images/saves? Those feel like they could quickly become clunky.

There is no single "best" practice.  Use whatever approach you and your team feel comfortable with and fits well with the rest of your infrastructure.

If you want to do the Docker approach, you might find this person's writeup worthwhile.

To answer your questions:

  If you were to use a Docker container, I know you can save all the jobs and configurations on external volumes. How would you go about keeping the updated plugins so you won't have to reinstall them if it falls?

Depends.  You may want to write a script or configuration management recipe that installs all your plugins for you and/or configures your jobs.  You can also create a custom Docker image containing Jenkins, all of your plugins, and all of your job configuration.  You can also combine these two approaches, using the aforementioned scripts as part of an automated image build process. Whatever you do, you should definitely ensure whatever files (e.g. Dockerfiles) and build steps (e.g. scripts) are part of the process are stored in configuration management in order to make deploys repeatable.

  Is there a better way than Docker commits/images/saves? Those feel like they could quickly become clunky.

Again, it's really just a matter of what you're comfortable with, there's no such thing as "best"; instead aim for "it does what I need it to do".


Equivalent of GitLab's "before_script" for Jenkins? I'm looking for Jenkins' equivalent of the before_script keyword provided by GitLab.

So far what I found is the input keyword that you can use before entering the steps part of the stage, but this feature needs a prompt, whereas I only want to perform a cd.

Is there a Jenkins equivalent of GitLab's before_scriptkeyword?

You can use the Jenkins provided step dir() which changes the directory to one relative to the Jenkins env.WORKSPACE (/var/lib/jenkins/workspace/JOB_NAME by default).

stage('Build') {
    steps {
        dir('path/to/dir') {
            // do stuff
        }

        dir('path/to/other/dir') {
            // do other stuff
        }
    }
}


how to set a variable from an ini file in ansible? I would like to set a variable value in a playbook reading it from an ini file, possibly failing the play if the file is not found or the value is missing in the file.
I have found the ini lookup plugin to read values from ini files but it is not clear to me how to put that into a variable that I can reuse in other tasks without having to read it again.
I could think of something like:
- name: lookup variable from ini
  ini:
    name: myParam
    section: mySection
    file: /path_to/ini_file
  register: my_variable

But since ini is not a module this does not work.
Moreover, the ini file to be read from is located on the remote system, the ini plugin howhever, seems to look on the control host for it (from the warning "Unable to find '/path_to/ini_file' in expected paths).

Since the ini lookup plugin only reads files local to the controlling host, the remote file needs to be fetched locally before it can be read and set_fact has to be used to put its content into a variable, which results in the following:
- block:
  - name: "Retrieve remote ini file"
    fetch:
      src: /path_to/ini_file
      dest: /tmp/ansible
  - name: "Read and store my value"
    set_fact:
      my_variable: "{{ lookup( 'ini', 'myParam section=mySection file=/tmp/ansible/{{ inventory_hostname }}/path_to/ini_file' ) }}"

Then the parameter is available as variable and can be used further in the playbook
- debug:
    var: my_variable


Validating certificates with get_url or yum on CentOS 7 using Ansible I'm pretty new to Ansible, but it seemed a better tool than a bash script for installing programs from source on remote servers...

Starting from my working-but-fragile bash script, I made an Ansible script to git clone, configure and compile what I need. However some of these sources are large, so I wanted to include the --depth 1 parameter. This is a problem because the CentOS 7 version of git is 1.8.3 and Ansible ignores the depth command if it can't use at least git 1.9.1. Obviously for non-system use, I wouldn't mind having the latest git 2.15 anyway so...

To make a newer version of git available than is on yum's base/extras/updates repos, I found the IUS repository which has a "safe replacement" package for a few things including git. Helpfully there's an Ansible task set to install their repo. Here's the relevant task:

- name: install IUS release package
   yum:
       name: "https://{{ ius_distribution_abbrev }}{{ ansible_distribution_major_version }}.iuscommunity.org/ius-release.rpm"
       state: present
   when: ansible_os_family == 'RedHat'

Ansible however fails this task:

fatal: [MY_REMOTE_IP_ADDRESS]: FAILED! => {"changed": false, "failed": true,
"msg": "Failed to validate the SSL certificate for centos7.iuscommunity.org:443.
Make sure your managed systems have a valid CA certificate installed.
You can use validate_certs=False if you do not need to confirm the servers
identity but this is unsafe and not recommended. Paths checked for this
platform: /etc/ssl/certs, /etc/pki/ca-trust/extracted/pem,
/etc/pki/tls/certs, /usr/share/ca-certificates/cacert.org, /etc/ansible.
The exception msg was: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:579)."}

Both yum's ca-certificates and python-urllib3 packages are installed and on the latest version from the yum repos.

Per this comment on github, it seems that Ansible can't handle SNI to get the right SSL certificate without python 2.7.9+. But CentOS 7 ships with python 2.7.5. You can't upgrade it because get_url has the same issue with SSL and so you would have to download the python source with validate_certs=no.

Worse, even when I did that, used make altinstall to put python 2.7.14 in /usr/local/bin/ without touching the OS python and then set ansible_python_interpreter: /usr/local/bin/python2.7, (at least allowing get_url to work) I'm then locked out of yum for the duration of the playbook because yum can't handle the newer version of python Ansible is trying to run it with.

Is there a way to make this all work together properly without removing a security feature?
Is Ansible the wrong tool for CentOS 7 because of the python version and yum?

While it's true that the version of python reported there doesn't have SNI support, I thought that RedHat backported it. But anyways, if not, you can do that yourself:

yum -y install gcc python-devel libffi-devel openssl-devel
pip install pyopenssl ndg-httpsclient pyasn1

(This works for requests at least, I'm not sure about Ansible.)

On a broader note:

  But CentOS 7 ships with python 2.7.5. You can't upgrade it because get_url has the same issue with SSL and so you would have to download the python source with validate_certs=no.

That's only the case if the server you're downloading Python from relies on SNI. I imagine most of their mirrors do not.

But really, you shouldn't be compiling Python on every individual server. You should do it once, either on a machine that you freeze into an image that you launch further machines off of, or you package it as an rpm and the machines just download that and install it. This allows you to control the download process.

Even simpler: download the package on your desktop (via verified https). Copy it to your file storage. Download from there onto your servers. There's no need to download directly from python.org every time, and it's more reliable to have a local copy anyways.


Docker in docker volume mount doesn't work I have the below command in my Jenkinsfile:
docker run --rm -v "${WORKSPACE}":/var/cache/de --entrypoint=/usr/local/bin/pytest ${IMAGE_NAME}:${IMAGE_VERSION} -m "not require_dss" -s -v --cov-report term --cov=app_dss_bundle --junitxml=/var/cache/de/rtest.xml tests

ls -l

I'm running inside docker slave, so it means its docker inside docker. I can't see the rtest.xml in the WORKSPACE afterward (it works for me in my private linux docker env). I already tried to run the suggestion of -v /var/run/docker.sock:/var/run/docker.sock but it didn't help. any other idea?

Single quotes in shell meant to leave the enclosed string 'as is'. That is the variable inside single quotes isn't going to be evaluated. Try changing '${WORKSPACE}' for "${WORKSPACE}".


Get changes of the parameterized pipeline on the fly in Jenkins Is it possible to get changes of the parameterized pipeline on the fly after making changes there?

For example I have a pipeline code:

properties([
        parameters([
                string(name: 'TARGET_HOST', defaultValue: 'test', description: 'Host on which artifact will be deployed'),
                string(name: 'ARTIFACT_ID', defaultValue: 'test', description: 'Name of artifact which will be doployed'),
                string(name: 'ARTIFACT_VERSION', defaultValue: '1', description: 'Version of artifact which will be deployed')
        ])
])

And I decided to change defaultValue for TARGET_HOST parameter from test to 1.

After such changes, I'll need to run job in order it apply changes and only running it for the second time after making changes, I'll obtain applied changes.

Is there a way to make job pick up changes during first run after making them?

So far I found a way to do it with Input structure:

def values = input id: 'Run-test-suites',
        message: 'Workflow Configuration',
        ok: 'Run',
        parameters: [
                [
                        $class      : 'BooleanParameterDefinition',
                        defaultValue: true,
                        name        : 'checkboxVal',
                        description : 'Testing of checkbox param'
                ],
                [
                        $class      : 'StringParameterDefinition',
                        defaultValue: "test",
                        name        : 'stringVal',
                        description : 'Testing of string param'
                ],
                [
                        $class     : 'ChoiceParameterDefinition', choices: '1\n2\n3',
                        name       : 'choiceVal',
                        description: 'Testing of choice parameter'
                ]
        ]

But I want to have something more flexible.

Currently what you are asking for is not possible. This is due to the way that the Jenkinsfile is loaded into the pipeline.
I think dolphy said it best on this StackOverflow answer (I saved this link a while ago to follow it for changes, because I had the same questions):

Jenkins does not know about the new parameters until it retrieves, parses, and runs the Jenkinsfile, and the only way to do that is to...run a build.
In effect, the build history will always be "one run behind" the Jenkinsfile; when you change something in the Jenkinsfile, the next build will run with the "old" Jenkinsfile, but pick up and process the new Jenkinsfile for the build after that.

There is also a related Jenkins Ticket you can follow