Why will my container run in Docker, but will not run in Kubernetes? Background

I have a container that I have defined with an CMD and an ENTRYPOINT in Docker. When I run the following command my container runs with no issues whatsoever.

docker run --rm -ti custom-container:latest

Here is the definition for my container:

FROM mcr.microsoft.com/powershell:6.2.3-ubuntu-bionic
COPY . /app
WORKDIR /app

CMD ["/app/Start-Test.ps1"]
ENTRYPOINT ["pwsh", "-f", "/app/Start-Test.ps1"]

The script Start-Test.ps1 is just an infinite loop that writes Hello World! to host.

Problem

When I go to run my container in Kubernetes, my container refuses to run.

Kubernetes is able to pull my image and it attempts to spin up the container, but I keep getting a status of BackedOff from Kubernetes.

I've gone through and tried to figure out why this is happening because the container runs perfectly fine in Docker.

I only have issues when I try to run the container in Kubernetes.

Here is the definition for my Kubernetes deployment.

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: my-custom-container
spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-custom-container
  template:
    metadata:
      labels:
        app: my-custom-container
    spec:
      containers:
      - name: my-custom-container
        image: repository-url.io/custom-container:latest
      imagePullSecrets:
      - name: regcred

Question

What is the issue preventing my container from running when it runs with no problems in Docker?

Additionally, yes I would like to use PowerShell as my entry point. It should be possible to use any scripting langauge (bash, Ruby, Python, Perl, etc) as an entry point.

To debug this, check the container logs and the container events.

kubectl get pods
Find your pod name.
kubectl logs
If logs don't make it obvious...
kubectl get events

If you deployed into a namespace, make sure to add -n  to all commands.

This should reveal basically everything. If you're stuck, you can also browse around the k8s dashboard if you haven't yet.  But that probably wouldn't reveal much more than the CLI is with these commands.


Can/should I use Jenkins to monitor whether a website is up and send out an email if it's down? I'm trying to set up a way to constantly monitor a website that is built and deployed via Jenkins. A simple curl command would tell me if everything is OK, but how I would set up a job to do this every ~10 minutes?

Or would this be a bad idea?

It's possible and pretty simple to do this with Jenkins. Here is a sample pipeline:

pipeline {
    agent any

    triggers {
        cron('*/5 * * * *')
    }

    stages {
        stage('Health checker') {
            steps {
                sh 'curl https://mywebsite.com'
            }

            post {
                failure {
                    mail to: 'notify-list@example.com',
                        from: 'jenkins@example.com',
                        subject: "${env.JOB_NAME} - Failed",
                        body: "Job Failed"
                }
            }
        }
    }
}


How to correlate Devops deploy on demand with Agile Sprint Process I'm not clear how a Sprint which is timed-box to a predetermined length (1/2/3 weeks) fits with a DevOps principle of being able to deploy on demand or as needed.

Is deploying to production part of the definition of done rather than a post activity to the overall sprint?

How does the sprint process capture that code was deployed to production in the middle of a sprint rather than at the end where the goal is to have "potentially shippable product"?

There is no "Deploy-once-a-day" rule in devops philosophy. It's more of: Deploy as soon as possible and as often as possible. Also it calls for decoupling architecture so different parts of it may be released separately and also for decoupling deploys from release.

The Deveops Handbook by Gene Kim et al.calls for modifying Definition of Done to include running in production-like environments. That is follow the devops principles all the time during the sprint.

What you want to avoid is ending a sprint with all the developers handing over their code that is tested, is working but is not integrated and is handed to other team to be deployed, somehow.

I'd argue that definition of done is : it's not done until it's ready to run in production. Once it's ready you may or may not release it depending on your specific business needs and the sprint goal but if it's not ready to be released right now it's not done.


Continuous deployment I have a nodejs application using expressjs. This server is handing a website built on vuejs, an application using angular and another angular app. Everything is working and I deployed to AWS eb using a pipeline. The problem is when a website change is made I build it and commit the change to AWS code commit.

Is there a way to make this automatic?

Should I keep all projects in the same repo?

What I want is: website change -> commit -> build website only -> push to AWS.

The only solution I can think of is to rebuild the 3 projects every time.

I would suggest adding a job that checks changes in the latest commit and build only that service where some files have been changed.


How can I access port 80 of another container, when port 80 is already exposed? I have a docker file that runs some integration tests between two containers. It looks a bit like this:
version: '3'
services:
    test_client:
        image: client:latest
        depends_on:
          - test_server
        command: bash -c "run_test.sh -p 80"
    test_server:
        image: server:latest

test_server launches a web server on port 80, but that same port is already exposed within test_client (I don't have control over the part of the client:latest image where the port is exposed. I tried using the port: 8080:80 to forward port 80 to 8080 in the server, but that only changes the port exposed on the host, not within the docker environment.
How can I connect test_client to test_server in such a scenario?

You can use a proxy to forward traffic between the two services, I typically do something like this which includes automated SSL with certbot. In the example below I have two services listening on port 80/443 (Sonarqube and Jenkins). You can also see I can reference container hosts by name (https://jenkins:8080)
nginx:
container_name: nginx-certbot
restart: unless-stopped
image: staticfloat/nginx-certbot
networks:
  - local
ports:
  - "0.0.0.0:80:80"
  - "0.0.0.0:443:443"
environment:
  CERTBOT_EMAIL: <insert_email>
volumes:
  - ./conf.d:/etc/nginx/user.conf.d:ro
  - letsencrypt:/etc/letsencrypt

And this is what my nginx config looks like
server {
listen              443 ssl;
server_name jenkins.hauntedmansion.io;
ssl_certificate     /etc/letsencrypt/live/jenkins.hauntedmansion.io/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/jenkins.hauntedmansion.io/privkey.pem;

client_max_body_size 20M;

  location / {
    proxy_set_header   Host $host;
    proxy_set_header   X-Real-IP $remote_addr;
    proxy_set_header   X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header   X-Forwarded-Proto $scheme;
    proxy_pass         http://jenkins:8080;
    proxy_read_timeout 90;
    proxy_redirect     http://127.0.0.1:8080 https://jenkins.hauntedmansion.io;
  }
}

If you want to see the full docker compose example check it out here:
https://gist.github.com/cvega/66e0e5a2815e9b923da2c616f74dea7e


convert declarative pipeline to scripted pipeline I already used the following declarative script as a part of my cleanup script, now I should convert it to the scripted pipeline, how I can convert the post to the scripted pipeline?

The scripted equivalent to the declarative
steps {
  // pipeline steps here
post {
  failure {
    // failure handler here
  }
  success {
    // success handler here
  }
  always {
    // always handler here
  }
}

is the native Groovy try { } catch { }:
try {
  // pipeline steps here

  // success handler here
} catch(Exception e) {
  // failure handler here
} finally {
  // always handler here
}

I'm not sure off the top of my head how to add an abort handler, but I think you might be able to do it by inspecting the currentBuild.result variable in the failure handler block.


How to write a dynamic declarative pipeline that contains sequential job inside parallel job I'm trying to write a declarative pipeline code that accepts a map and create a pipeline. I can able to achieve sequential stages or parallel stages but facing problems while making a pipeline that contains sequential stages inside parallel stages.

The input data would be Map. Each list in the map should run parallel and the items inside the list corresponding to each key should run in sequentially.

example data : [1:[11,12], 2:[21,22], 3:[31,32]]

Am using Jenkins 2.164.3

The output should be

This was covered in an official blog post when the feature to nest stages inside a parallel block was added.

Here is the example copy-pasted from the blog post.  In this example, the Windows and Linux blocks are run in parallel, but the stages within each block are run sequentially:

pipeline {
    agent none

    stages {
        stage("build and deploy on Windows and Linux") {
            parallel {
                stage("windows") {
                    agent {
                        label "windows"
                    }
                    stages {
                        stage("build") {
                            steps {
                                bat "run-build.bat"
                            }
                        }
                        stage("deploy") {
                            when {
                                branch "master"
                            }
                            steps {
                                bat "run-deploy.bat"
                            }
                        }
                    }
                }

                stage("linux") {
                    agent {
                        label "linux"
                    }
                    stages {
                        stage("build") {
                            steps {
                                sh "./run-build.sh"
                            }
                        }
                        stage("deploy") {
                             when {
                                 branch "master"
                             }
                             steps {
                                sh "./run-deploy.sh"
                            }
                        }
                    }
                }
            }
        }
    }
}

EDIT: I missed that you're trying to do this based on arbitrary input.  If you want to do that, you must use scripted Pipelines (or a script nested in a Declarative Pipeline).  Plain Declarative Pipelines don't support that kind of runtime-defined Pipelines.


Mail logs with Jenkins I am deploying my software on different OS ( Red hat & suse) and I need to know if my latest version of the software will work. In order to check if the installation works it's enough to check some log files generated by the software.

Now I am trying to integrate some CI/CD tools and I am trying Jenkins now. I have created a pipeline which is triggered by every commit in bitbucket and I need a way to get the log files and the build status from the servers and mail them to the rest of the team.

If you're using Jenkins pipeline, you can use the email notification post action, as a way to send the message:

post {
    failure {
        mail to: 'team@example.com',
             subject: "Failed Pipeline: ${currentBuild.fullDisplayName}",
             body: "Something is wrong with ${env.BUILD_URL}"
    }
}

(you would have to configure the server and credentials for your case)

The message itself can be constructed from the job data. The easiest way is perhaps to tee the output of the job to a file and use it as an artifact to attach to the message that gets sent.


VM Hosts: Single Large Host or Few Smaller Hosts? First, I apologize if this is the wrong forum to ask this question. I was wondering if this is a ServerStack or DevOps question. In either case here it goes:

I am building a small software shop and I am trying to figure out the infrastructure and machines required for it.

I bought MSDN Pro Subscription ($600/year) as well as JetBrains All Product Subscription ($600) for the development tools, desktop and server OS licenses, etc etc...

At the moment I have a budget of roughly $1500 (Walmart Credit Card) and I am entertaining the idea of buying 3x refurbished machines (each with Core i7-3770 cpu and 16GB of RAM) at $452 a pop or buying a large server at $1120 and maxing out it's RAM capacity for another $300-$400.

The background info on this is that I want to develop a product and have a full blown development environment (Source Control, Build/Deploy/Test and Release Management, Dev, Beta, and Prod environments). Cloud services (the cheapest) will run me about $200-$300 a month, and over period of 6 months it will become a recurring liability rather than an asset (which I pay out of pocket btw).

If I buy the hardware - it will have an initial cost of about $1500 but it will last me for the duration of the development process which is at least a year or so. Beyond that, if I outgrow it's capacity, I can move to docker containers and maybe a cloud solution for the Dev, Beta, and Prod environments.

So the question is - which is better - multiple dedicated cpu's or a single large server?

1x Large VM Host (64gb RAM) to have the following VMS (from here) (4GB used by Hyper-V 2016 Server):

1x - 8GB RAM - TFS - (Git, Active Directory, SQL2016, Sharepoint, IIS)
1x -  4GB RAM - TFS - Build Controller
1x -  4GB RAM - TFS - Release Agent, Test Agent
1x -  4GB RAM - DEV - Linux Mint (Postgre SQL)
1x -  4GB RAM - DEV - Linux Mint (Web Server/Apache)
1x -  4GB RAM - BETA - Linux Mint (Postgre SQL)
1x -  4GB RAM - BETA - Linux Mint (Web Server/Apache)
1x -  4GB RAM - PROD - Linux Mint (Postgre SQL)
1x -  4GB RAM - PROD - Linux Mint (Web Server/Apache)

Or having multiple smaller hosts:(from here)

1x Small Host - 16GB RAM (4GB used by Hyper-V 2016 Server):

1x - 6GB RAM - TFS - (Git, Active Directory, SQL2016, Sharepoint, IIS)
1x - 3GB RAM - TFS - Build Controller
1x - 3GB RAM - TFS - Release Agent, Test Agent

1x Small Host - 16GB RAM (4GB used by Hyper-V 2016 Server):

1x -  4GB RAM - DEV - Linux Mint (Postgre SQL)
1x -  4GB RAM - BETA - Linux Mint (Postgre SQL)
1x -  4GB RAM - PROD - Linux Mint (Postgre SQL)

1x Small Host - 16GB RAM (4GB used by Hyper-V 2016 Server):

1x -  4GB RAM - DEV - Linux Mint (Web Server/Apache)
1x -  4GB RAM - BETA - Linux Mint (Web Server/Apache)
1x -  4GB RAM - PROD - Linux Mint (Web Server/Apache)

A good rule of thumb is to try and avoid using one bit of logical kit (be it a single VM or a single physical host) for more than one thing.

If a physical server is going to be a VM host then that's all it should be. If a VM or a physical host is going to be a source code repository then that's all it should be etc. You can squeeze a bit extra out of your infrastructure if you consider everything to be a VM host and you then partition up those VMs to be single purpose hosts themselves.

The rationale behind this is that

A single host going down shouldn't impact your entire stack. Trust me this will hurt you badly as soon as you need downtime for an upgrade.
The cost to scale any individual component increases drastically if you want to try and scale on multiple similar servers (which will save you management dollars by reducing complexity) or it will cost you to scale out adhoc hardware as needed because you then need to deal with many different-sized hosts that are a pain to manage and require anyone managing them to know the platform intimately.

recommendation: split your infrastructure into as small a unit as is logical. If you can only afford to buy a single large bit of kit then make sure you split it into logically separate VMs or containers so that if you need to pop a bit of your infrastructure onto another server, or you need to scale stuff at different rates you can easily expand out but investing in a smaller upgrade.

Obviously the realities of software licencing, especially for windows OS's might make this cost prohibitive but, while I haven't caught up on windows licensing models in a while, I'm sure there are development centric licensing models that can suit your specific needs.


How to pin public root key when downloading an image with docker pull (Docker Content Trust)? How can I execute docker pull (with Docker Content Trust enabled) such that it fails if the image doesn't have a valid signature using the private key corresponding to (or subordinate to) the public key that I provide?
I just discovered that, in fact, DCT silently and dumbly downloads and trusts (TOFU) the root key when downloading an image for the first time.
My application for docker is a CI process that will use the debian:stable-latest docker image to build my application's releases in ephemeral cloud instances. Because the server used to build my application is ephemeral, DCT literally provides me zero security and it will be vulnerable to a MITM of the docker image payload on every execution of my build process.
Therefore, it's critical that I pin the public root key used by Debian to sign their docker images. So the questions are:

How do I download the public root key used by Debian to sign their docker images and
How do I tell docker pull to use the above key and fail if the image doesn't contain a valid signature from that key or a key that's downchain from the supplied root key?


Elastic Beanstalk give error Root Object doesn't exist in Jenkins I create a Jenkins job which uploads my war file to elastic beanstalk. But when I start to build it gives error java.lang.IllegalArgumentException: Root Object doesn't exist

Below is my configuration:

This error occurs because you used a variable ($) and wildcard name (*.war).

Elastic Beanstalk plugin does not support Variable or wildcard name in the Value.

I suggest a use full path of Jenkins job instead of ${WORKSPACE} and full war name like ROOT.war instead of *.war


How to move EC2 instance to another ECS cluster Assume I have two clusters devcluster and qacluster on ECS.
How to remove an EC2 instance (ecs-optimized amazon-linux) from qacluster  and add to devcluster?

You need to follow these steps:

Set ECS_CLUSTER=devcluster in /etc/ecs/ecs.config
Stop all tasks/containers
Remove checkpoint file - /var/lib/ecs/data/ecs_agent_data.json
Start ECS agent again as explained here - https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-agent-install.html

To make sure it uses correct cluster, check logs - docker logs ecs-agent. It should have a message like:

[INFO] Registration completed successfully. I am running as 'arn:aws:ecs:eu-west-1:ACCOUNT_ID:container-instance/CLUSTER_ID' in cluster 'devcluster'


Alarms management with Elasticsearch and Kibana I'm working on my first implementation of the Elasticsearch-Logstash-Kibana stack. I have everything working, but now I need to:

Set alarms on certain thresholds/events
Send notifications on alarms
(desirable) Manage alarms from within Kibana (i.e. alert/acknowledge)

I've read some partial solutions using another tools beside ELK, as Nagios, but I'm wondering what is the industry's "best" approach to this need.

Note: I've already taken a look at X-Pack and seems to be one of the acceptable solutions, but the licencing details and costs are still vague - not publicly available. I've contacted Elastic but need to answer a few questions with data I still don't have.

I have no experience with it myself, but Elastalert (http://elastalert.readthedocs.io/en/latest/elastalert.html) sounds like what you need (for the alarms/threshold part)

But besides that, it sounds like your interested in monitoring. In my opinion ELK is not a classic monitoring tool (more: data collection, data processing and data visualization). You should take a look at Icinga2 (https://www.icinga.com/products/icinga-2/) which can can be configured highly dynamical and is compatible with nagios plugins


AWS hosting of SPA Web App I have just recently started to consider using AWS for my SPA Web Apps. And I have read a lot about different types of how web apps could be hosted on AWS. I have looked into Lambda, S3 Static hosting. EC2. But can't seem to figure out how to exploit AWS and it services the best possible way.
My Web App consists of a front-end written in Vue js. Which uses a Backend API written in Laravel. The front-end and backend are isolation so they don't have to be placed on the same machine. So I guess that could bring some good opportunities for using the different AWS products? The Database is only connected to the Backend Api so the Frontend is static. I'm also openminded to changing to another type of database if that would fit better into the AWS environment. But I don't know if I should separate the Frontend and API into different services.
Thanks in advance!

The AWS answer would be to use appropriate services for each component.
E.g. Put the database on an RDS - they have multiple choices. Deploy the front-end using Fargate and Lambda.
There are 2 issues to consider:
1. The learning curve.
Getting up to speed on AWS technology - and keeping up with them as they make minor tweaks that break lagacy deployments - can get very expensive, time-wise.
OTOH, once you're an expert at these technologies, you can leverage them with amazing results.
2. The costs:
While you develop you will be using the free-tier mostly, and the costs will be minimal.
Once you move into production, you have have to start using "real" machinery, and even if you don't, most services have an expiration for their free tier: 12 months and/or a certain amount of usage.
At that point the costs start ballooning. from small-change to 3 or even 4-figure numbers for a small installation.

The other approach is to do it the way you'd do it without AWS.
You didn't give me much details, but if you're deploying something simple, you may want to spin up an EC2 instance, install a LAMP server and use it for both front & back end.
You can then use the Lifecycle Manager to back up the entire machine daily.
However, if you have to scale this up in a big way, you'll have to start from scratch.

So, what do you choose?
The answer depends on your resources: time, money and manpower, as well as how you see this growing from a one-off project to a long term heavily used installation that needs redundancy, load balancing and the ability to scale dynamically.
You could also look at something like AWS Beanstalk, which is easier to implement and redeploy, but not as dynamic, as their serverless approach.


Encrypting docker images (end to end); transporting over offline channels We are developing and building docker images in our intranet, and need to deploy them to several hosts belonging to us (developers, internal testing, external testing and so on). Some of these are in our intranet, some are accessible for 3rd parties on the internet.

The final deployment is inside the customer's intranet on several nodes (production, different test stages). These are behind firewalls that don't regularly let them access things outside their intranet, i.e., while some of them can be allowed to access an external registry for deployment, others can't and images have to be delivered manually through some arcane software upload tool.

I am looking for a way to have a registry on the internet (possibly ran by ourselves on some VM out there, preferably not) which allows the images to be stored encrypted (preferably GPG or similar, not a simple password). But then also being able to rather simply delivering "half" of the stuff via some upload mechanism manually. The customer is very paranoid, so keeping the end-to-end-encryption up is quite important.

Is there a tool that springs to mind, which is perfectly able to handle it? One solution would be to mirror the full registry to the customer premises on a dedicated host, keeping the encryption part intact.

A "standard" solution would be great, I'd be loath to hack something together if there is something lean/lightweight/established/stable around, already.

EDIT (+edit to the title): An extensive permission scheme like the one offered by Portus is a good start, but I am looking ideally for end-to-end encryption of the actual images. The customer is ultra paranoid and just getting started with Docker, cloud-based services etc.

Google Container Registry is a potential option.

Registry is private
Image layers are stored on a Google Cloud Storage bucket, which is encrypted by Google (RSA) and can be additionally encrypted with a client key (RSA)
Access control is granular down to each individual object stored on GCS bucket
Client-side authentication is enforced via oauth2. Clients configure docker to use the google cloud sdk as a credentials manager
Images are pulled via ssl connections

The fact that the registry is part of Google's Cloud Platform also means you can build up to some other interesting things. For example, you could set up a VM to serve as reverse-proxy and set up the GCP firewall to only allow 443 from a certain set of IPs (like your clients' offices).


Monitor program progress on multiple servers We have three servers that are running python programs that are running data analysis tasks inside a tmux session. The method we are using at the moment is ssh'ing into each of them connecting the tmux session and watching the output on the command line.

This method is tedious, so what we are looking for is a solution that automates monitoring of program progress(output on CLI) for multiple servers at the same time. We would ideally like a web UI solution but a CLI would also be perfectly suitable.

Thank you for reading.

Any time you're running ad-hoc long-running commands, you should step back and rethink your process, because that should be automated, including error handling.

Rather than connecting in to the servers to see status, a better approach is to push that information out. You can do a wide variety of things if your want to write a bunch of custom code, but the simplest thing is probably to start sending the output through syslog to a centralized logging system (syslog itself, or ELK, or whatever). That way you can monitor everything from a central location.

Thinking forward, if this isn't a one-off task, monitoring should be automated. That is, you should never have to just watch logs to see if things are progressing as they're supposed to. Instead, you should assume they are (and continue with other work) until your alerting fires off. This is an investment of time into getting reliable and wide-coverage alerting, but as your systems grow in complexity, it will pay off as you don't have to monitor everything any time you change anything.


Preventing accidental production deployments Last night I ran a playbook from a wrong branch that referenced an old inventory file with old IPs for each server, and deployed a dev build to a production machine. What would be a a good practice to avoid such mistake from happening in the future? How can I isolate environments better?

The best practice is to separate production and staging(testing) inventory.  See Staging vs Production.

  This way you pick with -i what you are targeting.

To make it more robust you might want to clone a separate structure for production and restrict access to it. Pull from staging before deploying to production.

In detail, this would mean to 1) put the directories into git, 2) make sure staging is committed, pushed and merged, and 3) then pull to the production and deploy. The workflow is up to you.


How to monitor ELK Stack Load? We have tried Prometheus and Grafana. Currently it is not set up on our cluster and we are tight with resources.

What are the best ways to monitor load on an ELK stack?
Is there anything built in?

Load can mean a lot with an ELK stack.

For ingestion load, the logstash parsing side, you're looking at a combination of one or two things:

CPU load on the ingestion boxes.
If used, back-pressure in your ingestion-queue (such as Redis, AMQP, or others)

This can be simple stuff like queue-size, or a fancy number like age of the oldest member of the queue.

If you're in a cloud-provider that gives you a way to auto-scale your ingestion nodes, keying on CPU functions pretty well in recent Logstash versions.

For ingestion and searching, the ElasticSearch side, it can depend on a couple of things due to your architecture. If you're using physical servers, it is very easy to have far more CPU capacity than I/O capacity to store the processed results. If you're using physical servers, keep an eye on I/O queue depth in addition to CPU loading. Also beware of the Compressed Object Pointer barrier, around 32GB of JVM HEAP. Stay under 32GB. Just do it.

If you're on cloudy instances, its harder to out-strip your I/O due to the reduced core-counts. Even so, queue-depth and cpu-loading are good things to keep an eye on. If your JVM monitoring tools are up to it, tracking Garbage Collection frequency is worth it until you have a feel for what you need for RAM.

For all of this, keeping an eye on consumed disk-space will tell you when you need to add more ES data-nodes or adjust your index retention settings.

For display and searching, such as with Kibana or Grafana, I've found this to be the part of the stack I never bother looking at. At least in any way other than up/down tracking. We track CPU, but we don't alarm anything since in our experience even a 2 core box with 7GB of RAM is enough to keep up without breaking a sweat. Both of these work OK with horizontal scaling if you do run into problems.

ELK stacks come in many sizes, from all in one boxes, all the way up to systems with routing logic to bring events to parsing and ES clusters owned by completely different parts of the organization. The monitoring needs of a small architecture (all in one) are rather different than medium architectures (discrete parsing tier, usually queue-mediated), and are different again from large architectures (multiple parsing-tiers with multiple event-stores).


How do I automate deployment to Docker with CircleCI? My circle.yml file and my CircleCI account are set up to use Docker for testing already, but I'd like to move to continuous deployment upon a successful build.

Here's what I have so far in circle.yml:

machine:
  services:
    - docker

dependencies:
  override:
    - docker info
    - docker build -t pgeiss/appname .

test:
  override:
    - docker run -d -p 8080:8080 pgeiss/appname; sleep 10
    - curl --retry 10 --retry-delay 5 -v http://localhost:8080/res.json

A few things need to happen for this to work properly. First, add a deployment section to circle.yml:

deployment:
  main: # or whatever your deployment is called
    branch: master # or whatever branch you want to deploy
    commands:
      - docker login -e (your email here) -u (your username here) -p (your password here)
      - docker push pgeiss/appname
      - ./start.sh

Thanks to this blog post for the following script. Then, make a file called start.sh in the top level of your repository (if you use a different name, change the circle.yml's last line) that contains the following:

#!/usr/bin/env bash

echo "stopping running application"
ssh $DEPLOY_USER@$DEPLOY_HOST 'docker stop dodsv'
ssh $DEPLOY_USER@$DEPLOY_HOST 'docker rm dodsv'

echo "pulling latest version of the code"
ssh $DEPLOY_USER@$DEPLOY_HOST 'docker pull pgeiss/appname-webapp:latest'

echo "starting the new version"
ssh $DEPLOY_USER@$DEPLOY_HOST 'docker run -d --restart=always --name dodsv -p 80:5432 pgeiss/appname:latest'

echo "success!"

exit 0

Finally, to allow the script to work you'll need to set the script to be executable with chmod and add the environment variables and your DEPLOY_USER's ssh key (preferably) or credentials (if no ssh key) to CircleCI. After doing that CircleCI should deploy your app after a successful build.


Best practices for CI for embedded platforms (ESP32/Arduino) Development processes for embedded software always lags behind that for web development. When looking at unit testing, automated builds and code analysis there are many limitations to popular tools.

From experience I have already dismissed the Arduino IDE to using platformIO with VS Code since, in regards to CI, it provides improved dependency installation and build configuration automation.

The question is what tools and processes already exist and which can be repurposed to be used within the Arduino framework to enable CI as with web development.

Meta: the next most relevant stack exchange group would be IoT, but not a single post mentions continuous integration

Since "deployment" to the IoT device, especially in bulk, will not be done using the CI system. Then the purpose of the CI system is mostly to make sure that it will work okay.

This means that you mostly want to run some tests using the CI. Automated testing is definitely possible, both for unit tests and for integration tests.

The most important thing for unit tests is usually very short runtime and a short feedback loop. For a large code base, if possible to compile parts of it using the native compiler of the host and run unit tests there - it is a great option. If you must, you can also cross-compile to the target platform and run the unit tests using some emulator to have the necessary speed. All of this can happen during a CI "build+test" job. Compiling for the target and running unit tests on the target will just be painfully slow if done properly, but can also be done - just make sure to only flash the smallest possible piece of code and have it report back via the Serial that it worked on the device.

For integration tests, if possible to install your code on an actual device and the have some chatter with the device using the serial interface - you can ensure that the code actually works on a target device. If for some reason the code fails or works incorrectly, maybe draws too much power and/or restarts then the integration tests will catch that. You can also have another test device performing measurements and reporting via the Serial interface to the CI job. Can use an Arduino, or another ESP to maybe measure voltage and/or amperage during the execution of your code parts and checking that against known good values. You don't expect the device to pull 400mA when in a deep-sleep mode, for example, this is easy to test with a "test harness".

All of these unit tests and integration tests can be triggered completely automatically on the CI server. Naturally, you will probably have this "server" running on-premises and not rely on a cloud solution that might or might not support your latest integration testing.

With hardware, smoke tests are more "real", since you can actually hook-up the device and see the magic smoke come out. Usually, you don't even need to run code before it burns since the problem is often in the hardware.

Since I mentioned testing, there is also integration testing which is done on the hardware level. If you have a good test harness that checks that the device works well by probing it, you can use the same test harness to do your QC cycle in the factory where they produce the devices later. There is quite a lot of information about making these testbeds. One example for this is here - https://www.sparkfun.com/tutorials/138.


Why is Rails (with Nginx/Passenger via Ansible) unable to see my RAILS_MASTER_KEY environment variable? I'm writing Ansible playbooks to provision an Ubuntu 18.04 server.

I want to provision the server for a Rails application using Passenger and Nginx. I used geerlingguy.passenger from Ansible Galaxy to configure Passenger and Nginx.

In addition, I used zzet.rbenv to install ruby 2.5.3.

The Ansible playbooks seem to have executed correctly, and when I log into the server via SSH, everything seems to be working (rbenv is on the path, RAILS_MASTER_KEY environment variable is defined).

Deployment of my application via Capistrano also appears to run successfully.

However, when I attempt to visit the landing page, I get a Passenger error: "Web application could not be started by the Phusion Passenger application server."

Looking in /var/log/nginx/error.log, I see:

Missing encryption key to decrypt file with. Ask your team for your master key and write it to /var/www/releases/20191230212823/config/master.key or put it in the ENV['RAILS_MASTER_KEY'].

I'm not sure why the above error is happening, because I've already configured a persistent environment variable using the weareinteractive.environment role.

Inspecting /etc/environment confirms that RAILS_MASTER_KEY is defined, and I can see if it I log in manually as my 'deploy' user. However, /etc/nginx/nginx.conf shows that the default user is www-data:

user www-data;

I think this should be fine, because the environment variables are configured in a way that is not associated with a user account. So, why does Rails fail to see this environment variable even though I see it when I log in via SSH?

I'm able to execute the following command on the server and see the contents of the encrypted credentials file:

RAILS_ENV=production EDITOR=vim bundle exec rails credentials:edit

So what is going wrong when Nginx/Passenger launch Rails?

Edit: It seems like Passenger/Nginx do not load environment variables from /etc/environment, which mostly explains my problem. I'm trying to figure out the appropriate way to configure environment variables for Passenger using Ansible now.

Root cause: environment variables from /etc/environment are not passed to Passenger. The standard method seems to be passenger_env_var in your nginx config as suggested by user joshaidan.


Installing phantomjs karma launcher in Ubuntu-Docker context Karma is a nodejs tool to automate browser testing. It seems that it depends on nodejs plugin fsevents.

But, fsevents is for processing Mac OS X events, therefore I get the error "not supported on your system" for the phantom karma launcher.

Questions I have found on GitHub seem to come from other Mac OS users dealing with upgrading npm.

My question is: how to install karma launcher in headless ubuntu/docker context?

Because fsevents is an optional dependency, the solution has been here:

npm install --no-optional karma-phantomjs-launcher


Exemplify CI/CD to a colleague from Microsoft world using product-neutral terms After a chat with a colleague who collaborates with quite high-end projects on Microsoft stack (for example, ASP.Net + MS SQL Server).

An interesting outcome was that although I believe that DevOps must be possible in the Microsoft world as well, my knowledge is too little so that I cannot give good examples how he could organize a CI/CD approach. Indeed in our community there are some hints about Team Foundation Server and PSBuildStack (?), but an official source does not reveal all that at a glance. The described hosted services make it even less transparent for a newcomer - is that a cloud build environment like CloudBees/BlueMix?

So, my question is now: how could I describe an example of mature toolchain using established DevOps terms which are valid more or less independent from specific environment/tooling - or do we have to admit that this engineering terminology is not there yet?

Something like this:

Source code version control system
Build agents
Automation and packaging scheduling system to encapsulate artefact composition logic
A binary repository (different classes of binaries: libraries, distros, container images)
A configuration repository for different environments
Package and configuration distribution system to encapsulate deployment logic

I would say that instead of describing a CI/CD system by the products/tools that you use, you should be describing the system by the purpose of each step in the system. Instead of saying "I have a Jenkins server that builds the artifacts, stores them in Artifactory, then Puppet deploys the artifacts to EC2." You could say, "Our build server creates and stores our artifacts in an artifact repository. Then our configuration management software detects the artifacts and deploys them to our cloud infrastructure." Your colleague can then make the connections and find the correct solution that Microsoft provides.


Is it possible to force to false or true the App.config transformation with arguments on Azure DevOps Pipeline? Is it possible to force the TransformOnBuild to false using MSBuild parameters in Azure DevOps?

So I Use Azure DevOps Pipeline to build a clickOnce application for a staging environment

Here is my MSBuild Arguments

/target:publish /p:ApplicationVersion=$(Build.BuildNumber) /p:InstallURL=http://app-staging.example.com/ /p:PublishURL=http://app-staging.example.com/ /p:UpdateEnabled=true  /p:UpdateMode=Foreground  /p:ProductName="App Staging" /p:OutputPath="$(build.ArtifactStagingDirectory)\Publish\\"

Is it possible to force to false or true the App.config transformation with arguments?

Short answer, yes.

Azure Pipelines supports conditional steps. Find these under Control Options in any build step.

How you build your custom conditions will depend on what triggers your step to true/false.

The screenshot has an example of how we control a build step based on a build variable (defined in the variables tab of the pipeline).


Serve an application built by the pipeline In the company that I work, the frontend application is very complex, so one of our layers of testing includes building/serving the application and testing with integrations mocked (The application isn't deployed anywhere on this step).

Right now we are trying to migrate from Jenkins to Gitlab CI, but I have no clue how to do this. I have the build artifact (Some html/css/js) but I couln't find any information how to share the artifact with some Gitlab CI service.

Basically what I want do achieve is Build -> Serve the application that was built -> Run some tests on it

Is it possible to do this?

Gitlab CI/CD offers Docker Engine. If you can create a docker image, you can serve the application the same way as you were doing with Jenkins. You can find more details on how to integrate both here.

Hope it helps.


Ansible SSH best practices What are the best practices to be followed wrt security when it comes to SSH authentication (password less, key based, etc.) when setting up Ansible in a new environment with 1000s of machines? Which is the best method of using passwordless SSH in an enterprise environment and why?

The answer is Key Based SSH

Reasons

You can SSH multiple host machines using the same key or Different keys.
If you reboot your machine while playbook running it again SSH using your provided key so your ansible-playbook runs without occurring any errors.
Also, you get authentication using the appropriate user.

use can use inventory.ini file

example

[hostMachine]
ubuntu@xx.xx.xx.xx  ansible_ssh_user=ubuntu ansible_ssh_private_key_file=/path of your pem file

[ABCDMachine]
ABCD@xx.xx.xx.xx ansible_ssh_user=ABCD ansible_ssh_private_key_file=/path of your pem file

Command for run ansible-playbook

  ansible-playbook -i inventory.ini main.yml


How to have kubernetes (re)build containers on deployment similar to openshifts ImageStream I am currently investigating how to manage the containers for our new data-processing pipeline.

What I would like to do is to be able to have whatever tool I choose rebuild the docker images when I have a new version ready in git (without me having to come up with a name for the docker image). I have found that OpenShifts ImageStreams does this but OpenShift feels like an intrusive dependency so I am searching for a well supported way of doing this with a more clean kubernetes install.

I have looked at helm but did not find this feature.

So

What is the most mature and widely supported tool that can build the containers automatically during the deployment to kubernetes?

You need something that will do basically CI/CD. There are few options (in no particular order):

Gitlab + autodeploy (or custom deployment)
skaffold + git-sync + kaniko

There are more, but basically, you need a component to watch for changes, something that knows how to build an image, and something else that can deploy it (some of these might be done by the same tools)

Helm will only do the deployment, but nothing else


What is recommended to document an IT technology stack, including their relationship to one another, in a graph database? Working for a large company with over 500 IT staff and over a 1,000 servers, with each server running its own business applications, we have a tremendous information and co-ordination challenge in knowing which IT staff member to contact for which server. The problem of co-ordination is compounded with different IT staff being responsible for different layers of the IT stack. E.g., there are different teams who are responsible for hardware, virtualization, operating systems, application servers and the applications themselves.

Considering this challenge, within DevOps there is a requirement to define and document all the components that constitute the various technology stacks within an IT environment. Traditionally this might have been accomplished with a propriety CMDB solution.

I've investigated typical CMDB solutions such as BMC Atrium and others for this purpose, the problem however is that they stop at the level of documenting IT assets themselves, at a high level, as per the ITIL framework, but to do not address documentation of the IT Technology Stack in detail. I have also investigated tools such as Puppet, Ansible and Salt, but these tools focus more on software deployment and configuration, and not on the people co-ordination around software.

A workable solution for example, would define the various concepts, along with the key attributes important to these concepts:

Hardware
Virtualisation
Operating Systems
Application servers
Applications

These concepts would then be associated with one another in terms of their relationships in order to form solutions. E.g., an application would depend on an application server, which would depend on an operating system, etc., Together this solution would be defined at the "Finance System". Having defined a system, all the metadata associated with these systems would be captured in order to facilitate co-ordination for each layer in the stack. I.e. the co-ordination of the technical support staff for each layer.

The purpose of such a solution would to do various queries against technology stacks, such as:

To determine who supports which components
Which components are out of date
Which components need to be patched

With this in mind, what open source tools exist to define all the components of an IT technology stack, including their relationship to one another, in a graph database such as Neo4J?

Considering your first paragraph, the organization you're describing is a highly siloed org, which is exactly what a DevOps organization tend to avoid.

  Considering this challenge, within DevOps there is a requirement to
  define and document all the components that constitute the various
  technology stacks within an IT environment. Traditionally this might
  have been accomplished with a propriety CMDB solution.

What you're describing here could be ITIL, which is a management system requiring documentation and you mix it with the fact a DevOps team will usually define the underlying layers as code, as such it gets back to any development documentation with the caveats of Code is Documentation often seen in Scrum methodology for an agile methodology of development (quick and short iterations aiming at minimal working solution at end of the iteration)

Disclaimer for the rest of this answer: I know more of chef and inspec and that's why I take them as exemple here; but they are not the only tools existing on the market, I won't open a debate on them as the better one is the one you're more comfortable with.

As such the rest of the question is a little biased and I, personally, didn't encounter an organization documenting the layer relation you describe more than the infrastructure as code and configuration management system code documentation. (Again, this doesn't mean no-one does it, I just never heard of it).
To illustrate from my company in a chef environment, an application cookbook will declare its dependencies (tomcat, jboss, nginx/php and on which OS, needed mount points for some shared datas and its DB schema name mostly) and expose its services URIs to be consumed by chef for other applications configuration, this sound like defining your 'Finance System' and the documentation for it is on this application cookbook README, with some more files if really needed.

Configuration management systems usually have a central reporting place, "chef-server" for datas an "manage UI" for presentation in chef world "ansible tower" for ansible world to name two of them, but they usually aim more at giving an oversight of the overall managed system than graphing the dependencies.

That said, for chef, the chef-server also act as a CMDB you can query with various tools (it return JSON datas from HTTP requests), the inter applications dependencies can be expressed in various way and there's no 'out of the box' method, each company will have its own way to declare them in the system for configuration purposes and as such you may leverage this to build your graph, but that's on your side.

In an infrastructure as code point of view, the infrastructure needs would be kept with the application, it is still the application who know what it needs under as middle-ware, which OS, with which locale, what are other services dependencies and what services this application offer).

Last thing I can think of if you want to manage those dependencies for documentation only are tools like glpi which is mainly a CMDB and a ticketing system, it takes advantage of documenting assets and their relation to be able to tell what is impacted when you open a ticket saying an application is down. coupled with ng-inventory it allows to query system states and as such can fulfill your query for patch needs, but in my opinion this is an audit system task, like could do inspect integrated within a chef run for exemple, as the next phase would be to fix the outdated systems by updating/patching them.


Ultimate docker permission solution I'm using docker some time and I like the conception very much. But I found a thing is a big problem for me.

I'm talking about permissions. While I'm not worrying about production, setting up my docker development environment becomes hell, because of file ownerships and permissions. For example:

www-data user from php container cannot create files in my directories (www-data uid is 82 while my uid is 1000)
I was trying to change www-data id to 1000 but I share this environment with macOS users, who typically have uid of 500
I cannot remove files that have been created within container - that makes it hard for me to jump between git branches sometimes (sudo rm somefile).

I spend hours, no, days looking for ultimate solution. Tried gosu in containers, mounting /etc/shadow as a volume etc. The problem is, I was unable to find a solution that can be considered multi-platform and proper (mounting local system files readonly just doesn't look good).

Thank you!

I'm a little confused about the exact use case. Are you asking about writing to volumes that are mounted inside the container?

You can change what the process in the docker container runs as by using --user on your run commands.

docker run --user 1000 --ti centos/7 /bin/bash

This answer may help you:
https://stackoverflow.com/questions/41100333/difference-between-docker-run-user-and-group-add-parameters

The manual explains this as well: https://docs.docker.com/engine/reference/run/#user


How to keep track of broken releases? I spent considerable time googling around, but found nothing that would answer my question, so here it is.

If you do a canary release, and you learn the the release is somehow broken: how to you denote that a certain tag is broken and should not be deployed anywhere else, but removed from your productions ASAP?

I had several ideas:

Using GitLab, I could add something like **broken** to tag release note. Deploy jobs would call GitLab API to ensure that the tag I want to deploy is not annotated as broken.
I could have a text file in repo, where broken tags would be listed. In this case deploy job checks if the tag you want to deploy is not listed in the file.
Do no save information about release problems: just deploy a new stable, patched version to production and never look back.

How do/would you handle this? Any tips or hyperlinks are welcome.

I am not sure about other approaches, but the one that I use is a modification of #3.

There's usually a classification of releases (for example think of Ubuntu LTS and regular release).

For the regular release, I would just release a new version and add a new tag.
For LTS/Stable releases I would delete a tag pointing to a faulty release or change a commit this tag points to


Options to assign a static ip to a service running in docker swarm Situation:

docker swarm with 3+ nodes
service running in the swarm with replicas=1 and restart=always
swarm hosts have multiple interfaces - one for each vlan, with the same name
service is deployed as a stack

I want to replace a service running in a VM on a static IP with the same one running docker, but I need to have the same SERVICE_IP on the ETH_SERVER interface.

What options do I have for one or more services to have static ips ?

A. added ipv4_address in docker stack yml - does not seem to be implemented in docker (at least not yet)

B. add a new host with reverse proxies to the docker stack services

C. use keepalived on the docker hosts and map SERVICE_IP - in this way if a host goes down the floating ip gets reassigned - but not sure if I could  expose ports with FLOATING_SERVICE_IP:: ...

D. ... ?

Are there any "better" ways of doing this?

I tried doing this:

on each docker host:

docker network create --config-only --subnet="<CIDR>" --ip-range="<CIDR>" -o parent="INTERFACE" "vlan_name_intermediate"

and then a global network for the whole swarm

docker network create -d macvlan --scope swarm --config-from "vlan_name_intermediate" "vlan_name"

Now if I attach a service to network "vlan_name" i can ping it from inside that vlan, but the IP is randomly assigned, not static. I tried adding this to the compose yaml file:

ipv4_address: "192.168.45.56"

where the ip is inside the CIDR ip-range, but the setting is ignored

On June 30, 2016 this issue was created on GitHub to request the implementation of static IP on docker-swarm services.

Since then multiple replies with thumbs up and down were added.

At the time of writing on December 20, 2019 this issue is open and the feature has not been implemented.

Discussion

After reading all comments I strongly doubt whether static IPs will ever be implemented in docker-swarm. On the other hand it is unclear why this issue has not been closed. If people would like to see that this feature gets implemented, one could create a Pull request or add a thumbs up to create more awareness.


Gitlab CI/CD fails to build docker image I have built a custom base image from alpine and use it as the base for ruby services that we develop.

FROM ruby:2.4.2-alpine
...

I use Gitlab and Gitlab's docker registry along with it. Each project has its own docker image storage.

I created a separate project for the alpine-ruby-service dockerfile. Then build and pushed the image to Gitlab.

In the ruby service project docker images, I reference it as follows

FROM gitlab.mycompany.dk/alpine-ruby-service-base
...

I have a CI/CD pipeline (.gitlab-ci.yaml) which builds these ruby service images when there is a commit to one of the branches.

The problem is that when the CI/CD pipeline is run it fails with the following message

 $ docker build --pull -t $CONTAINER_TEST_IMAGE .
   Sending build context to Docker daemon  709.6kB

 Step 1/13 : FROM gitlab.mycompany.dk:xxxx/alpine-ruby-service-base
 pull access denied for gitlab.mycompany.dk:xxxx/alpine-ruby-service-base,
 repository does not exist or may require 'docker login': denied:
 requested access to the resource is denied

I have an instruction to login to the Gitlab docker registry before the build command above using

 docker login --username=gitlab-ci-token --password=$CI_JOB_TOKEN gitlab.mycompany.dk:xxxx

the login succeeds but the following docker build instruction fails!!! can someone point me what is going wrong?

It looks like that the docker login and the docker build were done by different users. If a docker login is performed, then a ~/.docker directory will be created in the namespace of user X, if the build is done by user Y then the latter user does not have a .docker folder and is not authorized to pull the image. One could verify this by echo $USER in the pipeline, in this case gitlab-ci.yaml to check this.


Data processing infrastructure We are building a system for data science in AWS and our flow is pretty simple.

Get the data - From redis
get the model - From FSx/EBS
run.

This process script should take something like 5-10 seconds depends on the size of the data and model.
So far so good.
What we did so far is to build a python application (runs in kubernetes) that listen to a queue and runs the data science flow.
The problem we faced was Pythons memory leak. what ever we did and every approach we took with python a long running application that loads/unloads large objects will grow and grow, So we are trying a new approach , running the script on its own process each time a new process is created and closed when finished (still in kubernetes).
My question is
Is there a better way?
AWS Lambda - could be a good solution but it has a memory/cpu limits that doesn't fit.
Kubernetes Job - Inefficient when the script itself takes only a few seconds to run.
Are there any other solutions to run high load of requests with cpu/memory intensive work? (I'm looking for a resource/infrastructure solution, not a software one.)

The service can be always-on listening on the queue for jobs, or single-job started by the queue only when new work arrives.
The tradeoff is startup time for single-jobs vs cost for always-on service.
Define the requirements - number of jobs per day, max latency, ram/cpu needed, etc and then select the right technology that can deliver it in the most cost effective way.
From AWS Lambda, AWS Step Functions, through K8S Jobs and AWS Fargate to AWS EC2 on spot pricing, and more. Plenty of options, depends on the requirements.
Hope that helps :)


Angular config maps in kubernetes How can I integrate a config map into my angular application in kubernetes?

Any useful blogs or videos implementing this would be helpful.  I found a few, but they didn't work for me.

This one from medium should be fine for your purposes: https://medium.com/@ychetankumarsarma/how-to-manage-angular-application-configurations-for-different-environments-with-docker-and-8ff8c55a1c7d.

It is good because:

It shows a test / development scenario.
It notes that it is a good / accepted technique to use environment variables to drive config.
It shows you how to create a config map with a literal value (as an example).
It shows you how to read that environment variable into your app environment properly.

There is also a working sample app noted here -> https://github.com/chetanku/frontend-env which you should be able to "just start up" and work backwards from.


Terraform: how do you inspect sensitive data? In a resource of type aws_ecs_task_definition my plan has an line that looks like this:
~ container_definitions    = (sensitive) # forces replacement

Now in principle this could be doing anything - deleting all my containers, spinning up bitcoin mining containers, etc.
Is there any way I can look at the data, both the old and the new?

When your configuration or a provider marks an attribute as sensitive, Terraform will always hide that value in any output that's intended for human consumption.
The real values are available in machine-readable output though. This is primarily with the aim of integrating with external software, but if you need to then you can also inspect the machine-readable output directly yourself.
You can get a machine-readable (JSON) rendering of a plan like this:
terraform plan -out=tfplan
terraform show -json tfplan

If you intend to read it directly in the terminal then it can help to pipe it into jq, if you have that utility installed:
terraform show -json tfplan | jq


How to set Terraform Cloud environment variables from the command line? Terraform Cloud is a SaaS platform for managing Terraform runs, state and variables.  Once you have created a project you can set environment variables via the following page:

https://app.terraform.io/app/[organization]/workspaces/[workspace]/variables

These environment variables are then set when Terraform executes a run.  Out of an abundance of a desire to automate this process, I have scripted the process of generating these environments variables:

$SubscriptionId = "<subscription-id>"
$PrincipalName = "http://<service-principal-name>"
$SubscriptionUrl = "/subscriptions/$SubscriptionId"

az cloud set --name "AzureCloud" | Out-Null
az account set --subscription $SubscriptionId | Out-Null

$ServicePrincipal = az ad sp create-for-rbac --role="Owner" --scopes=$SubscriptionUrl --name $PrincipalName | ConvertFrom-Json

$env:ARM_CLIENT_ID=$ServicePrincipal.appId
$env:ARM_CLIENT_SECRET=$ServicePrincipal.password
$env:ARM_SUBSCRIPTION_ID=$SubscriptionId
$env:ARM_TENANT_ID=$ServicePrincipal.tenant

This works, however, I would like to extend the script to be able to push the variables to the appropriate Terraform Workspace.  I am aware that there is a RESTful Variables API that I can call although I was hoping for a solution that used the terraform command-line interface rather than relying on parsing backend.tf and calling a REST API from a script.

You can use the Terraform Cloud provider to manage workspace configuration in Terraform Cloud in the same way as you might manage any other infrastructure objects with Terraform. For historical reasons the provider is named tfe, but it can be used to manage both Terraform Enterprise and Terraform Cloud.

In particular, you can use the tfe_variable resource type to manage both Terraform variables and environment variables for a workspace:

variable "environment_variables" {
  type = map(string)
}

variable "workspace_id" {
  type = string
}

provider "tfe" {
  hostname = "app.terraform.io" # Terraform Cloud
}

resource "tfe_variable" "test" {
  for_each = var.environment_variables

  key          = each.key
  value        = each.value
  category     = "env"
  workspace_id = var.workspace_id
}

To integrate this with the script you've already written, you could perhaps use PowerShell's JSON encoder to produce a file azure.tfvars.json with content like this:

{
  "workspace_id": "organization/workspace",
  "environment_variables": {
    "ARM_CLIENT_ID": "...",
    "ARM_CLIENT_SECRET": "...",
    "ARM_SUBSCRIPTION_ID": "...",
    "ARM_TENANT_ID": "..."
  }
}

You can then run the above Terraform configuration using a command like this:

terraform apply -var-file=azure.tfvars.json

When considering automation of this sort it's worth considering where the Terraform state snapshots for this "early" Terraform configuration would live. You could potentially create an additional Terraform Cloud workspace for this special configuration and be sure to configure it to not use remote operations, so that a local run of this one workspace can then arrange for your other workspace to support remote operations.


Untracable glassfish docker container image or clean installation? For a customer I need to create a glassfish container, containing the glassfish 5.0 release.
I have been looking for standard glassfish images on docker hub, and other places, but can't manage to find a recent image containing a recent glassfish release. For some reason, I also can't find, glassfish isn't available on dockerhub anymore. Only thing I could find is a reference to glassfish server on dockerhub, but that's also gone for some unclear reason.
Now am I correct to just use a clean docker image and create a new one where I install glassfish myself?

Look into Payara.  It's a Glassfish fork/drop-in replacement that's better in most ways, including support.
There is a Payara 5 docker image available right on docker hub:
https://hub.docker.com/r/payara/server-full


Trigger Jenkins Job from Azure DevOps I want to trigger Jenkins Job when anyone pushes the code in Azure DevOps Repository. However, when I set up Jenkins Service Connection in Azure DevOps I am getting the following error:

  Failed to query service connection API: 'JenkinsURL'. Error Message:
  'A task was canceled.'

I have tried enabling port 8080 in the VM to resolve this.

The error message you have provided indicates that this is a problem with the configuration that could be permission related.  Here are some steps to help resolve this issue:

You can follow guides where other people have configured the two.  Pay special attention to the permission settings that are selected.
Double check your permissions on the servers.  While you may have configured Jenkins to communicate with Azure DevOps, you still need to make configuration changes in DevOps.
Double/Triple check your URLs on both systems.  I've been burned by dropping an s on http or missing a character on a copy paste.
If you are setting this up within Azure check the Azure Role Based Access Control to make sure proper permissions exist.


How to filter ELB logs by today's date? I've tried this query:

SELECT request_timestamp
FROM elb_logs
WHERE date_trunc('day', request_timestamp) = date_trunc('day', current_timestamp)
LIMIT 10

but it fails with:

  SYNTAX_ERROR: line 3:7: Unexpected parameters (varchar(3), varchar) for function date_trunc. Expected: date_trunc(varchar(x), date) , date_trunc(varchar(x), time) , date_trunc(varchar(x), time with time zone) , date_trunc(varchar(x), timestamp) , date_trunc(varchar(x), timestamp with time zone)

I've checked the Date and Time Functions and Operators docs page, but I can't see anything wrong.

I've created the elb_logs table in Amazon Athena according to this page.

What I am missing?

Either request_timestamp needs to be converted from the ISO 8601 formatted string into a timestamp with time zone, or current_timestamp needs to be converted into the string.

One method is to convert request_timestamp into a string in format %Y-%m-%d, e.g.

split_part(request_timestamp, 'T', 1)
substr(request_timestamp, 1, 10)

then compare it with current_timestamp which can be converted to the ISO 8601 formatted string.

Here is the final query:

SELECT *
FROM elb_logs
WHERE split_part(request_timestamp, 'T', 1) = split_part(to_iso8601(current_timestamp), 'T', 1)
LIMIT 10


How to log native commands run by Ansible Playbook Ansible internally uses SSH Commands to run on the target machine . Is there a way to log the commands that are being executed ?

Set the log file to use inline or in Ansible config, then run ansible-playbook with verbose option:
ANSIBLE_LOG_PATH="ansible-$(date +%Y%m%d%H%M%S).log" \
ansible-playbook -vvvv \
    --diff \
    -i inventory/myinventory \
    myplaybook.yml

ANSIBLE_LOG_PATH - is an environment variable that maps to the log_path configuration item. Sets the file to log to. When using an environment variable you can use shell substitution to add the date and time to the file name so the next run doesn't overwrite the log file.
-vvvv - be very very very verbose. Will show SSH commands executed.
--diff - bonus, when making a change show what has been done.
the rest are standard options.

The log will be in a file in the current directory named something like ansible-20200818155322.log


sshagent and jenkinsfile I am new to Jenkinsfile.
I have previously configured my Jenkins job from GUI, and we are now upgrading our server to use Jenkinsfile.
My job contains a section which needs to SSH to another server.
To do this, I previously used the sshagent plugin on the GUI to specify my keys.
However, on the jenkinsfile mechanism this does not work.

First things first:
Aside from using jenkinsfile, we have also trasitioned to building in docker. I read in some places that docker might conflict with sshagent plugin, so I am trying to remove docker by starting a clean job with "agent any" and trying to test.

So my new job looks like this:

    @NonCPS  // Necessary to allow .each to work.
def changelist() {
    def changes = ""
    currentBuild.changeSets.each { set ->
        set.each { entry ->
            changes += "${entry.author.id} "
        }
    }
    changes
}

pipeline {
    agent none
    stages {
            stage('test') {
            agent any
            steps {
                sshagent ( ['a-jenkins-credential']) {
    sh '''
ssh -vv myuser@myserver echo testing connection || true
ssh-add -L
echo done running remote windows test
'''
  }
            }
        }

}
}

On the same server, but using the jenkins GUI, this works flawlessly, but on the jenkinsfile pipeline this produces:

FATAL: [ssh-agent] Could not find specified credentials

and proceeds run and fail on the ssh command. The funny thing is, that this job uses the exact same key to succesfully pull from GIT, so it's not like it's not installed on the server.

Found the problem:

I was using the human readable key name from the GUI.
Need to use the key's UUId ID instead (this is specified next to the name in credentials screen)

steps {
            sshagent ( ['THIS-SHOULD-HAVE-BEEN-A-UNIQUE-ID-INSTEAD-OF-A-NAME']) {


How to deploy loki-stack helm chart with CRIO? I am trying to configure loki-stack using a helm chart to my k8s cluster.
Since my k8s cluster uses CRIO and not docker (which is common nowdays) I need to change the pipeline_stages: config map to use cri: {} instead of docker: {} to parse the logs properly.
How do I change the helm chart so that cri: {} is used by default instead of docker: {} in all the pipeline_stages:? many thanks.

Managed to solve this by changing the values.yaml in the loki-stack helm chart to the following:
promtail:
  enabled: true
  pipelineStages:
  - cri: {}

Enjoy.


Automatic periodic conditional checks I need a simple tool that will carry out some conditional checks periodically and give me a red/green status that I can use as an indication that something is wrong.

These are examples of checks that I would like to monitor automatically:

Does a row exist in a database that meets a certain criteria?
Does a file, with a maximum age, exist on some network share?
Does the response-body of a HTTP-request match a specific regex?

So basically, it would free me from checking these various indicators manually and just have a red/green dashboard that I could monitor.

I'm thinking something like this has got to exist in a ready-made solution.

All in all any monitoring tool would do for this case, monit, nagios, shinken, icinga, centreon, or even a crontab in bash could do...


Using docker with users leaves files belonging to the subuid/subgid Background

I wish to make sure all people and automated processes working with my project use the exact same version and environment of some tool which has to do work on the host file system.

Instead of running /usr/bin/xyz or whatever, people will then use a wrapper/shim script similar to:

docker run -ti --rm \
  -v $PWD:/workdir \
  -v $HOME/.ssh:/root/.ssh \
  dockerizedxyz "$@"

This is working splendidly but...

Problem

Obviously, if I stop right here, then files created by the xyz running inside the container will be ending up belonging to root:root on the host. It is clear why this happens, and as expected.

Hence, I have enabled user namespaces for the docker demon. This works fine as well. My host user has UID 1000 and the subuid 493216.

If I run the docker-xyz-shim in a directory (on the host) that belongs to 1000, and if that container starts to create files, my expectation at first had been that they would have ended up as belonging to 1000 on the host (i.e., forward- and backward-mapping).

But, lo and behold, they belong to 493216, i.e., they belong to the subuid instead of the host uid. There is only a forward-mapping happening (from host-uid to sub-uid, no backward-mapping from sub-uid to host-uid).

I understand why that is, and that this is working as intended. First, there is a 1:n mapping involved, and it would be non-trivial to map the potentialla 65536 sub-uid back to the one host-uid. It allows us to do all kinds of things that would be practical in other use-cases. But in my particular case, I actually do wish the files to end up with 1000 on the host.

Is there anything I can do about it?

What I tried

I added the host user 493216 to the host group 1000 and set the g+rwx permissions. If I do a su - 493216 on the host, I can now edit the files belonging to 1000. But from inside the container, the files are shown to be owned by nobody:nobody and I can read them, but not write them.

I can create new files inside the container if I do a chmod 777 on a directory on the host (it then belongs to nobody:nobdy with 777 permissions in the container), and they end up belonging to root:root in the container (fine) and to 493216:493216 on the host (to be expected after all this).

EDIT: replaced "git" with "xyz", this is not about git at all, but about how to modify host files from inside a docker container without having root on the host.

docker run -ti --rm \
  -v $PWD:/workdir \
  -v $HOME/.ssh:/root/.ssh \
  --user $(id -u):$(id -g) \
  dockerizedxyz "$@"

As long as you're not doing anything funny with the entrypoint in the container, this will leave files owned as the user that invoked Docker.


docker run vs docker-compose - different results? I have just started exploring Docker and am having trouble with docker commit/-compose.
I have copied a file into /root/.guacamole/lib (in the container) and then created an image using docker commit.
An ls of the relevant path in the container using
docker run -it --name="name" image bash

confirms that the file exists, but when the following docker-compose file sets up the container, the file is missing, even though it reports using the same image.
Any idea why that should be?
Thanks
$ cat docker-compose.yml
version: "2"
services:
  guacd:
    image: "guacamole/guacd"
    container_name: guacd
    hostname: guacd
    restart: always
    volumes:
      - "/data/shared/guacamole/guacd/data:/data"
      - "/data/shared/guacamole/guacd/conf:/conf:ro"
    expose:
      - "4822"
    ports:
      - "4822:4822"
    network_mode: bridge

  guac2:
    image: "guac2"
    container_name: guac2
    hostname: guac2
    restart: always
    volumes:
      - "/data/shared/guacamole/guacamole/guac-home:/data"
      - "/data/shared/guacamole/guacamole/conf:/conf:ro"
    expose:
      - "8080"
    ports:
      - "8084:8080"
    network_mode: bridge
    environment:
      - "GUACD_HOSTNAME=pc.lan"
      - "GUACD_PORT=4822"
      - "MYSQL_HOSTNAME=pc.lan"
      - "MYSQL_PORT=3306"
      - "MYSQL_DATABASE=guacamole_db"
      - "MYSQL_USER=guacamole_user"
      - "MYSQL_PASSWORD=password"
      - "GUACAMOLE_HOME=/data"

Here is the terminal text:
user@pc:~$ docker run -it --name="test" guac2 bash
root@b6dcd7dda985:/opt/guacamole# cd /root/.g*/lib
root@b6dcd7dda985:~/.guacamole/lib# ls
mariadb-java-client-2.6.2.jar  mysql-connector-java-5.1.46-bin.jar  mysql-connector-java-5.1.46.jar
***[NB 3 files listed above]***
root@b6dcd7dda985:~/.guacamole/lib# exit
exit
user@pc:~$ docker stop test
test
user@pc:~$ docker rm test
test
user@pc:~$ docker ps -a
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                    NAMES
ea92c2db77f6        guac2               "/opt/guacamole/bin/…"   51 minutes ago      Up 51 minutes       0.0.0.0:8084->8080/tcp   guac2
52aea98f7e2c        guacamole/guacd     "/bin/sh -c '/usr/lo…"   About an hour ago   Up 57 minutes       0.0.0.0:4822->4822/tcp   guacd
user@pc:~$ docker stop guacd
guacd
user@pc:~$ docker stop guac2
guac2
user@pc:~$ docker rm guacd
guacd
user@pc:~$ docker rm guac2
guac2
user@pc:~$ docker-compose up -d
Creating guac2 ... done
Creating guacd ... done
user@pc:~$ docker exec -it guac2 bash
root@guac2:/opt/guacamole# cd /root/.g*/lib
root@guac2:~/.guacamole/lib# ls
mysql-connector-java-5.1.46-bin.jar  mysql-connector-java-5.1.46.jar
***[NB only 2 files listed above]***
root@guac2:~/.guacamole/lib# exit

I'm a bit new to Docker as well, but are you 100% certain that you performed a Docker push and your commit of your docker image that you're building with composer? And you have matching versions on the remote and your local machine?
I'm also not seeing where you are mounting the '~/.guacamole/lib' path in your docker-compose.yml under volumes:


It is not allowed to use stages more than once [Jenkins_Declarative_Pipeline] I am completely new to the jenkins pipeline. I'm trying to create a project with multiple stages but in my Import stage , I want to execute parallel. For each stage in parallel , I have again multiple stages. I'm trying the following way, but getting syntax error as It is not allowed to use stages more than once.Can someone correct me how to achieve this, I have tried some online resources but unable to get clear syntax.

pipeline {
  agent any
  stages {
    stage('Build') {
      steps {
        echo 'from build'
      }
    }
  }

    stage('Test_A') {
      parallel {
        stages("GUI") { stage("Tests_A") { steps {echo 'from A'}} stage("Archive") {echo 'from Publish' } }
        stages("API") { stage("Tests_B") {steps {echo 'from B'} } }
        stages("CLI") {  stage("Tests_C") {steps {echo 'from C'} }}

      }
    }

  }

I want to create something like this

Got answer from StackOverflow and here is the link to the answer.

stage('Test') {
    parallel {
        stage("Test_A") {
            stages {
                stage("Tests_A") { steps { echo 'from A' } }
                stage("Archieve") { steps { echo 'from Archieve' } }
            }
        }
        stage("Test_B") {
            ...
        }

        ... and so on
    }
}


How to terminate EC2 instance in Jenkins via Vagrant when job is stopped manually? I've freestyle jobs which aims to run EC2 instances via Vagrant (vagrant up), however when I cancel the job manually (by clicking X button), the instances on EC2 are still running. How can I make sure that the instances are terminated when the job is cancelled?

This can be achieved by using Post build task plugin and by adding the following Script command in Post-build Action, Post build task solved the problem:

vagrant destroy -f

Plus adding rule with Log text as "Build was aborted" make sure that above command is only invoked on aborted builds.

For example:


How to make a goreleaser script to build a deb file locally? I am an absolute novice in Golang but I want to modify a Go build script to build a file locally only instead of publishing it to GitHub.

https://github.com/dahendel/docker-machine-driver-cloudstack/blob/master/.goreleaser.yml

How to proceed?

I've created a fork with a Dockerfile to capture the release environment I can so far "reverse-engineer" but as I run it I get the following error and don't know how what and how to modify in the .goreleaser.yml.

   • releasing using goreleaser 0.117.1...
   • loading config file       file=.goreleaser.yml
   • RUNNING BEFORE HOOKS
   • LOADING ENVIRONMENT VARIABLES
      • pipe skipped              error=publishing is disabled
   • GETTING AND VALIDATING GIT STATE
      • releasing v1.0.5, commit d47a87ff8c671ec70b99a125fd6aadc45949905d
      • pipe skipped              error=disabled during snapshot mode
   • PARSING TAG
   • SETTING DEFAULTS
      • LOADING ENVIRONMENT VARIABLES
      • SNAPSHOTING
      • GITHUB/GITLAB/GITEA RELEASES
      • PROJECT NAME
      • BUILDING BINARIES
      • ARCHIVES
            • DEPRECATED: `archive` should not be used anymore, check https://goreleaser.com/deprecations#archive for more info.
      • LINUX PACKAGES WITH NFPM
            • DEPRECATED: `nfpm` should not be used anymore, check https://goreleaser.com/deprecations#nfpm for more info.
      • SNAPCRAFT PACKAGES
      • CALCULATING CHECKSUMS
      • SIGNING ARTIFACTS
      • DOCKER IMAGES
      • ARTIFACTORY
      • S3
      • BLOB
      • HOMEBREW TAP FORMULA
         • optimistically guessing `brew[0].installs`, double check
      • SCOOP MANIFEST
   • SNAPSHOTING
   • CHECKING ./DIST
   • WRITING EFFECTIVE CONFIG FILE
      • writing                   config=dist/config.yaml
   • GENERATING CHANGELOG
      • pipe skipped              error=not available for snapshots
   • BUILDING BINARIES
      • running hook              hook=dep ensure
   ⨯ release failed after 0.01s error=pre hook failed:

I have cloned your repo and try that in my local machine. Here is the steps :

Git clone
executing Dry run (testing everything before doing a release "for real" :

$ goreleaser release --skip-publish
show there is no error

 SIGNING ARTIFACTS
  • pipe skipped              error=artifact signing is disabled
• DOCKER IMAGES
  • pipe skipped              error=docker section is not configured
• PUBLISHING
  • pipe skipped              error=publishing is disabled
• release succeeded after 20.75s

execute goreleaser for release

$ goreleaser release
goreleaser will created dist folder inside project and this folder will consist of distribution packages (deb, rpm).

I have encounter some issues and here is what I do :

error=missing GITHUB_TOKEN, GITLAB_TOKEN and GITEA_TOKEN

create github or gitlab token ( https://github.com/settings/tokens) and put it as environment variabel

export GITHUB_TOKEN=xxxxyyyyyzzzzz

resolve the issue.

pre hook failed: xxxx is not within a known GOPATH/src

as I see in your goreleaser.yaml

hooks:
pre: dep ensure

you're using dep ensure, checking $GOPATH and make sure $GOPATH pointing to right path of your Go project.

error=dist is not empty, remove it before running goreleaser or use the --rm-dist flag

dist folder has been created before, you can either manually delete the folder or add flags --rm-dist when executing goreleaser command

$ goreleaser release --skip-publish --rm-dist

error=nfpm failed: rpmbuild not present in $PATH

this error occured as I was running on mac machine so there is no rpmbuild installed, installing rpm, rpmbuild solve the issue

$brew install rpm

error=git is currently in a dirty state, please check in your pipeline what can be changing the following files:
M Gopkg.lock

Goreleaser seems to check file diff, so as because running hook (dep ensure) updating the Gopkg.lock and this changes/updates are not pushed to git. The solution is always pushing the changes to git.

error=git tag v1.0.5 was not made against commit 3ae83eebd904d33cc549117254e857ebea04df90

reading from GoReleaser documentation which is "GoReleaser enforces semantic versioning and will error on non-compliant tags. Your tag should be a valid semantic version. If it is not, GoReleaser will error."

after pushing to git, make sure you have to update the tags, in this case I updates the tags to v1.0.6 (previously v1.0.5).

error=GitHub/GitLab/Gitea Releases: failed to publish artifacts POST http://xxxyyyzzz/releases: 404 Not Found []

make sure release text is there.


Azure Devops Pipelines Triggers not working I have a Repository in Azure Repos which has branches as master, build, feature. Build Branch and Feature branches have their own pipelines code in the azure-pipelines.yaml in the branch itself respectively. So that they can have their dedicated pipelines

The problem is that When I commit to the Build branch or in the feature branch then, both the pipelines are triggered.

I have written in my azure-pipelines.yml

trigger:
  branches:
    include:
        - build
    exclude:
       - feature

If you have set a CI trigger make sure you have set the filter to the exact branch which you need the build to be triggered and not the entire repository which is the default case.


How to manage services inside the container? Systemctl/service commands are not working on a container which is built by using centos:centos7 image.

Can anybody help me here?

Using systemd in Docker is not really that straightforward.

However, according to the doc on the docker hub, systemd is now included in both the centos:7 and centos:latest base containers.
However, you will need to include text similar to the example Dockerfile :

FROM centos:7
ENV container docker
RUN (cd /lib/systemd/system/sysinit.target.wants/; for i in *; do [ $i == \
systemd-tmpfiles-setup.service ] || rm -f $i; done); \
rm -f /lib/systemd/system/multi-user.target.wants/*;\
rm -f /etc/systemd/system/*.wants/*;\
rm -f /lib/systemd/system/local-fs.target.wants/*; \
rm -f /lib/systemd/system/sockets.target.wants/*udev*; \
rm -f /lib/systemd/system/sockets.target.wants/*initctl*; \
rm -f /lib/systemd/system/basic.target.wants/*;\
rm -f /lib/systemd/system/anaconda.target.wants/*;
VOLUME [ "/sys/fs/cgroup" ]
CMD ["/usr/sbin/init"]

I strongly recommend to read the full description tab documentation on the hub. This will help you make it run correctly.


Jenkins file that uses docker-compose gives ambigous error I'm new to Jenkins and I want to automatize my current workflow a little bit. Right now I push it to the raspberry. Then via SSH I pull the changes, run docker-compose build and docker-compose up afterwards.

My jenkins file is:

pipeline{
    agent any

    stages{
        stage('build'){
            steps{
                script{
                    docker.withTool("default") {
                        sh 'docker-compose build'
                    }
                }

            }

        }

        stage('deploy'){
            steps{
                script{
                    docker.withTool("default") {
                        sh 'docker-compose up'
                    }
                }

            }
        }
    }
}

And when clicking 'build now' I get the following console-output:

[Pipeline] }
[Pipeline] // stage
[Pipeline] withEnv
[Pipeline] {
[Pipeline] stage
[Pipeline] { (build)
[Pipeline] script
[Pipeline] {
[Pipeline] tool
[Pipeline] }
[Pipeline] // script
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (deploy)
Stage "deploy" skipped due to earlier failure(s)
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
ERROR: No org.jenkinsci.plugins.docker.commons.tools.DockerTool named default found
Finished: FAILURE

I also tried running it with docker.withTool("docker") but that gave me the same results.

You need to install the Docker Commons plugin? https://plugins.jenkins.io/docker-commons/

Edit: According to this link for I would like to use Docker CLI inside of a Pipeline job

  You need to make sure that “default” points to a docker tool inside of
  Manage Jenkins> Configure System which will also run on the os of the
  slave.


Why is docker container recreated I'm testing Wordpress docker on an Amazon EC2 machine.
yesterday, all file based changes (like plugins, themes, etc) were gone. The Ec2 machine is up for 8 days, other containers are also up for 8 days, but the Wordpress container is like created 20 hours ago.

I'm not asking about volume mapping etc, I know that to really persist file changes in a container we should use volumes, my question is:

1-On what events the container is re-created? We've not used docker run when it's bee re-created
How can we see a log of all the events happening on the docker (and not in specific containers)?

EDIT:
Log from journalctl -u docker.service:

Jun 11 16:05:44 ip-172-31-39-113 dockerd[906]: time="2019-06-11T16:05:44.328303723Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete
Jun 13 04:05:30 ip-172-31-39-113 dockerd[906]: time="2019-06-13T04:05:30.306067658Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete
Jun 13 13:40:31 ip-172-31-39-113 systemd[1]: Stopping Docker Application Container Engine...
Jun 13 13:40:31 ip-172-31-39-113 dockerd[906]: time="2019-06-13T13:40:31.557328738Z" level=info msg="Processing signal 'terminated'"

Only the second line is almost at the time container has been deleted and re-created

EDIT2:
output from docker ps -a:

CONTAINER ID        IMAGE                              COMMAND                  CREATED             STATUS                    PORTS               NAMES
3aebfdef77c8        wordpress:latest                   "docker-entrypoint.s…"   24 hours ago        Exited (0) 15 hours ago                       site-wordpress

First, see Docker daemon’s log. Refer docker docs

If some request or command executed to the Wordpress container, it might had restarted through docker daemon.


Accessing k8s outside of VM I am running k8s cluster for evaluation purposes inside some vagrant box. I have external kubectl access. I was able to get PostgreSQL helm template running on the cluster. But now let say I want to access it outside of the k8s(or VM actually).

There is a problem that I am not very educated in the niceties of Host to Quest communication in VM not to speak about the k8s stack. By empirical research I have found that turning service to NodePort makes it available outside the k8s. But as I said the installed PostgreSQL service is ClusterIP so I want to access it outside aswell without finding a way to turn it to NodePort. So there is couple of noob questions here

what is difference between kubectl proxy and kubectl port-forward?
what is different when executing kubectl proxy from inside VM running k8s cluster and running the same kubectl command on the host?
what is the usual way to make services available outside of the cluster, when using the cluster just for development?
is this quote true?

  port forwarding is not needed when using private network and accessing guest from host with guest private ip, port forwarding is used to access guest with localhost/127.0.0.1 the ports are then forwarded

kubectl port-forward forwards connections to a local port to a port on
  a pod. Compared to kubectl proxy, kubectl port-forward is more generic
  as it can forward TCP traffic while kubectl proxy can only forward
  HTTP traffic. Generally speaking, using port forwarding you could get
  on your ‘localhost’ any services launched in your cluster.

https://stackoverflow.com/questions/58360323/difference-between-kubectl-port-forwarding-and-proxy
It does not make a lot of sence to run 'kubectl proxy' from inside VM running k8s cluster. Kubectl is your client tool, so stick with it outside of cluster range.
The way you would access production services, for the most part if your service is DB oriented stick with ClusterIP. If you want to externally access some Web/HTTP service expose it as a Load Balancer or an Ingress ( in case u have N of them ). If you want to test the DB you could just run a dummy pod and execute some sql-client commands from there.


Improving deployment process of agent software I'm a developer on an agent software team, researching ways to streamline the mass deployment of our agent on clients' machines. Currently the process involves running a script to determine the exact package needed, downloading a bsx from our company's FTP and running it, followed by running an executable with flags for registration.
I've looked into Ansible as a possible improvement, however I'm unclear on the value it provides in such a case. What's more, I'm not sure I'm the right person to be authoring the clients' playbooks for such a thing. At this point it would appear to me that developing our own script to further automate the process would be a better solution, and let the customer DevOps team deploy it in whichever way they see fit.
Before I present these findings to my managers, I would appreciate any opinions on the matter the community could provide.

ansible-pull would work well for this, though it makes the assumption that your clients are using ansible. That's not an assumption I would be comfortable making in your shoes.

At this point it would appear to me that developing our own script to further automate the process would be a better solution

I agree with that. Since you already have a script to figure out what needs to be downloaded, I'd extend that script to download the needed file(s) and trigger the installation/registration.


Prometheus-operator - detection of expiring certs failing I had a situation yesterday where my K8S master api cert has expired and I had Prometheus-operator active in the past weeks, but neither the 24 hours, neither the 7 days alert has triggered. I went to the dashboard and made the original querry, I got nothing. I cut it out and mod it to apiserver_client_certificate_expiration_seconds_count{job="apiserver"} > 0, I got the result of all nodes, I checked the timers and moded it to apiserver_client_certificate_expiration_seconds_count{job="apiserver"} > 14000000 so I wanted one that is off to be cut and it was, that doesn't help me, but I think there is a problem with the alert, anyone else fought that?
https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeclientcertificateexpiration
If this is supposed to monitor all endpoints that make a request to the api server for their certificates then this failed as I didn't get an alert for the expired certificate.

The alert is broken, it is fixed in further release version of the mixins, we need to apply on(job) infront of the histogram to fix it. Example: apiserver_client_certificate_expiration_seconds_count{job="apiserver"} > 0 and on(job) histogram_quantile(0.01, sum by (job, le) (rate(apiserver_client_certificate_expiration_seconds_bucket{job="apiserver"}[5m]))) <  60432131800 and yes, that means your certs are not being tracked...


How to run Ansible with sudo no password commands with become syntax When I run ansible against a host machine that has some commands I need to run as root, I can't run these with "become" because it runs the commands as root instead of just running them with sudo.

What I've been doing is just using command: sudo ...  and then suppressing warnings with

args:
  warn: no

is there a better, more standardized, way to run a linux commandline with sudo sans password?

When I run ansible against a host machine that has some commands I need to run as root, I can't run these with "become" because it runs the commands as root instead of just running them with sudo.

That's not true; become uses a variety of methods for privilege escalation, and defaults to using sudo.
Are you perhaps running into one of the known limitations with the become module, like trying to limit sudo access to certain commands?


Restart agent for each build How do I configure jenkins to restart the agent for each build?

I tried to use "launch method: launch agent via execution of command on the master" in combintation with "availability: take this agent online when in demand, and offline when idle" but it has two drawbacks:

(not critical) build starts with a delay of about one minute.
(critical) the node goes offline only a minute after the end of the build, and if another build starts at that time, the same still running agent will be used (even if there are several similarly configured free nodes).

There is even a plugin (scripted cloud plugin) that, according to the description, should be suitable, but it does not work, probably because its code is outdated.

Slave restarts the agent itself after each build? You can use Conditional BuildStep plugin to restart it on a defined condition.


Docker Hub automated builds and Travis CI I'm wanting to create a basic 'on new commits to master, rebuild the images and deploy to kubernetes' pipeline.

Docker Hub has its concept of automated builds on new commits, but I'd still need to use Travis CI to do the kubernetes deployment.

Is there a way to synchronise these - so that Travis doesn't start its scripts until Docker Hub has built the image?

Alternatively - I could just do all of the building on Travis CI. But Docker Hub doesn't seem to provide service accounts - or have I got that wrong? It seems like the only way to create a service account is to actually manually create a new username/password for service account. Is that the case?

What are the generally established ways of doing this?

If you're married to Docker Hub, you could use a webhook to notify Travis of the need to start it's job.

However, I solve this problem using Quay.io - it has the robot account (you refer to "service account") you need. It also has a richer set of notifications.

You could maintain your current workflow using webhooks to

create the repository on Quay or Docker Hub, have the image built on git events.
Create a webhook in the repository to send data to your Travis job when builds succeed

Or, you could build your images in the Travis job.

Push the initial image to a repository on Quay
Add a robot account to your repository in Quay
Add the necessary secrets to your Travis build (token and robot user name)
Configure Travis to build the image, run whatever tests you need and then push on green.
Add an after_success task to redeploy your app in k8s after the push.