How do I bootstrap an alpine image to work with a private repo proxy behind TLS? I am in a corporate environment behind a gateway that does DPI and often replaces the TLS certificate of most online resources with our corporate certificate. Many of our developers use images based on alpine linux to do their work and at peak times the network gets overloaded, builds crash because of failed apk add commands and whatnot. So I set up an instance of Nexus with a caching proxy for alpine apk packages.

However, the Nexus is served behind TLS with a certificate signed by our internal CA which is obviously not in the list of default trusted CAs. Without doing apk add ca-certificates I cannot do an update-ca-certificates and without trusting our internal CA I cannot do apk add. I would like to avoid exposing Nexus via insecure HTTP.

I can do a wget our.cert.server.com/CA.crt, but /etc/ssl/certs is empty and behavior doesn't change when I create /etc/ssl/certs/ca-certificate(s).crt and copy the certificate into there.

How can I (or our developers) bootstrap alpine images for work in our network without too much of a change to our workflow?

It turns out that /etc/ssl/cert.pem is the bundle scanned by apk to verify a repo behind HTTPS, so the following works:

wget our.cert.server.com/CA.crt
cat CA.crt >> /etc/ssl/cert.pem
sed -i 's,http://dl-cdn.alpinelinux.org/alpine,https://nexus-endpoint.company.com/repository/alpine-proxy,' /etc/apk/repositories
apk update


ansible aws windows ec2 - Internal Error: this connection module does not support running commands via sudo I am getting below error when I am trying to run one playbook which connects to Windows machine hosted on AWS EC2. I enabled winrm and win_ping worked fine but when I am invoking ansible-playbook I am not sure why it is using sudo as if the target machine is Linux host.

I tried disabling become=True in ansible.cfg but same issue occurred.

Is it compulsory to use ansible_winrm_transport=ntlm for the win_ping to work. I feel that is causing this issue.

Any help would be appreciated.

fatal: [10.0.150.18]: FAILED! => {"failed": true, "msg": "Internal Error: this connection module does not support running commands via sudo"}

[root@localhost ]# cat ansible.cfg
[defaults]
log_path=/var/log/ansible.log
ansible_winrm_operation_timeout_sec = 200
ansible_winrm_read_timeout_sec = 500

[privilege_escalation]
become=True

[root@localhost ]# ansible --version
ansible 2.2.0.0

[root@localhost ]# cat hosts
[localhost]
localhost ansible_connection=local

[marcel]
10.0.150.18  ansible_user=Administrator ansible_password=xxxxxxxxxx ansible_port=5985 ansible_connection=winrm ansible_winrm_server_cert_validation=ignore ansible_winrm_transport=ntlm

According to the documentation:

  Prior to Ansible 2.4, become would only work when ansible_winrm_transport was set to either basic or credssp, but since Ansible 2.4 become now works on all transport types.

You're using Ansible 2.2. Might that be it? The error message might be a bit misleading.


Amazon S3 bucket does not redirect folder URLs to index.html I have uploaded my HUGO static site into AWS S3 Bucket. I have also added the domain in cloudrfront.

The home index.html page works fine, but I cannot access my resources.
For example https://www.example.com works fine, but I cannot access https://www.example.com/posts/. I have to redirect links to the html file every time.

On localhost https://www.example.com/posts/ displays automatically the page ( without redirecting https://www.example.com/posts/ to https://www.example.com/posts/index.html.

Is there some property in cloudfront or Amazon S3 to replicate the behaviour of localhost and not have to change the links of the site?

CloudFront does not actually have this feature. You can redirect the root object https://www.example.com/ to https://www.example.com/index.html by setting DefaultRootObject, but there's nothing for subdirectories such as https://www.example.com/posts/.

However, you can run arbitrary code on CloudFront using Lambda@Edge, so it's possible for us to add this feature ourselves. With the "origin request" trigger we can rewrite the request to S3, changing /posts/ to /posts/index.html. https://github.com/CloudUnder/lambda-edge-nice-urls has some good handling of paths that works well with Hugo.

Another option is to enable uglyurls in your config so that all links within you site have index.html hardcoded in to them.


How to deal with docker compose's naming convention of named volumes? When defining named volumes in docker-compose.yml, their names are prepended with their parent folder name. This causes a problem when scripts outside of docker compose have to interact with them. The question is, what is the best way to deal with this?

An example scenario would include the following docker-compose.yml:

version: "3"
services:
  nginx:
    build: ./nginx
    container_name: nginx
    ports:
      - "80:80"
    volumes:
      - jekyll-data:/usr/share/nginx/html:ro

networks:
  backend:

volumes:
  jekyll-data:

Where the jekyll-data named volume is populated by the following bash script:

docker run \
  --name helper \
  --volume="parent_folder_jekyll-data:/web" \
  -it busybox \
  true
docker cp jekyll/web/. helper:/web
docker rm helper

In the above case, parent_folder is the name of the parent folder. This means that moving the contents to a different folder would break the application. Is there a proper way to deal with this situation?

The abridged output of docker volume ls where unnamed volumes have been removed:

DRIVER              VOLUME NAME
local               flaskthymedata_grafana-data
local               flaskthymedata_influxdb-data
local               flaskthymedata_postgres-data
local               veleda_grafana-data
local               veleda_influxdb-data
local               veleda_jekyll-cache
local               veleda_jekyll-data
local               veleda_postgres-data
local               veledaio_grafana-data
local               veledaio_influxdb-data
local               veledaio_jekyll-cache
local               veledaio_jekyll-data
local               veledaio_postgres-data

Docker Prepends the current folder name with all the components name created using docker compose file.

Eg : If the current folder name containing the docker-compose.yml file is test, all the volumes,network and container names will get test appended to it. In order to solve the problem people earlier proposed the idea of using -p flag with docker-compose command but the solution is not the most feasible one as a project name is required just after the -p attribute. The project name then gets appended to all the components created using docker compose file.

The Solution to the above problem is using the name property as in below.

volumes:
  data:
    driver: local
    name: mongodata

networks:
  internal-network:
    driver: bridge
    name: frontend-network

This volume can be referred in the service section as

services:
  mongo-database:
      volumes:
        - data:/data/db
      networks:
        - internal-network

The above name attribute will prevent docker-compose to prepend folder name.

Note : For the container name one could use the property container_name

services:
  mongo-database:
    container_name: mongo


Getting a 502 Bad Server Request on Nginx I'm trying run the frontend (react) and a backend (nodejs, express) from a same ubuntu 20.4` machine but for some reason my nginx is not mapping correctly. I don't know if I have a typo somewhere or just configured something wrong?
Inside /etc/nginx/sites-enabled/ I have created 2 files one for the frontend and one for the backend
The contents inside the backend file are the following:
 server {
    # server_name  ## add server name
      server_name  backend.develop.mysite.be; // Dummy name for SO, but actual file contains the right address

    location / {
        proxy_read_timeout 1800;
        proxy_connect_timeout 1800;
        proxy_send_timeout 1800;
        send_timeout 1800;

      proxy_pass http://localhost:9000;  // [ASK]: Is this is what is causing the problem? Maybe I should change it to 3000 ?
      proxy_http_version 1.1;
      proxy_set_header Upgrade $http_upgrade;
      proxy_set_header Connection 'upgrade';
      proxy_set_header Host $host;
      proxy_cache_bypass $http_upgrade;
    }

    listen 443 ssl; # managed by Certbot
    ssl_certificate /etc/letsencrypt/live/frontend.develop.mysite.be/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/frontend.develop.mysite.be/privkey.pem; # managed by Certbot
    include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot

}
 server {
    if ($host = backend.develop.mysite.be) {
        return 301 https://$host$request_uri;
    } # managed by Certbot

    listen 80;
      server_name  backend.develop.mysite.be;
    return 404; # managed by Certbot

}

the contents for the frontend are very similar just with a different name. In the past this setup worked for me, but for some reason right now I'm getting a 502 BAD Gateway response error but only for the backend. The frontend seems to work just fine.

Never mind, I just figured it out . I just needed to change the port i was refering inside the backend file /etc/nginx/sites-enabled/backend
Changed this line proxy_pass http://localhost:9000; to this line proxy_pass http://localhost:3000; and it worked.


Hosted CI/CD for iOS that's not in the US? TL;DR: Do you know of any hosted CI/CD providers for iOS, who have their datacenter/build-boxes in Asia, or at least Europe? (Bonus points if they provide both build and deploy, but build is the MVP.)

Back-story:

We're running CI/CD at-scale for iOS and Android: we run 10+ simultaneous builds both to test/verify our Merge-Request branches and to deploy trunk builds to our testers and stakeholders. We're using a SaaS/cloud provider we're pretty happy with... except for their location.

We're in Asia, as is our source. Client is in a highly-regulated industry and fighting hard to keep source on-prem, as they think their regulator can't handle source in the cloud yet. Please accept this premise. I understand why they need to let go of it. But for now... assume they can't.

What that means is: source is in Asia, but the CI/CD providers who build it are all seemingly in the US (Circle, Buddybuild, etc). Trans-pacific bandwidth sucks, particularly during the Asia business day. The time taken by every clone before every build spikes to >60 minutes for much of the business day.

Going on-prem for the CI/CD would be pretty easy for Android builds, on docker containers. But iOS is the problem. You either need to put yourself in the business of teaching people to admin OSX and keep a fleet of build-boxes operational, or you need to let specialists solve that problem for you.

NOTES:

I am not asking for recommendations, SE community! This is a factual technical question: are certain services, that meet certain technical requirements, available in a certain geography?
We know that MacStadium can provide bare MacOS VM's in Ireland. But that means you have to admin your own whole CI process, plus a lot of low-level system admin tasks we'd rather avoid. It also means separating build from deployment, for sure. But the latency appears to be acceptable.
We know of other people who have cloud CI/CD platforms closer to us... but without iOS/MacOS support.
We know that shallow clones require less bandwidth, which would mitigate the problem, but they also have other complications that mean our current provider doesn't support it yet. And they don't entirely solve the problem in any case.
We've experimented with using off-premise GitHub mirror, which solves some of the problem, but doesn't address the regulatory issues. It also doesn't work with many webhooks, and specifically the Merge-Request webhooks that are critical to our CI pipeline for new code. We could create an agent that would monitor the webhooks, then imperatively relay API commands to the other service-providers... but that's really jumping through hoops, plus we'd have created substantial new code to maintain.

One could run Jenkins on MacOS.

https://medium.com/@dixitakansha15/continuous-integration-in-ios-using-jenkins-8e081753cc72

https://savvyapps.com/blog/continuous-integration-ios-jenkins


How do I setup an OpenShift route that will allow both 80 and 443 traffic to my app? I'd like to setup a route that will pass both 80 and 443 traffic to the same service in OpenShift. When I try to create a second route it errors out. I want this to be in a yaml like would be used with oc create -f route.yaml so I can submit it to our IaC code repo.

You'll want to be sure your spec.tls section has the following two items to tell the router you want both secure and insecure traffic allowed in the edge.

apiVersion: route.openshift.io/v1
kind: Route
metadata:
  name: my-app-ui
  namespace: my-namespace
spec:
  host: my-host
  port:
    targetPort: http
  tls:
    insecureEdgeTerminationPolicy: Allow
    termination: edge
  to:
    kind: Service
    name: my-app-ui
    weight: 100
  wildcardPolicy: None


Azure Devops - workflow/strategy for uploading only changed files to shared hosting provider after a build of static website (Jekyll) Scenario:
I'm using the Ruby/Jekyll build process to generate a static website from markdown etc files via a templating process -- the same flow as the one used by Github Pages but instead of hosting it on Github I want to be able to upload it to my (existing) shared hosting provider and serve the files from there. (I already have DNS and everything in place with the web host.)
The hosting provider offers FTP and SSH via cPanel but no direct hook in to Git.
The build process (which I am doing manually for now, but will eventually become an automated build process and no doubt generate more questions!) creates/updates the static files in a '_site' directory within my repo.
My local "CI" build (on my own pc) uses a development branch and then when happy with the changes I push them up into master.
Where I'm stuck:
I want to be able to use an Azure Devops release pipeline to get the files into my web host, triggered by a merge into master.
I have had this running successfully (triggered manually, but I don't think the trigger part is a problem) using an inbuilt FTP task to grab the contents of the '_site' directory as an artifact and FTP them up to the web host.
However the FTP task (and the artifact) isn't very 'smart', in that it will re-upload everything every time I want to deploy even if only a few files have changed. This obviously takes longer and eats into my (metered) bandwidth for uploading (and seems like it is not good DevOps!)
Most often, only a small subset of files are actually changed by the build. This is usually where a new page is created or a stylesheet changed for example. Occasionally, all files are actually changed, such as if an "included" component changes so that all the static pages have to be re-rendered.
What I'm looking for is an approach or workflow to just upload the changed (and new) files, rather than all the files. It sounds like it ought to be easy, so perhaps I am missing something.

I don't think you are missing any easy solutions because

the easy solutions probably rely on you having full access to the webserver -- which you don't have? The challenge is what is the best you can do given limited server access.
FTP does not have a reliable, built-in way to only upload changed files.

I think:

your only solution for problem 1 is to write your own shell script

problem 2 is usefully discussed here: https://stackoverflow.com/questions/424680/trying-to-find-a-simple-way-to-do-upload-only-modified-files-through-ftp .

The viable solutions both amount to 'save hashes of the files on the server, then compare before upload'. However if you can get your pipeline to install and use git-ftp then git-ftp will do all that for you.


Logging & Monitoring on multiple servers I lead a small project with 2 groups of servers (servers in a single group are clones with the same microservices, like pods in the k8s), a balance loader, and a DB. Docker & k8s aren't used because of small scale of the project (but there are plans for migration). Instead, our cloud provider allows to set dynamic servers creation/removal depending on the load using filesystem image. This is fine for now, but there is a lack of logging and monitoring, I tried to structure the requirements:

A dashboard with server parameters such as CPU, RAM, disk space, network, etc usage for each server. Some dashboards with such technical information are available by our provider, but a provider-independent solution is desired.
A dashboard with numbers of responses by HTTP status code, for each server group in total and each server separately. We use NginX and it provides a status page, but it works on separate servers only.
A logs analysing & digging window, with support of multiline messages (like exception tracebacks).
Programming language independence. Our server apps write logs to several different files with a date in filename, probably this will have to be somehow changed.
Ability to set email (or even launching a shell script?) notifications with custom triggers.
Maybe there are some other important features?

Additional wishes:

Logging/monitoring setup should be free to use.
Simplicity is preferred. I hope a single solution can be flexible enough to cover all the needs of logging & monitoring.
Solution maturity / popularity and availability of docs,examples is important.
The solution should persist when migration to containers and orchestration will come.

I imagine the final result to be 1) a server that accumulates all the logs, provides dashboards, and logs digging panel, and 2) a script that is installed on each server, is run by SystemD timer, and periodically sends local logs to the server. But how the script will know which exact log files are presented on the current server install, using some config? Though I can be wrong about this algorithm.
I've heard about LogStash, ElasticSearch, Grafana, Kibana, Zabbix, Loki, Prometheus, etc, tried to google everything out, but found a total mess of recommendations. I'd to see modern explanations of different technologies, stacks, what software can be used together or is analog of another, and a comparison by the mentioned needs.
I also hope that the answers will be very helpful to many other developers and beginning project leads for the topic is highly important.

Well, I spent hours reading lots of articles, documentation, and finally figured everything out. See the comparative table:

Brief description
Zabbix is a full-fledged technology that can deal with both metrics and logs processing, yet can be extended. It also supports automatic hosts detection & network maps building.
Prometheus is a metrics monitoring system. It is Kubernetes native, thus it can automatically detect hosts for data extracting. It also has myriads of available adapters: docs: Exporters And Integrations. To visualize the data, Grafana is used. Speaking about data storage, Prometheus has its own local system for that, but there are integrations with other technologies for sharding, see docs: Storage.
PLG is a stack for log aggregation. Logs get collected with PromTail, Loki is a core processing engine, and Grafana is used as an interface. Loki has no own storage system, so some other solution is required. It stores data as chunks and indexes, which are usually stored separately docs: Storage. As I understood, PLG has no trigger/notification system, but the data extracting part also can parse logs into metrics and export them into Prometheus, so we get alerts. Also, I met a notice that PromTail doesn't support multiline logs, but FluentD can be used instead as it is more flexible.
ELK / EFK is a technology stack for logs parsing, indexing, searching, so it can perform complex queries. For data extracting, LogStash was originally used, but FluentD is becoming popular.


No Such Host: Kubernetes/Docker cannot pull from private k8 registry I have a private docker registry in k8 in the default namespace with tls at https://docker-registry.default:5000. By doing local port forwarding to it(at port 5000) and adding docker-registry.default to my /etc/hosts file, I have been able to pull and push images to it.

However, I'm using a serverless framework(Nuclio) which manages the deploy for me.
The error I get is:

Normal   Scheduled  7m52s                   default-scheduler                                       Successfully assigned nuclio/helloworld1-96876fcf9-7c87w to gke-your-first-cluster-1-pool-1-fe915942-gnkx
  Normal   Pulling    6m14s (x4 over 7m51s)   kubelet, gke-your-first-cluster-1-pool-1-fe915942-gnkx  Pulling image "docker-registry.default:5000/docker/nuclio/processor-helloworld1:latest"
  Warning  Failed     6m14s (x4 over 7m51s)   kubelet, gke-your-first-cluster-1-pool-1-fe915942-gnkx  Failed to pull image "docker-registry.default:5000/docker/nuclio/processor-helloworld1:latest": rpc error: code = Unknown desc = Error response from daemon: Get https://docker-registry.default:5000/v2/: dial tcp: lookup docker-registry.default on 169.254.169.254:53: no such host
  Warning  Failed     6m14s (x4 over 7m51s)   kubelet, gke-your-first-cluster-1-pool-1-fe915942-gnkx  Error: ErrImagePull
  Normal   BackOff    6m2s (x6 over 7m51s)    kubelet, gke-your-first-cluster-1-pool-1-fe915942-gnkx  Back-off pulling image "docker-registry.default:5000/docker/nuclio/processor-helloworld1:latest"
  Warning  Failed     2m38s (x21 over 7m51s)  kubelet, gke-your-first-cluster-1-pool-1-fe915942-gnkx  Error: ImagePullBackOff

It gives a no such host error.

A Nuclio developer has confirmed that it's not a nuclio issue since I cannot even do a docker login from the Nuclio Admin pod(not the failed pod above) to this internal docker-registry.default though I can do a docker login to the Docker hub and a wget to docker-registry.default successfully. The Nuclio Admin pod is Alpine Linux if it helps.

It seems your private DNS is misconfigured. Check your host or make sure that docker-registry.default is the a valid hostname.
You can run a temporary Ubuntu pod to check if that hostname is resolving correctly from your cluster, e.g.
$ kubectl run ubuntu -it --image ubuntu --rm=true -- bash
root@shell:/# apt update && DEBIAN_FRONTEND=noninteractive apt --yes install dnsutils
root@shell:/# dig docker-registry.default
docker-registry.default.    IN  A D0.y0u.h4ve.1P?

Additional suggestions:

If you're using a cloud provider (such as Azure), make sure your Private DNS zone is linked with your cluster's virtual network. Read: Connect privately to an Azure container registry using Azure Private Link
If your container registry endpoint isn't correct, edit the deployment manifest and fix the image path.
Test pulling the image from another VM within the same private network.


How do I SSH into an EC2 instance from AWS gitlab-runner? I have setup my own gitlab-runner on AWS following GitLab's walkthrough (Autoscaling GitLab Runner on AWS EC2).
One step in my deployment is to SSH into another EC2 instance. The SSH connection fails every time (see below for the one exception).
I have added an Inbound rule to my security group for the EC2 instance that allows SSH connections from the security group for both the gitlab-runner manager and the gitlab-runner machines. Still SSH connections are refused. Allowing SSH traffic from 0.0.0.0/0, however, causes the SSH connection from the gitlab-runner to the EC2 instance to succeed.
Why doesn't allowing my gitlab-runner security group in the inbound rules work, and how do I fix this so that I don't have to open my EC2 instance to the world to get SSH access from gitlab-runner?

When you use public IP the traffic exits your VPC and comes back in, at which point it is no longer identified as coming from that security group.
So you can adjust your ssh connection and use private IP


When your micro service in production needs a secret, where is the secret stored and retrieved from? The context of the question is Secrets management.

How do you manage storing and retrieving secrets for your micro service which is in production?

Short answer

Ideally, you should store secrets as environment variables, and retrieve them from a secrets management system like Hashicorp's Vault or AWS Parameter store.

Long answer

I saw your questions out of turn, and kinda touched on this in your other question:

  Again, there are many perfectly valid options for handling secrets: Chef vault, 1password, environment variables injected by your build instances, home-grown solution using encrypted files stored on object storage, etc....

It really depends on how you run and deploy your apps. I worked at a company that used Amazon Beanstalk for production, and they shipped all their secrets and configs as a 'secrets.conf' file that was created by the Jenkins build server, and shipped with the app when it deployed. The problem there was that it was unruly to manage the same secrets across multiple jobs in Jenkins. So, I came in and moved them towards using Chef to create the file instead, and Chef pulled the file from Hashicorp Vault.

At my current company, they are heavy users of GitLab with it's built-in CI/CD build system. Because the the apps are all Docker instances running in Kubernetes, all secrets are actually set as environment variables set during the deployment. The build server gets those from 'secret variables' that is set from the GitLab UI.

These fit the business need at the time, and that's what really matters at the end of the day. However, you if want to be proper, you should store secrets as environment variables because you specifically mention micro services. It is highly recommend that you follow the 12 Factor App approach to configs: https://12factor.net/config

  The twelve-factor app stores config in environment variables (often shortened to env vars or env). Env vars are easy to change between deploys without changing any code; unlike config files, there is little chance of them being checked into the code repo accidentally; and unlike custom config files, or other config mechanisms such as Java System Properties, they are a language- and OS-agnostic standard.

So you'll definitely want to leverage environment variables. How you set those environment variables could be your Configuration Management system (Ex: Chef), or your build server when it does the deploy, or your apps could be coded to pull directly from your secret storage and set the variables internally.

For actual place where the secrets live (where they are retrieved from), you have a few options. But the 2 you mentioned in your comment would be really great ones to use: Hashicorp's Vault and AWS Parameter store. Others would be Chef Vault, home-grown solution using encrypted files stored on object storage, 1password, etc.... Ideally, you'd want something that can be programmatically access from your build pipeline or apps.


How to avoid the flakyness that comes with "gpg --recv-key"? If you want to verify a signature of a downloaded file with gpg, you first have to import the key. Unfortunately, this operation is extremely slow and flaky in practice.

For example, here I tried it out:

$ gpg --keyserver pgpkeys.mit.edu --recv-key A0E98066
gpg: keyserver receive failed: No data

$ gpg --keyserver pgpkeys.mit.edu --recv-key A0E98066
gpg: key B550E09EA0E98066: public key "Yichun Zhang (agentzh) <agentzh@gmail.com>" imported
gpg: marginals needed: 3  completes needed: 1  trust model: pgp
gpg: depth: 0  valid:  14  signed:   0  trust: 0-, 0q, 0n, 0m, 0f, 14u
gpg: next trustdb check due at 2019-01-13
gpg: Total number processed: 1
gpg:               imported: 1

The first run failed. When I repeated, it took very long but eventually succeeded. Especially, in a CI environment, it breaks a lot, that is why I am looking for alternatives.

Question:

How do you avoid the dependence to the remote keyserver?
Is there an easy way to export the key as a file and use it later to import it in the CI environment without making a request to the keyserver?

Download the key once:

gpg --keyserver pgpkeys.mit.edu --recv-key A0E98066
gpg --export A0E98066 > openresty-agentzh-A0E98066.gpg

Then later you can import the key without having to access the keyserver:

gpg --import openresty-agentzh-A0E98066.gpg


Jenkins returns 502 gateway error and (104: Connection reset by peer) So my server got maintenance schedule from cloud provider, after maintenance process, the server rebooted, then my jenkins cannot be accessed from the web anymore, I am using nginx and checking the logs I found this.

  /var/log/nginx/error.log

2018/03/26 16:11:00 [error] 21689#21689: *1 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 139.139.139.139, server: jenkins.domain.com, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:8080/", host: "jenkins.domain.com"

2018/03/26 16:11:00 [error] 21689#21689: *1 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 139.139.139.139, server: jenkins.domain.com, request: "GET /favicon.ico HTTP/1.1", upstream: "http://127.0.0.1:8080/favicon.ico", host: "jenkins.domain.com", referrer: "http://jenkins.domain.com/"

here is my nginx conf file:

upstream app_server {
server 127.0.0.1:8080 fail_timeout=0;
}

server {
  listen 80;
  server_name jenkins.domain.com;
  root /var/www/jenkins.domain.com;

  location ~ /\.well-known/acme-challenge {
        default_type "text/plain";
  }

  location / {
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Host $http_host;
        proxy_redirect off;

        if (!-f $request_filename) {
            proxy_pass http://app_server;
            break;
        }
    }
}

What is the cause of this and maybe what can I do to fix this? Really need your help guys.

Thanks

If the system recently rebooted and your proxy server is giving 502/503s, it's most likely that the backend service failed to start.

Using whatever tools are most appropriate for your OS/distro (e.g. systemctl, service/chkconfig, stop/start/restart/etc.), do the following things:

Verify that the backend (in this case Jenkins) is configured to start on boot.  Enable it if not.
Start the backend.
If the backend fails to start, check the application logs to find the source of the error.  For Jenkins on Linux, this would be /var/log/jenkins/jenkins.log.


Azure DevOps Different Post Deployment script per environment I have a visual studio database project being built and released through Azure DevOps.

I need to execute a different post deployment for Dev vs. Prod.

Is there a way to achieve this either through the release pipeline or the visual studio project?

You can define multiple stages in a release pipeline. Each stage (i.e. Dev, Prod) can have its own set of tasks and/or other factors that differ. If you want the same set of tasks, you can duplicate a stage then change what's needed. Alternatively, you can configure the tasks using pipeline variables. The variables can be defined to have different values based on the stage scope (or applicable to the release scope). This way, each stage is essentially the same set of tasks but at runtime each behaves differently based on the variable values applicable to it at runtime.
See the MS docs on stages based on your pipeline type preference:

YAML
Classic


RUN fails while installing Angular on Ubuntu Image I am trying to create an image for my angular application for running unit and e2e tests. I started from the ubuntu image and then installed the other components. My dockerfile has the following statements.
FROM ubuntu:focal-20201008

RUN apt update && apt install -y \
    nginx \
    nodejs \
    npm \
    && npm install -g @angular/cli@9.1.12 \
    && rm -rf /var/lib/apt/lists/*

When I build the image using
sridhar@sridhar-HP-Laptop-15-H:~/abui/$ docker build --tag abui .

I get the following message (last few lines pasted here)
npm WARN deprecated request@2.88.2: request has been deprecated, see https://github.com/request/request/issues/3142
npm WARN deprecated har-validator@5.1.5: this library is no longer supported
/usr/local/bin/ng -> /usr/local/lib/node_modules/@angular/cli/bin/ng

> @angular/cli@9.1.12 postinstall /usr/local/lib/node_modules/@angular/cli
> node ./bin/postinstall/script.js

The command '/bin/sh -c apt update && apt install -y     nginx     nodejs     npm     && npm install -g @angular/cli@9.1.12     && rm -rf /var/lib/apt/lists/*' returned a non-zero code: 1

P.S. I don't want to start from the node docker image as of now

Finally managed to generate the docker image for Angular. Here are is the screnshot.

I am listing the contents of the Docker File along with the comments that explain the reasoning and my learnings.
# Created by Sridhar Pandurangiah - 29 October 2020
# This Docker file helps create the containers for the Angular User Interface
#1 We would like to create from the node image and then add the layers. I had initially started from a base image and proceeded to add Ubuntu, it did not work out as creating a ubuntu from the base involves a lot of work and its rather baseless to start from a base image. Then switched to starting from the Ubuntu image and the problems I faced is documented in this question. So had to start from the node image. Also remember that docker containers have to address a single concern.

FROM node:15.1.0 as ab-suite-image

#Add Information about this image
LABEL maintainer="sridhar@st.in"
LABEL version="1.0"
LABEL decsription="This is a docker image built from the node image and includes angular9 and metronic theme"

#2 Add Angular and other peer dependencies that do not get installed by default to avoid the build from exiting with errors
RUN npm install -g @angular/cli@9.1.12 @angular/core @angular/common eslint chokidar jquery popper.js tslib --save

# Set environment variables for file locations if required

#3 Set the environment Variable to determine the environment where the application is deployed development | testing | staging | production
ENV AB_COMPLIANCE_ENV=development

#4 Set the working directory to copy the angular dependencies
WORKDIR /opt/ng

#5 We don't pull the code from GIT as it would mean storing the private ssh keys on a docker image that can be publicly accessed. This is is not advisable.
COPY package.json ./

#6 Update project dependencies using package.json. This stage is where we encountered a lot of issues with build failing as the shell scripts were returing errors. the flag --legacy-peer-deps ensures that the npm doesn't throw errors so the docker build continues
RUN npm install --legacy-peer-deps

#7 Copy all the other project files and build
COPY . ./
RUN ng build

#8 Configure Volumes if required

#9 Copy start Script if required
#COPY start.sh /start.sh
#CMD ["./start.sh"]

#10 Expose Port for the application
EXPOSE 4200 443

This docker image can be used to for running unit and E2E tests.
Some literature on the web ask you to install nginx but not advisable. Run nginx on a seperate dedicated container. On production you can build the angular application with the "prod" flag and copy the files so that nginx can serve them.


Build FreeBSD Docker image using Ubuntu host OS I know that in generally it is impossible to build FreeBSD Docker image on Linux host OS due to different kernels and sys calls. Relevant discussion.

However, I would like to know maybe there is a simple workaround to overcome this limitation?

P.S. I don't want to consider Vagrant usage inside Docker container and I don't want to consider usage of FreeBSD as host OS.

As you can't build a docker image for an arm kernel from a x86 one neither.

Nor can't you build an image with a glibc lib targetting a 4.x kernel on a host running a 3.X kernel.

Docker isolate processes and give a virtual filesystem but that's still the host kernel who own the process, so obviously without a layer of hardware virtualization like virtualbox, vmware, xen, kvm, there's no chance to build an image with binaries made for a different kernel type.


docker mariadb access from remote I created a docker container with docker-compose in my VPS and I want to access it from my local computer but the connection will refuse. also, I allowed the 3306 port in the VPS.
my docker-compose.yml file:
version: "3"

services:
  mariadb:
    image: mariadb
    container_name: mariadb
    restart: 'always'
    volumes:
      - ./conf:/etc/mysql
    environment:
      MYSQL_ROOT_PASSWORD: ${password}
      MYSQL_DATABASE: ${database}
      MYSQL_USER:  ${username}
      MYSQL_PASSWORD: ${password}
    ports:
      - 3306:3306

how may I access the MariaDB server from my local computer?

Check netstat -tlpn on the VPS, can you see 0.0.0.0:3306 or something like that? If so, check the firewall. Both the firewall in the operating system and the firewall from the VPS provider (if there is any). As a last resort (and not a bad idea) - you can connect to your VPS via VPN - there are good openvpn tutorials on the internet. The reason why the VPN is a good idea - especially for a development server is that you may need to allow the user from everywhere (if you do not have a static IP address) and if you accidentally create some 'testuser' with 'testpassword' anybody from the internet will be able to connect to your db. So use good passwords, limit access to your public IP and use VPN if possible.


Container runtime Vs orchestrator The terms container runtime and orchestrator always confuse me. Can someone please put these in simple terms?

Thanks

A container runtime is the part of your container environment in charge of the creation and basic features of your containers. The obvious one is Docker, but you can also find Containerd, CRI-O and other as runtime for your containers.

An orchestrator, by contrast, will not exactly create your container (ie, an orchestrator is not the technology used to create them). An orchestrator will use your container runtime to manage them. In a way, it's a layer around your container runtime wich allow finer usages, such as scaling, multi host deployments, load balancing...

The most famous orchestrators would be Kurbenetes, Docker Swarm...

I hope this help! Have fun.


Istio Virtual service - enforce specific api version I am having troubles creating an object within specific api group it gets switced from alpha to beta, why? networking.istio.io/v1beta1 vs networking.istio.io/v1alpha3

Note: I can control in the crd virtualservice.

I have managed to solve it myself later that day, I had to disable one of the apigroups within the specific crd that is responsible for the given object.
In that case it was editing:
kubectl edit crd virtualservices.networking.istio.io
And then change:

  versions:
  - name: v1alpha3
    served: true
    storage: true
  - name: v1beta1
    served: true
    storage: false

to

  versions:
  - name: v1alpha3
    served: true
    storage: true
  - name: v1beta1
    served: false
    storage: false

Then on redeploy I had the correct apigroup which I wanted.


Should I run CI on merge state or HEAD of branch in a PR? I'm using Jenkins Pipeline with Github for our CI purposes and I'm trying to think what the strategy should be for what is checked out to run.

Jenkins provides three options when discovering pull requests:

 1. Merging the pull request with the current target branch revision
 2. The current pull request revision
 3. Both the current pull request revision and the pull request merged with the current target branch.

I'm looking for opinions on what others choose. To run just the branch head because then it's clear exactly what is being run, or to run the merge state because then it makes you more confident when you merge.

Currently, I'm using just the PR branch HEAD and enabled Github to require a master merge if the branch is out of date.

Usually you want to test your code as closely as possible to how it's going to be deployed.  This means you're going to want to run your tests against code that's been merged with master, so choose the first option (merge first, then test).  There's no point in the 2nd option if you're never going to deploy the code in that state (i.e. unmerged).

Or for another perspective: good CI practices involve merging soon and merging often.  This would naturally lead one to the first option.


You're looking at a hex-escaped string.  After some googling, I found this entry on server-fault:  https://serverfault.com/a/758511/90491.

You're looking at a non-decrypted HTTPS request.

From Link:

  You are most likely seeing this because you are making an HTTPS request to an HTTP endpoint. For example, you're sending an HTTPS request to port 80 of your web server instead of 443. As a result, the HTTP endpoint gets a bunch of encrypted data, makes no effort to decrypt it (since HTTP is supposed to be plaintext), and so you get a bunch of gibberish in the log file.


Is it allowed to remove mounted directory from inside Docker container? I have a container which is based on tomcat image and I have mounted 'webapps' folder to hosted machine directory '/app/webapps'. Now for some updations I want to remove this webapps folder from inside the container but when I execute command: rm -rf webapps below error is produced:

  rm: cannot remove 'webapps': Device or resource busy

So I want to know is it possible to remove mounted directory by executing a Linux command from inside the container or not?

No, you sadly can't remove a volume that is in use by a container which is stated under 'Extended Description' here: https://docs.docker.com/engine/reference/commandline/volume_rm/

However, I'd suggest using docker for its strength and restarting the docker container after updating the directory externally.


How to handle a new Dev Ops position? At my current job, I started as a Software Developer, and I've worked myself into a DEVOPS position. I am worried about getting fired, not because I'm doing something wrong, but because I'm used to proactively doing things(writing code), whereas most of my time now is sitting around waiting for things to happen....
My daily duties(and I asked my manager in a detailed email listing these same ones) are,

Monitor 5 different servers' performance.
Monitor 2 of those servers Active Batch execution agents
Handle requests as they are created.
Handle project tasks as they are created.
Monitor Email/Skype/Phone
Build whatever is requested that isn't directly assigned thru the ticketing system.

It seems like a lot at first glance, but what is actually involved is little more than watching 1 monitor for new errors, and another monitor for messages. I can usually complete tickets in minutes-1 hour. Is this normal for Dev Ops-just sitting around waiting? If so, what can I do to be more useful? If not, am I in danger of getting fired?

I agree with those who have commented that the role you describe doesn't really match a typical DevOps role. However, there are certain things you won't be able to change in your organization. Instead, focus on the things you can do.
Understand the Objectives
The best place to start is with a clear understanding of Service Level Objectives. What kinds of things do external customers actually care about? If they don't care about uptime, don't make uptime measurements a high priority. Perhaps latency, software updates, content accuracy, or some other measure is more important. Also be aware of your internal customers' needs (your boss, coworkers, etc). What things do they care about?
Once you understand what is most important to your customers, measure, measure, measure! Get as much data as you can about those items that matter. How long does it take to deploy code? When you close a ticket, what percentage of the time does it come back unresolved? And so on. Use those measurements to improve over time.
Here are some areas you can focus your efforts (varies, depending on the company you work for):
Configuration
If someone logged into a server and broke the configuration, how hard would it be to correct the error?

You should be able to consistently build and replace resources with minimal effort.
Configuration management should be complete so that all meaningful changes are tracked.
Ideally, build up processes or tooling as needed so that developers can modify configuration themselves in a controlled and collaborative way (:D DevOps!).

CI/CD
When new code is released, how does it get delivered?

When code is pushed to source control, it should trigger automated testing and deployment. If possible, this should happen in a pipeline that deploys multiple environments in sequence as tests pass (dev -> qa -> uat -> prod).
Improve visibility so that you can easily see where different versions of the code are running, what errors are occurring, and how performance is changing.
Enable your software developers to deliver code themselves (:D DevOps!).
Collaborate with developers to improve this process (:D DevOps!)

Logging/Metrics
When something goes wrong, how quickly can you find out what the problem was? If something went wrong 2 months ago, can you still find out what the problem was?

Collect logs in a central place with tools that help you search and visualize.
Aggregate metrics with tools that help visualize.
Reduce noise! If it's not meaningful, don't keep it (but be a little conservative, sometimes things turn out to be meaningful later).
Work with developers to understand how logging and metrics would help them deliver better software (:D DevOps!).

Alerting/Incident Response
Ideally, you should not ever need to look at a graph, dashboard, or log message, unless there's an actual problem to solve. Sitting and watching a dashboard is not a good task for humans.

Set up alerting so that you get notified when problems arise.
Alerts should go to ONE person, who is the RIGHT person.
Alerts should be urgent (needs to be addressed immediately), and actionable (the person getting the alert can fix it).
In the spirit of paging the right person, you may be able to work with developers on getting them into some kind of on-call rotation (even just M-F 8-5 for a subset of code issues). (DevOps?)

Disaster Recovery
If your servers were destroyed today, how long would it take to be up and running again?

Take backups of data and set up backup infrastructure.
Test backup recovery regularly.
Test failover to secondary infrastructure regularly.
Work with developers to better understand how code will operate in a failover scenario and improve as needed (:D DevOps!).

Documentation
Heaven forbid something happen to you, but if it does, how well can someone else replace you. Document what you've built and improved so clearly that another person can replace you. Don't worry, this makes you less likely to be fired.
Education
Always, educate yourself. Take online courses, read books, work on certifications. I don't know what the path looks like for you. I believe that spending a few hours every week specifically dedicated to improving your own skills is a valuable use of your time (even for your employer!).
A book like Site Reliability Engineering (Beyer, Jones, Petoff & Murphy) or The Phoenix Project (Kim, Behr & Spafford) might be good to read on this topic.

Totally possible I've entirely missed the point of your question. If so, I apologize. However, I think this ought to give you some ideas of how you can spend your time in ways that is worthwhile to both your employer and yourself.


Should Jenkins apply round robin if nodes have the same label? In Jenkins it is possible to assign labels to nodes, e.g. somenode to nodes and then one could call somenode in the pipeline and then Jenkins will run the build on some of the nodes.

Problem

The assumption was that Jenkins would apply round robin, i.e. the number of builds that is run on node 1 2 3 4 and 5 is equal, but actually the majority, I.e. more than 90 percent is run on node four.

@jayhendren indicated that:

  I think you might find the answer to your question in this SO post:
  https://stackoverflow.com/q/11024007/2777965


Jenkins installed on aws unable to launch on browser I have a fresh aws instance with Ubuntu 14 installed.

I am trying to install and run jenkins on it. I did the following steps.

Installed java8

wget -q -O - https://pkg.jenkins.io/debian/jenkins-ci.org.key | sudo apt-key add -
sudo sh -c 'echo deb http://pkg.jenkins.io/debian-stable binary/ > /etc/apt/sources.list.d/jenkins.list'
sudo apt-get update
sudo apt-get install jenkins

sudo apt-get update
sudo apt-get install jenkins

After that I checked the status of the Jenkins server

sudo service jenkins status

The response was:

Jenkins Automation Server is running with the pid 8355

sudo netstat -ntulp | grep 8080
tcp6       0      0 :::8080                 :::*                    LISTEN      8356/java

However, when I go to ipaddress of <aws_instance_ip_address>:8080 the page does not load.

Going to <aws_instance_ip_address> opens the Ubuntu Apache default page.

What do I need to do for Jenkins to launch from browser?

It seems your have not opened your 8080 port in Security Groups in AWS.

Steps:

Go to AWS console
Click Security Group and select the security group that you have assigned to your EC2 instance.
Click Edit Inbound Rules
Add a Custom TCP Rule and add 8080 in the Port Range.
You will now see Jenkins rise from the grave :) i.e. first time installation screen.

P.S.:
My Friend, if you have reached till Step 5, you know what to do next ;)


Prevent ${PIPELINESTATUS[0]} from being evaluated in Jenkinsfile Below is a script I am running in the deploy of a docker container. The docker ps -a | grep $APP_CONTAINER_NAME command is returning non-zero when the container was not correctly built/tested, hence the || true.

How can I prevent Jenkins from interpreting the ${PIPELINESTATUS[0]} as a variable replacement in the script?

sh label: 'Stop and Remove Old Docker Container', script: '''
                    docker ps -a
                    echo $APP_CONTAINER_NAME

                    docker ps -a | grep $APP_CONTAINER_NAME || true
                    DOCKERCODE="${PIPESTATUS[0]}"
                    GREPCODE="${PIPESTATUS[1]}"
                    echo "Docker Command Code: $DOCKERCODE"

                    echo "Grep Command Code: $GREPCODE"

                    if [ $DOCKERCODE -eq 0 ] then;
                        docker stop $APP_CONTAINER_NAME
                        docker rm $APP_CONTAINER_NAME
                    else
                        echo "WARNING: Docker command was empty or had an error"
                    fi
                '''

EDIT1:
To clarify - this command gives the error /mnt/data/jenkins/workspace/Test@tmp/durable-4c7b4748/script.sh: 7: /mnt/data/jenkins/workspace/Test@tmp/durable-4c7b4748/script.sh: Bad substitution

I've tried triple double quotes, escaping the quotes in the shell script, and single quoting the PIPESTATUS command prevents execution.

UPDATE:
I tried escaping quite a few things. A single slash results in the above error as well. A double slash results in printing the line and not running the bash command.

        sh label: 'Stop and Remove Old Docker Container', script: '''
                docker ps -a
                echo $APP_CONTAINER_NAME

                docker ps -a | grep $APP_CONTAINER_NAME || true
                echo "HERE"
                echo \\${PIPESTATUS[@]}
                DOCKERCODE="\\${PIPESTATUS[0]}"
                GREPCODE="\\${PIPESTATUS[1]}"
                echo "Docker Command Code: \$DOCKERCODE"

                echo "Grep Command Code: \$GREPCODE"

                if [ \\$DOCKERCODE -eq 0 ] then;
                    docker stop $APP_CONTAINER_NAME
                    docker rm $APP_CONTAINER_NAME
                else
                    echo "WARNING: Docker command was empty or had an error"
                fi
            '''

Output from above version:

TEST
+ docker ps -a
+ grep TEST
+ true
+ echo HERE
HERE
+ echo ${PIPESTATUS[@]}
${PIPESTATUS[@]}
+ DOCKERCODE=${PIPESTATUS[0]}
+ GREPCODE=${PIPESTATUS[1]}
+ echo Docker Command Code: ${PIPESTATUS[0]}
Docker Command Code: ${PIPESTATUS[0]}
+ echo Grep Command Code: ${PIPESTATUS[1]}
Grep Command Code: ${PIPESTATUS[1]}
/mnt/data/jenkins/workspace/Test@tmp/durable-4a5608fa/script.sh: 17: /mnt/data/jenkins/workspace/Test@tmp/durable-4a5608fa/script.sh: Syntax error: "else" unexpected (expecting "then")

FINAL ANSWER:
The shebang needed to be on the same line as the script: ''' because of how just the first few characters are read by the system to check for #!.

sh label: 'Stop and Remove Old Docker Container', script: '''#!/usr/bin/env bash
                docker ps -a
                echo $APP_CONTAINER_NAME

                docker ps -a | grep $APP_CONTAINER_NAME || true
                status=("${PIPESTATUS[@]}")
                DOCKERCODE=${status[0]}
                GREPCODE=${status[1]}
                echo "Docker Command Code: $DOCKERCODE"

                echo "Grep Command Code: $GREPCODE"

                if [ $DOCKERCODE -eq 0 ]; then
                    docker stop $APP_CONTAINER_NAME
                    docker rm $APP_CONTAINER_NAME
                else
                    echo "WARNING: Docker command was empty or had an error"
                fi
            '''

There seems to be some syntax issues and also I would suggest to use bash instead if using sh shell.

Try the below code and see if this works properly.
(EDIT - Put shebang on first line to accept as answer)

sh label: 'Stop and Remove Old Docker Container', script: '''#!/usr/bin/env bash
                docker ps -a
                echo $APP_CONTAINER_NAME

                docker ps -a | grep $APP_CONTAINER_NAME || true
                status=("${PIPESTATUS[@]}")
                DOCKERCODE=${status[0]}
                GREPCODE=${status[1]}
                echo "Docker Command Code: $DOCKERCODE"

                echo "Grep Command Code: $GREPCODE"

                if [ $DOCKERCODE -eq 0 ]; then
                    docker stop $APP_CONTAINER_NAME
                    docker rm $APP_CONTAINER_NAME
                else
                    echo "WARNING: Docker command was empty or had an error"
                fi
            '''

In the above code, I have made changes in the way you were trying to get output of command docker ps -a | grep $APP_CONTAINER_NAME || true using PIPESTATUS variable.
Since PIPESTATUS is a special variable, it's value gets updated after every command, so you were not getting any value of ${PIPESTATUS[1]}

The other change I have made is fixing the syntax issue in the line if [ $DOCKERCODE -eq 0 ] then; to if [ $DOCKERCODE -eq 0 ]; then

You can test out the code further.


how to prevent Ansible from showing output on screen? I have this in my playbook which execute a command on remote server and enters its log on local server.

 - name: run script
   shell: runuser -l testuser -c "/tmp/test.sh"
   register: myshell_output
 - name: copy output to a local file
   lineinfile:
     dest: /thesaurus/output
     line: "{{ item }}"
     insertafter: EOF
   with_items:
    - "#####################Beginning##########################"
    - "{{ myshell_output.stdout }}"
    - "########################END#############################"
   delegate_to: localhost

the output is always shown on screen and troubleshooting gets hard and distracting.

TASK [run script] ************
changed: [1.1.1.1]
changed: [2.2.2.2]
TASK [copy output to a local file] ***********
changed: [1.1.1.1 -> localhost] =>
(item=#####################Beginning##########################)
 ok: [2.2.2.2 -> localhost] =>
(item=#####################Beginning##########################)
{
  REALLY LONG AND BIG OUTPUT!
}
changed: [1.1.1.1 -> localhost] =>
(item=########################END#############################)
ok: [2.2.2.2 -> localhost] =>
(item=########################END#############################)

is it possible to prevent Ansible from printing these output on screen and just insert output to a file?

This can be done by judicious use of one of the Ansible callback plugins. Set the relevant configuration in your ansible.cfg file.

You have a few options...

In order to print nothing to the screen, you can use the null callback:

  This callback prevents outputing events to screen

However, this may be overkill and doesn't do what you want, which is to write the output to a file. This can be done with either of the

tree or
log_plays

modules.

I would probably suggest the log_plays module in your case.

Note: This is going straight from the documentation. I haven't used the tree or log_plays plugin myself, opting either for the yaml or profile_tasks one, to make things more readable. It also seems like you want to filter just a part of the output, not all of it. You may want to use a combination of filters in order to get the desired effect.


How to automate installing an ISO in a VM? We're trying to develop CI/CD processes to automate testing and installing our builds. Ya know, as one does in DevOps.

The end result of our build process are ISOs that use Red Hat Kickstart to install RHEL + our software. Today we burn the ISOs to DVD and install them on bare metal hardware. I would like to some how install and test these ISOs on virtual machines as part of our CI process. Something like:

Push commit to GitLab.
GitLab CI process runs.
Generates an ISO.
Creates a half dozen VMs via Ansible/Vagrant/something else??
Installs ISO in each VM.
Runs a smoke test on each VM to verify that it boots and our software runs.

I'm not really sure how to do 4, 5, and 6. What tools should we be using? Welcome any tips or advice... even just names of automation tools to look into.

If you're into RHEL you could look into running oVirt ( https://ovirt.org/ ). This will enable you to manage host machines, VMs and virtually anything in between through Ansible.

What you need is oVirt on top of CentOS or similar, on a baremetal machine. This/these machine(s) will host your new VMs and your so-called Hosted Engine (i.e. the oVirt virtualization management engine run from within a VM running on the hosts themselves to ensure high availability and flexibility), and then you just need Ansible and the oVirt Python SDK.

Once you have this, it's "as simple as" creating Ansible playbooks and using the oVirt Ansible modules https://docs.ansible.com/ansible/latest/modules/list_of_cloud_modules.html#ovirt


What are the advantages of using Jenkins over Hudson? Jenkins is a fork of Hudson.

What are the benefits/advantages of using Jenkins over Hudson for a small private business?

Number of commits.

Jenkins is a fork of Hudson (as commented here) but has a more active development, Hudson's last commit is quite old. And look at Hudson's open bugs


How to calculate burn rate for SLOs? I've read the Google SRE book a few times but I need some clarifications on exactly how to set up the burn rate and understanding how long it'll take to trigger an alert.

Most of my questions are specifically from this section on the book: https://landing.google.com/sre/workbook/chapters/alerting-on-slos/#4-alert-on-burn-rate

In Table 5-4, the error rate for 99.9% SLO for burn rate of 1 is equal to 0.1%. I just want to confirm - this 0.1% is coming from 100% - 99.9% where 99.9% is the SLO. Does this mean if some service has a ridiculously lower SLO, (say 90% SLO), then their error rate for burn rate of 1 would be 10%. Am I interpreting this properly?
Couple sentences below, the book says

  For burn rate-based alerts, the time taken for an alert to fire is (1-SLO/error ratio) * alerting window size * burn rate.

So if my SLO is 95%, and my error rate is 1 (let's assume all the requests coming up in last 1 hour are errors). Let's assume my burn rate is 1. If I plug these values into the formula, I am getting,

(100-95/1) * 1 hour * 1 = 5.

This is where my confusion is. Is this 5 hours? Do you convert 1 hour into 60 minutes to get the minutes? Is that how long it'll take Prometheus to fire the first alert?

Furthermore, isn't the detection time too late if it will take 5 hours to get an alert? Maybe some concrete examples on how you use this formula to calculate some real values would be really helpful.

The next formula,

  The error budget consumed by the time the alert fires is: (burn rate * alerting window size)/period.

Just want to clarify this - if my burn rate is 1 and my alerting window size is 1 hour, this means, within 5 hours, I would have consumed, (1 * 1)/5 = 20% of my error budget.

Is this right?

The table 5-8 has recommendations to send an alert if the long window is 1 hour and the short window is 5 minutes and if the burn rate is 14.4 and the error budget consumed is 2%.

In the graph above, it shows that the error rate for 5 minutes (10%) is higher than the error rate for 1 hour. Why would their error rates be different if the burn rate is same = 14.4? I'm having difficulty understanding this bit.

It also says it will take 5 minutes to alert based on this information. Will this be true for any SLO? Or is it only true for 99.9% SLO?

And lastly, just wanted to clarify: The examples in the chapter use this following recording rule, which is for ratio of errors per request.

  job:slo_errors_per_request:ratio_rate1h{job="myjob"} > (14.4*0.001)

If I want to something similar for identifying latency - i.e. ratio of all requests which have a latency of more than 2 seconds, then this would be the following PromQL:

(sum by (job, le) (rate(latency_quantile{job="myjob", le="2"}[1h]))
/
sum by (job, le) (rate(request_count{job="myjob"}[1h]))) > (14.4 * 0.001)

Does this look right?

Bottom line, I wish this particular chapter, which the authors acknowledge that it has some complex implementations, had some more concrete examples especially around the formulae and the burn rate. Some examples specified in the tables and chart make sense. But some need a little more clarification to understand the nuances (e.g. specifying that a burn rate of 14.4 translates into 2% of the error budget and this is because when you divide by 30/14.4, you get 50 hours of error budget, and 1 hour is 2% of those 50 hours.)

Yes, you are interpreting that correctly.
When you express your SLO as a percentage, you should express your error rate as a percentage too. So, in you example your error rate of 1 is 100%. That would make the equation:

(100-95/100) * 1 hour * 1 = 0.05 hours = 3 minutes

The period in that equation is the reporting period that you have chosen for your SLO. You have not mentioned what time period you chose. The examples in the workbook are 30 days. If you use a 30 day (720 hour) period in your example, you would consume approximately 0.14% of your error budget in the 3 minutes it takes for the alert to fire:

( 1 * 1 hour ) / 720 hours = 0.14%

The rates are different because they are measured over different time periods. (Note that the graph is using a logarithmic scale, so the actual error rate and the peak of the 5m measurement is 15%, not 10%.)

Consider what you would measure one minute after the errors started:

The 5m measurement would see 4 minutes of no errors plus one minute of 15% errors, which gives an overall error rate of 3%.
The 60m measurement would see 59 minutes of no errors plus one minute of 15% errors, which is just an overall error rate of 0.25%.

Then 5 minutes after the errors started:

The 5m measurement would see 5 minutes of 15% errors, so the overall rate would be 15%.
The 60m measurement would see 55 minutes of no errors plus 5 minutes of 15% errors, so the overall rate would be 1.25%. This is actually still too low to trigger the alert, so it will actually take more than 5 minutes to alert.

Using the equation from before, the actual time taken to alert is 5.76 minutes:

5.76 = (0.001/0.15) * 60 * 14.4

Your PromQL seems to be looking at the good events instead of the bad events, as the le="2" selector is for the bucket counting latencies less than or equal to (le) 2 seconds. You may want to create a separate "good_latency_rate1h" recording rule for the good events, and then use something like "(1-good_latency_rate1h) > (14.4*0.001)" for your alerts.


Use one Helm chart for ALL microservices? This is a follow up question to my post Parameterized Kubernetes manifest files?.
I understand more about Helm now, after lots of reading, including the Learn Helm book (which I recommend). I still have this understanding that each microservice will have its OWN Helm chart? This defeats the purpose of what I wanted, where I just have ONE helm chart, and every time I want to use it, I just update the Chart.yml and values.yml files for each microservice, everything else is the same. For example, the microservice name, versions, and repo will change of course per microservice. Seems like I'm looking for a "template" for Chart.yaml and values.yaml. Is this possible to do? I don't know if it's recommended or not (I may not care).
I come from a CloudFoormation background, where I just have ONE parameterized CloudFormation template, and I just pass it parameters pertinent to each microservice.
UPDATE: I'm aware of Helm library charts, and I don't think they can provide the functionality I want?

Generally, a Helm chart would cover all your microservices - ideally, you should be able to deploy whole application in one chart. It may get a little messy if you include external 3rd party dependencies, where you would need to handle dependent charts, but I assume this is out of scope of your question.
More so, if your microservices are very similar you could write for loops, such as:
{{- range $index, $service := .Values.myservices}}
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp-{{ $service.name }}
# --- etc ---
{{- end }}

Now, where to put your chart - in most cases a preferred way is to have a separate repo for all Ops CD files, including Helm chart - so this is where Helm chart would live. This is not to be mixed with CI files, such as Dockerfile - those should live alongside microservice repositories themselves.


Azure DevOps/Pipelines seed job equivalent? I'm looking at moving to Azure and using Azure Pipelines (I've used Jenkins before now) - I plan to deploy to Azure Kubernetes Service and store my Docker images in an Azure Container Registry.  Using Azure DevOps/Pipelines to integrate with Azure's own services would seem to be simplest approach.

I've got a small collection of applications to deploy - about 30 or so.  In the Jenkins world I had a Git repository containing my job definitions written in Jenkins Job DSL language and a seed job that would create all 30 Jenkins "pipelines" for me.

Can I do this with Azure DevOps?  Is there an equivalent to a seed job process in Azure DevOps?

--

Jenkinsfile equivalent in Azure repos looks to be a similar question (but not identical).

Yes this can be done, Azure Pipelines can ingest a yaml file, configuring the pipeline as code. These yaml files should be source controlled and the pipeline can be set up to build in an automated way.

You can also reuse steps within the pipeline


Jenkins - Scripted Pipeline - Input String I am struggling to set up an input step in a scripted Jenkins pipeline that will let me as the user to input a string.

Scenario:
I am building a Docker image. I want to ask the user what tags to tag the image with once it's built. I want the user to supply a comma separated list like so:
latest,0.0.4,some-tag-name

All I have right now is an input asking the user if the core image should be built:

  def buildCoreImage = input(
    message: 'Build Core Image?',
    ok: 'Yes',
    parameters: [
      booleanParam(defaultValue: true, description: 'Push the button to build core image.',name: 'Yes?')
    ]
  )

  echo "Build core?:" + buildCoreImage

So in theory it should be as easy as changing the booleanParam to a stringParam, BUT the documentation does not have it:
Jenkins - Pipeline: Input Step

Thanks for your time!

After some more research I found the answer:

  def coreImageTags = input(
    id: 'coreImageTags', message: 'Enter a comma separated list of additional tags for the image (0.0.1,some-tagname,etc):?',
    parameters: [
      [$class: 'StringParameterDefinition', defaultValue: 'None', description: 'List of tags', name: 'coreImageTagsList'],
    ]
  )

  echo ("Image tags: "+coreImageTags)

Important note

It is stated in the documentation but I want to point it out specifically: If you copy/paste my solution here, coreImageTags will contain a string directly. If you do multiple parameters like so:

    parameters: [
      [$class: 'StringParameterDefinition', defaultValue: 'None', description: 'List of tags', name: 'coreImageTagsList'],
      [$class: 'StringParameterDefinition', defaultValue: 'None', description: 'Something else', name: 'somethingElse'],
    ]

You will not get a single values but an array of values:

  echo ("Image tags: "+coreImageTags['coreImageTagsList'])
  echo ("Image tags: "+coreImageTags['somethingElse'])

The following posts will help:

stackoverflow.com read-interactive-input-in-jenkins-pipeline-to-a-variable

CloudBees - Pipeline-How-to-manage-user-inputs

A list of classes you can use in this way and what they do is in the Jenkins Input-Step documentation:
Pipeline - Input Step


Preserving ownership of a folder after mounting it I have dockerized an application which accesses a folder "Folder1" inside the container. The folder is owned by a user "user1". When I bring the container up, I need to mount this folder over my host machine. Upon doing so, the ownership of "Folder1" inside the container changes to root as the corresponding mapped folder is under root's ownership on the host machine.

This change of ownership makes the folder inaccessible to my application inside the container. Any idea how do I fix this?

At least in the *nix world the ownership (i.e. uid and gid) of a file/folder on a mounted filesystem doesn't change, it is the actual ownership from the host exporting that filesystem (if ownership is supported by that filesystem).

Note that if the container has different user/group names matching the uid/gid of the file/folder they will appear under the container's names in ls -l, not as the host's names (or as numbers if no container names match the respective IDs).

So if on the host machine the folder is owned by root, it will also be owned by root in the container (in other words it's nothing "special" about it being owned by root vs other users). The only way to allow the container app to write to the filesystem is to  change the ownership accordingly on the host system (or give it other write permissions).

Maybe worth noting that if the host's filesystem itself doesn't support ownership, it can still be "faked" inside the container using the uid and gid mount options (in the *nix world). See Ownership of a mount point.


Cloudformation issue with target groups and ASG I am trying to use cloudformation to setup a ASG and I have a target group which I would like to associate with the ASG. The target group is created okay but when I go to attach it to the ASG I get an error message. I'd like to be able to attach this as I want a ALB in front of the ASG

Error message:

Value of property TargetGroupARNs must be of type List of String

Cloudformation code

TargetGroup:
    Type: AWS::ElasticLoadBalancingV2::TargetGroup
    Properties:
      VpcId:
        Fn::ImportValue:
          !Sub "${Prefix}-VpcId"
      TargetType: instance
      Port: 443
      Protocol: HTTPS
AutoScalingGroup:
    Type: AWS::AutoScaling::AutoScalingGroup
    Properties:
      AutoScalingGroupName: Autoscaling Group Alpha
      Cooldown: 120
      DesiredCapacity: 1
      LaunchConfigurationName: AlphaLaunchConfiguration
      MaxSize: 1
      MinSize: 1
      Tags:
        -
          Key: Name
          Value: Alpha ASG
          PropagateAtLaunch: 'true'
      TargetGroupARNs:
        Ref: TargetGroup

Resources:
    TargetGroup:
        Type: AWS::ElasticLoadBalancingV2::TargetGroup
        Properties:
          VpcId:
            Fn::ImportValue:
              !Sub "${Prefix}-VpcId"
          TargetType: instance
          Port: 443
          Protocol: HTTPS
    AutoScalingGroup:
        Type: AWS::AutoScaling::AutoScalingGroup
        Properties:
          AutoScalingGroupName: Autoscaling Group Alpha
          Cooldown: "120"
          DesiredCapacity: "1"
          LaunchConfigurationName: AlphaLaunchConfiguration
          MaxSize: "1"
          MinSize: "1"
          Tags:
            -
              Key: Name
              Value: Alpha ASG
              PropagateAtLaunch: true
          TargetGroupARNs:
            - Ref: TargetGroup

As the error message states, TargetGroupARNs value should be of type List.
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-as-group.html#cfn-as-group-targetgrouparns


Jenkins is running as a docker image in a kubernetes cluster and won't come up due to plugin version incompatibility We are running Jenkins as a docker image in a kubernetes cluster. However it does not come up any more because of some plugin version incompatibility.
How can we upgrade the plugins, or disable them to bring up Jenkins?

Note that because Jenkins is not up, we cannot upgrade / disable plugins with the Jenkins UI.

If you know what plugins are causing the issue, you can use the Jenkins CLI to remove or downgrade those plugins.  For instance:

java -jar jenkins-cli.jar -s http://localhost:8080/ install-plugin SOURCE ... [-deploy] [-name VAL] [-restart]

Installs a plugin either from a file, an URL, or from update center.

 SOURCE    : If this points to a local file, that file will be installed. If
             this is an URL, Jenkins downloads the URL and installs that as a
             plugin.Otherwise the name is assumed to be the short name of the
             plugin in the existing update center (like "findbugs"),and the
             plugin will be installed from the update center.
 -deploy   : Deploy plugins right away without postponing them until the reboot.
 -name VAL : If specified, the plugin will be installed as this short name
             (whereas normally the name is inferred from the source name
             automatically).
 -restart  : Restart Jenkins upon successful installation.

I believe you can also just drop an .hpi file into a plugins directory in your Jenkins installation, but I've never needed to resort to this solution before, so I can't say how well it works.


Load balancing Nexus artifact repository Load balancing Nexus is currently not supported by Sonatype, except by putting a Nexus instance in front of two with smart-proxying enabled or via a newer feature.

Still, I gave it a shot, by sharing the filesystem with GlusterFS for the /storage, and trying to enable stickyness in the Apache Load Balancer config and in Nexus config for the UI.

With the UI I am failing miserably, as it seems the Nexus container does not honor the properties for setting the cookie value; using the following configuration at the moment:

For the storage it seems to work, by limiting to GET and HEAD requests only, still have to try opening to POSTs. I was also unsure about the possible collateral effects of scheduled jobs, so all were disabled in the "secondary" node.

Has anybody achieved some level of trustworthy configuration for load-balancing nexus? The UI is not actually important to be load-balanced, I would be happy enough with the storage.

Sonatype's Nexus 3 Pro supports High Availability through a couple of mechanisms that are collectively known as Component Fabric:

Peer-to-peer Repository Managers means there is no one master, also known as a single point of failure.  Packages are replicated between the nodes to ensure they are eventually consistent.
Storage Backends mean you can use high durability storage such as S3.
Dynamic Nodes enable auto-scaling support to increase capacity when demand is high and decrease it when demand is low to reduce costs.

It's not really in Sonatype's interests to support HA for the community project as it would cannibalize some of the enterprise customers from their paid product.


Central repository hosting various binaries/applications for different platforms/architectures I would like to improve the automation and performance when managing VMs using Ansible. One of my use cases is to install custom software on a variety of operating systems i.e Red-Hat, Ubuntu, Centos. This software is distributed using single-file installers.  For instance, installing a mySQL database on Red-Hat and Centos.

I recently learned about "repository managers" (or something similar). I don't know exactly where to look/learn more about what are the tools out there. I also don't know what the official terminology is.

This repository is ideally straightforward to use i.e, I just put a binary there, classify it with some tags[?] maybe like 64-bit, version 3.3, Centos.

Currently, I have this. But I don't really like hosting the binary on an NFS server

name: Get mySQL binary
command: "find {{ nfs }} -name \"*mysql*\""
register: mySQL_location

name: Install mySQL
command: "{{mySQL_location.stdout}} --option 1 --option 2
become: yes

A typical use case would look like this

I want to install Software A on Centos
I use Ansible to get the appropriate OS specific version of Software A from a "central repository"
I use an Ansible module and specify version, OS, etc, and that downloads that binary (or binaries) to the VM. This should be fast since it will be over the local network, even for binaries of 100 MB+. Afterwards, Ansible installs it.
The Ansible role would be configurable to affect the installation settings, etc.
Next, I want to reuse the same role to easily do the same for, let's say Red-Hat

My goal is to have the Ansible playbooks be easily maintainable when a new version comes out, without, having to change anything in a playbook, just swap/add a new binary in the repository for a new version of Software A

I understand that the YUM package manager does this in a way, however, I am wondering what other options (if any) are out there.

Your scenario is a perfect use case for a binary repository manager. A BRM's primary feature is to act as a central storage system for your binaries, but there are many other features including:

Versioning
Tagging
Permission management
Logging/auditing

Using a BRM is more lightweight than a typical source code repository system such as Git or SVN. Those systems will have much more metadata than is necessary for binary files

I have personally used Artifactory by JFrog, but I have also heard many good things about Nexus.


How to add my own columns to Jenkin's Dashboard I want to add my custom columns, and I also want to add some popups on those columns in Jenkins dashboard.

How does one add new columns, which are not present in job's configuration page?

you can use the Jenkins Dashboard View to add the layout of your choice. The page says:

  The dashboard view comes with a number of portlets that you can configure your view with

so you can add top, bottom, or column portlets as you seee fit.


Why shouldn't I try to hire a 'DevOps Engineer'? The idea of having a DevOps Engineer has become quite popular recently, and it seems appealing to just have a person who can slot in and provide many of the benefits of DevOps, as described in the Puppet blog:

  Organizations using DevOps practices are overwhelmingly high-functioning: They deploy code up to 30 times more frequently than their competitors, and 50 percent fewer of their deployments fail, according to our 2015 State of DevOps report.

However, I've noticed a lot of vocal opposition to the idea of a DevOps Engineer to try and make these improvements:

  Even with broad agreement about core DevOps attributes, controversy surrounds the term “DevOps engineer.” Some say the term itself contradicts DevOps values. Jez Humble, the co-author of Continuous Delivery, points out that just calling someone a DevOps engineer can create a third silo in addition to dev and ops — "...clearly a poor (and ironic) way to try and solve these problems."

Why might it not be such a great idea for a business to hire a DevOps Engineer to try and 'implement DevOps', as opposed to the organisational change advocated by blogs like this? Will the benefits be negated by just having an isolated DevOps role?

TL;DR: You should never try to hire a DevOps Team

There are essentially three most common roles to hire for:

DevOps Architect / Evangelist
DevOps Engineer
CI/CD Engineer

These roles are distinct from your 6 essential software development roles that traditionally compose the software engineering organization:

Product Management
Software Development
Tools Development
Security and Compliance
Quality and Testing
System Operations (SRE)

Lets go through the three roles one by one and see how they fit

DevOps Architect or Evangelist

Why: If you are lost, slow, broken and don't know what to do.
When: At the start of the process in planing stages.
What: Management level role to guide all managers and leads in the entire Software Engineering org. This person will plan the entire transformation of your engineering organization to a highly functioning state.
Who: Consulting member well versed in theory, management practices, culture topics and operations who reports directly to VP of Software Engineering.

In some cases and for smaller and mid-sized companies you might start the process instead with hiring a consulting organization, like DORA.

DevOps Engineer

Why:

To bridge the gaps between teams if they are organized along the functional roles mentioned above to ensure cross functional level cooperation.
To embed with product oriented teams, which have each of the 6 traditional roles included in the team, to help bridge the knowledge gaps and to help with implementation and adoption of the novel practices and tools.

When: Once you've laid out your plans and the organizational transformation starts and the entire management team is on board.
What: Enable cross function cooperation, ensure that team boundaries are broken down, that local optimizations inside teams are not creating a barrier to high throughput of work throughout the value chains all the way from customer wishes to customer deliveries.
Who: Experienced engineer with skills both in software development and system operations. He should be skilled in the best practices, process and culture changes related to DevOps transformation.

CI/CD Engineer

Why: To help implement CI/CD pipelines, integrate your tool chain, bring in the tools that will enable better working of the company.
When: During the transition in larger organization, while the above roles have been already filled.
What: Engineer, which is essentially part of the tools team that will be able to setup CI/CD pipelines and start integrating internal systems in a way that will remove friction from the throughput of work.
Who: Engineer experienced with Tools, Integration process, Release Management and DevOps practices. Someone who understands they are replacing human gating in release process by Automation.


How to store Jenkins job results and produce a summary report? I'd like to store custom job result data into some format (e.g. in XML) that Jenkins can read and produce the summary report on the job page (ideally in the form of table).

I'd like to know how this can be achieved?

Have a look at the Summary Display plugin. Some more details about it (= quote from the linked page):

... allows an easy and fully customized build report display.
Features

Allow a rich summary report visible from both project and build page
Reports must be written in an XML format according to the syntax described in section Syntax Description
This plugin allow the parsing of several XML files
Reports displayed are ordered according to the XML file names
Element displayed are based on

In general, the answer to such questions is usually found by searching the endless Jenkins plugin index.


How to use kubernetes_service data provider in Terraform? I have simple workflow. Create a cluster, install ingress with helm and once it's ready, get load balancer's IP address to update CF dns records. It's all in one script.
After creating nginx-ingress controller, I was trying to get IP address of load balancer.
resource "helm_release" "nginx_ingress" {
  name       = "ingress-nginx"
  ...
}

data "kubernetes_service" "this" {
  metadata {
    name = "ingress-nginx-controller"
    namespace = "ingress-nginx"
  }
  depends_on = [ helm_release.nginx_ingress ]
}

This never worked for me and the reason was that it was trying to connect to localhost.
Then I found this limitation.

WARNING When using interpolation to pass credentials to the Kubernetes
provider from other resources, these resources SHOULD NOT be created
in the same Terraform module where Kubernetes provider resources are
also used. This will lead to intermittent and unpredictable errors
which are hard to debug and diagnose. The root issue lies with the
order in which Terraform itself evaluates the provider blocks vs.
actual resources. Please refer to this section of Terraform docs for
further explanation.

and moved kubernetes provider to another module.
However, I have now a new problem.
module "cluster" {
  source = "./modules/cluster"

  cluster_node_count = var.cluster_node_count
  age_path = var.age_path
}

module "network" {
  source = "./modules/network"

  host = module.cluster.host
  token = module.cluster.token
  cluster_ca_certificate = module.cluster.cert
  cloudflare_email = var.cloudflare_email
  cloudflare_api_key = var.cloudflare_api_key
  main_domain = var.main_domain
  cluster_subdomains = var.cluster_subdomains

  depends_on = [module.cluster]
}

This also doesn't work and giving

Providers cannot be configured within modules using count, for_each or depends_on.

Without depends_on the network module is called at the same time as cluster, which of course leads to fail, since for the network module cluster should be ready.
So it's kind of dead-loop which I am not sure how to break.

The provider documentation you quoted here seems to have a mistake. It talks about using the two providers together in the same module but it seems like it really means in the same configuration.
The problem arises because Terraform needs to configure a provider in order to plan with it, and so everything inside a provider "kubernetes" block must be known at the time of planning. This means you can't use anything in there which isn't created yet.
That documentation is trying to tell you that you'll need to split the configuration that declares your Kubernetes cluster from all of your configurations that configure or make use of the Kubernetes cluster, and then you can terraform apply the creation of the cluster first, and then separately terraform apply your other configurations that interact with it using the kubernetes provider.


How to install curl in conjunction with --no-install-recommends in Debian correctly? I am building a Docker image based on debian:buster-slim and installing cUrl in this image. According to the recommendations made by Hadolint, I install all packages via apt-get with the option --no-install-recommends to avoid the installation of unneeded packages.
Installed in this way, cUrl is not able to get any file via HTTPS due to it is unable to find the "local issues certificate", as shown in the extract from Docker output below.
Does someone know which additional package I have to install to solve this problem?
Step 15/19 : RUN set -ex;     curl --output distribution.zip $FULL_URL
 ---> Running in 9aa176a788d7
+ curl --output distribution.zip https://www.host.domain/file.zip
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

This is my current version of the use Dockerfile:
FROM debian:buster-slim

RUN set -ex;
    apt-get -y update

RUN set -ex; \
    apt-get -y install --no-install-recommends \
        curl=7.64.0-4+deb10u1 \
        unzip=6.0-23+deb10u1 \
        coreutils=8.30-3 \
        lsb-release build-essential ssh-client apt-transport-https \
        python gnupg

RUN set -ex;
    curl --output distribution.zip https://www.host.domain/file.zip

you need just to install ca-certificates so update your apt-get command to :
RUN set -ex; \
    apt-get -y install --no-install-recommends \
        ca-certificates \
        curl=7.64.0-4+deb10u1 \
        unzip=6.0-23+deb10u1 \
        coreutils=8.30-3 \
        lsb-release build-essential ssh-client apt-transport-https \
        python gnupg


Can't make MySQL 5.7 work with Docker Compose, no matter what I'm configuring a docker compose file, and the first step is to configure the MySQL service (so I can further use in my app service), but no matter what, I cannot enter MySQL within the container, even when I set all environments. That's how my docker-compose.yml file looks like:

version: "3.3"

services:
  db:
    image: mysql:5.7
    volumes:
      - ./tmp/db:/var/lib/mysql
    restart: always
    environment:
      MYSQL_ROOT_PASSWORD: root
      MYSQL_DATABASE: wordpress
      MYSQL_USER: wordpress
      MYSQL_PASSWORD: wordpress

I've tried SEVERAL (I literally lost count of how many) other configurations, and none of them work, it just doesn't work.

If I go inside the container with docker exec it ${CONTAINER_NUMBER} bash
and try to access MySQL, it will ALWAYS return:

ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: YES)

Or if I try with the user wordpress:

ERROR 1045 (28000): Access denied for user 'wordpress'@'localhost' (using password: YES)

I'm pretty confident that I spent more than 8 hours only changing the configuration, so the only thing I can think right now is: the image is broken.

I'm in a Ubuntu 19, not sure if this makes any difference.

First of all: your docker-compose.yml file is absolutely fine. I just tested it, created the necessary dir for bind mount, started the compose project (docker-compose up -d) and was able to login into mysql (docker-compose exec db bash -c "mysql -u root -proot" - worked with wordpress as well).

So now I'll take a guess (which is usually good...): you already have an initialized mysql filesystem in ./tmp/db from a previous run (or copied from somewhere else). In this case, here is what you might have missed in the mysql docker image documentation in the section on environment variables:

  Do note that none of the variables below will have any effect if you start the container with a data directory that already contains a database: any pre-existing database will always be left untouched on container startup

If you want to have a look under the hood how this happens, you can study the image entrypoint script

So, as you may have now understood, your container starts but it is using the existing mysql filesystem with the previous settled dbs, data, users and passwords.

Simply stop your container, delete any files and directories in ./tmp/db and restart from scratch: you should get your blank db install and be able to login in mysql with the users/pass defined in docker-compose.yml


Bare-Metal Kubernetes storage in enterprise Problem in a nutshell

What kind of storage for dynamic provisioning in a bare-metal kubernetes should i use.

Our configuration

Each of our kubernetes nodes (worker and master) are running on VMs. The VMs themselves run on a VCenter on top of an Hyperconverged architecture
So far the cluster have no persistent storage configured (other than the host storage)
We do have an HDFS (Hadoop Distributed File System) running in bare metal as well, to make it short, hdfs is a distributed and Highly Available storage.

What do i want

I've seen Ceph&Rook being a popular solution. But if i understand it properly it required each kubernetes nodes (if i want it to be fault resistant) to have an attached physical storage, which will all contains more or less the same datas.

What's the problem with that

If each nodes have 1Tb of storage, i will have 1Tb of storage as a ceph&rook storage, and any time i want to add a node to the cluster i will be forced to add 1To of storage to the new node as well.

Ceph will duplicate data across the cluster to make it available in case of failure. But since the cluster is running on top of a hyperconverged architecture, the cepth duplication will be duplicated again by the hyperconverged architecture... you may start to see the problem.

The above example makes no sense to me, especially when i have an HDFS next to this cluster.

My question is what is my best option to use the HDFS as a dynamic storage provisioner?

A storage solution for kubernetes is a pain for those of us who chose to run kubernetes on premises.

What storage solution one picks would certainly depend on potential usage, a solution that will perform well for simple file operations would not pan out very well if you want to run a database in kubernetes.

Given your configuration, I think "vSphere Cloud Provider" is your best option

  VMware offers a Cloud Provider known as the vSphere Cloud Provider (VCP) for Kubernetes which allows Pods to use enterprise grade persistent storage.

architecture overview of vSphere Cloud Provider

If you don't want to run databases you can use NFS using HDFS setup you have or any other NFS server.

Disclaimer: I have not used vSphere Cloud Provider or ceph myself. Personally, I have used NFS from our storage server for production and it works fine for moderate database loads. We also have a large gluster implementation supporting close to thousand volumes, however, based on issues we have seen with gluster I will not suggest that to anyone.

On a side note, you don't "have" to use all of your cluster nodes as "storage nodes" for Ceph & Rook. Only storage nodes will have restrictions on them. If you are going with Ceph & Rook, I will suggest to use dedicated storage nodes otherwise it may impact performance of your workloads.


Is Azure's confidential computing essentially encrypted RAM? I'm trying to wrap my head around the Azure confidential computing offering.

It appears that that AWS does not offer encryption at the application level (see diagram for what I mean by this:)

My question is: Is Azure's confidential computing essentially encrypted RAM?

The article you are referencing mentions encrypted RAM but that was only a reference to possibilities of enhancing the technology more in the future, not about the current state of confidential computing.
Confidential computing means, that when data is in use these things are prevented:

Malicious insiders with administrative privilege or direct access to hardware on which it is being processed

Hackers and malware that exploit bugs in the operating system, application, or hypervisor

Third parties accessing it without their consent

according to Microsoft own announcement
So in other words, it is kind of equivalent to encryption and firewall that are seen in communication but this time applied to running programs in a Trusted Execution Environment (TEE), which the term for the area which is protected when server runs a program.


Is there an overview of current resource usage in a GCP region? When a container cluster was created on GCP in the morning it took less than 10 minutes until it was created. Now, after more than half an hour the cluster is still not created. It turned out that the resources in a certain zone were insufficient. How to check whether there are enough resources in a GCP zone before creating for example a kubernetes cluster?

Instance 'X' creation failed: The zone 'projects/Y/zones/Z' does not have
enough resources available to fulfill the request. Try a different zone,
or try again later.

After choosing another zone and deploying the cluster took less than 4 minutes to deploy it.

This page indicates something about regions and zones, but does not show any resource usage. I think that the resource usage of a zone is not visible.