Kubernetes: Custom second scheduler container unable to start I tried to add a new scheduler to the master node as a static POD.
I create the yaml file and save it.
The pod as been created and his state is running but the container is unable to start.
the kubectl output:
NAME                                   READY   STATUS    RESTARTS   AGE
my-scheduler                           0/1     Running   5          28m

I tried to see reason in the events and saw the reason cannot join network of a non running container and the last event with error:
Startup probe failed: Get "https://127.0.0.1:10251/healthz": http: server gave HTTP response to HTTPS client

I tried to send GET request to http URL and got OK answer but https answer was a error:
curl: (35) error:1408F10B:SSL routines:ssl3_get_record:wrong version number

also I tried to get the scheduler log and got:
    I1016 18:47:15.779350       1 registry.go:173] Registering SelectorSpread plugin
    I1016 18:47:15.779753       1 registry.go:173] Registering SelectorSpread plugin
    I1016 18:47:15.791161       1 registry.go:173] Registering SelectorSpread plugin
    I1016 18:47:15.791191       1 registry.go:173] Registering SelectorSpread plugin
    W1016 18:47:15.794300       1 authorization.go:47] Authorization is disabled
    W1016 18:47:15.794518       1 authentication.go:40] Authentication is disabled
    I1016 18:47:15.794695       1 deprecated_insecure_serving.go:51] Serving healthz insecurely on [::]:10251

scheduler file:
apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    component: my-scheduler
    tier: control-plane
  name: my-scheduler
  namespace: kube-system
spec:
  containers:
  - command:
    - kube-scheduler
    - --authentication-kubeconfig=/etc/kubernetes/scheduler.conf
    - --authorization-kubeconfig=/etc/kubernetes/scheduler.conf
    - --bind-address=127.0.0.1
    - --kubeconfig=/etc/kubernetes/scheduler.conf
    - --leader-elect=false
    - --scheduler-name=my-scheduler
    - --port=10251
    - --secure-port=0
    image: k8s.gcr.io/kube-scheduler:v1.19.0
    imagePullPolicy: IfNotPresent
    livenessProbe:
      failureThreshold: 8
      httpGet:
        host: 127.0.0.1
        path: /healthz
        port: 10251
        scheme: HTTPS
      initialDelaySeconds: 10
      periodSeconds: 10
      timeoutSeconds: 15
    name: my-scheduler
    resources:
      requests:
        cpu: 100m
    startupProbe:
      failureThreshold: 24
      httpGet:
        host: 127.0.0.1
        path: /healthz
        port: 10251
        scheme: HTTPS
      initialDelaySeconds: 10
      periodSeconds: 10
      timeoutSeconds: 15
    volumeMounts:
    - mountPath: /etc/kubernetes/scheduler.conf
      name: kubeconfig
      readOnly: true
  hostNetwork: true
  priorityClassName: system-node-critical
  volumes:
  - hostPath:
      path: /etc/kubernetes/scheduler.conf
      type: FileOrCreate
    name: kubeconfig
status: {}

what am I doing wrong? What is the right way to config the scheduler?

In your manifest, you should set the spec.containers[0].livenessProbe.httpGet.scheme to HTTP not HTTPS.
This is because you use the option --port=10251. As the documentation said, you should not use the deprecated --port option anymore, because it expose the API insecurely.
You should set the following options: --port=0 --secure-port=10251 and then you would be able to set the spec.containers[0].livenessProbe.httpGet.scheme to HTTPS.


How to list the containers attached to a given user-defined network? I'm reworking a bunch of scripts using the deprecated docker run --link option in favor of user-defined bridges.

By searching in the doc, I didn't find any command to list all containers attached to a given network. For now, I am parsing the JSON output of docker network inspect through the jq tool; I've posted that as an answer below since it may help someone. But, do you know another option to achieve that? Maybe using a dedicated Docker command, option or a custom Go format?

The docker network inspect command produces a JSON output that can be parsed using jq to extract the relevant piece of information:

sudo docker network inspect app-bridge | jq '.[].Containers[].Name'

"app2"
"app3"
"app1"
"laughing_easley"
"database"
"thirsty_wing"

If you are already familiar with JSON processing with jq, the advantage is you can reinvest your knowledge without having to bother with Go templates. On the other hand, it relies on an external tool which may or may be not available at your site.


Docker - Mount a volume from a container to an other (equivalent volumes_from) in docker-compose 3 I've two containers : nginx & angular. The angular container contains the code and is automatically pulled from the registry when there is a new version (with watchtower).

I set up a Shared Volume between angular & nginx to share the code from angular to nginx.

  ### Angular #########################################
  angular:
    image: registry.gitlab.com/***/***:staging
    networks:
      - frontend
      - backend
    volumes:
      - client:/var/www/client

  ### NGINX Server #########################################
  nginx:
    image: registry.gitlab.com/***/***/***:staging
    volumes:
      - client:/var/www/client
    depends_on:
      - angular
    networks:
      - frontend
      - backend

volumes:
  client:
networks:
  backend:
  frontend:

When I build & run for the first time the environment, everything works.
The problem is when there is a new version of the client, the image is pulled, the container is re-built and the new code version is inside the angular container, but in the nginx container it still the old code version of the client.

The shared volumes does not let me do what i want because we can not specify who is the host, is it possible to mount a volumes from a container to an other?

The Problem: Named volumes are initialized on container creation when the volume is empty, and from the first container that starts up using that volume. That initialization includes all files, directories, and file metadata like owners and permissions. If there is content in the named volume, docker will not initialize it again since that would defeat the purpose of persistence.

Option A: When you have multiple containers sharing a volume to share updated build time content like you have, my first suggestion is to modify the build to place the relevant data in each image. Depending on what is being shared and between what images, you may be able to use shared image layers to avoid ever needing to copy the data.

Option B: When placing the data in each image and avoiding the shared volume is infeasible, then you likely want a process to copy data from the image into the volume as part of your container entrypoint. This has two advantages. First it works when the containers startup in either order, so you can eliminate dependencies between container startup order (though you may find it necessary to include some wait step in the other container entrypoints if they fail when run before the data is copied). And second, your copy can update the volume when existing data is already there.

I've implemented a load-volume and save-volume pair of scripts in my docker base image that you can reuse in your own images to implement this second option. I typically ran into this issue when using tmpfs volumes to support read-only containers, but the scripts have been made generic enough to support your use case. Note that your scenario likely involves deleting existing data from the volume, and I've implemented that using rsync for performance, so that needs to be installed in your image to work.


Take automatic screenshots of a website and send it to WhatsApp or Facebook Messenger I am trying to automate the process of "sending the screenshots of a monitoring tool dashboard to my WhatsApp number" at regular intervals.

So far I have achieved the following:

Automated the screenshot taking using pageres-cli
Integrated the same with SMTP server using postfix

Now I also want to send these screenshots to my Whatsapp group or Facebook messenger group. How can I achieve this?

In case the requirement for Whatsapp/Facebook is not mandatory, you could consider using a solution offering an API for such things and targeted specifically for IT teams.

For example, Slack channels allow uploading files through Slack's API and sharing files in channels.

Documentation: https://api.slack.com/methods/files.upload

For a ready CLI stub, consider for example slackcat.


Is cross-region Kubernetes a thing, and if so, what role does VPN place in this context? Given you are going to overcome risks of a whole cloud region going offline, you would implement some cross-site distribution mechanisms.
In terms of Kubernetes, cluster communication would happen across public network.
Therefore, probably, you need to setup end to end VPN connections?
Does this all make sense or how do you approach the problem?

First, etcd is a core component in Kubernetes clusters. It is based on the Raft Consensus Algorithm and a weaker point with it is when the latency between the members (e.g. in the distribued etcd database) in the cluster is larger and vary more. It might work to deploy it with longer latency between the nodes (e.g. regions as in cross-region) but it might not be the best idea, at least not out of the box.
But if you use cross-region deployments, one of the main reason to do it is probably to tolerate a full region outage - but then I think it is even a better idea to deploy independent (e.g. can work without the other) clusters.
What might be interesting in relation to this is Kubernetes Federation Evolution and kubefed.
I don't see VPN (e.g. a single network?) very related to this (this is more its own topic) - at least not when you want independent clusters. SSL/TLS might also be related. And if you do cross-regions, e.g. Software Defined Network or cloud VPC might be related as well. E.g. Amazon VPC is located on a single region, but Google Cloud VPC is global (multi-region).


Best tool for Docker Container test automation? In my project i am building about 200 containers I want to check in a  smoke test if the containers are still running after a rebuild in pipeline.

I want to  bring up the container with certain parameters and check if it is responding on ports (service being up), or run simple test inside the container.
Best would be to use docker-compose for starting containers, so that paramateres can easily be configured.

Example for a nginx container

run  "nginx -t" inside the container to check if the container still has a valid configuration file
check if nginx  is exposing services on port 80 and 443

Similar for other containers, main topic is checking ports for rest services. After bringing up the container(s), the tests should be run and then the whole thing shoudl be shutdown again.

What is the best tool for doing such a thing automated - and not writing everything manually?

Although we're deploying to Kubernetes, we've found that docker-compose is a nice way of running tests in CI. Either set the IS_TEST flag in your container or use a separate Dockerfile where tests scripts are wired up. That way you can just run docker-compose build to test, and pull in any dependencies you might need as well.


Process guidance for new development team leader I've just recently taken over leadership of a small team of developers within our group. The previous Team Leader was very much into micro-management and ran the development stream in a six week waterfall approach. The developers were assigned the work items to they had to work on to the exclusion of other work. He was very controlling in that aspect. Needless to say, this didn't really meet the required objectives from the business users.

Moving forward, we want to change this work process. The applications being supported are very mature, with only small to medium feature improvements. The focus needs be much more agile in the delivery fixes/changes whilst allowing for the feature development in parallel.

Our team and environment is as follows.

Applications/Sites

ASP.Net Websites
SQL Server 2012 databases
SQL Server Integration Services
SQL Server Reporting Services
Vendor Supplied 2-tier fat client application (has inbuilt customisation ability, all changes have to be manually migrated)
Vendor Supplied Silverlight Web application (has inbuilt customisation ability, all changes have to be manually migrated)

Team

4 dedicated developers
1 first level support member (probably 15% dev work)
1 application support member (probably 30% support, 50% dev, 20% analysis)
1 DBA\Team Leader (probably 15% support, 60% dev, 25% analysis)
1 Department Manager (probably 20% dev work)

Definite Microsoft shop (mandated from head office)

TFS 2017 (TFVC)
All source code is in one TFS collection with no branching
Visual Studio 2015 (planning to upgrade 2017)
Multiple solutions/projects with some cross project dependencies
We have control and buy-in for server setup and configuration

Current Workflow methodology

Developers work directly against individual TFS Work Items
No "Sprint" or Release date limitations. There would be significant resistance against this. From a business point of view, that as one of the main restrictions to the timely delivery of fixes/changes in the past.
Management and Administration of processes needs to be very, very minimal
Source code needs to remain on-premises (for the short term at least)
All deployment should be as hands-off as possible (with the exception of the vendor applications)
When an item of work is ready for next environment it should be deployed.
Deployments into production environment requires sign off from Dev team and Business team.

Open to all suggestions on how we can best re-structure things to allow us to deliver what's required.

Cheers
Phil

I suspect I'm late to the party, but I've survived a few similar scenarios and might have some tips for you.  It sounds like your team has been over-managed and has grown hostile to process or rules in general.

Do not go directly to Agile and sprint based development.  Consider going with Kanban, and link work items to an epic/larger context.  Sometimes that context will be "break/fix" and sometimes it will be features.  Linking will help you make it clearer where time is spent.  Kanban is preferable because it's likely the absolute minimum process necessary to get work done - it won't require you to bludgeon your team with buzzwords, burndowns, estimates, and other distractions.  It WILL allow you to limit the amount of work in progress - I suggest you start with 3-4 things per team member and try to whittle it down until you can reach a state of continous flow.  Lead your standup using this kanban board.
Look for opportunities to incentivize smaller delivery batches, and run them through an automated testing process.  Use this process to gradually improve the code base and build trust with the release managers who currently introduce friction to your delivery process, which causes an increase in change/batch size, which causes increased risks, which "validates" the need for an outside gate to production.
Break up monolithic repositories, then break up monolithic applications.  Smaller applications again mean smaller batch sizes and reduced risk.
LISTEN a lot.  Your team has been traumatized by a toxic manager, and you have damaged goods on your hands.  If you come in too hot, you'll break trust.  ESPECIALLY through the early phases of this transition, building trust and rapport are the keys to you ever getting a devops practice off the ground.
DO NOT switch tools simply because other people, who seem to be better at this than your team, are using them.  Do I think there are better workflows than your team has?  You bet!  But don't switch for superstitious or personal reasons.  Understand WHAT your current toolchain prevents you from achieving and make sure that the underlying concepts are understood by every single member of your team before you begin.
Be prepared for this to take a year or two to really turn around.  You will need to apply slow, predictable, steady pressure on the organization in order to drive meaningful or longterm change.


TeamCity web interface not accessible on Chrome I've installed teamcity on two machines now. The first, I tried reinstalling multiple times because the web interface would launch in Chrome and fail with:

  This site can’t be reached The webpage at http://localhost:6666/ might be temporarily down or it may have moved permanently to a new web address.

  ERR_UNSAFE_PORT

The teamcity-server service was running, but not accessible on the port I'd set up. Eventually I tried curl localhost:6666, and got a response, so tried again in Edge browser. In Edge, everything works fine.

On the second machine, same deal. Installed fine, not found in Chrome, works fine in Edge.

Is this a known issue with TeamCity? Is there some kind of setting in Chrome that may need changing? How should I go about diagnosing and fixing the problem?

You will need to explicitly allow that port.  From the guide here:

You can modify the shortcut to Chrome so that it explicitly allows that port when chrome launches:  "C:\Documents and Settings\User\Local Settings\Application Data\Google\Chrome\Application\chrome.exe --explicitly-allowed-ports=6666"


TravisCI: Pandas version advanced (Jan 2021) numpy is now incompatible I am very much a TravisCI beginner.  That said, I maintain python package mplfinance and we run a TravisCI test suite for every Pull Request.  We run the same tests for py36, py37 and py38.   Beginning in January 2021, the Py37 tests began failing consistently with the error:
ERROR: pandas 1.2.0 has requirement numpy>=1.16.5, but you'll have numpy 1.16.4 which is incompatible.

Here's the thing:  Some of the tests run using pytest under tox, and those tests are fine.  But we also run a test from a shell script.  In that test, for each PR, the script checks out and installs the package (using pip3 install .) first from the PR itself, and then from the version currently in the repository, and does some comparison of the two versions.  This works fine for py36 and py38, and before January worked fine also for Py37.  Here is an example of it working fine for Py37 prior to January:
https://travis-ci.org/github/matplotlib/mplfinance/jobs/751128518
In the above link you can see the following
Collecting pandas
  Using cached pandas-1.1.5-cp37-cp37m-manylinux1_x86_64.whl (9.5 MB)
Collecting python-dateutil>=2.1
  Using cached python_dateutil-2.8.1-py2.py3-none-any.whl (227 kB)
Collecting cycler>=0.10
  Using cached cycler-0.10.0-py2.py3-none-any.whl (6.5 kB)
Requirement already satisfied: numpy>=1.15 in /home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages (from matplotlib->mplfinance==0.12.7a4) (1.16.4)

Notice that in installs pandas 1.1.5 and numpy 1.16.4
Beginning in January this is what I see:
Example:
https://travis-ci.org/github/matplotlib/mplfinance/jobs/753110408
Collecting pandas
  Using cached pandas-1.2.0-cp37-cp37m-manylinux1_x86_64.whl (9.9 MB)
Collecting kiwisolver>=1.0.1
  Using cached kiwisolver-1.3.1-cp37-cp37m-manylinux1_x86_64.whl (1.1 MB)
Collecting pillow>=6.2.0
  Using cached Pillow-8.1.0-cp37-cp37m-manylinux1_x86_64.whl (2.2 MB)
Requirement already satisfied: numpy>=1.15 in /home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages (from matplotlib->mplfinance==0.12.7a5) (1.16.4)

Now it is installing pandas 1.2.0, and initially it appears satisfied with numpy 1.16.4, but then later in the file I see
ERROR: pandas 1.2.0 has requirement numpy>=1.16.5, but you'll have numpy 1.16.4 which is incompatible.

(Oddly, further up in the file, it appears that tox is installing numpy==1.19.4 and pandas==1.1.5 ??)
Does anyone have any ideas what I can do about this?
Much appreciated.  Thank you.

This turned out to be a problem with pip determining the correct dependency: for some reason it thought that the pandas requirement was for numpy>=1.15
The error message
ERROR: pandas 1.2.0 has requirement numpy>=1.16.5 ...

was coming from pandas itself (upon import).
The fix is to upgrade to a newer version of pip (inside the script running on travisCI).  The newer version of pip is able to corrently detect that pandas needs a newer version of numpy.
Just put pip install --upgrade pip near the top of the script.
Not sure why TravisCI doesn't just use the latest pip by default.
Reference: https://travis-ci.community/t/11214


How to override default aws region credentials for Terraform? I am using Terraform to create aws resources. I practiced with a single region and with my aws cli credentials being used. worked great!

however when I tried to override the region in the Terraform .tf config file it hangs indefinitely.

So how does one either set my aws credentials for the CLI to allow any region, OR how does one set Terraform config files (.tf) to allow multi region infrastructure?

I believe you would be using multiple modules to create various resources(EC2,ELB, Etc) in AWS using Terraform. You can create a custom module having output value of each variable you want to configure per your requirement like aws-region, etc.

Then source(import) this custom module into your resource module and use its values using $.

custom_module

output "aws_region" {
  value = "us-xyz-1"
}
output "aws_anyproperty" {
  value = "abc"
}

Add above custom_module as source in your resource module

module "custom_module" {
  source="../../../modules/custom_module"
}

provider "aws" {
  version="~> y.x"
  profile="${module.custom_module.aws_anyproperty}"
  region="${module.custom_module.aws_region}"
}


How to have multiple log streams in docker We have an application that writes three types of logs into three separate files: access logs, generic application logs and system logs. The format (and purpose) of these logs are very different. And we have separate logforwarders that send them separately to our centralised logging system.

Based on the treat logs as event streams principle, we are thinking about moving from using files to stdout. While we know some of the benefit of this approach, this would also mean that we'd get a merged stream of the differently formatted logs, which we would need to split again either before we can send them to our central system (Kibana / Splunk / etc.), or inside there.

We're wondering whether there are any tools or recommendations on how we should approach this situation.

I'm still looking for a merging/splitting approach myself, but meanwhile this approach recommended by Kubernetes documentation seems like a sound solution: Use a sidecar container for each of your separate logs.

A "sidecar" is any docker container that you use alongside another docker container to work with it in some way. In this case for each of your three logs you would have a separate container that scans or tails the logs and outputs to stdout.

This way each of your log-sidecar-containers has its own docker-log from its own stdout. Being separate like this, you can use standard docker (and kubernetes, etc) practices for separating or aggregating. Here's what the Kubernetes page has to say:

  This approach allows you to separate several log streams from different parts of your application, some of which can lack support for writing to stdout or stderr. The logic behind redirecting logs is minimal, so it’s hardly a significant overhead. Additionally, because stdout and stderr are handled by the kubelet, you can use built-in tools like kubectl logs.

The "separate log streams" stem from the built-in tagging that docker applies to logs from different containers, described in the docker documentation here:

  The tag log option specifies how to format a tag that identifies the
  container’s log messages. By default, the system uses the first 12
  characters of the container id. To override this behavior, specify a
  tag option


Jenkins Health Check URL I'm planning to supervise services by periodically checking if they are up and running, Jenkins is our central orchestration tool and must be running with near zero downtime.

How can I check health status of a Jenkins server - By health status I mean service is up and web app application is operational ?

Does Jenkins provide a native health check URL ? Or does I need to make additional scripts?  if so what factors should be included in beside checking jenkins.service and curl web app home page.

The complication reside in the definition of the health status (and its usefulness).

You mentioned the definition being service is up and web app application is operational and for that indeed checking the homepage should suffice.

However that doesn't mean the service is properly configured for and correctly performing all the jobs that it's supposed to be doing. Without a consistent, universally acceptable definition of a health status there is no point of providing such information as a single, one-stop location to check.

But Jenkins offers via its Remote access API various checking points that users can combine to derive their interpretation of the overall service health status. For example maybe checking the status of all the configured jobs or at least that of the critical ones would be more useful, which could be done using the .../job_id/lastSuccessfulBuild/api/ or .../job_id/api/ checkpoints.

The JenkinsAPI could also be of interest (better docs at least), found via Where can I find jenkins restful api reference?


Trigger a GitlabCI pipeline after it detects a change via a curl callback I would like to trigger a GitlabCI pipeline that builds a Packer image, whenever there is a new version of a binary released in Github.

How could I do that via the ci yaml configuration?
Should I somehow use the API for pipeline triggering?

One could set up a webhook between github and gitlab, see settings>integrations>github in gitlab or trigger a pipeline in gitlab by configuring github.

https://docs.gitlab.com/ee/ci/ci_cd_for_external_repos/github_integration.html

If one does not have a github account and one cannot fork the github project, one could create a script that runs on daily basis and triggers the gitlab pipeline by calling the pipeline API if "new release in github" is applicable.


What tools do you use to monitor applications and business-specific data? (apart from Jenkins) Currently, we use Jenkins to run small node.js scripts in order to check if applications are doing what they're supposed to. Like, we have some routines which should run in an interval and we need to check if these routines have run and have done everything we expect.

Why Jenkins? Mostly because we can just login and look at the dashboard and see the health status of all our jobs at a glance; we can set up notifications; and because we don't know other tools to execute our scripts in a "monitoring" way.

But we know we're misusing Jenkins and the problems we have faced (like an attack we suffered which used our Jenkins instance to attack a credit card company and the constant instability) are making us look for another tool (or tools).

But all I could find was performance monitoring tools. What tools do you guys use to monitor things other than performance?

We use Datadog, with metrics pushed from important operational processes using the statsd protocol ( http://docs.datadoghq.com/guides/metrics/ ), so we aren't too deeply tied to Datadog on the application side.

In Datadog's interface we can then configure monitors to alert on any number of things, including anomaly detection if defining thresholds is impossible.

So for instance, we have many tasks that run on a hourly basis, they'll post a datapoint to a given metric (app.task.task_name), and we will then monitor that this metric is consistently posting at one data point per hour. If the rate changes, we get alerted.


What is the biggest Docker image on Docker Hub and how to find it? The lowest boundary of a Docker image can be very small i.e. under 1 MB (busybox) or even zero bytes (scratch).

Now for research purporses, I need to reference the biggest Docker image on the Docker Hub.

How to find it?

If you're looking for a master list of all images and sizes, kind of a 'statistics' pages, Docker Hub doesn't have that. You'll need to use their HTTP API to:

get all the repositories, stepping through the paginated results. API reference: https://docs.docker.com/registry/spec/api/#listing-repositories
pull the Image Manifest for each repo. API reference: https://docs.docker.com/registry/spec/api/#pulling-an-image-manifest
Store those manifests in local database so you can sort them based on image size

You'll specifically need to pull the newest Image Manifest Version 2, Schema 2, which apparently not all repos support yet. This newer manifest has the image sizes in it: https://docs.docker.com/registry/spec/manifest-v2-2/

All in all, this is much faster than trying to actually download all the images from Docker Hub, but it will still take quite a long time and a lot of processing. The database for storing this all might get rather large and difficult to process, so I'm not sure how much you really want to pursue this route for a simple research project. That's up to you.

You may want to take @Vish's advice and simply go for obvious large images such as the Microsoft or Java ones.


How to copy a file locally to remote pod before it starts? # Copy /tmp/foo local file to /tmp/bar in a remote pod in namespace
<some-namespace>
  kubectl cp /tmp/foo <some-namespace>/<some-pod>:/tmp/bar

works, but then a pod should be running. The problem is that the file has to be copied before the pod starts. The jar file has to be loaded by an Application server. Destroying the pod after the copy will not work as the docker image is deployed again and the content will be wiped out.

A configMap was misused to copy the jar before the pod starts, but that only seems to work for configuration files. This was confirmed by comparing the checksum, e.g. local vs. remote.

Ok. Maybe you're not gonna like it but there are some things you can do.

One and most simple is to use a volume mount, eventually on a persistent storage but I think an emptyDir can work as well (or even better depending on the nature of the file you want to copy) and use an init container  to copy the file to the volume before the app container start. I don't know how you generate that file and I can't speculate about the security implications but the file needs to be in the docker image used for the init container. So, for your init container, you will have a Docker file  like

FROM ubuntu
COPY localfile /container/file
CMD cp /container/file /mnt/kube-volume

replacing localfile, container/file and /mnt/kube-volume with appropriate values; you build the container image locally -on your machine, so it can contain the file-  and upload it to the (secure) docker repo along with the app container image.

Another method is to upload the file to a (secure) location and have an init container or a bootstrap process that download the file when you start your main application container  in the pod.

The third option is to create a docker image that contains that file and use it for the app container in your pod.


How to parse command output (docker log) through goaccess? How to parse docker logs output / tail output through goaccess? When I do:

docker container logs mycontainer > logs

and then goaccess -f logs it works perfectly, but if I do docker logs --tail all mycontainer | goaccess - it doesn't work :( but in the manual  you can do this via tailing command: https://goaccess.io/man

tail -f access.log | goaccess -

Pls help :( I think the problem is that the native tailing differs from docker's?

What do you mean by "it doesn't work"? I got an error suggesting that the log format wasn't correct:

Fatal error has occurred
Error occured at: src/goaccess.c - set_curses - 1394
No time format was found on your conf file.

When I edited /etc/goaccess.conf (i.e. uncommented/selected time-format, date-format and log-format) the docker logs mycontainer | goaccess - command works as expected.

As an aside, for my nginx proxy inside my docker container, I had to set my log_format as follows:

log-format %v %h %^ %^ [%d:%t %^] "%r" %s %b "%R" "%u"


'[[0;1;31mFAILED[0m] Failed to start' when running cloud-init? I am trying to contextualize a docker container using cloud-init.
Why do I see [[0;1;31mFAILED[0m] Failed to start in the OpenStack log?
What is a likely cause of this error?

From my rather limited experience, a likely reason is that the command sudo docker image build does not find the docker file.
(But a virtual machine may have been successfully created.)


Ansible: register variable with loop What’s the best way to use registered variable ‘audit_tools’ to check if all items are own by root ?
Do I need to use Jinja2 filter or something ?
Thanks
- name: Verify audit tools are own by root user.
  block:
    - name: check if audit tools are own by root user.
      become: true
      stat:
        path: "/sbin/{{ audit_loop }}"
      loop:
        - auditctl
        - aureport
        - ausearch
        - autrace
        - auditd
        - audispd
        - augenrules
      loop_control:
        loop_var: audit_loop
      register: audit_tools

    debug:
      msg: “one or more tools are not own by root.”
    When: .....

  Rescue
     ......

Q: "Check if all items are own by root."
A: Put the list of the tools into the variable audit_tools. Compare the length of the lists. For example
- hosts: localhost
  vars:
    audit_tools:
      - auditctl
      - aureport
      - ausearch
      - autrace
      - auditd
      - audispd
      - augenrules
  tasks:
    - block:
        - stat:
            path: "/sbin/{{ item }}"
          loop: "{{ audit_tools }}"
          register: result
        - assert:
            that: no_audit_tools == no_owner_root
            fail_msg: "One or more tools are not own by root."
          vars:
            no_audit_tools: "{{ audit_tools|length }}"
            no_owner_root: "{{ result.results|
                               json_query('[?stat.pw_name==`root`]')|
                               length }}"
      rescue:
        - debug:
            msg: "Rescue: audit tools not owned by root."

If not all items are owned by root assert will fail and the block will proceed to the rescue section
TASK [assert] ****
fatal: [localhost]: FAILED! => changed=false
  assertion: no_audit_tools == no_owner_root
  evaluated_to: false
  msg: One or more tools are not own by root.

TASK [debug] ****
ok: [localhost] =>
  msg: 'Rescue: audit tools not owned by root.'

Q: "This solution requires JMESPath to be installed. Is there an alternative solution?"
A: Yes. It is. Use Jinja filter selectattr
            no_owner_root: "{{ result.results|
                               selectattr('stat.pw_name', 'eq', 'root')|
                               list|length }}"


Best Practice Authenticating Jenkins to Bitbucket to Access Team Repository What's the "best-practice" approach to authenticate a CI tool like Jenkins or Bamboo to BitBucket? I started with App Passwords because it's cheap and easy, but I hate that it's tied to my account because if I leave the company, the CI/CD integration with Bitbucket gets killed.

Is there anything that isn't scoped to a specific user?

In an enterprise environment (but even in a personal one, I assume tech savvy enough users to do that) it's generally a good idea to have dedicated accounts created specifically for automated tools execution.

I'd even recommend one account per tool because different tools may have different permission requirements and it's generally a good idea to limit the permission scope to the minimum required, for security reasons. In large enterprises this is often mandatory, prescribed by policies.

In some cases it might be necessary to have different tools executed under the same account, in which case such combo accounts would be required. Again, one per specific use case.

If you want a parallel - take a look at a Linux system, for example - different system accounts used for different purposes. Or service accounts, at Google for example.


How do you reference a code commit tag in IAM If I add a tag to a codecommit repo with
code commit => the repo => Settings => repository tags => add tags
how do i reference this tag in IAM

I have in the policy document a statment i am trying to use to reference the tag Im not sure what i use in the resource section to refrence a tag I add.

statement {
  sid = "AllowCodeCommitAccess"
  actions = [
    "codecommit:GitPull",
  ]
resources = ["?????????"]
}

Also I haven't actually added the tag yet to codecommit repo as something on aws said it could impact access. I assume that just means If IAM uses it to denny resources but wasn't %100 sure and haven't added it yet.

I found what i needed to allow or the eqivalent StringNotEquals to deny useing the tag
aws:ResourceTag

"Condition": {
                "StringEquals": {
                    "aws:ResourceTag/Key_-1": "Value_-1"
                }
            }


Can you use `aws-sdk` to store objects in Google Storage? Google Storage claims they support interoperability in their solution with Amazon AWS S3 and Eucalyptus Systems APIs.

source: https://cloud.google.com/storage/docs/interoperability

Can the aws-sdk for Java, or Node.js, or Python, be used to store and retrieve objects with the Google Storage APIs?

Most of the AWS SDKs mentioned support constructing a custom endpoint, for example as described here for Node.JS.

Setting the endpoint to storage.googleapis.com and using your GCP access credentials should enable you to operate the Google storage backend service "like S3".


How do we move to a modern CICD approach This may also be a general pulse; but here lies the issue

We use EKS, we have Jenkins, we have Ansible, we have Terraform, we have BitBucket; but none of these really form a pipeline and I'm stuck with answering; do we need more (FluxCD/AgroCD/drone.io) etc... or what we have is enough and it's simply about stringing these things together.

Currently, Jenkins really just does mostly app deployment stuff of business app for our devs; but if I wanted to deploy like Fluentd to our clusters or pod security policies, it'll usually require me to manually deploy it.

Really trying to think of a modern CICD approach with security as the forefront, most of our usescases is deploying stuff to kubernetes; but that is not all of our usecase (some scragglers like firewall automation)

You have all the necessary tools for the 2 use cases you mentioned. For application development, it looks like you have something already set up.  I'm assuming, for time's sake, you have automated both code quality and security scanning of your application and that is all working.  If you haven't already started packaging your applications into helm to deploy into kubernetes I suggest you do that.  What your looking to do is automate your deployments of pre-built applications and infrastructure into EKS or cloud (aws).

To deploy applications onto your EKS cluster terraform allows you to deploy via helm.  So your Jenkins pipeline can deploy your terraform infrastructure which utilize the providers for kubernetes, eks, helm, AWS and more to provision and configure your infrastructure and applications. There is a separate pipeline usually for custom apps, pre-built apps, and infrastructure.  Each has it's own quality, security, and resiliency checks needed so you may need some additional tools as you go forward.

For example here is a link to the Hashicorps website on using terraform in automated pipelines.  Note it's considered advanced https://learn.hashicorp.com/terraform/development/running-terraform-in-automation

At this point I'm not sure what you're using Ansible for as that's not necessary for EKS and it's services but if you still need it in parallel to terraform there are plugins to Jenkins https://plugins.jenkins.io/ansible/.   There's also one for terraform https://plugins.jenkins.io/terraform//

So in short, no need to start adding tools but exploring the ones you have.  The way to look at it is this:

Task Runner and orchestrator for CI/CD: Jenkins

this will orchestrate all CI/CD

Application deployment: Helm

this would deploy and version all applications
applications can be deployed via jenkins or terraform

Infrastructure deployment: Terraform

initial testing done manually but Jenkins can run terraform to deploy infrastructure and base services.


System monitoring and reporting with Jenkins Are there any plugins that would enable me to distribute system monitoring and reporting via Jenkins, if I provided it with data in a required format. E.g. I would like to write a simple script that would display the status of every server and maybe some metrics. If that runs, say every hour, and dumps data in an XML or CSV or whatever conventional format of the plugin, can I distribute reports with that data using Jenkins using a special plugin?

I understand there are separate apps that do system monitoring but the fact of the matter I am trying to make Jenkins my one-stop-shop for what my less technical usership needs so, because I enabled them to run builds and deployments using Jenkins as they wish, I would prefer not to give them a different interface with a different login and other overhead.

having "ab"used Jenkins to be many things, the most sensible (IMHO) you can do is expose monitoring dashboards via build report page.
I think you should not even put Jenkins and monitoring under the same url, due to the "who will guard the guard" problem.

Also, nowadays monitoring tools are too versatile and have too many "faces" (a face for each characteristic user, and some specific users prefer to have their own personal "face" to monitoring)
About "reporting" I don't know what do you mean. if its reporting of builds statuses - READ ON.
Also there are other things to let user go through (logs, alerts, test reports)

So what I would suggest to address monitoring visibility is:

monitor your Jenkins in your monitoring :) (builds, statuses, etc.)
define monitoring url inside Jenkins (e.g. as Global Env Variable)
let's assume you have a job that deploys service $svc on environment $env. using Groovy you can generate:

generate a url to point at that monitoring service "dashboard" for that service in that env. let's call it serviceMonitorUrl
environmentMonitorUrl, and others
using addSummary and related stuff you can add to the build report a list of links to monitoring
you can decorate your build history column with "labels" pointing at branch, env, hash, whatever
your userdom will get build page with those links, and will be able to "click" them.

Having said that, there are Jenkins pipelines visualizations you can also use to visualize Jenkins jobs statuses, etc. e.g. this one.

Generally:
Jenkins is good for CI related scheduling.
But the truth is - CI should not be the focal point of users.
CI should be "transparent" until something is broken.
Thus making users "aware" of Jenkins (CI), and doing everything through it, unless business requirement is met by it (like company's product is being controlled/operated though Jenkins) - you should not do this. This kind of "knowledge" is not really serving the users.
They need to be able to see where to look for information easily, not be specific application's advanced users.

Think what happens when you decide to migrate from Jenkins to something else. you are hooked now.

(it isn't necessarily bad, just be aware of it)


archive contents of a reviewboard server as read-only My team was using an Ubuntu box to host an instance of Reviewboard 2.5.2 that we used for a few years before migrating to a Bitbucket service in the cloud. The machine was maintained by my predecessor who has long-since left the company.

Edit: to clarify, the new service is not Reviewboard, but Bitbucket, and it's hosted on an internal cloud service.

Now that we have a service elsewhere that we don't need to maintain, we want to take this old server offline permanently, but would like to preserve the contents site itself -- that is all of the reviews -- for posterity.

I'd rather not just spin up another Reviewboard instance and I'd also rather not be in the business of maintaining a physical web server -- the idea is that I'd put this in our company cloud.

Now I've done a dump of the json-formatted data via RBTool, however this isn't really useful to me in its current form. Edit: it's not useful because I was hoping for an off-the-shelf way to dump the website in static form.

What I ultimately want is to have these old, archived reviews be searchable and for all the links to work more or less as expected for the sake of convenience.

Has anyone had luck doing something like this?

What I ultimately want is to have these old, archived reviews be
  searchable and for all the links to work more or less as expected for
  the sake of convenience.

As already indicated in one of the comments, wget could be used to retrieve the content:

wget -r --no-parent devops.com

will create a devops.com folder and put all files in this directory.


Jinja syntax error: missing endif, in spite of providing it I am using Salt 2019.2.0 version and below is my state file where I am using if condition to check the values and run states if it's a match. Even if I have provided endif, it says Jinja looking for missing endif tag. Any help please?
CODE:
{% if salt['pillar.get']('spade:Corename') == 'FI' and salt['pillar.get']('spade:NodeType') == 'master' %}
dataimport-script:
  file.managed:
    - name: /opt/bin/DataImport.py
    - source: salt://files/DataImport.py

dataimport-crontab:
  file.managed:
    - name: /etc/cron.d/solr-dataimport
    - contents: |
                # # set up data import for FI every 2 minutes
                */2 * * * * root /usr/bin/python /opt/bin/DataImport.py
{% elif salt['pillar.get']('spade:NodeType') in ['slave','ds'] and salt['pillar.get']('spade:Corename') == 'FI' %}
update-fi-solrconfig:
  file.replace:
    - name: {{ salt['pillar.get']('solr:home_dir') }}/data/{{ salt['pillar.get']('spade:Corename') }}/conf/solrconfig.xml
    - pattern: '"autoDeletePeriodSeconds">30'
    - repl: '"autoDeletePeriodSeconds">-1'
{% endif %}

ERROR:
local:

Data failed to compile:
----------
Rendering SLS 'base:solr.install-solr' failed: Jinja syntax error: Unexpected end of template. Jinja was looking for the following tags: 'endif'. The innermost block that needs to be closed is 'if'.; line 205
---

[...]

update-fi-solrconfig:
file.replace:
- name: {{ salt['pillar.get']('solr:home_dir') }}/data/{{ salt['pillar.get']('spade:Corename') }}/conf/solrconfig.xml
- pattern: '"autoDeletePeriodSeconds">30'
- repl: '"autoDeletePeriodSeconds">-1'
{% endif %} <======================

There is no error in what you have posted here. The error lies elsewhere in your code which you did not include in the OP. Using the Jinja live parser, when pasting in:
{% if spade.corename == 'FI' and spade.nodetype == 'master' %}
dataimport-script:
  file.managed:
    - name: /opt/bin/DataImport.py
    - source: salt://files/DataImport.py

dataimport-crontab:
  file.managed:
    - name: /etc/cron.d/solr-dataimport
    - contents: |
                # # set up data import for FI every 2 minutes
                */2 * * * * root /usr/bin/python /opt/bin/DataImport.py
{% elif spade.nodetype in ['slave','ds'] and spade.corename == 'FI' %}
update-fi-solrconfig:
  file.replace:
    - name: {{ salt['pillar.get']('solr:home_dir') }}/data/{{
salt['pillar.get']('spade:Corename') }}/conf/solrconfig.xml
    - pattern: '"autoDeletePeriodSeconds">30'
    - repl: '"autoDeletePeriodSeconds">-1'
{% endif %}

With a pillar of:
spade:
  corename: FI
  nodetype: master

Everything renders just fine. This tells me there is nothing wrong with your snippet. The fact that your error indicates you have at least 205 lines of code, but the example has far less tells me the issue was with another "if" block further up in your code.


How to run experimental Docker features on CircleCI When a build is started on CircleCI one of the first things that is done is:

  Setup a remote Docker engine

Specified reusable docker engine, but build has not been whitelisted.
Contact CircleCI to be whitelisted
Allocating a remote Docker Engine
...
Remote Docker engine created. Using VM 'prealloc-wrjtu1qd-1491949826270'
Created container accessible with:
  DOCKER_TLS_VERIFY=1
  DOCKER_HOST=tcp://<IP>:2376
  DOCKER_CERT_PATH=/tmp/docker-certs615987123
  DOCKER_MACHINE_NAME=51123

Subsequently when docker build --squash -t imagename . is run on CircleCI it results in:

Error response from daemon: squash is only supported with experimental mode
Exited with code 1

Discussion

It works to use experimental features locally after changing the /etc/docker/daemon.json as follows:

{
    "experimental": true
}

and restarting the docker systemctl service, but how to do that on CircleCI? It seems impossible as the build output indicates that a connection is made to a remote docker that reside on the CircleCI systems.

CircleCI now allows you to use a machine executor which provisions a separate VM for you, with Ubuntu 14.04 and Docker version 17.06.0-ce installed. This allows you to turn on experimental features for the Docker daemon.

You need to use the machine key instead of the docker key, to run your job in a separate virtual machine instead of just a Docker container.

You can only choose 2 images for the machine:

circleci/classic:latest: Ubuntu 14.04 with Docker 17.03.0-ce, or
circleci/classic:edge: Ubuntu 14.04 with Docker 17.06.0-ce - the one with experimental features.

You'll also need to install dependencies on the machine yourself, as it is quite bare. For example, if you need PHP for your tests, you'll need to run sudo apt-get install -y php5.

Here is a sample .circleci/config.yml that builds a Docker image using experimental feature docker build --squash:

  .circleci/config.yml

version: 2
jobs:
  build:
    # Run in a separate virtual machine instead of a Docker container.
    machine:
      enabled: true
      # Use Ubuntu 14.04 with bleeding edge Docker daemon 17.06.0-ce.
      image: circleci/classic:edge
    steps:
      - checkout
      - run:
          command: |
            # Restart Docker with experimental features on.
            sudo sh -c 'echo '\''DOCKER_OPTS="--experimental=true"'\'' >> /etc/default/docker'
            sudo service docker restart

            # Install dependencies for tests etc.
            sudo apt-get update
            sudo apt-get install -y php5

            # Build image with experimental feature --squash.
            docker build --squash -t myuser/myimage .

            # Login and push Docker image to registry.
            docker login -u $DOCKER_USER -p $DOCKER_PASS
            docker push myuser/myimage


How to generate Code Churn in VSTS? In this article I noticed that is it possible to generate code churn reports for TFS:

I would like to generate the similar report using Visual Studio Team Services (VSTS). Is that possible and, if so, how?

For now you can't.
But you have Analytics and there is a Code part that is in the roadmap.
Take a look there anyway. It has some awesome widgets so far.


Google Cloud Storage vs Redis / ElasticSearch We're building an application that reads data from an API and displays it in a frontend. Due to API limits, we have to cache the API calls meaning that we save the result of the API calls (both user information (first name, country, email etc.) and big sets of activity log data) in a database and only request new data from the API if the stored data is older than x hours.

This application should also work for multiple users that can log in and see their relevant data. For each user, the application connects to a different API endpoint (e.g. "user1.api.com/data/whatever" and "user2.api.com/data/whatever").

So, in the end there will be two databases. One for caching the API calls of an external tool and another one for the application's user management. However, we are not sure what would be best to use.

Is it generally a good idea to use cloud databases for either caching or user management? As the application will run on Google Container Engine, we could probably use Google Cloud Storage (for caching) and Firebase (for user management). Or should we rather use something like Redis / ElasticSearch (for caching) and something different for the different user data?

I don't know much about Redis/ElasticSearch, but GCS is not really a database-like solution, it is closer to a file storage solution.

If you're looking for database-like storage Google Cloud offers:

Cloud SQL
Cloud Bigtable
Cloud Datastore
Cloud Spanner

If you expect a modest app traffic or alternating high/low traffic it might be more cost-effective (and simpler) to have a Google App Engine app instead of a GCE-based solution.


How to automatically publish git releases/comments to the site? Sorry if title isn't correct.  I'm trying to figure out what to search for on this matter.

I'm looking for the easiest way to automatically have my semantic versioning and commits visible on my site when commits / releases happen.

I recently started using Git, GitHub, GitFlow, Axosoft, and GitKraken.    Are there any options, plugins, or RSS type feeds in those products that do this already or is this something I would have to create using Git webhooks which would call my site and insert into my DB?

I can handle the PHP/MySQL.  I'm just looking for recommendations on how to setup the communication between the products I listed above so that I can create a PHP page showing this to my customers...

[timestamp] - RELEASE v2.1.1
[timestamp] - [axof: 2] ENHANCED XYZ [...a sentence or two of details...]
[timestamp] - RELEASE v2.0.1
[timestamp] - [axof: 1] BUG ABC FIXED [...a sentence or two of details...]

We did have a similar situation and did try out a few options. Let me suggest them and you can choose if they suit your need.
We had Jira for User Story (requirements), Bitbucket for SCM, Jenkins for CI orchestration, to name a few. Am not going into the other components since they may not be relevant from what I understand based on the description.
So, to start with, Jira would hold our (developer) user stories, and the developers start analysis based on that. Once they are clear, they pull the relevant code base from Bitbucket straight into the IDE (Eclipse) using git plugins, complete the work and push code back into Bitbucket. Two things were needed to be followed 'as part of the comment given during commit':

The Jira ID for the user story for which the change is being made
The actual comments for the change
E.g., [JIR-BBA]:: Code updated to pull correct URL from database.

Then, in the Jenkins job, we strip down the same into 'JIR-BBA', i.e. Jira ID, and the comment. The Jira ID would then be fed into the Bitbucket plugin which would directly update the corresponding Jira story. And this would also provide a hyperlink in Bitbucket which would directly take you to the job run in Jenkins, thus providing a 2-way communication.
Now the other part in your question, about the status of the build. For this we followed the below steps:

Created a Python script that would connect to Jira and can be run from the command prompt. This is easily available if you search for those 2 terms. The script would take in 4 parameters: i. Jira URL, ii. Jira credential, iii. Jira ID, iv. Comments
In Jenkins, we called the python script which would connect to Jira using the above credentials, and update the correct Jira task with the comments the developer provided (from the build job) and the job completion status (from SIT Deploy, SIT Test, UAT Deploy, etc.)
Now, inside Jira, once you go to the task, you shall find comments being updated for each of the jobs like 'Dev done', 'QA done', etc. depending on how you would want to get it update for each of the Jenkins jobs.
That way, you are maintaining a traceability for each of the requirements so that you know which stage each of them are deployed into or test failed.
Thus you do not need to go into Jenkins each time for the status as the latest is already available in Jira.


Running services on different ports in development environment and production environment A bit of an odd question I think but here goes: I have a couple of services running in an application that I'm building. In production each service is containerised and lives on its own server. But on my development computer I like to keep local versions all running. Of course, if I have multiple services running on localhost, the only way to do this is to have them each on their own port. Now, to maintain parity between development and production, I figure that I must now also get them all running on those different ports in production.

So I guess my question is is it okay to have all these services running on "non-standard" ports (eg, a webserver that isn't on 80 or 443) in production. My understanding of ports is that:
1) There are some ports usually used for specific services.
2) This isn't mandatory.
3) Some people deliberately change them for security through obscification, but this doesn't achieve much.

So it should be fine? Or is there a better practice for this problem?

This should be fine.  It's something you may even have to do if ports are already in use on localhost.  Even development languages like .net core are now supporting different environments that can be used to configure the different ports.

Example: 1433 is the standard sql server port.  If I wanted to test a local sql server container, but did not want to stop my locally installed sql server, I could run it on another port (e.g. 1431).


How to Remove a Kubernetes (GKE) Cluster node-pool without down-time I am using Google Kubernetes Engine.
I have a multiple Kubernetes cluster node-pools.
I want to delete one of them.

Is there a way that I can first migrate all the pods on one node-pool to another node-pool through the same kind of rolling update methods found in other features?

I thought it would do this automatically for me.
But when I deleted the node-pool I got downtime, luckily this was not a production environment.

I would like to know if there is a way to automatically migrate and move all pods from one node-pool to another node-pool and slowly roll the change when a node-pool is removed?

Thanks

If you cordon and drain the nodepool before deleting it, then you can avoid downtime. I use the following script (shamelessly taken from the lazyweb elsewhere and adapted to my needs):

oldpool=pool-1
oldnodes=$(k get no --selector='cloud.google.com/gke-nodepool='$oldpool -o json | jq .items[].metadata.name -r | xargs)

kubectl cordon --selector='cloud.google.com/gke-nodepool='$oldpool

for n in $oldnodes; do
  echo "draining $n"
  kubectl drain --delete-local-data --ignore-daemonsets $n
  echo "----------------------------"
done;


What's the way to deploy a resource once on a helm chart that is deployed multiple times? in my use case i've got a helm chart for my application.

One of the resources is actually a Grafana Dashboard for the application.

This dashboard actually monitors all instances of the application, and i want it to be deployed only once.

Considering i'm performing 2 helm install commands for the same apps, which will cause the dashboard to be deployed twice, what's the best practices to have it deployed only once?

Should it be a totally different "release"? (and be deployed individualy?)
Should I specify in the 2nd installation not to deploy the dashboard? (seems bad)

What's the best practice?

I've eventually created an umbrella chart with alias:

dependencies:
  - name: app
    repository: file://./app
    version: 1.0.0
    alias: child1
  - name: app
    repository: file://./app
    version: 1.0.0
    alias: child2

And in values.yaml i've changed the needed values,
then added the grafana dashboard in the bundled yaml file.


Ensure a line is uncommented in an XML file using ansible How can I use ansible to ensure that a line is uncommented in an XML file?   In /etc/tomcat8/context.xml I want to ensure that the section:

<!-- Uncomment this to disable session persistence across Tomcat restarts -->
<!--
<Manager pathname="" />
-->

Is changed to:

<!-- Disable session persistence across Tomcat restarts -->
<Manager pathname="" />

The lineinfile module doesn't appear to be able to help because this is a multi-line section
The xml module might be able to ensure that the <Manager... is in place, but it doesn't appear to be able to delete the commented out version.
I could write a script to do this with sed if I have to, but I find it is better to use ansible modules when possible.

As mentioned in the question and the comments on it, this isn't an area in which Ansible excels.  The "best" solution depends a bit on the type of file you're dealing with, and the structure of it.

The cleanest way to deal with these sorts of changes in Ansible is to templatize the entire thing; this allows you to see the entire file at once, rather than piecing it together through many places, and is much less fragile than pattern- or line-based approaches.  When you need to add configuration in multiple places (for instance, different roles), that's when dynamic-loading directives become very useful, like Apache httpd's Include:

Include /etc/httpd/conf.d/*.conf

This then allows you to write individual configuration files from different places, and let the client software concatenate them.  However, as far as I can tell from a quick search, Tomcat doesn't support this, so you would need to operate with a single template file.

lineinfile is a hacky solution that's very fragile, and template is almost always a better option.  However, in this case, the node that you're adding is a self-closing one, so as long as it is top-level, it can go anywhere and lineinfile will work (I know you say it won't, but I'll get back to that).  xml is a similar option that's a bit less fragile.

Those two you've ruled out because they can't delete the comment you have in the file.  However, I'll argue that that's an improper requirement (in most cases, and probably yours).

When using a configuration management tool, the source of truth shifts from the files themselves on servers to the configuration that's checked in to master.  If someone wants to edit the file, they'll do so by editing the configuration management scripts, not by editing the file directly on the server.  Therefore, there is no need for comments in the resultant file that lives on the server, except perhaps one at the top that says "this is auto-generated, go look at source control instead".  The comment should be in your templating code - and as long as the result compiles and works correctly, you shouldn't care at all what it looks like.


Trigger cicd pipeline after merge request accept on master, Gitlab Trying to run some task after merge request get accept (only on master branch).
I am aware of only: - master -merge_requests config options, but want to specifically run pipeline only when code get merged in master.

https://docs.gitlab.com/ee/ci/yaml/

https://gitlab.com/gitlab-org/gitlab-ce/issues/31296#note_34944784

integration-testing:
  stage: test
  only:
    - master
  script:
    - ./gradlew clean build asciidoctor

Another option is to create a webhook

https://gitlab.scm.webanywhere.co.uk/help/user/project/integrations/webhooks.md

If the "Merge request events" is clicked one could trigger a pipeline if a Pull Request gets updated, created or merged.


Is it possible to view all changes made on a Git feature branch in Azure Devops? Azure DevOps offers the overview of changes made in a branch when a PR is created. However, sometimes I'd like to have access to that overview when viewing a branch without the need to create a PR (or check out the branch locally). In the 'branches' section it's possible to view the changes on per commit basis, but the overview of all changes seems to be missing.
Is it possible to have this kind of overview of changes made in one branch in Azure DevOps without creating a PR?

You're looking for the branch compare functionality.  In the list of branches, click the overflow ellipses on your feature branch and select compare branches.

Azure DevOps will diff you branch with the default branch which is changeable if necessary.


Nginx routing setup within Docker container, request timing out with 502 I have a React application where a user can enter in some number, there is an algorithm that runs some calculation and spits out a result, but before we get there, Nginx is looking at the request and deciding whether to route it to React or Express server.

Well when the number is submitted, there is an xhr request that remains pending until it just times out with a 502 bad gateway error.

All this is happening inside of a Docker container.

I ran a docker logs <container-id> and kept seeing this:

upstream server temporarily disabled while connecting to upstream

I thought it might be a typo in my Express server or in my worker process or even in the React application but I find none, so I am left wondering if the issue has to do with how I built my default.conf file:

upstream client {
    server client:3000;
}

upstream api {
    server server:5000;
}

server {
    listen 80;

    location / {
        proxy_pass http://client;
    }

    location /sockjs-node {
        proxy_pass http://client;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "Upgrade";
    }

    location /api {
        rewrite /api/(.*) /$1 break;
        proxy_pass http://api;
    }
}

I feel like my knowledge of Nginx configurations and upstream is limited to where something may not be correct about this setup. After running for awhile the whole application crashed.

To get this configuration to override the default default.conf inside of Nginx I setup a Dockerfile.dev inside of nginx/ folder:

FROM nginx
COPY ./default.conf /etc/nginx/conf.d/default.conf

The application overall is using this docker-compose.yml file:

version: "3"
services:
  postgres:
    image: "postgres:latest"
  redis:
    image: "redis:latest"
  nginx:
    restart: always
    build:
      dockerfile: Dockerfile.dev
      context: ./nginx
    ports:
      - "3050:80"
  api:
    build:
      dockerfile: Dockerfile.dev
      context: ./server
    volumes:
      - /app/node_modules
      - ./server:/app
    environment:
      - REDIS_HOST=redis
      - REDIS_PORT=6379
      - PGUSER=postgres
      - PGHOST=postgres
      - PGDATABASE=postgres
      - PGPASSWORD=postgres_password
      - PGPORT=5432
  client:
    build:
      dockerfile: Dockerfile.dev
      context: ./client
    volumes:
      - /app/node_modules
      - ./client:/app
  worker:
    build:
      dockerfile: Dockerfile.dev
      context: ./worker
    volumes:
      - /app/node_modules
      - ./worker:/app

I am also wondering if the default.conf file is fine, but its my Express API that is taking too long to respond.

Then towards the bottom of the docker-compose.yml file I added some Redis environment variables I thought it may have needed below worker:

 worker:
    build:
      dockerfile: Dockerfile.dev
      context: ./worker
    volumes:
      - /app/node_modules
      - ./worker:/app
    environment:
      - REDIS_HOST=redis
      - REDIS_PORT=6379

I rebuilt the image with docker-compose up --build and restarted the application, and I can see the previous xhr requests still pending and they still give me a 502 and 504 error after a non-zero amount of time pending.

You will notice in this upstream block:

upstream api {
    server server:5000;
}

I called it api but then I say server server:5000;. That was the problem because I originally had it as:

upstream server {
    server server:5000;
}

but then thought that server might be a reserved operator in the Nginx world so I changed it to api but did not change it inside the block.

So it should have been:

upstream api {
    server api:5000;
}

After changing it, the xhr pending and timing out 502 errors went away.


Chef username/passwords best practice for multiple nodes I've been playing around with Chef to evaluate it as a potential CD tool for the company I work for.

So far I've been using it with Vagrant to provision a VM. In my recipes I've been retrieving the username/password from data bags.

I'm now looking to re-use these recipes across multiple nodes which all have different credentials so pointing to one data bag for credentials is no longer sufficient.

The only way I can see to get around the issue is to store the username and password in the node attributes, however is this best practice/secure?

I've done some googling around this and can't get a clear answer...

Storing sensitive data in attributes is definitely not a good practice. I once used a strategy of a chef-vault per environment (dev,stg,prod), and it worked well. If you use the sensitive data within templates you can set the sensitive property on the template resource to true to keep the password in plain text form from showing up in the run logs and associated template file diffs

We eventually moved to Hashicorp vault, and while it took more time upfront to learn and get up and running, it was definitely a better long term solution for secrets management and automation around managing sensitive data


How do I make read only mount in Docker container writable? When developing code on a very large repository, I would like to run multiple builds/tests in parallel in containers. I need to get my repository or some other large directory into the container. If I mount it read/write, the parallel processes will fight each other and I will need to be able to clean after them, there is also issue with privilege escalations on those files. If I mount it read/only then I cannot run processes that write files inside the directory.

Docker suggest to build a new container with a COPY of the repository, save it as new layer and then start several containers from this new image. Having to make a copy of megabytes of data every time you want to run tests is making it take too long though.

How can I solve this efficiently?

Solution to this problem is to create an overlay filesytem over a read-only mount, but if you try to do it directly, overlay will refuse to put upper and work directories on another overlay filesystem. The trick is to create a tmpfs for the upper and work directories like this:

Create a script called run-in-c.sh

#!/bin/bash
NAME=$1
shift
HOSTNAME=dock${NAME}
CONTAINER=dc-${USER}-${NAME}
REPOSITORY=${HOME}/repository
BASEIMAGE=hub.docker.io/my-org/my-base-container
OVERLAY=/mnt/overlay
LOWERDIR=/mnt/lower
UPPERDIR=${OVERLAY}/upper
WORKDIR=${OVERLAY}/work
TARGET=/mnt/repository
# PRIVILEGED="--privileged"
PRIVILEGED="--cap-add SYS_ADMIN"
docker container create --name $CONTAINER $PRIVILEGED --hostname $HOSTNAME --volume ${CO}:${LOWERDIR}:ro $BASEIMAGE
docker container start $CONTAINER
docker container exec $CONTAINER mkdir -p $OVERLAY
docker container exec $CONTAINER mount -t tmpfs tmpfs $OVERLAY
docker container exec $CONTAINER mkdir -p $WORKDIR $UPPERDIR $TARGET
docker container exec $CONTAINER mount -t overlay overlay -o lowerdir=${LOWERDIR},upperdir=${UPPERDIR},workdir=${WORKDIR} $TARGET
docker container exec -it --workdir $TARGET $CONTAINER $*
docker container stop $CONTAINER
docker container rm $CONTAINER

Then you can run commands in the container as:

run-in-c.sh test01 'cd dir && command args'

or simply get interactive shell with:

run-in-c.sh naming-stuff-is-too-hard bash


What are feasible quality criteria for Git repositories? If you manage a lot of Git repositories (or other, but Git is a specific and common technology), it could happen that depending on users there will be quite different usage profiles. On the worst case end you could face issues with broken repositories (unplanned work, risks) and wasted storage (which is claimed to be cheap but is still accounted for).

Who is responsible for imposing minimum acceptance criteria for "quality of usage" (I cannot find better term for now) and what are realistic quantifiable measures you could install and link to actionables?

Examples of anti-patterns and related risks you might be dealing with:

Folks upload files, also binaries, which are neither part of the software nor test documents
Binary libraries instead of using dependency management
Repos become too big (over 15G as stated here [1])
Repos are below 15G but the history is many gigabytes
Other?

Or, should anybody care about these things at all? (formal objections, not opinions, why not care)

[1] https://www.quora.com/What-is-the-practical-maximum-size-of-a-Git-repository-full-of-text-based-data

Who is responsible for checking that people don't check in commits that have large files? The same people who are responsible for checking that the commits aren't bad in other ways: everyone.

If git is new in your company, make sure that the git education includes this sort of thing. After that, it's up to code reviewers to notice mistakes and correct them.

If it becomes a repeated problem, then whoever is in charge of developer tooling can write some checks that run every night or whatever and let people know that something slipped through. But that seems like overkill in most situations.


Ansible - when statement task always return false value I've got 1 master node from which I would like to instal NewRelic agents remotely depedning on the OS I have on them. Some of them are Amazon 2 Amis and some Amazon 1 Ami.
I put when statement in the Ansible Playbook's task in the following way:

- name: Enable repo for Amazon Linux 2/Red Hat 7v
  yum_repository:
    name: newrelic-infra
    description: NewRelic repo
    baseurl: https://download.newrelic.com/infrastructure_agent/linux/yum/el/7/$basearch
  when: (ansible_distribution == "Amazon" and ansible_distribution_version == "(Karoo)") or
        (ansible_distribution == "RedHat" and ansible_distribution_major_version == "7")

- name: Enable repo for Amazon Linux 1/Red Hat 6v
  yum_repository:
    name: newrelic-infra
    description: NewRelic repo
    baseurl: https://download.newrelic.com/infrastructure_agent/linux/yum/el/6/$basearch
  when: (ansible_distribution == "Amazon" and ansible_distribution_version == "2018.03") or
        (ansible_distribution == "RedHat" and ansible_distribution_major_version == "6")

I understand that when statement should check the if statement and proceed further. Nevertheless, whenever I run playbook I receive that if statement is false and it's skipping above tasks. What might be the root cause?

I've already checked on remote host ansible_distribution and ansible_distribution_version if strings match. I suppose it must be some syntax problem as I'm new in ansible.
Thanks in advance.

Your statements look ok. If I create a test case from the sparse info you provided and what I can observe when gathering facts from my own hosts, I can detect old/recent distributions correctly using your conditions. See my example below.

I suggest you double check the data gathered from your host to understand why it doesn't match. The easiest way for that: from the same host your use to launch your playbook, run

ansible -i path/to/your_inventory.ini your_target_group_name -m setup -a "filter=ansible_distribution*"

Here is the test.yml playbook I wrote to test your (known-so-far) values and your conditions

---
- name: Statements for checking old/recent distribs
  hosts: localhost
  gather_facts: false

  vars:
    os_list:
      - ansible_distribution: "Amazon"
        ansible_distribution_version: "(Karoo)"
        ansible_distribution_major_version: "(Karoo)"
      - ansible_distribution: "Amazon"
        ansible_distribution_version: "2018.03"
        ansible_distribution_major_version: "2018"
      - ansible_distribution: "RedHat"
        ansible_distribution_version: "6.x"
        ansible_distribution_major_version: "6"
      - ansible_distribution: "RedHat"
        ansible_distribution_version: "7.x"
        ansible_distribution_major_version: "7"

  tasks:

    - name: Check conditions for recent distribs
      debug:
        msg: "This is a recent distrib"
      loop: "{{ os_list }}"
      loop_control:
        label: "{{ item.ansible_distribution }}-{{ item.ansible_distribution_version }}"
      when: (
              item.ansible_distribution == "Amazon"
              and
              item.ansible_distribution_version == "(Karoo)"
            )
            or
            (
              item.ansible_distribution == "RedHat"
              and
              item.ansible_distribution_major_version == "7"
            )

    - name: Check conditions for old distribs
      debug:
        msg: "This is an old distrib"
      loop: "{{ os_list }}"
      loop_control:
        label: "{{ item.ansible_distribution }}-{{ item.ansible_distribution_version }}"
      when: (
              item.ansible_distribution == "Amazon"
              and
              item.ansible_distribution_version == "2018.03"
            )
            or
            (
              item.ansible_distribution == "RedHat"
              and
              item.ansible_distribution_major_version == "6"
            )

And the result

PLAY [Statements for checking old/recent distribs] ************************************************************************************************************************************************************************

TASK [Check conditions for recent distribs] *******************************************************************************************************************************************************************************
ok: [localhost] => (item=Amazon-(Karoo)) => {
    "msg": "This is a recent distrib"
}
skipping: [localhost] => (item=Amazon-2018.03)
skipping: [localhost] => (item=RedHat-6.x)
ok: [localhost] => (item=RedHat-7.x) => {
    "msg": "This is a recent distrib"
}

TASK [Check conditions for old distribs] **********************************************************************************************************************************************************************************
skipping: [localhost] => (item=Amazon-(Karoo))
ok: [localhost] => (item=Amazon-2018.03) => {
    "msg": "This is an old distrib"
}
ok: [localhost] => (item=RedHat-6.x) => {
    "msg": "This is an old distrib"
}
skipping: [localhost] => (item=RedHat-7.x)

PLAY RECAP ****************************************************************************************************************************************************************************************************************
localhost                  : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

As you can see it does the job as expected. So either your data is not gathered correctly or your check the wrong values.

Note: When @Vladimir Botka asks for an MCVE in the comments, this is a typical example.


What tools should I deploy to support AWS for a Managed Services company? I work for a managed services company and am working out a plan to retrofit our customer support machines to support AWS in a general way.

We use Ansible and lots of ssh port forwarding to connect to traditional, non-cloud, boxes (Windows, Linux, AIX, Firewalls, routers, UC, other Cisco stuff).

Beyond the AWS CLI, what tools should I deploy to my Linux and Windows support boxes for managing, monitoring and troubleshooting AWS that are different than the traditional tools?

Installing a box is probably not the best solution, mostly because there's no real need for it.

You should ask for an access to AWS console in customer account with proper rights fitting your responsibilities and maybe setup AWS config, Cloudwatch for the monitoring and alerting part, Cloudtrail and maybe Amazon Inspector for the audit part.

To ensure machines and AWS services themselves are properly configured you may rely on AWS Config/Amazon Inspector or tools like inspec (and here you may or may not need a bastion host on your customer VPC to check other machines, this mostly depends on the agreement you have with customer about direct links/VPN)


How to track changes to AWS settings? Is there a way to track changes we make to the AWS system?

For example, changes to a subnet setting, from using nat to iwg - these display a message, and then disappear.

Is there a way to get AWS to create a log so that one can track which changes were made to what and when?

The closest thing we have now is the ElasticBeanstalk events - but even that only tells you what AWS did, not which settings were changed to cause the events.

AWS has the CloudTrail service that tracks most of the API calls done in your account. Then stores these in files in your specified S3 bucket.

https://aws.amazon.com/cloudtrail/

Using CloudTrail you can see who, or which service, called which change - in many cases also including which arguments were used.

Unfortunately it doesn't support ALL services at the moment.


AWS Setting instance to use domain from another account We're transitioning between accounts and we have an app on our new account but we want to use the domain from our old account. We plan on shutting off the old account soon and transitioning everything over, but there's services running on the old account so we can't do that yet.

The best idea I had is to try and use RAM and share the VPC instance and then set a new load balancer in the old account and have a DNS entry in Route 53 pointing to that load balancer. Whenever I try to share the resource it's giving me and error that I don't have permissions to share with an external account, even when I use a root account, I've tried writing policies to give myself permissions but that didn't work either, but I could have written the permissions incorrectly.

Would anyone know how to fix this permissions issue? Or know of a better way to achieve my goal?

Let me know if you need any other info.

If I'm getting this right; Maybe you can just create on your old account's Route53 public hosted zone a CNAME record (if you have a load balancer on your new account) or an A record for the instance on your new account. Just don't use the Alias functionality from Route53 as you wont find the resources from the new account.


How can I use a buildah script for image generation using GitHub Actions? Buildah scripts typically use shell. You can see an example of a script here,
#!/bin/sh
ctr=$(buildah from alpine:3)
buildah commit "$ctr" myAlpineImage

Let's say I have such a shell script that produces an image "myAlpineImage". How can I use GitHub CI to automate the creation of this image, and preferably uploading it to GitHub Container Registry?

Here is what I did, I create a Makefile that includes my own method of image creation,
.PHONY: image clean image-release
image: image-release

image-release:
    cargo build --release
    buildah unshare ./image/distroless.sh release

clean:
    cargo -v clean

For this purpose the only thing that matters is the image created by ./image/distroless.sh matches the action (redhat-actions/push-to-registry) in the following flow. We'll continue using the name myAlpineImage as in the question,
name: Distroless Image Creation

on:
  push:
    branches: [ master ]
  pull_request:
    branches: [ master ]

env:
  CARGO_TERM_COLOR: always

jobs:
  build:

    runs-on: ubuntu-latest

    steps:
    - uses: actions/checkout@v2
    - name: Download and create image.
      run: make image-release
    - name: Push To Registry
      uses: redhat-actions/push-to-registry@v2.1.1
      with:
        image: myAlpineImage
        tags: latest
        registry: ghcr.io/evancarroll/project
        username: evancarroll
        password: ${{ secrets.GHCR_TOKEN }}

You can see here we make the call to make image-release which calls the buildah script and makes the image rootless.
Then I added the secret for GHCR_TOKEN to my repo. The token is generated in the "Settings / Developer settings / Personal access" tokens on github..


Using Jenkins to control docker image build flow I'm trying to figure out the proper way to get build environments into a CI pipeline.

What I want to achieve is:

someone pushes to the docker source repo (not registry, a repo with the docker folder)
jenkins kicks off a build of that docker image
if successful docker then goes and runs a software build using the newly built docker image
if successful the docker image is tagged and pushed to the docker registry

All I have so far is a Jenkins file that has four stages

checks out out docker source repo
builds docker image from source
Checks out software source code (code that builds in the docker environment)
builds software source code in docker container
If successful tags and pushes new image to docker registery

My main concern is everything I'm doing is going to be in the 'script' blocks, except the git checkouts, and it feels super hacky to be building and kicking off docker builds with jenkins using bash...

are there any tools/already defined procedures to do things like this?  Is this the best way to do it?  I am looking for a best practice approach as this will be put into production and I want it to be as resilient and long lasting as possible.

I have found a solution so I'll leave it here for others.

using docker

#!/usr/bin/env groovy

pipeline{
agent buildsvr

  stages{
    stage('Checkout and build Docker Source'){
      steps{
        checkout([
          $class: 'GitSCM',
          branches: [[name: '*/master']],
          doGenerateSubmoduleConfigurations: false,
          submoduleCfg: [],
          userRemoteConfigs: [[
            credentialsId: 'asdf787c-5fd4-4dfb-81ad-873jhf107223',
            url:
'git@mydockersourcerepo.git']]])
        sh 'pushd docker && docker build -t ${image}:latest . && popd '
      }
    }
    stage('Checkout Target Source'){
      steps{
        script{
          checkout([
            $class: 'GitSCM',
            branches: [[name: 'mybranch']],
            doGenerateSubmoduleConfigurations: false,
            extensions: [[
              $class: 'SubmoduleOption',
              disableSubmodules: false,
              parentCredentials: false,
              recursiveSubmodules: true, reference: '',
              trackingSubmodules: true
              relativeTargetDir: 'source']],
            submoduleCfg: [],
            userRemoteConfigs: [[
              credentialsId: 'Bitbucket-Jenkins',
              url: 'ssh://git@sourcerepo.git']]])
        }
      }
    }
    stage('Build Source Target'){
      steps{
       script{
          sh 'docker-compose up'
        }
      }
    }
  }
  post{
    success{
      script{
        sh 'docker tag ${image} 10.90.239.223:5000:/${image} && docker push
10.90.239.223:5000:/${image}'
      }
    }
  }
}


How to explain devops to non-techy managers? I'm totally excited about DevOps. I know that DevOps is the methodology that will move us into building an IT infrastructure that will streamline and move our company forward.
But how do I sell this to my bosses, especially non-technical bosses?

We are going to implement an automation project which will include, automated deployment , infrastructure cloudification , continuous integration process .. we definitely need to persuade our bosses to invest at higher levels in this.

Note: we started to improve our process by automating tests, releases and supervision , it's a step towards adopting devOps but the automation project itself is in stand by as we need more investment.

Being a consultant I am contractually obliged to answer, "it depends".  With that out of the way, I can actually answer your question.

What does it depend on? Well, that could come down to what your boss thinks about DevOps:

If your boss has heard of the term, via their obsession with CIO.com maybe, then ask them what they think it means.  From there work out what the difference is and if their view is compatible.  Identify a suitable project to trial DevOps on and pitch it to them.  Remember that at the core DevOps is culture so consider how that could be applied to a project.
If your boss has never heard of the term then build a business case for DevOps. Use the Puppet Labs State of DevOps and material from books such as The Phoenix Project to write the business case.  Find a problem that your boss has and DevOps could solve and use it as a conversation starter. As kenchew said you don't have to mention DevOps, you might for example, suggest that Operations be more involved in a project or more test automation to be planned as part of project delivery.
If your boss thinks DevOps is just another buzz word then do one of the above but absolutely don't mention DevOps, look at other similar models such as Site Reliability Engineering, Platform Engineering or Continuous Deployment and find out how they could solve the problem.

The key is to focus on understanding what your boss is motivated by, then carve out a small bit of time, money and people to take tangible steps to solving that problem.

I highly recommend the book To Sell Is Human by Daniel H. Pink, fundamentally Daniel Pink talks about how selling something is a very human thing to do, all we need to do is tap into the needs and align our "pitch" by proposing a solution that serves those needs.


What is a good strategy to keep my site online when S3 goes offline? What is a good strategy to keep my site online when S3 goes offline?

If S3 US East 1 goes offline, how should I have my app configured/structured to prevent that taking my entire site offline?

What are the best strategies to diversify in this sort of situation?

In March 2015, Amazon AWS announced they support S3 replication across regions. When a certain region in S3 goes offline, you can serve files from your mirror in another region.

source: https://aws.amazon.com/blogs/aws/new-cross-region-replication-for-amazon-s3/

The practice of keeping your infrastructure online by doing a switch over to another region is a complex one, but S3 is a relatively small and simple component. Netflix has a great article on their experience with Chaos Gorilla.

This also applies to service degradation, like increased latency. Not just when a service you depend upon is completely offline. Netflix has an article on this as well: Chaos Engineering Upgraded.


Applying manifests in Azure DevOps pipeline Basically, I want the pipeline to apply some manifests before deploying images from ACR. No images need to be made. It would just be things like applying the issuer.yaml, ingress.yaml, and maybe a few other things that need to be run before the client and api images are pulled from ACR.

Was looking at the documentation here:

https://docs.microsoft.com/en-us/azure/devops/pipelines/tasks/deploy/kubernetes-manifest?view=azure-devops#overview

But just not really seeing what I need.

Suggestions for how to do this?

Once again, come across the answer right after asking the question.

Needed to look at the Kubectl task and not the Kubernetes manifest task. Particularly the Commands section:

- task: Kubernetes@1
  displayName: kubectl apply using arguments
  inputs:
    connectionType: Azure Resource Manager
    azureSubscriptionEndpoint: $(azureSubscriptionEndpoint)
    azureResourceGroup: $(azureResourceGroup)
    kubernetesCluster: $(kubernetesCluster)
    command: apply
    arguments: -f mhc-aks.yaml


Understanding contents and workwise of a Terraform plan If you export a Terraform plan, it is a binary file.

Now, calls to a cloud management interface are HTTP calls following corresponding REST APIs.

Does this mean this internal Terraform format somehow encodes these calls through some enumerations?

Like, given in the plan it would be, "B#3$1$2" which then translate "call the API function we have given number 3 passing values 1 and 2 as arguments"?

How does it work?

Terraform use protobuf to store the plans according to its code

  // writeTfplan serializes the given plan into the protobuf-based format used
  // for the "tfplan" portion of a plan file.

You can find here the code generating the plan file.

So in brief that's a binary file with a serialized version of what to do in a portable format (protocol buffers) for which you find more information here


Why unable to find the exact path for grains storage in master? I am trying to completely remove the grains from the system. I have one master and one minion setup(minion id= minion1). I am doing

  salt minion1 grains.ls

that is listing me all the grains

Now I removed data.p which is the cache memory for the grains in master

path : /var/cache/salt/master/minions/minion1/data.p

Without running a salt_sync or any other command for sync I am again doing.

 `salt minion1 grains.ls`,

It's returning the same result.(The grains are still somewhere stored.

Where exactly it is storing the grains and how to delete that?

These are built-in grains, which are collected at minion startup. Here's the code responsible for generating them.

You cannot remove them, but you can overwrite them, for example to an empty value in /etc/salt/grains:

osfinger: null

or with grains execution module: salt minion1 grains.delval osfinger.

Keep in mind that some of the core grains are essential for the correct functioning of salt, e.g. module providers for pkg module rely on them.


What are best and comprehensive practices to consider when running docker in production? Finally, you are so much in love with Docker that you want to move your online business-critical production systems with sensitive customer data to a Docker Swarm. Some might even already have done so. The other organization can't afford it by a policy forbidding production processes running in root mode.

What could be a checklist of building blocks to consider for a Docker production environment? One does not need all of them, but all of them should be important to be assessed.

Disclaimer: I know there is a SE policy to avoid "large endless lists" but I think this checklist cannot be very big... and endless noway.

So - what are these buildings blocks?

If not already deployed, consider running a Linux host system with advanced
security settings - hardened kernel, SELinux etc.
Consider using a tiny Docker base image, like alpine, busybox or even scratch e.g. start with an empty base image
Use USER setting other than root
Carefully assess to further reduce the already shrinked set of kernel capabilities granted to container
Consider having only one executable binary per container to launch your process, ideally statically linked
Those who want to break your system to get a shell access might wonder if they found out your container has all shells disabled
Mount read-only volumes where only possible

Question: what else?

The host on which the containers are running

Run the docker security bench on every node that runs docker containers https://github.com/docker/docker-bench-security

Running the following command on a node that runs docker containers:

docker run -it --net host --pid host --cap-add audit_control \
    -e DOCKER_CONTENT_TRUST=$DOCKER_CONTENT_TRUST \
    -v /var/lib:/var/lib \
    -v /var/run/docker.sock:/var/run/docker.sock \
    -v /usr/lib/systemd:/usr/lib/systemd \
    -v /etc:/etc --label docker_bench_security \
    docker/docker-bench-security

returns a list of checks:

[INFO] 1 - Host Configuration

[WARN] 1.1  - Ensure a separate partition for containers has been created

[NOTE] 4.2  - Ensure that containers use trusted base images

[PASS] 4.6  - Ensure HEALTHCHECK instructions have been added to the container image

Quote from the repository README:

  The Docker Bench for Security is a script that checks for dozens of
  common best-practices around deploying Docker containers in
  production. The tests are all automated, and are inspired by the CIS
  Docker Community Edition Benchmark
  v1.1.0.

Some of the issues that are reported by the security bench could be solved by reading the official docker security article and comparing it with the bullets that are defined in the question the following things are important as well:

protect the docker daemon socket by implementing ssl
content trust using notary and DOCKER_CONTENT_TRUST variable


Correct way to handle 'optional' array of resources in Terraform In one of our our Terraform modules we have a variable to optionally create a load balancer. As part of this we create some subnets thusly:

resource "aws_subnet" "subnet_loadbalancer" {
  count = var.create_loadbalancer ? length(var.az_names) : 0
  vpc_id            = aws_vpc.vpc_main.id
  cidr_block        = var.lb_subnet_cidrs[count.index]
  availability_zone = var.az_names[count.index]
}

Now, if we are creating a loadbalancer I want to output the first subnet for later use in other modules:

output primary_LB_Subnet {
  value = aws_subnet.subnet_loadbalancer[0]
}

This is fine if we create the loadbalancer, but not if var.create_loadbalancer is false. In that case I get the following error:

Error: Invalid index

  on ..\..\modules\vpc\loadbalancer.tf line 19, in output "primary_LB_Subnet":
  19:   value = aws_subnet.subnet_loadbalancer[0]
    |----------------
    | aws_subnet.subnet_loadbalancer is empty tuple

The given key does not identify an element in this collection value.

I can see why this would be a problem, as we haven't create the subnets, so my immediate thought was to do something like:

output loadbalancer_sg_id {
  value = var.create_loadbalancer ? aws_subnet.subnet_loadbalancer[0] : null
}

But this doesn't help the issue and I still get the same issue. What is the correct way to address this within Terraform?

Could you wrap the assignment to value using the try type conversion function?

output primary_LB_Subnet {
  value = try(aws_subnet.subnet_loadbalancer[0], null)
}